search

IDT-ITI / MMFusion-IML

Code and trained models for our paper: K. Triaridis, V. Mezaris, "Exploring Multi-Modal Fusion for Image Manipulation Detection and Localization", Proc. 30th Int. Conf. on MultiMedia Modeling (MMM 2024), Amsterdam, NL, Jan.-Feb. 2024.
61 stars 5 forks source link

Getting error while using late_fusion_detection.pth for inference #14

Open man0007 opened 4 months ago

man0007 commented 4 months ago

HI, Getting the below error while using the late_fusion_detection.pth for inference.

Traceback (most recent call last): File "C:\Users\manan\Desktop\projects\MMFusion-IML\inference.py", line 48, in model.load_state_dict(ckpt['state_dict']) File "C:\Users\manan\Desktop\projects_env\MMFusion-IML\env_312\Lib\site-packages\torch\nn\modules\module.py", line 2153, in load_state_dict raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format( RuntimeError: Error(s) in loading state_dict for CMNeXtWithConf: Missing key(s) in state_dict: "backbone.extra_downsample_layers.0.proj.module.weight", "backbone.extra_downsample_layers.0.proj.module.bias", "backbone.extra_downsample_layers.0.norm.ln_0.weight", "backbone.extra_downsample_layers.0.norm.ln_0.bias", "backbone.extra_downsample_layers.1.proj.module.weight", "backbone.extra_downsample_layers.1.proj.module.bias", "backbone.extra_downsample_layers.1.norm.ln_0.weight", "backbone.extra_downsample_layers.1.norm.ln_0.bias", "backbone.extra_downsample_layers.2.proj.module.weight", "backbone.extra_downsample_layers.2.proj.module.bias", "backbone.extra_downsample_layers.2.norm.ln_0.weight", "backbone.extra_downsample_layers.2.norm.ln_0.bias", "backbone.extra_downsample_layers.3.proj.module.weight", "backbone.extra_downsample_layers.3.proj.module.bias", "backbone.extra_downsample_layers.3.norm.ln_0.weight", "backbone.extra_downsample_layers.3.norm.ln_0.bias", "backbone.modal_mix.blocks.0.convs.0.0.weight", "backbone.modal_mix.blocks.0.convs.0.0.bias", "backbone.modal_mix.blocks.0.convs.0.1.weight", "backbone.modal_mix.blocks.0.convs.0.1.bias", "backbone.modal_mix.blocks.0.convs.0.1.running_mean", "backbone.modal_mix.blocks.0.convs.0.1.running_var", "backbone.modal_mix.blocks.0.convs.1.0.weight", "backbone.modal_mix.blocks.0.convs.1.0.bias", "backbone.modal_mix.blocks.0.convs.1.1.weight", "backbone.modal_mix.blocks.0.convs.1.1.bias", "backbone.modal_mix.blocks.0.convs.1.1.running_mean", "backbone.modal_mix.blocks.0.convs.1.1.running_var", "backbone.modal_mix.blocks.0.convs.2.0.weight", "backbone.modal_mix.blocks.0.convs.2.0.bias", "backbone.modal_mix.blocks.0.convs.2.1.weight", "backbone.modal_mix.blocks.0.convs.2.1.bias", "backbone.modal_mix.blocks.0.convs.2.1.running_mean", "backbone.modal_mix.blocks.0.convs.2.1.running_var", "backbone.modal_mix.blocks.0.final.weight", "backbone.modal_mix.blocks.0.final.bias", "backbone.modal_mix.blocks.1.convs.0.0.weight", "backbone.modal_mix.blocks.1.convs.0.0.bias", "backbone.modal_mix.blocks.1.convs.0.1.weight", "backbone.modal_mix.blocks.1.convs.0.1.bias", "backbone.modal_mix.blocks.1.convs.0.1.running_mean", "backbone.modal_mix.blocks.1.convs.0.1.running_var", "backbone.modal_mix.blocks.1.convs.1.0.weight", "backbone.modal_mix.blocks.1.convs.1.0.bias", "backbone.modal_mix.blocks.1.convs.1.1.weight", "backbone.modal_mix.blocks.1.convs.1.1.bias", "backbone.modal_mix.blocks.1.convs.1.1.running_mean", "backbone.modal_mix.blocks.1.convs.1.1.running_var", "backbone.modal_mix.blocks.1.convs.2.0.weight", "backbone.modal_mix.blocks.1.convs.2.0.bias", "backbone.modal_mix.blocks.1.convs.2.1.weight", "backbone.extra.2.block1.2.mlp.fc2.weight", "backbone.extra.2.block1.2.mlp.fc2.bias", "backbone.extra.2.norm1.weight", "backbone.extra.2.norm1.bias", "backbone.extra.2.block2.0.norm1.weight", "backbone.extra.2.block2.0.norm1.bias", "backbone.extra.2.block2.0.attn.q.weight", "backbone.extra.2.block2.0.attn.q.bias", "backbone.extra.2.block2.0.attn.kv.weight", "backbone.extra.2.block2.0.attn.kv.bias", "backbone.extra.2.block2.0.attn.proj.weight", "backbone.extra.2.block2.0.attn.proj.bias", "backbone.extra.2.block2.0.attn.sr.weight", "backbone.extra.2.block2.0.attn.sr.bias", "backbone.extra.2.block2.0.attn.norm.weight", "backbone.extra.2.block2.0.attn.norm.bias", "backbone.extra.2.block2.0.norm2.weight", "backbone.extra.2.block2.0.norm2.bias", "backbone.extra.2.block2.0.mlp.fc1.weight", "backbone.extra.2.block2.0.mlp.fc1.bias", "backbone.extra.2.block2.0.mlp.dwconv.dwconv.weight", "backbone.extra.2.block2.0.mlp.dwconv.dwconv.bias", "backbone.extra.2.block2.0.mlp.fc2.weight", "backbone.extra.2.block2.0.mlp.fc2.bias", "backbone.extra.2.block2.1.norm1.weight", "backbone.extra.2.block2.1.norm1.bias", "backbone.extra.2.block2.1.attn.q.weight", "backbone.extra.2.block2.1.attn.q.bias", "backbone.extra.2.block2.1.attn.kv.weight", "backbone.extra.2.block2.1.attn.kv.bias", "backbone.extra.2.block2.1.attn.proj.weight", "backbone.extra.2.block2.1.attn.proj.bias", "backbone.extra.2.block2.1.attn.sr.weight", "backbone.extra.2.block2.1.attn.sr.bias", "backbone.extra.2.block2.1.attn.norm.weight", "backbone.extra.2.block2.1.attn.norm.bias", "backbone.extra.2.block2.1.norm2.weight", "backbone.extra.2.block2.1.norm2.bias", "backbone.extra.2.block2.1.mlp.fc1.weight", "backbone.extra.2.block2.1.mlp.fc1.bias", "backbone.extra.2.block2.1.mlp.dwconv.dwconv.weight", "backbone.extra.2.block2.1.mlp.dwconv.dwconv.bias", "backbone.extra.2.block2.1.mlp.fc2.weight", "backbone.extra.2.block2.1.mlp.fc2.bias", "backbone.extra.2.block2.2.norm1.weight", "backbone.extra.2.block2.2.norm1.bias", "backbone.extra.2.block2.2.attn.q.weight", "backbone.extra.2.block2.2.attn.q.bias", "backbone.extra.2.block2.2.attn.kv.weight", "backbone.extra.2.block2.2.attn.kv.bias", "backbone.extra.2.block2.2.attn.proj.weight", "backbone.extra.2.block2.2.attn.proj.bias", "backbone.extra.2.block2.2.attn.sr.weight", "backbone.extra.2.block2.2.attn.sr.bias", "backbone.extra.2.block2.2.attn.norm.weight", "backbone.extra.2.block2.2.attn.norm.bias", "backbone.extra.2.block2.2.norm2.weight", "backbone.extra.2.block2.2.norm2.bias", "backbone.extra.2.block2.2.mlp.fc1.weight", "backbone.extra.2.block2.2.mlp.fc1.bias", "backbone.extra.2.block2.2.mlp.dwconv.dwconv.weight", "backbone.extra.2.block2.2.mlp.dwconv.dwconv.bias", "backbone.extra.2.block2.2.mlp.fc2.weight", "backbone.extra.2.block2.2.mlp.fc2.bias", "backbone.extra.2.block2.3.norm1.weight", "backbone.extra.2.block2.3.norm1.bias", "backbone.extra.2.block2.3.attn.q.weight", "backbone.extra.2.block2.3.attn.q.bias", "backbone.extra.2.block2.3.attn.kv.weight", "backbone.extra.2.block2.3.attn.kv.bias", "backbone.extra.2.block2.3.attn.proj.weight", "backbone.extra.2.block2.3.attn.proj.bias", "backbone.extra.2.block2.3.attn.sr.weight", "backbone.extra.2.block2.3.attn.sr.bias", "backbone.extra.2.block2.3.attn.norm.weight", "backbone.extra.2.block2.3.attn.norm.bias", "backbone.extra.2.block2.3.norm2.weight", "backbone.extra.2.block2.3.norm2.bias", "backbone.extra.2.block2.3.mlp.fc1.weight", "backbone.extra.2.block2.3.mlp.fc1.bias", "backbone.extra.2.block2.3.mlp.dwconv.dwconv.weight", "backbone.extra.2.block2.3.mlp.dwconv.dwconv.bias", "backbone.extra.2.block2.3.mlp.fc2.weight", "backbone.extra.2.block2.3.mlp.fc2.bias", "backbone.extra.2.norm2.weight", "backbone.extra.2.norm2.bias", "backbone.extra.2.block3.0.norm1.weight", "backbone.extra.2.block3.0.norm1.bias", "backbone.extra.2.block3.0.attn.q.weight", "backbone.extra.2.block3.0.attn.q.bias", "backbone.extra.2.block3.0.attn.kv.weight", "backbone.extra.2.block3.0.attn.kv.bias", "backbone.extra.2.block3.0.attn.proj.weight", "backbone.extra.2.block3.0.attn.proj.bias", "backbone.extra.2.block3.0.attn.sr.weight", "backbone.extra.2.block3.0.attn.sr.bias", "backbone.extra.2.block3.0.attn.norm.weight", "backbone.extra.2.block3.0.attn.norm.bias", "backbone.extra.2.block3.0.norm2.weight", "backbone.extra.2.block3.0.norm2.bias", "backbone.extra.2.block3.0.mlp.fc1.weight", "backbone.extra.2.block3.0.mlp.fc1.bias", "backbone.extra.2.block3.0.mlp.dwconv.dwconv.weight", "backbone.extra.2.block3.0.mlp.dwconv.dwconv.bias", "backbone.extra.2.block3.0.mlp.fc2.weight", "backbone.extra.2.block3.0.mlp.fc2.bias", "backbone.extra.2.block3.1.norm1.weight", "backbone.extra.2.block3.1.norm1.bias", "backbone.extra.2.block3.1.attn.q.weight", "backbone.extra.2.block3.1.attn.q.bias", "backbone.extra.2.block3.1.attn.kv.weight", "backbone.extra.2.block3.1.attn.kv.bias", "backbone.extra.2.block3.1.attn.proj.weight", "backbone.extra.2.block3.1.attn.proj.bias", "backbone.extra.2.block3.1.attn.sr.weight", "backbone.extra.2.block3.1.attn.sr.bias", "backbone.extra.2.block3.1.attn.norm.weight", "backbone.extra.2.block3.1.attn.norm.bias", "backbone.extra.2.block3.1.norm2.weight", "backbone.extra.2.block3.1.norm2.bias", "backbone.extra.2.block3.1.mlp.fc1.weight", "backbone.extra.2.block3.1.mlp.fc1.bias", "backbone.extra.2.block3.1.mlp.dwconv.dwconv.weight", "backbone.extra.2.block3.1.mlp.dwconv.dwconv.bias", "backbone.extra.2.block3.1.mlp.fc2.weight", "backbone.extra.2.block3.1.mlp.fc2.bias", "backbone.extra.2.block3.2.norm1.weight", "backbone.extra.2.block3.2.norm1.bias", "backbone.extra.2.block3.2.attn.q.weight", "backbone.extra.2.block3.2.attn.q.bias", "backbone.extra.2.block3.2.attn.kv.weight", "backbone.extra.2.block3.2.attn.kv.bias", "backbone.extra.2.block3.2.attn.proj.weight", "backbone.extra.2.block3.2.attn.proj.bias", "backbone.extra.2.block3.2.attn.sr.weight", "backbone.extra.2.block3.2.attn.sr.bias", "backbone.extra.2.block3.2.attn.norm.weight", "backbone.extra.2.block3.2.attn.norm.bias", "backbone.extra.2.block3.2.norm2.weight", "backbone.extra.2.block3.2.norm2.bias", "backbone.extra.2.block3.2.mlp.fc1.weight", "backbone.extra.2.block3.2.mlp.fc1.bias", "backbone.extra.2.block3.2.mlp.dwconv.dwconv.weight", "backbone.extra.2.block3.2.mlp.dwconv.dwconv.bias", "backbone.extra.2.block3.2.mlp.fc2.weight", "backbone.extra.2.block3.2.mlp.fc2.bias", "backbone.extra.2.block3.3.norm1.weight", "backbone.extra.2.block3.3.norm1.bias", "backbone.extra.2.block3.3.attn.q.weight", "backbone.extra.2.block3.3.attn.q.bias", "backbone.extra.2.block3.3.attn.kv.weight", "backbone.extra.2.block3.3.attn.kv.bias", "backbone.extra.2.block3.3.attn.proj.weight", "backbone.extra.2.block3.3.attn.proj.bias", "backbone.extra.2.block3.3.attn.sr.weight", "backbone.extra.2.block3.3.attn.sr.bias", "backbone.extra.2.block3.3.attn.norm.weight", "backbone.extra.2.block3.3.attn.norm.bias", "backbone.extra.2.block3.3.norm2.weight", "backbone.extra.2.block3.3.norm2.bias", "backbone.extra.2.block3.3.mlp.fc1.weight", "backbone.extra.2.block3.3.mlp.fc1.bias", "backbone.extra.2.block3.3.mlp.dwconv.dwconv.weight", "backbone.extra.2.block3.3.mlp.dwconv.dwconv.bias", "backbone.extra.2.block3.3.mlp.fc2.weight", "backbone.extra.2.block3.3.mlp.fc2.bias", "backbone.extra.2.block3.4.norm1.weight", "backbone.extra.2.block3.4.norm1.bias", "backbone.extra.2.block3.4.attn.q.weight", "backbone.extra.2.block3.4.attn.q.bias", "backbone.extra.2.block3.4.attn.kv.weight", "backbone.extra.2.block3.4.attn.kv.bias", "backbone.extra.2.block3.4.attn.proj.weight", "backbone.extra.2.block3.4.attn.proj.bias", "backbone.extra.2.block3.4.attn.sr.weight", "backbone.extra.2.block3.4.attn.sr.bias", "backbone.extra.2.block3.4.attn.norm.weight", "backbone.extra.2.block3.4.attn.norm.bias", "backbone.extra.2.block3.4.norm2.weight", "backbone.extra.2.block3.4.norm2.bias", "backbone.extra.2.block3.4.mlp.fc1.weight", "backbone.extra.2.block3.4.mlp.fc1.bias", "backbone.extra.2.block3.4.mlp.dwconv.dwconv.weight", "backbone.extra.2.block3.4.mlp.dwconv.dwconv.bias", "backbone.extra.2.block3.4.mlp.fc2.weight", "backbone.extra.2.block3.4.mlp.fc2.bias", "backbone.extra.2.block3.5.norm1.weight", "backbone.extra.2.block3.5.norm1.bias", "backbone.extra.2.block3.5.attn.q.weight", "backbone.extra.2.block3.5.attn.q.bias", "backbone.extra.2.block3.5.attn.kv.weight", "backbone.extra.2.block3.5.attn.kv.bias", "backbone.extra.2.block3.5.attn.proj.weight", "backbone.extra.2.block3.5.attn.proj.bias", "backbone.extra.2.block3.5.attn.sr.weight", "backbone.extra.2.block3.5.attn.sr.bias", "backbone.extra.2.block3.5.attn.norm.weight", "backbone.extra.2.block3.5.attn.norm.bias", "backbone.extra.2.block3.5.norm2.weight", "backbone.extra.2.block3.5.norm2.bias", "backbone.extra.2.block3.5.mlp.fc1.weight", "backbone.extra.2.block3.5.mlp.fc1.bias", "backbone.extra.2.block3.5.mlp.dwconv.dwconv.weight", "backbone.extra.2.block3.5.mlp.dwconv.dwconv.bias", "backbone.extra.2.block3.5.mlp.fc2.weight", "backbone.extra.2.block3.5.mlp.fc2.bias", "backbone.extra.2.norm3.weight", "backbone.extra.2.norm3.bias", "backbone.extra.2.block4.0.norm1.weight", "backbone.extra.2.block4.0.norm1.bias", "backbone.extra.2.block4.0.attn.q.weight", "backbone.extra.2.block4.0.attn.q.bias", "backbone.extra.2.block4.0.attn.kv.weight", "backbone.extra.2.block4.0.attn.kv.bias", "backbone.extra.2.block4.0.attn.proj.weight", "backbone.extra.2.block4.0.attn.proj.bias", "backbone.extra.2.block4.0.norm2.weight", "backbone.extra.2.block4.0.norm2.bias", "backbone.extra.2.block4.0.mlp.fc1.weight", "backbone.extra.2.block4.0.mlp.fc1.bias", "backbone.extra.2.block4.0.mlp.dwconv.dwconv.weight", "backbone.extra.2.block4.0.mlp.dwconv.dwconv.bias", "backbone.extra.2.block4.0.mlp.fc2.weight", "backbone.extra.2.block4.0.mlp.fc2.bias", "backbone.extra.2.block4.1.norm1.weight", "backbone.extra.2.block4.1.norm1.bias", "backbone.extra.2.block4.1.attn.q.weight", "backbone.extra.2.block4.1.attn.q.bias", "backbone.extra.2.block4.1.attn.kv.weight", "backbone.extra.2.block4.1.attn.kv.bias", "backbone.extra.2.block4.1.attn.proj.weight", "backbone.extra.2.block4.1.attn.proj.bias", "backbone.extra.2.block4.1.norm2.weight", "backbone.extra.2.block4.1.norm2.bias", "backbone.extra.2.block4.1.mlp.fc1.weight", "backbone.extra.2.block4.1.mlp.fc1.bias", "backbone.extra.2.block4.1.mlp.dwconv.dwconv.weight", "backbone.extra.2.block4.1.mlp.dwconv.dwconv.bias", "backbone.extra.2.block4.1.mlp.fc2.weight", "backbone.extra.2.block4.1.mlp.fc2.bias", "backbone.extra.2.block4.2.norm1.weight", "backbone.extra.2.block4.2.norm1.bias", "backbone.extra.2.block4.2.attn.q.weight", "backbone.extra.2.block4.2.attn.q.bias", "backbone.extra.2.block4.2.attn.kv.weight", "backbone.extra.2.block4.2.attn.kv.bias", "backbone.extra.2.block4.2.attn.proj.weight", "backbone.extra.2.block4.2.attn.proj.bias", "backbone.extra.2.block4.2.norm2.weight", "backbone.extra.2.block4.2.norm2.bias", "backbone.extra.2.block4.2.mlp.fc1.weight", "backbone.extra.2.block4.2.mlp.fc1.bias", "backbone.extra.2.block4.2.mlp.dwconv.dwconv.weight", "backbone.extra.2.block4.2.mlp.dwconv.dwconv.bias", "backbone.extra.2.block4.2.mlp.fc2.weight", "backbone.extra.2.block4.2.mlp.fc2.bias", "backbone.extra.2.norm4.weight", "backbone.extra.2.norm4.bias", "backbone.extra.2.FRMs.0.channel_weights.mlp.0.weight", "backbone.extra.2.FRMs.0.channel_weights.mlp.0.bias", "backbone.extra.2.FRMs.0.channel_weights.mlp.2.weight", "backbone.extra.2.FRMs.0.channel_weights.mlp.2.bias", "backbone.extra.2.FRMs.0.spatial_weights.mlp.0.weight", "backbone.extra.2.FRMs.0.spatial_weights.mlp.0.bias", "backbone.extra.2.FRMs.0.spatial_weights.mlp.2.weight", "backbone.extra.2.FRMs.0.spatial_weights.mlp.2.bias", "backbone.extra.2.FRMs.1.channel_weights.mlp.0.weight", "backbone.extra.2.FRMs.1.channel_weights.mlp.0.bias", "backbone.extra.2.FRMs.1.channel_weights.mlp.2.weight", "backbone.extra.2.FRMs.1.channel_weights.mlp.2.bias", "backbone.extra.2.FRMs.1.spatial_weights.mlp.0.weight", "backbone.extra.2.FRMs.1.spatial_weights.mlp.0.bias", "backbone.extra.2.FRMs.1.spatial_weights.mlp.2.weight", "backbone.extra.2.FRMs.1.spatial_weights.mlp.2.bias", "backbone.extra.2.FRMs.2.channel_weights.mlp.0.weight", "backbone.extra.2.FRMs.2.channel_weights.mlp.0.bias", "backbone.extra.2.FRMs.2.channel_weights.mlp.2.weight", "backbone.extra.2.FRMs.2.channel_weights.mlp.2.bias", "backbone.extra.2.FRMs.2.spatial_weights.mlp.0.weight", "backbone.extra.2.FRMs.2.spatial_weights.mlp.0.bias", "backbone.extra.2.FRMs.2.spatial_weights.mlp.2.weight", "backbone.extra.2.FRMs.2.spatial_weights.mlp.2.bias", "backbone.extra.2.FRMs.3.channel_weights.mlp.0.weight", "backbone.extra.2.FRMs.3.channel_weights.mlp.0.bias", "backbone.extra.2.FRMs.3.channel_weights.mlp.2.weight", "backbone.extra.2.FRMs.3.channel_weights.mlp.2.bias", "backbone.extra.2.FRMs.3.spatial_weights.mlp.0.weight", "backbone.extra.2.FRMs.3.spatial_weights.mlp.0.bias", "backbone.extra.2.FRMs.3.spatial_weights.mlp.2.weight", "backbone.extra.2.FRMs.3.spatial_weights.mlp.2.bias", "backbone.extra.2.FFMs.0.cross.channel_proj1.weight", "backbone.extra.2.FFMs.0.cross.channel_proj1.bias", "backbone.extra.2.FFMs.0.cross.channel_proj2.weight", "backbone.extra.2.FFMs.0.cross.channel_proj2.bias", "backbone.extra.2.FFMs.0.cross.cross_attn.kv1.weight", "backbone.extra.2.FFMs.0.cross.cross_attn.kv2.weight", "backbone.extra.2.FFMs.0.cross.end_proj1.weight", "backbone.extra.2.FFMs.0.cross.end_proj1.bias", "backbone.extra.2.FFMs.0.cross.end_proj2.weight", "backbone.extra.2.FFMs.0.cross.end_proj2.bias", "backbone.extra.2.FFMs.0.cross.norm1.weight", "backbone.extra.2.FFMs.0.cross.norm1.bias", "backbone.extra.2.FFMs.0.cross.norm2.weight", "backbone.extra.2.FFMs.0.cross.norm2.bias", "backbone.extra.2.FFMs.0.channel_emb.residual.weight", "backbone.extra.2.FFMs.0.channel_emb.channel_embed.0.weight", "backbone.extra.2.FFMs.0.channel_emb.channel_embed.0.bias", "backbone.extra.2.FFMs.0.channel_emb.channel_embed.1.weight", "backbone.extra.2.FFMs.0.channel_emb.channel_embed.1.bias", "backbone.extra.2.FFMs.0.channel_emb.channel_embed.3.weight", "backbone.extra.2.FFMs.0.channel_emb.channel_embed.3.bias", "backbone.extra.2.FFMs.0.channel_emb.channel_embed.4.weight", "backbone.extra.2.FFMs.0.channel_emb.channel_embed.4.bias", "backbone.extra.2.FFMs.0.channel_emb.channel_embed.4.running_mean", "backbone.extra.2.FFMs.0.channel_emb.channel_embed.4.running_var", "backbone.extra.2.FFMs.0.channel_emb.channel_embed.4.num_batches_tracked", "backbone.extra.2.FFMs.0.channel_emb.norm.weight", "backbone.extra.2.FFMs.0.channel_emb.norm.bias", "backbone.extra.2.FFMs.0.channel_emb.norm.running_mean", "backbone.extra.2.FFMs.0.channel_emb.norm.running_var", "backbone.extra.2.FFMs.0.channel_emb.norm.num_batches_tracked", "backbone.extra.2.FFMs.1.cross.channel_proj1.weight", "backbone.extra.2.FFMs.1.cross.channel_proj1.bias", "backbone.extra.2.FFMs.1.cross.channel_proj2.weight", "backbone.extra.2.FFMs.1.cross.channel_proj2.bias", "backbone.extra.2.FFMs.1.cross.cross_attn.kv1.weight", "backbone.extra.2.FFMs.1.cross.cross_attn.kv2.weight", "backbone.extra.2.FFMs.1.cross.end_proj1.weight", "backbone.extra.2.FFMs.1.cross.end_proj1.bias", "backbone.extra.2.FFMs.1.cross.end_proj2.weight", "backbone.extra.2.FFMs.1.cross.end_proj2.bias", "backbone.extra.2.FFMs.1.cross.norm1.weight", "backbone.extra.2.FFMs.1.cross.norm1.bias", "backbone.extra.2.FFMs.1.cross.norm2.weight", "backbone.extra.2.FFMs.1.cross.norm2.bias", "backbone.extra.2.FFMs.1.channel_emb.residual.weight", "backbone.extra.2.FFMs.1.channel_emb.channel_embed.0.weight", "backbone.extra.2.FFMs.1.channel_emb.channel_embed.0.bias", "backbone.extra.2.FFMs.1.channel_emb.channel_embed.1.weight", "backbone.extra.2.FFMs.1.channel_emb.channel_embed.1.bias", "backbone.extra.2.FFMs.1.channel_emb.channel_embed.3.weight", "backbone.extra.2.FFMs.1.channel_emb.channel_embed.3.bias", "backbone.extra.2.FFMs.1.channel_emb.channel_embed.4.weight", "backbone.extra.2.FFMs.1.channel_emb.channel_embed.4.bias", "backbone.extra.2.FFMs.1.channel_emb.channel_embed.4.running_mean", "backbone.extra.2.FFMs.1.channel_emb.channel_embed.4.running_var", "backbone.extra.2.FFMs.1.channel_emb.channel_embed.4.num_batches_tracked", "backbone.extra.2.FFMs.1.channel_emb.norm.weight", "backbone.extra.2.FFMs.1.channel_emb.norm.bias", "backbone.extra.2.FFMs.1.channel_emb.norm.running_mean", "backbone.extra.2.FFMs.1.channel_emb.norm.running_var", "backbone.extra.2.FFMs.1.channel_emb.norm.num_batches_tracked", "backbone.extra.2.FFMs.2.cross.channel_proj1.weight", "backbone.extra.2.FFMs.2.cross.channel_proj1.bias", "backbone.extra.2.FFMs.2.cross.channel_proj2.weight", "backbone.extra.2.FFMs.3.channel_emb.channel_embed.4.running_var", "backbone.extra.2.FFMs.3.channel_emb.channel_embed.4.num_batches_tracked", "backbone.extra.2.FFMs.3.channel_emb.norm.weight", "backbone.extra.2.FFMs.3.channel_emb.norm.bias", "backbone.extra.2.FFMs.3.channel_emb.norm.running_mean", "backbone.extra.2.FFMs.3.channel_emb.norm.running_var", "backbone.extra.2.FFMs.3.channel_emb.norm.num_batches_tracked". size mismatch for decode_head.linear_c1.proj.weight: copying a param with shape torch.Size([512, 192]) from checkpoint, the shape in current model is torch.Size([512, 64]). size mismatch for decode_head.linear_c2.proj.weight: copying a param with shape torch.Size([512, 384]) from checkpoint, the shape in current model is torch.Size([512, 128]). size mismatch for decode_head.linear_c3.proj.weight: copying a param with shape torch.Size([512, 960]) from checkpoint, the shape in current model is torch.Size([512, 320]). size mismatch for decode_head.linear_c4.proj.weight: copying a param with shape torch.Size([512, 1536]) from checkpoint, the shape in current model is torch.Size([512, 512]). size mismatch for conf_head.linear_c1.proj.weight: copying a param with shape torch.Size([512, 192]) from checkpoint, the shape in current model is torch.Size([512, 64]). size mismatch for conf_head.linear_c2.proj.weight: copying a param with shape torch.Size([512, 384]) from checkpoint, the shape in current model is torch.Size([512, 128]). size mismatch for conf_head.linear_c3.proj.weight: copying a param with shape torch.Size([512, 960]) from checkpoint, the shape in current model is torch.Size([512, 320]). size mismatch for conf_head.linear_c4.proj.weight: copying a param with shape torch.Size([512, 1536]) from checkpoint, the shape in current model is torch.Size([512, 512]).

kostino commented 4 months ago

It seems like you are trying to use the late fusion model. Please see: https://github.com/IDT-ITI/MMFusion-IML/issues/6#issuecomment-2126445872