Keys mismatch when loading weights of Co-DETR with ViT-L

[function2-llx]

Dear authors,

I tried testing with the config https://github.com/Sense-X/Co-DETR/blob/221f1a38767f9456f2efe7d2e7f23f69b9c7cf84/projects/configs/co_dino_vit/co_dino_5scale_lsj_vit_large_lvis.py and the corresponding checkpoint. When loading the checkpoint, it produces the following messages:

The model and loaded state dict do not match exactly unexpected key in source state_dict: query_head.loss_cls.fed_loss_weight, bbox_head.0.loss_cls.fed_loss_weight missing keys in source state_dict: backbone.rope_glb.freqs_cos, backbone.rope_glb.freqs_sin, backbone.blocks.2.attn.rope.freqs_cos, backbone.blocks.2.attn.rope.freqs_sin, backbone.blocks.5.attn.rope.freqs_cos, backbone.blocks.5.attn.rope.freqs_sin, backbone.blocks.8.attn.rope.freqs_cos, backbone.blocks.8.attn.rope.freqs_sin, backbone.blocks.11.attn.rope.freqs_cos, backbone.blocks.11.attn.rope.freqs_sin, backbone.blocks.14.attn.rope.freqs_cos, backbone.blocks.14.attn.rope.freqs_sin, backbone.blocks.17.attn.rope.freqs_cos, backbone.blocks.17.attn.rope.freqs_sin, backbone.blocks.20.attn.rope.freqs_cos, backbone.blocks.20.attn.rope.freqs_sin, backbone.blocks.23.attn.rope.freqs_cos, backbone.blocks.23.attn.rope.freqs_sin, roi_head.0.bbox_head.fc_cls.weight, roi_head.0.bbox_head.fc_cls.bias, roi_head.0.bbox_head.fc_reg.weight, roi_head.0.bbox_head.fc_reg.bias, roi_head.0.bbox_head.shared_fcs.0.weight, roi_head.0.bbox_head.shared_fcs.0.bias, roi_head.0.bbox_head.shared_fcs.1.weight, roi_head.0.bbox_head.shared_fcs.1.bias

There are three parts of issues:

1. Unexpected fed_loss_weight in query_head.loss_cls and bbox_head.0.loss_cls, which seems to be the loss weight for federated loss.
2. Missing RoPE frequencies in the backbone, which does not matter in theory. However, it is still strange that only keys related to backbone.rope_glb.* are reported here, while keys related to backbone.rope_win are not.
3. Missing keys for roi_head.

It seems that there may be a mismatch between the code of the repository and the weights. Could you please take a look at this?

Best regards

[TempleX98]

These unexpected and missing weights do not affect the model performance as they do not participate in the inference process.

[function2-llx]

Thanks for the reply. But I would like to finetune the model, so I think it may be better to fully utilize the pre-trained weights?

[TempleX98]

If you want to finetune the model on conventional downstream datasets (e.g., COCO), it would be better to adopt the COCO model and config. If you want to finetune it on datasets with long-tail distribution, please email me to obtain the LVIS training codes.

[function2-llx]

I see, thanks!