Nanboy-Ronan
commented
3 months ago Hello, thank you for your interest in this work. From what I see in your error, your model has name "vision_model.model.xxx", "vision_model.model.xxx", while our checkpoint's key name is "vision_model.encoder.xxx". This is the reason that you get this error.
The MedCLIP model is initialized from this line https://github.com/ubc-tea/Backdoor_Multimodal_Foundation_Model/blob/main/medclip/modeling_medclip.py#L94. The transformers module is used to initialize the model. Thus, I suspect that you may have a different version of transformers module than what MedCLIP repo used, which initializes your model with key "vision_model.model.xxx" rather than "vision_model.encoder.xxx".
We use the transformers version of 4.23.1 as detailed in this file: https://github.com/ubc-tea/Backdoor_Multimodal_Foundation_Model/blob/main/environment.yml#L342. Can you please try this version?
Why does this error occur? How do I solve this?
$ python zero_shot.py /home/cr/miniconda3/envs/backdoor_Medclip/lib/python3.8/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead. warnings.warn( /home/cr/miniconda3/envs/backdoor_Medclip/lib/python3.8/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or
Nonefor 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passingweights=ResNet50_Weights.IMAGENET1K_V1. You can also useweights=ResNet50_Weights.DEFAULTto get the most up-to-date weights. warnings.warn(msg) Some weights of the model checkpoint at emilyalsentzer/Bio_ClinicalBERT were not used when initializing BertModel: ['cls.predictions.transform.LayerNorm.bias', 'cls.predictions.transform.dense.bias', 'cls.predictions.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.dense.weight', 'cls.predictions.decoder.weight', 'cls.seq_relationship.bias', 'cls.seq_relationship.weight']When I asked GPT, they answered Error Summary The following issues occurred while loading the model checkpoint:
Warning Messages: Deprecation warnings from torchvision regarding the 'pretrained' parameter and other arguments.
Bio_ClinicalBERT Initialization Warning: Some weights were not used during the initialization of the BERT model. This typically happens when initializing a BERT model from a checkpoint trained on a different task or architecture.
RuntimeError: Errors encountered when loading the state dictionary for MedCLIPModel, with missing keys as follows:
Missing key(s) in state_dict: "vision_model.model.conv1.weight", "vision_model.model.bn1.weight", ..., "vision_model.model.layer3.3.output.dense.bias". Unexpected key(s) in state_dict:
"vision_model.projection_head.weight", "vision_model.model.embeddings.patch_embeddings.projection.weight", ..., "vision_model.model.layernorm.bias". These errors indicate a mismatch between the model architecture and the checkpoint.