Pretrained Models do not load with pip release

[paulgavrikov]

I installed the latest version of medclip via pip and am trying to load the pretrained models. However, when loading ResNet or ViT I get the following error

RuntimeError: Error(s) in loading state_dict for MedCLIPModel:
    Unexpected key(s) in state_dict: "text_model.model.embeddings.position_ids".

It works if I install the module directly from git.

Reproduce:

from medclip import MedCLIPModel, MedCLIPVisionModelViT, MedCLIPVisionModel

model = MedCLIPModel(vision_cls=MedCLIPVisionModel)
model.from_pretrained()

[Weixiang-Sun]

same problem! Do u find any solution?

[paulgavrikov]

same problem! Do u find any solution?

Yes, it works if you install directly from this repo pip install git+https://github.com/RyanWangZf/MedCLIP.git

[Spring24ch]

Why do I use pip install git+https://github.com/RyanWangZf/MedCLIP.git to produce ![Uploading 微信图片_20240121232226.png…]()

[Spring24ch]

same problem! Do u find any solution?

[EastMage]

same problem! Do u find any solution?

you may delete this key from the dict using 'del text_encoder_checkpoint["text_model.model.embeddings.position_ids"]'. This is because of the Mismatched transformers version.

[StefanDenn3r]

A fix would also be to not be strict with loading the models weights. for that you can just add strict=False in this file: https://github.com/RyanWangZf/MedCLIP/blob/main/medclip/modeling_medclip.py#L185