search

ibrahimethemhamamci / CT-CLIP

Developing Generalist Foundation Models from a Multimodal Dataset for 3D Computed Tomography
197 stars 21 forks source link

Issue about checkpoint of CT-CLIP #10

Closed zhi-xuan-chen closed 7 months ago

zhi-xuan-chen commented 7 months ago

Dear author,

Thank you for your amazing work! I was trying to reproduce your results recently, but I met the following problem when I load the checkpoint of your model. image How can I solve this problem? Looking forward for your response!

sezginerr commented 7 months ago

Hello @zhi-xuan-chen, thank you very much for your interest in our work. Are you using our zero-shot inference code to load checkpoints? Also, could you please check your torch version, which should be 2.0.1? It's unexpected because it works in my case. Discr blocks are not used in CT-CLIP anyway, as the encoder is from another work of ours that require discr blocks. Therefore, it would be better if I remove them from the checkpoints and the model code (as it takes some unnecessary GPU memory). I will send the PR for edited codes and update the weight files in hugging face today after ensuring they do not break anything.

sezginerr commented 7 months ago

Hello @zhi-xuan-chen, I have updated model weights and the transformer_maskgit code. Discr blocks should be removed now and it should work in your case as well. Please let me know if you can run the model.

zhi-xuan-chen commented 7 months ago

OK! Only the transformer_maskgit code was changed, so I only need to update the code of this part?

zhi-xuan-chen commented 7 months ago

I have noticed only ctvit.py was modified. I will try your new code immediately!

sezginerr commented 7 months ago

Yes, I only modified ctvit.py to remove weights that are not used in the forward run of CT-Clip. Let me know if you have any further inquiries on this!

zhi-xuan-chen commented 7 months ago

The model checkpoint can be successfully loaded. Thank you very much!