search

winycg / CLIP-KD

[CVPR-2024] Official implementations of CLIP-KD: An Empirical Study of CLIP Model Distillation
29 stars 0 forks source link

Missing teacher pretrained models #9

Open mrpeerat opened 6 days ago

mrpeerat commented 6 days ago

Hi! I'm trying to run KD methods, but the training script requires t-model-checkpoint. I downloaded the teacher weights (https://github.com/winycg/CLIP-KD/releases/download/CLIP-KDv0.1/ViT_B_16_cc3m_12m_ep32.pt), changed the file name to vit_b_16.pt (similar to your training script).

However, I found this error: RuntimeError: Error(s) in loading state_dict for CLIP: Missing key(s) in state_dict: "positional_embedding", "text_projection", "logit_scale", "visual.class_embedding", "visual.positional_embedding", "visual.proj", "visual.conv1.weight", "visual.ln_pre.weight", "visual.ln_pre.bias", "visual.t Unexpected key(s) in state_dict: "epoch", "name", "state_dict", "optimizer", "scaler" Any suggestions? Thank you in advance.

winycg commented 6 days ago

Thanks for your attention. The form checkpoint of : checkpoint_dict = { "epoch": completed_epoch, "name": args.name, "state_dict": model.state_dict(), "optimizer": optimizer.state_dict(), } You should use the "state_dict" key to index the model weights. Please refer to https://github.com/winycg/CLIP-KD/blob/f3ae18b9da8ac8570796c4e113cb8a9006dab18a/src/training/main_kd.py#L190

If you have further questions, please feel free to ask me.

winycg commented 6 days ago

If you uses the teacher ViT-B-16 pretrained on Laion (https://github.com/winycg/CLIP-KD/releases/download/CLIP-KDv0.1/ViT_B_16-laion400m_e32.pt) provided by open-clip, you do not need to use "state_dict" to index, because this .pt file only includes model weights.