Evaluation of Trained SigLIP Checkpoints

[work4cs]

I train a SigLIP model without problem, but when I load the training checkpoint to evaluate on the downstream task, I get error: RuntimeError: Error(s) in loading state_dict for CLIP: Unexpected key(s) in state_dict: "logit_bias".

on incompatible_keys = model.load_state_dict(state_dict, strict=strict)

state_dict from checkpoint has the key "logit_bias", but model does not.

I can assign a value to init_logit_bias, passing to create_model_and_transforms() during evaluation, to make the model.state_dict() notice the key logit_bias. But is there a more elegant way to do so, which I do not need specify a model when create it and pass a specific variable to it but I can load the necessary keys based on the checkpoint? (I tried model.load_state_dict(state_dict, strict=**False**), but it did not load the correct value from the checkpoint. Also, I am not sure if it is secure/reliable enough to change the strict to False. )

[rwightman]

@work4cs a SigLIP model using one of the SigLIP model configs (e.g. https://github.com/mlfoundations/open_clip/blob/main/src/open_clip/model_configs/ViT-B-16-SigLIP-256.json) will not have this issue, but if you train a 'non-siglip' model by enabling siglip, yes you need to pass the logit_bias OR create a new config for that model with logit bias in it, it's technically no longer the original model config that you used