jiazhou-garland / EventBind

MIT License
19 stars 0 forks source link

Problem about loading the corresponding model #4

Open Hoantrbl opened 3 weeks ago

Hoantrbl commented 3 weeks ago

When I try to load the checkpoint about the N-Caltech101, it occurs the following error:

raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for DataParallel:
    Unexpected key(s) in state_dict: "module.prompt_leranable.ctx", "module.prompt_leranable.clip_model.positional_embedding", "module.prompt_leranable.clip_model.text_projection", "module.prompt_leranable.clip_model.logit_scale", "module.prompt_leranable.clip_model.visual.class_embedding", "module.prompt_leranable.clip_model.visual.positional_embedding", "module.prompt_leranable.clip_model.visual.proj", "module.prompt_leranable.clip_model.visual.conv1.weight", "module.prompt_leranable.clip_model.visual.ln_pre.weight", "module.prompt_leranable.clip_model.visual.ln_pre.bias", "module.prompt_leranable.clip_model.visual.transformer.resblocks.0.attn.in_proj_weight", "module.prompt_leranable.clip_model.visual.transformer.resblocks.0.attn.in_proj_bias", "module.prompt_leranable.clip_model.visual.transformer.resblocks.0.attn.out_proj.weight", "module.prompt_leranable.clip_model.visual.transformer.resblocks.0.attn.out_proj.bias", "module.prompt_leranable.clip_model.visual.transformer.resblocks.0.ln_1.weight", "module.prompt_leranable.clip_model.visual.transformer.resblocks.0.ln_1.bias", "module.prompt_leranable.clip_model.visual.transformer.resblocks.0.mlp.c_fc.weight", "module.prompt_leranable.clip_model.visual.transformer.resblocks.0.mlp.c_fc.bias", "module.prompt_leranable.clip_model.visual.transformer.resblocks.0.mlp.c_proj.weight", "module.prompt_leranable.clip_model.visual.transformer.resblocks.0.mlp.c_proj.bias", "module.prompt_leranable.clip_model.visual.transformer.resblocks.0.ln_2.weight", "module.prompt_leranable.clip_model.visual.transformer.resblocks.0.ln_2.bias", "module.prompt_leranable.clip_model.visual.transformer.resblocks.1.attn.in_proj_weight", "module.prompt_leranable.clip_model.visual.transformer.resblocks.1.attn.in_proj_bias", "module.prompt_leranable.clip_model.visual.transformer.resblocks.1.attn.out_proj.weight", "module.prompt_leranable.clip_model.visual.transformer.resblocks.1.attn.out_proj.bias", "module.prompt_leranable.clip_model.visual.transformer.resblocks.1.ln_1.weight", "module.prompt_leranable.clip_model.visual.transformer.resblocks.1.ln_1.bias", "module.prompt_leranable.clip_model.visual.transformer.resblocks.1.mlp.c_fc.weight", "module.prompt_leranable.clip_model.visual.transformer.resblocks.1.mlp.c_fc.bias", "module.prompt_leranable.clip_model.visual.transformer.resblocks.1.mlp.c_proj.weight", "module.prompt_leranable.clip_model.visual.transformer.resblocks.1.mlp.c_proj.bias", "module.prompt_leranable.clip_model.visual.transformer.resblocks.1.ln_2.weight", "module.prompt_leranable.clip_model.visual.transformer.resblocks.1.ln_2.bias", "module.prompt_leranable.clip_model.visual.transformer.resblocks.2.attn.in_proj_weight", "module.prompt_leranable.clip_model.visual.transformer.resblocks.2.attn.in_proj_bias", "module.prompt_leranable.clip_model.visual.transformer.resblocks.2.attn.out_proj.weight", "module.prompt_leranable.clip_model.visual.transformer.resblocks.2.attn.out_proj.bias", "module.prompt_leranable.clip_model.visual.transformer.resblocks.2.ln_1.weight", "module.prompt_leranable.clip_model.visual.transformer.resblocks.2.ln_1.bias", "module.prompt_leranable.clip_model.visual.transformer.resblocks.2.mlp.c_fc.weight", "module.prompt_leranable.clip_model.visual.transformer.resblocks.2.mlp.c_fc.bias", "module.prompt_leranable.clip_model.visual.transformer.resblocks.2.mlp.c_proj.weight", "module.prompt_leranable.clip_model.visual.transformer.resblocks.2.mlp.c_proj.bias", "module.prompt_leranable.clip_model.visual.transformer.resblocks.2.ln_2.weight", "module.prompt_leranable.clip_model.visual.transformer.resblocks.2.ln_2.bias", "module.prompt_leranable.clip_model.visual.transformer.resblocks.3.attn.in_proj_weight", "module.prompt_leranable.clip_model.visual.transformer.resblocks.3.attn.in_proj_bias", "module.prompt_leranable.clip_model.visual.transformer.resblocks.3.attn.out_proj.weight", "module.prompt_leranable.clip_model.visual.transformer.resblocks.3.attn.out_proj.bias", "module.prompt_leranable.clip_model.visual.transformer.resblocks.3.ln_1.weight", "module.prompt_leranable.clip_model.visual.transformer.resblocks.3.ln_1.bias", "module.prompt_leranable.clip_model.visual.transformer.resblocks.3.mlp.c_fc.weight", "module.prompt_leranable.clip_model.visual.transformer.resblocks.3.mlp.c_fc.bias", "module.prompt_leranable.clip_model.visual.transformer.resblocks.3.mlp.c_proj.weight", "module.prompt_leranable.clip_model.visual.transformer.resblocks.3.mlp.c_proj.bias", "module.prompt_leranable.clip_model.visual.transformer.resblocks.3.ln_2.weight", "module.prompt_leranable.clip_model.visual.transformer.resblocks.3.ln_2.bias", "module.prompt_leranable.clip_model.visual.transformer.resblocks.4.attn.in_proj_weight", "module.prompt_leranable.clip_model.visual.transformer.resblocks.4.attn.in_proj_bias", "module.prompt_leranable.clip_model.visual.transformer.resblocks.4.attn.out_proj.weight", "module.prompt_leranable.clip_model.visual.transformer.resblocks.4.attn.out_proj.bias", "module.prompt_leranable.clip_model.visual.transformer.resblocks.4.ln_1.weight", "module.prompt_leranable.clip_model.visual.transformer.resblocks.4.ln_1.bias", "module.prompt_leranable.clip_model.visual.transformer.resblocks.4.mlp.c_fc.weight", "module.prompt_leranable.clip_model.visual.transformer.resblocks.4.mlp.c_fc.bias", "module.prompt_leranable.clip_model.visual.transformer.resblocks.4.mlp.c_proj.weight", "module.prompt_leranable.clip_model.visual.transformer.resblocks.4.mlp.c_proj.bias", "module.prompt_leranable.clip_model.visual.transformer.resblocks.4.ln_2.weight", "module.prompt_leranable.clip_model.visual.transformer.resblocks.4.ln_2.bias", "module.prompt_leranable.clip_model.visual.transformer.resblocks.5.attn.in_proj_weight", "module.prompt_leranable.clip_model.visual.transformer.resblocks.5.attn.in_proj_bias", "module.prompt_leranable.clip_model.visual.transformer.resblocks.5.attn.out_proj.weight", "module.prompt_leranable.clip_model.visual.transformer.resblocks.5.attn.out_proj.bias", "module.prompt_leranable.clip_model.visual.transformer.resblocks.5.ln_1.weight", "module.prompt_leranable.clip_model.visual.transformer.resblocks.5.ln_1.bias", "module.prompt_leranable.clip_model.visual.transformer.resblocks.5.mlp.c_fc.weight", "module.prompt_leranable.clip_model.visual.transformer.resblocks.5.mlp.c_fc.bias", "module.prompt_leranable.clip_model.visual.transformer.resblocks.5.mlp.c_proj.weight", "module.prompt_leranable.clip_model.visual.transformer.resblocks.5.mlp.c_proj.bias", "module.prompt_leranable.clip_model.visual.transformer.resblocks.5.ln_2.weight", "module.prompt_leranable.clip_model.visual.transformer.resblocks.5.ln_2.bias", "module.prompt_leranable.clip_model.visual.transformer.resblocks.6.attn.in_proj_weight", "module.prompt_leranable.clip_model.visual.transformer.resblocks.6.attn.in_proj_bias", "module.prompt_leranable.clip_model.visual.transformer.resblocks.6.attn.out_proj.weight", "module.prompt_leranable.clip_model.visual.transformer.resblocks.6.attn.out_proj.bias", "module.prompt_leranable.clip_model.visual.transformer.resblocks.6.ln_1.weight", "module.prompt_leranable.clip_model.visual.transformer.resblocks.6.ln_1.bias", "module.prompt_leranable.clip_model.visual.transformer.resblocks.6.mlp.c_fc.weight", "module.prompt_leranable.clip_model.visual.transformer.resblocks.6.mlp.c_fc.bias", "module.prompt_leranable.clip_model.visual.transformer.resblocks.6.mlp.c_proj.weight", "module.prompt_leranable.clip_model.visual.transformer.resblocks.6.mlp.c_proj.bias", "module.prompt_leranable.clip_model.visual.transformer.resblocks.6.ln_2.weight", "module.prompt_leranable.clip_model.visual.transformer.resblocks.6.ln_2.bias", "module.prompt_leranable.clip_model.visual.transformer.resblocks.7.attn.in_proj_weight", "module.prompt_leranable.clip_model.visual.transformer.resblocks.7.attn.in_proj_bias", "module.prompt_leranable.clip_model.visual.transformer.resblocks.7.attn.out_proj.weight", "module.prompt_leranable.clip_model.visual.transformer.resblocks.7.attn.out_proj.bias", "module.prompt_leranable.clip_model.visual.transformer.resblocks.7.ln_1.weight", "module.prompt_leranable.clip_model.visual.transformer.resblocks.7.ln_1.bias", "module.prompt_leranable.clip_model.visual.transformer.resblocks.7.mlp.c_fc.weight", "module.prompt_leranable.clip_model.visual.transformer.resblocks.7.mlp.c_fc.bias", "module.prompt_leranable.clip_model.visual.transformer.resblocks.7.mlp.c_proj.weight", "module.prompt_leranable.clip_model.visual.transformer.resblocks.7.mlp.c_proj.bias", "module.prompt_leranable.clip_model.visual.transformer.resblocks.7.ln_2.weight", "module.prompt_leranable.clip_model.visual.transformer.resblocks.7.ln_2.bias", "module.prompt_leranable.clip_model.visual.transformer.resblocks.8.attn.in_proj_weight", "module.prompt_leranable.clip_model.visual.transformer.resblocks.8.attn.in_proj_bias", "module.prompt_leranable.clip_model.visual.transformer.resblocks.8.attn.out_proj.weight", "module.prompt_leranable.clip_model.visual.transformer.resblocks.8.attn.out_proj.bias", "module.prompt_leranable.clip_model.visual.transformer.resblocks.8.ln_1.weight", "module.prompt_leranable.clip_model.visual.transformer.resblocks.8.ln_1.bias", "module.prompt_leranable.clip_model.visual.transformer.resblocks.8.mlp.c_fc.weight", "module.prompt_leranable.clip_model.visual.transformer.resblocks.8.mlp.c_fc.bias", "module.prompt_leranable.clip_model.visual.transformer.resblocks.8.mlp.c_proj.weight", "module.prompt_leranable.clip_model.visual.transformer.resblocks.8.mlp.c_proj.bias", "module.prompt_leranable.clip_model.visual.transformer.resblocks.8.ln_2.weight", "module.prompt_leranable.clip_model.visual.transformer.resblocks.8.ln_2.bias", "module.prompt_leranable.clip_model.visual.transformer.resblocks.9.attn.in_proj_weight", "module.prompt_leranable.clip_model.visual.transformer.resblocks.9.attn.in_proj_bias", "module.prompt_leranable.clip_model.visual.transformer.resblocks.9.attn.out_proj.weight", "module.prompt_leranable.clip_model.visual.transformer.resblocks.9.attn.out_proj.bias", "module.prompt_leranable.clip_model.visual.transformer.resblocks.9.ln_1.weight", "module.prompt_leranable.clip_model.visual.transformer.resblocks.9.ln_1.bias", "module.prompt_leranable.clip_model.visual.transformer.resblocks.9.mlp.c_fc.weight", "module.prompt_leranable.clip_model.visual.transformer.resblocks.9.mlp.c_fc.bias", "module.prompt_leranable.clip_model.visual.transformer.resblocks.9.mlp.c_proj.weight", "module.prompt_leranable.clip_model.visual.transformer.resblocks.9.mlp.c_proj.bias", "module.prompt_leranable.clip_model.visual.transformer.resblocks.9.ln_2.weight", "module.prompt_leranable.clip_model.visual.transformer.resblocks.9.ln_2.bias", "module.prompt_leranable.clip_model.visual.transformer.resblocks.10.attn.in_proj_weight", "module.prompt_leranable.clip_model.visual.transformer.resblocks.10.attn.in_proj_bias", "module.prompt_leranable.clip_model.visual.transformer.resblocks.10.attn.out_proj.weight", "module.prompt_leranable.clip_model.visual.transformer.resblocks.10.attn.out_proj.bias", "module.prompt_leranable.clip_model.visual.transformer.resblocks.10.ln_1.weight", "module.prompt_leranable.clip_model.visual.transformer.resblocks.10.ln_1.bias", "module.prompt_leranable.clip_model.visual.transformer.resblocks.10.mlp.c_fc.weight", "module.prompt_leranable.clip_model.visual.transformer.resblocks.10.mlp.c_fc.bias", "module.prompt_leranable.clip_model.visual.transformer.resblocks.10.mlp.c_proj.weight", "module.prompt_leranable.clip_model.visual.transformer.resblocks.10.mlp.c_proj.bias", "module.prompt_leranable.clip_model.visual.transformer.resblocks.10.ln_2.weight", "module.prompt_leranable.clip_model.visual.transformer.resblocks.10.ln_2.bias", "module.prompt_leranable.clip_model.visual.transformer.resblocks.11.attn.in_proj_weight", "module.prompt_leranable.clip_model.visual.transformer.resblocks.11.attn.in_proj_bias", "module.prompt_leranable.clip_model.visual.transformer.resblocks.11.attn.out_proj.weight", "module.prompt_leranable.clip_model.visual.transformer.resblocks.11.attn.out_proj.bias", "module.prompt_leranable.clip_model.visual.transformer.resblocks.11.ln_1.weight", "module.prompt_leranable.clip_model.visual.transformer.resblocks.11.ln_1.bias", "module.prompt_leranable.clip_model.visual.transformer.resblocks.11.mlp.c_fc.weight", "module.prompt_leranable.clip_model.visual.transformer.resblocks.11.mlp.c_fc.bias", "module.prompt_leranable.clip_model.visual.transformer.resblocks.11.mlp.c_proj.weight", "module.prompt_leranable.clip_model.visual.transformer.resblocks.11.mlp.c_proj.bias", "module.prompt_leranable.clip_model.visual.transformer.resblocks.11.ln_2.weight", "module.prompt_leranable.clip_model.visual.transformer.resblocks.11.ln_2.bias", "module.prompt_leranable.clip_model.visual.ln_post.weight", "module.prompt_leranable.clip_model.visual.ln_post.bias", "module.prompt_leranable.clip_model.transformer.resblocks.0.attn.in_proj_weight", "module.prompt_leranable.clip_model.transformer.resblocks.0.attn.in_proj_bias", "module.prompt_leranable.clip_model.transformer.resblocks.0.attn.out_proj.weight", "module.prompt_leranable.clip_model.transformer.resblocks.0.attn.out_proj.bias", "module.prompt_leranable.clip_model.transformer.resblocks.0.ln_1.weight", "module.prompt_leranable.clip_model.transformer.resblocks.0.ln_1.bias", "module.prompt_leranable.clip_model.transformer.resblocks.0.mlp.c_fc.weight", "module.prompt_leranable.clip_model.transformer.resblocks.0.mlp.c_fc.bias", "module.prompt_leranable.clip_model.transformer.resblocks.0.mlp.c_proj.weight", "module.prompt_leranable.clip_model.transformer.resblocks.0.mlp.c_proj.bias", "module.prompt_leranable.clip_model.transformer.resblocks.0.ln_2.weight", "module.prompt_leranable.clip_model.transformer.resblocks.0.ln_2.bias", "module.prompt_leranable.clip_model.transformer.resblocks.1.attn.in_proj_weight", "module.prompt_leranable.clip_model.transformer.resblocks.1.attn.in_proj_bias", "module.prompt_leranable.clip_model.transformer.resblocks.1.attn.out_proj.weight", "module.prompt_leranable.clip_model.transformer.resblocks.1.attn.out_proj.bias", "module.prompt_leranable.clip_model.transformer.resblocks.1.ln_1.weight", "module.prompt_leranable.clip_model.transformer.resblocks.1.ln_1.bias", "module.prompt_leranable.clip_model.transformer.resblocks.1.mlp.c_fc.weight", "module.prompt_leranable.clip_model.transformer.resblocks.1.mlp.c_fc.bias", "module.prompt_leranable.clip_model.transformer.resblocks.1.mlp.c_proj.weight", "module.prompt_leranable.clip_model.transformer.resblocks.1.mlp.c_proj.bias", "module.prompt_leranable.clip_model.transformer.resblocks.1.ln_2.weight", "module.prompt_leranable.clip_model.transformer.resblocks.1.ln_2.bias", "module.prompt_leranable.clip_model.transformer.resblocks.2.attn.in_proj_weight", "module.prompt_leranable.clip_model.transformer.resblocks.2.attn.in_proj_bias", "module.prompt_leranable.clip_model.transformer.resblocks.2.attn.out_proj.weight", "module.prompt_leranable.clip_model.transformer.resblocks.2.attn.out_proj.bias", "module.prompt_leranable.clip_model.transformer.resblocks.2.ln_1.weight", "module.prompt_leranable.clip_model.transformer.resblocks.2.ln_1.bias", "module.prompt_leranable.clip_model.transformer.resblocks.2.mlp.c_fc.weight", "module.prompt_leranable.clip_model.transformer.resblocks.2.mlp.c_fc.bias", "module.prompt_leranable.clip_model.transformer.resblocks.2.mlp.c_proj.weight", "module.prompt_leranable.clip_model.transformer.resblocks.2.mlp.c_proj.bias", "module.prompt_leranable.clip_model.transformer.resblocks.2.ln_2.weight", "module.prompt_leranable.clip_model.transformer.resblocks.2.ln_2.bias", "module.prompt_leranable.clip_model.transformer.resblocks.3.attn.in_proj_weight", "module.prompt_leranable.clip_model.transformer.resblocks.3.attn.in_proj_bias", "module.prompt_leranable.clip_model.transformer.resblocks.3.attn.out_proj.weight", "module.prompt_leranable.clip_model.transformer.resblocks.3.attn.out_proj.bias", "module.prompt_leranable.clip_model.transformer.resblocks.3.ln_1.weight", "module.prompt_leranable.clip_model.transformer.resblocks.3.ln_1.bias", "module.prompt_leranable.clip_model.transformer.resblocks.3.mlp.c_fc.weight", "module.prompt_leranable.clip_model.transformer.resblocks.3.mlp.c_fc.bias", "module.prompt_leranable.clip_model.transformer.resblocks.3.mlp.c_proj.weight", "module.prompt_leranable.clip_model.transformer.resblocks.3.mlp.c_proj.bias", "module.prompt_leranable.clip_model.transformer.resblocks.3.ln_2.weight", "module.prompt_leranable.clip_model.transformer.resblocks.3.ln_2.bias", "module.prompt_leranable.clip_model.transformer.resblocks.4.attn.in_proj_weight", "module.prompt_leranable.clip_model.transformer.resblocks.4.attn.in_proj_bias", "module.prompt_leranable.clip_model.transformer.resblocks.4.attn.out_proj.weight", "module.prompt_leranable.clip_model.transformer.resblocks.4.attn.out_proj.bias", "module.prompt_leranable.clip_model.transformer.resblocks.4.ln_1.weight", "module.prompt_leranable.clip_model.transformer.resblocks.4.ln_1.bias", "module.prompt_leranable.clip_model.transformer.resblocks.4.mlp.c_fc.weight", "module.prompt_leranable.clip_model.transformer.resblocks.4.mlp.c_fc.bias", "module.prompt_leranable.clip_model.transformer.resblocks.4.mlp.c_proj.weight", "module.prompt_leranable.clip_model.transformer.resblocks.4.mlp.c_proj.bias", "module.prompt_leranable.clip_model.transformer.resblocks.4.ln_2.weight", "module.prompt_leranable.clip_model.transformer.resblocks.4.ln_2.bias", "module.prompt_leranable.clip_model.transformer.resblocks.5.attn.in_proj_weight", "module.prompt_leranable.clip_model.transformer.resblocks.5.attn.in_proj_bias", "module.prompt_leranable.clip_model.transformer.resblocks.5.attn.out_proj.weight", "module.prompt_leranable.clip_model.transformer.resblocks.5.attn.out_proj.bias", "module.prompt_leranable.clip_model.transformer.resblocks.5.ln_1.weight", "module.prompt_leranable.clip_model.transformer.resblocks.5.ln_1.bias", "module.prompt_leranable.clip_model.transformer.resblocks.5.mlp.c_fc.weight", "module.prompt_leranable.clip_model.transformer.resblocks.5.mlp.c_fc.bias", "module.prompt_leranable.clip_model.transformer.resblocks.5.mlp.c_proj.weight", "module.prompt_leranable.clip_model.transformer.resblocks.5.mlp.c_proj.bias", "module.prompt_leranable.clip_model.transformer.resblocks.5.ln_2.weight", "module.prompt_leranable.clip_model.transformer.resblocks.5.ln_2.bias", "module.prompt_leranable.clip_model.transformer.resblocks.6.attn.in_proj_weight", "module.prompt_leranable.clip_model.transformer.resblocks.6.attn.in_proj_bias", "module.prompt_leranable.clip_model.transformer.resblocks.6.attn.out_proj.weight", "module.prompt_leranable.clip_model.transformer.resblocks.6.attn.out_proj.bias", "module.prompt_leranable.clip_model.transformer.resblocks.6.ln_1.weight", "module.prompt_leranable.clip_model.transformer.resblocks.6.ln_1.bias", "module.prompt_leranable.clip_model.transformer.resblocks.6.mlp.c_fc.weight", "module.prompt_leranable.clip_model.transformer.resblocks.6.mlp.c_fc.bias", "module.prompt_leranable.clip_model.transformer.resblocks.6.mlp.c_proj.weight", "module.prompt_leranable.clip_model.transformer.resblocks.6.mlp.c_proj.bias", "module.prompt_leranable.clip_model.transformer.resblocks.6.ln_2.weight", "module.prompt_leranable.clip_model.transformer.resblocks.6.ln_2.bias", "module.prompt_leranable.clip_model.transformer.resblocks.7.attn.in_proj_weight", "module.prompt_leranable.clip_model.transformer.resblocks.7.attn.in_proj_bias", "module.prompt_leranable.clip_model.transformer.resblocks.7.attn.out_proj.weight", "module.prompt_leranable.clip_model.transformer.resblocks.7.attn.out_proj.bias", "module.prompt_leranable.clip_model.transformer.resblocks.7.ln_1.weight", "module.prompt_leranable.clip_model.transformer.resblocks.7.ln_1.bias", "module.prompt_leranable.clip_model.transformer.resblocks.7.mlp.c_fc.weight", "module.prompt_leranable.clip_model.transformer.resblocks.7.mlp.c_fc.bias", "module.prompt_leranable.clip_model.transformer.resblocks.7.mlp.c_proj.weight", "module.prompt_leranable.clip_model.transformer.resblocks.7.mlp.c_proj.bias", "module.prompt_leranable.clip_model.transformer.resblocks.7.ln_2.weight", "module.prompt_leranable.clip_model.transformer.resblocks.7.ln_2.bias", "module.prompt_leranable.clip_model.transformer.resblocks.8.attn.in_proj_weight", "module.prompt_leranable.clip_model.transformer.resblocks.8.attn.in_proj_bias", "module.prompt_leranable.clip_model.transformer.resblocks.8.attn.out_proj.weight", "module.prompt_leranable.clip_model.transformer.resblocks.8.attn.out_proj.bias", "module.prompt_leranable.clip_model.transformer.resblocks.8.ln_1.weight", "module.prompt_leranable.clip_model.transformer.resblocks.8.ln_1.bias", "module.prompt_leranable.clip_model.transformer.resblocks.8.mlp.c_fc.weight", "module.prompt_leranable.clip_model.transformer.resblocks.8.mlp.c_fc.bias", "module.prompt_leranable.clip_model.transformer.resblocks.8.mlp.c_proj.weight", "module.prompt_leranable.clip_model.transformer.resblocks.8.mlp.c_proj.bias", "module.prompt_leranable.clip_model.transformer.resblocks.8.ln_2.weight", "module.prompt_leranable.clip_model.transformer.resblocks.8.ln_2.bias", "module.prompt_leranable.clip_model.transformer.resblocks.9.attn.in_proj_weight", "module.prompt_leranable.clip_model.transformer.resblocks.9.attn.in_proj_bias", "module.prompt_leranable.clip_model.transformer.resblocks.9.attn.out_proj.weight", "module.prompt_leranable.clip_model.transformer.resblocks.9.attn.out_proj.bias", "module.prompt_leranable.clip_model.transformer.resblocks.9.ln_1.weight", "module.prompt_leranable.clip_model.transformer.resblocks.9.ln_1.bias", "module.prompt_leranable.clip_model.transformer.resblocks.9.mlp.c_fc.weight", "module.prompt_leranable.clip_model.transformer.resblocks.9.mlp.c_fc.bias", "module.prompt_leranable.clip_model.transformer.resblocks.9.mlp.c_proj.weight", "module.prompt_leranable.clip_model.transformer.resblocks.9.mlp.c_proj.bias", "module.prompt_leranable.clip_model.transformer.resblocks.9.ln_2.weight", "module.prompt_leranable.clip_model.transformer.resblocks.9.ln_2.bias", "module.prompt_leranable.clip_model.transformer.resblocks.10.attn.in_proj_weight", "module.prompt_leranable.clip_model.transformer.resblocks.10.attn.in_proj_bias", "module.prompt_leranable.clip_model.transformer.resblocks.10.attn.out_proj.weight", "module.prompt_leranable.clip_model.transformer.resblocks.10.attn.out_proj.bias", "module.prompt_leranable.clip_model.transformer.resblocks.10.ln_1.weight", "module.prompt_leranable.clip_model.transformer.resblocks.10.ln_1.bias", "module.prompt_leranable.clip_model.transformer.resblocks.10.mlp.c_fc.weight", "module.prompt_leranable.clip_model.transformer.resblocks.10.mlp.c_fc.bias", "module.prompt_leranable.clip_model.transformer.resblocks.10.mlp.c_proj.weight", "module.prompt_leranable.clip_model.transformer.resblocks.10.mlp.c_proj.bias", "module.prompt_leranable.clip_model.transformer.resblocks.10.ln_2.weight", "module.prompt_leranable.clip_model.transformer.resblocks.10.ln_2.bias", "module.prompt_leranable.clip_model.transformer.resblocks.11.attn.in_proj_weight", "module.prompt_leranable.clip_model.transformer.resblocks.11.attn.in_proj_bias", "module.prompt_leranable.clip_model.transformer.resblocks.11.attn.out_proj.weight", "module.prompt_leranable.clip_model.transformer.resblocks.11.attn.out_proj.bias", "module.prompt_leranable.clip_model.transformer.resblocks.11.ln_1.weight", "module.prompt_leranable.clip_model.transformer.resblocks.11.ln_1.bias", "module.prompt_leranable.clip_model.transformer.resblocks.11.mlp.c_fc.weight", "module.prompt_leranable.clip_model.transformer.resblocks.11.mlp.c_fc.bias", "module.prompt_leranable.clip_model.transformer.resblocks.11.mlp.c_proj.weight", "module.prompt_leranable.clip_model.transformer.resblocks.11.mlp.c_proj.bias", "module.prompt_leranable.clip_model.transformer.resblocks.11.ln_2.weight", "module.prompt_leranable.clip_model.transformer.resblocks.11.ln_2.bias", "module.prompt_leranable.clip_model.token_embedding.weight", "module.prompt_leranable.clip_model.ln_final.weight", "module.prompt_leranable.clip_model.ln_final.bias", "module.prompt_leranable.meta_net.linear1.weight", "module.prompt_leranable.meta_net.linear1.bias", "module.prompt_leranable.meta_net.linear2.weight", "module.prompt_leranable.meta_net.linear2.bias". 

It seems like something (maybe the "SpecificTextualPrompt" module) can't not be loaded into the model. Then I try to correct the error by utilizing the following code:

EventCLIP.load_state_dict(torch.load(cfg['MODEL']['Load_Path'], map_location={'cuda:2': 'cuda:0'}), strict=False)

The errors disappear. However, I wonder whether it will influence the performance of the model?

Can you help me fix the code?

Hoantrbl commented 3 weeks ago

When I try to retrain the mdoel based on your checkpoint and save the better model pickle, I find that I can load my pickle successfully. I think it might be your checkpoint wrong.

Can you upload the lastest checkpoint?

jiazhou-garland commented 2 weeks ago

Hello, this error comes from the config file, where the learnable ['MODEL']['TextEncoder']['leranable_ctx'] is needed to be True. I have updated the config file, please try to load the checkpoint again.

Hoantrbl commented 2 weeks ago

Thanks a lot!

In the meantime, I find that some errors occurs in your paper experiments.

In EventCLIP, it utilizes the VIT-B/16 for fine-tuning experiments.

image

However, in your paper, you said it's VIT-L/14, which may causes the unfair comparision and wrong analyisis in Model parameter size analysis.

image

I strongly recommend the author should make more clear explanation about it.

jiazhou-garland commented 2 weeks ago

Please refer to the latest version of EventCLIP, page 5, Our Implementation Details, where they said 'For the pre-trained CLIP, we adopt the variant with the ViT-L/14 [10] image encoder'.

Hoantrbl commented 2 weeks ago

Please refer to the latest version of EventCLIP, page 5, Our Implementation Details, where they said 'For the pre-trained CLIP, we adopt the variant with the ViT-L/14 [10] image encoder'.

You can refer to https://github.com/Wuziyi616/EventCLIP/issues/5#issuecomment-2306389902 for EventCLIP's response.