About the evaluation of the provided SLT checkpoints

[ycmin95]

Thanks for sharing the SLRT codebase, and I want evaluate the provided checkpoints in this repo.

I following the readme to setup the environment and download the checkpoints, and when I evaluate the checkpoints, it reports the weights size mismatch error, as shown below:

Similar problem also appears in the two-stream setting:

I try to locate the origin of this error, based on the prune_embedding.ipynb, the final classifier should be torch.Size([6386, 1024]) and the gloss_embeddings should be 2009. However, the load weights are ([6308, 1024]) and 2004, is anything here I misunderstand?

Thanks for your time and looking for your reply~

[ChenYutongTHU]

Thanks for raising this issue.

We changed the prune_embedding.ipynb a bit when preparing the released code, increasing the number of glosses from 2004 to 2009.

For evaluating the downloaded model, please use the gloss_embeddings.bin provided here.

You need to retrain a model if you use the gloss_embeddings.bin generated by yourself. Try again and see if you can reproduce the results. Thanks.

[ycmin95]

Thanks for your quick reply, I successfully conducted the evaluation with the provided embedding.