Can not reproduce results on msr-vtt dataset

[qimw]

Hi, I am trying to reproduce the msr-vtt retrieval benchmark results on the full test set. I download the pre-trained weighted and config from the msr-vtt benchmark sheet and only change the split_name to full-test in data_loader. Then I perform evaluation by python test.py --config output/config.json --resume output/trained_model.pth --device 0 However, I get lower results:

How can I get the same results as the benchmark sheet ? Thank you!

[ioanacroi]

Hi,

Can you please give us more details? Which config and model are you using? Also one of the problem might be that you are missing the argument --eval_from_training_config.

Have a nice day!

[qimw]

Hi, Sorry for the delay, I used the config and model from first row of msrvtt benchmark form and specify config with --config argument. I didn't use --eval_from_training_config argument. I have solved this problem by retraining the model 😀

[qimw]

Hi, where can I find the jsfusion_val_caption_idx.pkl?

[ioanacroi]

Hi,

Sorry for the late reply! Until we will update the archives, please find it here: wget https://www.robots.ox.ac.uk/~vgg/research/teachtext/data-hq/high-quality/jsfusion_val_caption_idx.pkl

Cheers, Ioana