Unable to get the correct performance with model zoo models

[justin557]

Hi ruotian,

I did a couple of experiments by evaluating your model zoo models. For example, by running: python tools/eval.py --dump_images 0 --num_images 5000 --model models/trans_self_crit/model-best.pth --infos_path models/trans_self_crit/infos_trans_nscl-best.pkl --language_eval 1 --beam_size 5 I'm only able to a CIDEr score of 124 (130 reported).

Similar things also happen for other model zoo models. Around 5-7 points difference. (For both the transformer models and butd models) BTW, beam size of 2-3 is better than beam size 5.

I have been debugging for a few days. It's not the pytorch version problem. I tried both pytorch 1.7 and 1.4. Am I using the right command? What do you think could be the cause?

[ruotianluo]

That is weird. I have tested again, with freshly pulled repo, and got the same result.

Can you compare to your result with the result here https://drive.google.com/file/d/1Twl1nmacsmXtYGq_NZMWoh0hPT_cUtqh/view?usp=sharing.

[justin557]

My result is different. For BLEU1 I got 0.79 something.

The predicted sentences are different also.

My version is also freshly pulled and almost all files are from your google drive.

I'm using pytorch 1.7 python 3.8 in docker.

[ruotianluo]

I am using 1.5 and py3.7. I doubt that would bring so much difference.

[justin557]

Alright, thank you. I'll try to downgrade my python version and test again.

[justin557]

I finally find out why.

It's because I used 36 features instead of 10-100 features. I guess the model zoo models are trained with 10-100 features That's why if 36 features are used, I get lower performance.

ruotian, you might add one or two lines in the model zoo descriptions, such that other people won't have similar problem. Thank you for your help again.

[ruotianluo]

Good point.