Evaluation result on OK-VQA

[gyjung975]

Hi, thanks for your interesting work.

I wonder how you evaluated your final results of OK-VQA.

Paper says it was evaluated with soft accuracy of VQAv2, so I tried to evaluate your OKVQA_val_gpt3.json in Evaluation Logs using VQA evalutation code. https://github.com/GT-Vision-Lab/VQA

It shows 58.89 score, but 60.4 in paper. I used "mscoco_val2014_annotations.json" file in OK-VQA website for annotation file.

Didn't you use the VQA evaluation code? or the log files are not final result?

Thank you

[Yushi-Hu]

Hi,

Thanks for pointing out! We have uploaded the wrong log. I have uploaded the correct one. Would you please help check if the number is correct this time? Thanks!

[gyjung975]

It works well! Thank you

Can I also get logs for VQAv2 dataset?

[Yushi-Hu]

Sorry that we are using the VQAv2 test-dev set, it does not come with gold answers (it is evaluated on Eval.ai) so we cannot share that in Github. I can send it via email if you need it. Would you please drop me an email so that I know where to send it?

[gyjung975]

gyjung975@uos.ac.kr Thank you for sending it here.

[Yushi-Hu]

Just sent!