About generating VQA test result files

salesforce / ALBEF

Code for ALBEF: a new vision-language pre-training method

BSD 3-Clause "New" or "Revised" License

1.53k stars 195 forks source link

About generating VQA test result files #87

Open yang178908 opened 2 years ago

yang178908 commented 2 years ago

Hello! How do I generate the VQA test result file? Do I add --evaluate to the finetune command, and replace the checkpoint file with the finetuned weights? Why all the answers in my generated result file are no in this way?

LiJunnan1992 commented 2 years ago

"Do I add --evaluate to the finetune command, and replace the checkpoint file with the finetuned weights?"

Yes that is how to generate result.

sanyalsunny111 commented 2 years ago

After running --evaluate over the finetuned VQA checkpoint the result's json file is saved in output/vqa/result/vqa_result_epoc0_rank0. How to get a score from this json file. The VQA section of the repo point 5 says "evaluate the result using evaluation server" what does this mean? How to get VQA evaluation score with and without evaluation server.