search

GT-Vision-Lab / VQA

Other
363 stars 139 forks source link

evaluation script gives wrong accuracy #7

Open ovidiunitu opened 6 years ago

ovidiunitu commented 6 years ago

While testing my solution I noticed this odd behavior. (See the picture below)

image As you can see, my generated answer is 'none'. According to the evaluation metric the correct accuracy should be 30% because there is one answer the same as mine. I think this is happening because of the processing done before evaluation. In file vqaEval.py line 42, the answer 'none' is replaced with '0' . Because there is not any '0' in ground truth answers, the accuracy is set to 0.00%. If I remove 'none': '0', from manualMap dictionary I get the right accuracy for this question (30%).

If it helps, the id of this question is 411188011, and the name of the picture is COCO_val2014_000000411188.jpg

Can you look more into it? I hope I didn't miss anything.

AishwaryaAgrawal commented 6 years ago

Thanks for bringing up the issue and looking into potential reason! I will look more into it and get back to you.

guoyang9 commented 5 years ago

BTW, should we replace all the number words to digits, e.g., 'one on left' to '1 on left'?

Another concern is that some of the answers in the Annotations are just 'a' or 'the', is it appropriate to just delete these ones?

Hope I didn't mistake them.

baiyuting commented 5 months ago

did it get solved?