search

Cyanogenoid / pytorch-vqa

Strong baseline for visual question answering
238 stars 97 forks source link

Mismatch in Computing Accuracy #22

Open nikhilkarnwal opened 4 years ago

nikhilkarnwal commented 4 years ago

According to the code from official VQA api , accuracy is accounted only when agreeing answer is not same as discarded because in line 98, it is computing other GT answers which are not same as discarded one. But it varies from what you have done. you are assuming that it will be matched to all but discarded answer while in their API, they are computing it as if it will be matched to all but discarded and other which are same as discarded answers. So, seems like it will (10-agreeing)*min(agreeing/3,1).

Please correct me if I am wrong.

@Cyanogenoid @pplantinga @guoyang9

Cyanogenoid commented 4 years ago

I'm having a hard time understanding what you mean. Can you give a concrete example?