search

allenai / aokvqa

Official repository for the A-OKVQA dataset
Apache License 2.0
58 stars 5 forks source link

there is different performance from the paper #7

Closed huangsiyong closed 4 months ago

huangsiyong commented 1 year ago

Hi, it is an interesting work. However, I find something different from the your paper. I try to evaluate the model loaded the provided pretrained weight. I find the DA acc higher a little than the paper show one but all the MC acc is one point lower. What's wrong? Is it any change about the dataset?

apoorvkh commented 1 year ago

Hi @huangsiyong, thanks for bringing this up. We did actually make some small changes to the dataset between our last post to arxiv and publication. Apologies for the delay in revision, but we will update the arxiv release very soon.

apoorvkh commented 4 months ago

I think we can defer to the published ECCV paper, with updated numbers, instead:

https://www.ecva.net/papers/eccv_2022/papers_ECCV/papers/136680141.pdf

And I've updated the link in this repo's readme.