search

OFA-Sys / OFA

Official repository of OFA (ICML 2022). Paper: OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework
Apache License 2.0
2.41k stars 248 forks source link

referring expression comprehension number do not match #289

Open joellliu opened 1 year ago

joellliu commented 1 year ago

Hi,

Thanks for sharing the great work. I ran the inference code for referring expression comprehension, but it looks like the numbers I got do not match what's reported here on RefCOCO+/g, although the numbers are basically the same for RefCOCO.

RefCOCO: 90.05/92.93/85.26 (reported) vs. 90.05/92.91/85.28 RefCOCO+: 85.8/89.87/79.22 (reported) vs. 84.45/90.08/77.75 RefCOCOg: 85.89/86.55 (reported) vs. 84.52/85.2

I followed the instruction in readme https://github.com/OFA-Sys/OFA#visual-grounding-referring-expression-comprehension and I'm using Tesla T4 gpus.

Do you know what might be wrong? Thank you so much for your help!

logicwong commented 1 year ago

@joellliu Thanks for the reminder, these ckpts correspond to the results of the first version of the paper, and we didn't update these ckpts in time when we updated the paper. We will upload the latest ckpts later.

joellliu commented 1 year ago

Ok thanks