Reproducibility issue: RefCOCO Test B has lower accuracy than reported

[OscarWang114]

Hi, thank you for the great work!

Similar to the last comment in #10 , we only achieved ~75 accuracy on RefCOCO TestB following the instructions in this repo and using the provided checkpoints. Details:

RefCOCO Reported 80.32 (Val) 82.67 (Test A) 78.12 (Test B) RefCOCO Reproduced 80.18 (Val) 82.53 (Test A) 75.24 (Test B)

Could you provide the training instructions/checkpoints that achieved ~78 accuracy, or update the result in your paper?

[djiajunustc]

Hi @OscarWang114 ,

That's a nice suggestion since I re-implement this repository before releasing it. The results of TransVG on RefCOCO and RefCOCO+ are somewhat different from those reported in our paper. I'll update the new performance, and suggest the following work to compare with these results.

Best Regards, Jiajun Deng

[OscarWang114]

Hi @djiajunustc,

Thank you for the reply. I look forward to the update! 😃

[djiajunustc]

Hi @OscarWang114 ,

Please wait for another week. I'll update the arXiv version together with the re-produced performance of that with R-101 backbone. I'm training these models now.

Best, Jiajun Deng