search

sibeiyang / sgmn

Graph-Structured Referring Expressions Reasoning in The Wild, In CVPR 2020, Oral.
MIT License
116 stars 15 forks source link

accuracy of DGA #9

Open freedom6927 opened 3 years ago

freedom6927 commented 3 years ago

I use your released code to train dga model on the environment of python3 and torch 1.7. Then, i used the trained paratmeters to test your released dataset(test_expressions.json), the accuracy is 53.22%. However, the accuracy your write in papar is 86.64%, 84.79%, 78.31%, 80.21%, 80.26%(testA and testB of Refcoco and refcoco+ and test of recocog) , avg is about 79%. The number of test samples in your released test dataset is 34000+, and the number of all test samples of recoco,recoco+ and recocog is 5657+5059 +5726+4889+9602=30933(section 4 in paper dga). So i guess all test samples of recoco,recoco+ and recocog are included in your released test dataset. But the accuracy 53.22% is much lower than 79%. why is that. For the training, all parameters are the same as your proposed paper(Dynamic Graph Attention for Referring Expression Comprehension)

zpfinley commented 3 years ago

@freedom6927 @sibeiyang this project, the author provided the Ref-Reasoning data set. 53% of the results were on the Ref-Reasoning data set. The author did not provide RefCOCO, RefCOCO+ and RefCOCOg Datasets, but provided a link with bottom-up-attention to process . Similarly, I also hope that the author can provide processed RefCOCO, RefCOCO+ and RefCOCOg Datasets, and their split

wowxqh commented 3 years ago

Hello, author, I found that your codes cannot reproduce the results in your paper. I can't reproduce refcoco's paper data with different pytorch versions and different parameter settings. Hope explanation. Thanks.

bareblackfoot commented 3 years ago

I couldn't reproduce the sgmn on the ref-reasoning dataset, too. The accuracy was 39.64% on the ref-reasoning test set after 10 epochs of training, while the paper says it produces 51.39% accuracy.