search

aioz-ai / MICCAI21_MMQ

Multiple Meta-model Quantifying for Medical Visual Question Answering (MICCAI 2021)
https://blog.ai.aioz.io/research/vqa-mmq/
MIT License
35 stars 13 forks source link

The problem of the accuracy #3

Closed PengPeixi closed 3 years ago

PengPeixi commented 3 years ago

Hello! I have tried to run the command line "sh run_vqa_PathVQA.sh" and then run "sh run_test_PathVQA.sh", but the total accuracy is 46.6%. I have tried it for many times, but the total accuracy I get is always around 46.6%.Can you please tell me how can I get the total accuracy on 'MMQ + MEVF' which is 49.0%?

xuanbinh-nguyen96 commented 3 years ago

Dear PengPeixi,

After reading your concern, I have cloned and re-run the codes by following all instructions in our git repo on two different hardware setups. (1) Intel Xeon E5-2690 v4 CPU and Nvidia GeForce GTX 1080 Ti GPU 11Gb RAM. (2) AMD Ryzen Threadripper 1950X 16-Core Processor CPU and Nvidia GeForce TITAN V GPU 12 Gb RAM.

The evaluation logs in the train set and the validate set, the pretrained models on PathVQA of both mentioned setups can be found in: (link) The testing result in the PathVQA test set is 48.96% in setup (1) and 48.23% in setup (2), respectively. The results in the test set still meet the claim in our previous issue response: the experimental error is approximately 1.58%. We hope that some comments of us in this issue can help: https://github.com/aioz-ai/MICCAI21_MMQ/issues/2

PengPeixi commented 3 years ago

Thank you very much for your reply! I will try to change the initial seed. And when I unzip or open the zip file which is downloaded from the link you supply, some errors occur. Could you please upload the zip file again?

xuanbinh-nguyen96 commented 3 years ago

Thank you very much for your reply! I will try to change the initial seed. And when I unzip or open the zip file which is downloaded from the link you supply, some errors occur. Could you please upload the zip file again?

I have updated the zip file. You can download and use them.

PengPeixi commented 3 years ago

Thank you very much for your generous help!