search

baeseongsu / ehrxqa

EHRXQA: A Multi-Modal Question Answering Dataset for Electronic Health Records with Chest X-ray Images
MIT License
57 stars 3 forks source link

Access to the prediction results on the test set #10

Closed nooralahzadeh closed 2 weeks ago

nooralahzadeh commented 3 weeks ago

Hi, We want to compare the results from our method to the results that you have in the test set. Is there any possibility of sharing the output of your method on the test set?

Thanks, Farhad

baeseongsu commented 3 weeks ago

Hi, @nooralahzadeh

Do you want the experimental results for the executed NeuralSQL (i.e., the answer itself) generated by prompting BM25 (train), which is the third row in Table 5?

Best, Seongsu

nooralahzadeh commented 3 weeks ago

Hi @baeseongsu Thanks for the prompt response. It would be great to have the generated NeuralSQL and the execution results for ChatGPT+ M3AE (both Fixed and BM25) in the test set!

Best, Farhad

baeseongsu commented 3 weeks ago

Hi @nooralahzadeh,

I will provide those generated and executed results within this week. I need to double-check whether the results are exactly the same as the ones used in the paper.

Best, Seongsu

baeseongsu commented 2 weeks ago

Hi @nooralahzadeh,

Sorry for the delayed response :(

I've uploaded the generated NeuralSQL and execution results for ChatGPT + M3AE (both fixed and BM25) in the test dataset via Dropbox link. Here is the link: https://www.dropbox.com/scl/fo/2tuz46h0nv4s4felfuu7j/AC5iaC5ooh4xY0HCR2S8gQc?rlkey=1zsfvkj4jpy3qbbogp3l3g9qk&st=2e14k2zj&dl=0

Note that this is the file I used in the arXiv submission. However, since execution results depend on the syntax in SQL grammar (e.g., "How many days" can be computed as Julian day (float) or just integer), I cannot guarantee 100% identical execution results. Nevertheless, they are very similar to the original experimental results (actually even better because I updated some postprocessing and executor parts after the publication).

If you have any questions about the detailed implementation, please feel free to ask.

nooralahzadeh commented 2 weeks ago

Dear @baeseongsu Thanks for your effort and for providing the experiment results. It is indeed valuable in our work. I recommend putting a link to this in your GitHub repository to share with other researchers.

Best, Farhad