search

Stanford-AIMI / RRG24

Shared task on Large-Scale Radiology Report Generation @ BioNLP ACL'24
https://stanford-aimi.github.io/RRG24/
MIT License
8 stars 0 forks source link

Question about generating empty findings/impression #8

Closed oscarloch closed 6 months ago

oscarloch commented 6 months ago

Hi! Given the newly released test set:

Q1: When we are evaluated for the findings section of the leaderboard, do we have to generate findings for all the reports in the test, or only those for which findings are present?

Q2: In the case of all reports, should our models predict an empty findings section?

Thank you so much for clarifying!

jbdel commented 6 months ago

For test-public and test-hidden, only generate findings or impression for an image when the findings or impression string exists. For public, you have the ground truth, for hidden, the string is "to_generate".

Example:

import datasets
dataset = datasets.load_dataset("StanfordAIMI/interpret-cxr-test-hidden/")
findings_to_generate = [s for s in dataset["test"] if s["findings"]]