search

jbdel / vilmedic

ViLMedic (Vision-and-Language medical research) is a modular framework for vision and language multimodal research in the medical field
MIT License
151 stars 20 forks source link

Possibilities for including the evaluation and some data pre-processing code in vilmedic #2

Closed tzcskys closed 2 years ago

tzcskys commented 2 years ago

Hi Jean-Benoit Delbrouck @jbdel ,

Really thanks for your sharing of this wonderful job. It helps a lot for beginners like me to learn and test some baseline works quickly. Also, the code is well organized and I learned so much from it. Thanks!

I want to ask you if there is any possibility to include the code for the evaluation of these works? For example, the linear evaluation and img-text cross-modal retrieval evaluation in ConVIRT and GLoRIA. I know we can find those codes in other natural img-text repositories and modify them to fit into this project. But it would be great and super cool to have this official support from you. I think this may help all vilmedic users to try the whole vilmedic pipeline (including the pre-training and the evaluation) more easily and make vilmedic a more fundamental project for medical img-text research.

Additionaly for the evaluation, some may need to pre-process the data, e.g., the CheXpert 8x200 used in ConVIRT and CheXpert 5x200 used in GLoRIA. I noticed that the vilmedic kindly provides the code to plot learned representations but it requires using the dataset that returns labels. For example, the official doc use label.tok when plotting convirt representations on mimic-cxr. However, I can not find a certain way to generate this label.tok. It would be great if you can also include this part in vilmedic.

Thanks again for your great job and generous sharing. Regards.

jbdel commented 2 years ago

Hello!

Thank you for your interest. I do not intend to code (for now) the retrieval and linear evaluation of the pretrained cross-modal methods.

Thanks for pointing out the missing label files. You can now generate the label file (compatible with vilmedic) of any vilmedic report file here: https://github.com/jbdel/vilmedic/tree/main/data/make_datasets/mimic_cxr Please refer to the README on how to use the get_chexbert_label script

Do not hesitate to reach out by email if you have any further inquiries.

Best,

JB

tzcskys commented 2 years ago

Thanks for your reply and update. Cheers.