How to use Mammo-CLIP model easily

[Iamnotphage]

Hi, Thank you so much for your great work on this project.

I am a computer science undergraduate working on my school project.

Anyway, I want to ask how to use the trained model easily.

I want to design a program that will input a mammogram (PNG format) and then give it to the model, and then output the probability of each label (such as mass and suspicious_calcification for vindr dataset).

Let me talk about my current progress:

I downloaded the preprocessed PNG dataset (vindr), and modified the path of zs_clip.yaml and vindr_zs.yaml (I commented rsna_zs). I also downloaded the Efficient-Net B5 model, and finally ran eval_zero_shot_clip.py successfully.

But I still have questions, even though it runs smoothly.

1. Why is there only one line in the output results-vindr.json: {"D:\\MyDownload\\b5-model-best-epoch-7.tar": {"mass": 0.48817351418390853, "suspicious_calcification": 0.4631887699366397}}
2. If I want to achieve what I said before, take the image as input and output probability of various labels, how should I do it?

Because I have not officially entered this field, there may be some basic questions, please understand.

Thank you for your time to read this issue

[shantanu-ai]

@Iamnotphage You get one line in the output of eval_zero_shot_clip.py because you are calculating the model's zero-shot accuracy. If you want to give a mammogram and get a logit or proba, you must use Mammo-CLIP as a classifier. Follow instructions here and linear probe/finetuning here. Basically, create a classifier and train it using linear probes or fine-tuning. Follow the hugging face paths for pretrained mammo-clip classifiers; we have uploaded classifier checkpoints. Also, preprocess the mammograms as per mammo-clip as instructed here. If you do not complete preprocessing, you will get random results.