Low Quality of Pseudo Masks in Zero-Shot Setting for Breast Tumor Dataset

[lgy112112]

Hello team,

Firstly, I’d like to thank you for sharing your impressive work and code! I’m reaching out because I encountered an issue while reproducing the zero-shot pseudo masks, specifically for the breast tumor dataset.

Steps Taken

1. I followed your instructions to set up the dataset (breast tumor) and placed it in the expected directories.
2. I downloaded the weights and ensured that the zeroshot.sh script ran without any issues.
3. For prompts, I used those suggested in the paper to ensure alignment with your methodology.

Issue Details

Despite following the prescribed setup, I’m observing that the quality of the generated pseudo masks is quite low:

• Dice Similarity Coefficient (DSC): ~0.25 on average, with a high variance.

Given the above, I wanted to confirm if there’s anything I might have missed in the setup. Additionally, would it be possible for you to provide a Docker environment or a complete pipeline configuration to ensure all dependencies and parameters are in sync?

It is easy to see it in my studio: https://lightning.ai//vision-model/studios/fucking-medclip-samv2/code?turnOn=true

cd MedCLIP-SAMv2
bash zeroshot.sh

Thank you for your support!

[lgy112112]

Still can not figure it out. Out come:

Config:

Already set prompt: breast tumor. I notice that when I change it as essay did to be longer, outcome would be worse :(

Files all set:

[TahaKoleilat]

Hi @lgy112112,

Thank you for your interest in our work.

Could you clarify if the results you're seeing are from the BUSI or UDIAT dataset? In our paper, all results were reported on the testing sets across the four imaging modalities. Specifically for breast tumors, we evaluated on the UDIAT testing set, while BUSI served as the training set for our weakly supervised method.

For BUSI, we experimented with six prompt designs and selected the best-performing result for each image. The average DSC after doing this was around 56%. The resulting pseudo-masks were used to train nnUNet, and testing was subsequently conducted on the UDIAT dataset.

It's worth noting that the zero-shot approach is sensitive to text prompts and M2IB hyperparameters. I recommend experimenting with different text prompts for each image, as a single prompt may not yield optimal results across an entire dataset.

Let me know if you have any further questions.

[YeongHyeon]

@TahaKoleilat Hi again!

This is the most helpful project for my recent interests. Then, can I ask you to share know-how or recipe for easy tuning of M2IB? It will be helpful information to utilize your work.

Thanks.

[lgy112112]

Hi @lgy112112,

Thank you for your interest in our work.

Could you clarify if the results you're seeing are from the BUSI or UDIAT dataset? In our paper, all results were reported on the testing sets across the four imaging modalities. Specifically for breast tumors, we evaluated on the UDIAT testing set, while BUSI served as the training set for our weakly supervised method.

For BUSI, we experimented with six prompt designs and selected the best-performing result for each image. The average DSC after doing this was around 56%. The resulting pseudo-masks were used to train nnUNet, and testing was subsequently conducted on the UDIAT dataset.

It's worth noting that the zero-shot approach is sensitive to text prompts and M2IB hyperparameters. I recommend experimenting with different text prompts for each image, as a single prompt may not yield optimal results across an entire dataset.

Let me know if you have any further questions.

Thx soooo much for your reply. I want to offer details for this issue:

1. I am working on BUSI dataset. In the beginning I just want to check the pipeline so I did not introduce any other dataset, instead I splitted BUSI into train/val/test. In this way, a good result is expected (cuz val/test was originally train set you know), but it turns out BAD :(
2. FYI, I did not do WSS part. As I said I just want to check out the output of zeroshot.

Hparams search results:Hparams search results:

Final Result:

Outcome Vis:

3. I try different CLIP/sam-vit/ensemble-prompts but just can not get rid of this outcome.

[lgy112112]

Hi team,

Could you please give me a runable pipeline? I just want to check the quality of pseudo-masks but, as I said above, it seem un-reproducible.