search

med-air / 3DSAM-adapter

Holistic Adaptation of SAM from 2D to 3D for Promptable Medical Image Segmentation
170 stars 12 forks source link

Questions about multiple organ (tumor) segmentation tasks #29

Open SnaKey0u0 opened 1 year ago

SnaKey0u0 commented 1 year ago

Thanks for your great work, it’s given me a lot of inspiration.

I have a question about multi-organ segmentation, which is training the model using multiple different organ (tumor) datasets. In your use case, it seems like you’ve done 4 separate trainings and evaluations for kits, lits, pancreas, and colon.

I’ve trained on Task03_Liver, Task06_Lung, Task07_Pancreas, Task08_HepaticVessel, Task09_Spleen, Task10_Colon datasets all at once. For Task03_Liver, Task07_Pancreas, Task08_HepaticVessel, I used seg = (seg == 2).astype(np.float32) and for the rest, I used seg = (seg == 1).astype(np.float32). But the results don’t seem to be very good. Have you done any experiments in this area?

peterant330 commented 1 year ago

Thanks for your great work, it’s given me a lot of inspiration.

I have a question about multi-organ segmentation, which is training the model using multiple different organ (tumor) datasets. In your use case, it seems like you’ve done 4 separate trainings and evaluations for kits, lits, pancreas, and colon.

I’ve trained on Task03_Liver, Task06_Lung, Task07_Pancreas, Task08_HepaticVessel, Task09_Spleen, Task10_Colon datasets all at once. For Task03_Liver, Task07_Pancreas, Task08_HepaticVessel, I used seg = (seg == 2).astype(np.float32) and for the rest, I used seg = (seg == 1).astype(np.float32). But the results don’t seem to be very good. Have you done any experiments in this area?

Hi, We didn't do this kind of experiment as the main focus of the paper is task-specific fine-tuning. One issue might be the data preprocessing. Different tasks may require different preprocessing, I am not sure if you have addressed it properly. Another issue is that you are actually trying to build a medical imaging foundation model, with relatively large datasets. In such a case, you need a large model to accommodate the variability of the data, so freezing most of the pre-trained parameters might not be a good choice. Here is a recent paper in this area that you can refer to, https://github.com/uni-medical/SAM-Med3D, they are using a similar method to adapt the encoder from 2D to 3D, but they fine-tune all the parameters and trained on larger datasets.