search

bowang-lab / MedSAM

Segment Anything in Medical Images
https://www.nature.com/articles/s41467-024-44824-z
Apache License 2.0
2.78k stars 377 forks source link

Any plan to deal with full 3D volumn directly instead of slicing #160

Closed billhhh closed 11 months ago

billhhh commented 11 months ago

Hi Jun and Bo,

Thank you for your great work. As far as I know the current repo deals with 3D data by slicing them into 2D images via a preprocessing step. May I know do you have any plan (or do you know if there is any Medical SAM repo) to extend your model to deal with full 3D data directly (not by slicing)?

Thanks again. Best

JunMa11 commented 11 months ago

Hi @billhhh ,

Thanks for raising this great question.

Our primary goal is to develop a general segmentation method that can handle all kinds of images. Thus, 2d is a proper way to achieve this goal. I think it is still an open question for direct 3D inference with 2D models since there are many factors that can affect the final performance, such as image size and spacing, et al. I have not seen any thorough study that applies 2D models to 3D images with comprehensive evaluation.

Another possible solution is to develop dedicated 3D SAM-likes models for 3D data. There have been some 3D transformer-based networks in the community, such as UNeter and Swin UNeter. It could be an interesting direction to explore their potential under promptable segmentation settings.

Hope the answer helps and please feel free to raise any questions.

Best, Jun

billhhh commented 11 months ago

Thank you!