search

MedicineToken / Medical-SAM2

Medical SAM 2: Segment Medical Images As Video Via Segment Anything Model 2
Apache License 2.0
479 stars 58 forks source link

Question about the 3D segmentation of organs #30

Closed AlexanderZeilmann closed 2 months ago

AlexanderZeilmann commented 2 months ago

Hey :) the paper looks really impressive and consider applying it in my workflow. To do this I have a quick question:

The Medical SAM 2 paper states in the abstract:

That allows users to provide a prompt for just one or a specific image targeting an object, after which the model can autonomously segment the same type of object in all subsequent images, regardless of temporal relationships between the images.

And later

In this scenario, users only need to provide a prompt for the first or a specific frame targeting an area, such as the optic cup in a fundus image. Subsequently, the model can autonomously segment the optic cup in subsequent images, even in the absence of temporal relationships between these images.

Does that mean the if I annotate some organ in the center only half of the organ is segmented?

This in turn would mean, that

  1. in order to segmented the organ I have to find the very first pixel of that organ in a specific direction,
    • which is difficult to annotate and
    • since this annotation is right at the border of the organ this segmentation is really difficult to do as the border pixel is not a typical pixel of the organ.
  2. Organs like the colon which go back and forth in the volume can not be segmented as the information is flowing only in one direction.

Am I missing something here? Thank you!

heyoeyo commented 2 months ago

From what I've seen from the original SAM2 (and I assume Medical-SAM2), the model is pretty robust to changes to the frame ordering (you can give frames out-of-order and it still works), though giving continuous looking sequences (whether regular or reversed order) should work best.

So it should be fine to pick a frame where it's easy to get a good segmentation, then propagate 'forward in time' until the segmentation is lost, then go back to the prompted frame (and reset the memory bank) and run 'backwards in time' to get the segmentations for frames prior to the prompted frame.

jiayuanz3 commented 2 months ago

yes, I agree the comments provided by @heyoeyo and thanks for the well explanation. additionally, the two quotes we are referring to the One-prompt Segmentation capability in 2d cases, regardless of temporal relationships between the images.

AlexanderZeilmann commented 2 months ago

Thank you to both of you! I understand now.

So the complete workflow would be to

  1. Go to the first labelled slice and segment forward and backward in the volume to create a (partial) segmentation of the volume.
  2. Go to the second labelled slice and create another partial segmentation. Similarly for all other labeled slices.
  3. Combine the partial segmentations to a "complete" segmentation of the volume.
  4. Show to the user for further annotations.
  5. Repeat.