Scribbles - Add ROI select feature to prune background for coldstart

[masadcv]

This is a nice-to-have feature request. At the moment scribbles coldstart requires a lot of user interaction especially to indicate all background regions.

Using an ROI, will enable putting everything outside as background and hence improve the histogram-based likelihood model used within this approach.

This possibly requires the use of vtkMRMLAnnotationROINode for selecting ROI and then passing this as an extra argument from client to relevant transforms at server.

[tvercaut]

For the record, providing a bounding box ROI is a standard approach for classical scribble based segmentation algorithms such as GrabCut. Having this feature in MONAI Label would allows us to provide a reference user-facing implementation of GrabCut in 3D and could take advantage of related work in MONAI: https://github.com/Project-MONAI/MONAI/issues/1601

Additionally, bounding boxes have found a few use cases in interactive deep learning and weakly supervised training:

• BIFSeg https://arxiv.org/abs/1710.04043
• DeepCut https://arxiv.org/abs/1605.07866
• https://arxiv.org/abs/2004.06816
• etc

[lassoan]

In 3D Slicer, we initialize grow/grab/graph-cut based methods using a ROI that is generated by expanding the scribbles by 10% of the ROI size on each side (see here). This dramatically reduces the image size for small structures (can make segmentation 100x faster), without requiring users to define an ROI. The only disadvantage is that you need to paint scribbles on at least two planes.

I agree that explicitly specifying an ROI may be useful in the long term, but you can choose this as an immediate solution, as it is simpler to implement (no need for additional GUI) and very convenient to use.

[tvercaut]

Indeed, thanks @lassoan. A related option is to specify "extreme points" and deduce the bounding box from it. This can be even faster than scribbles for the user and has the advantage of providing additional information with respect to a bbox.

This strategy is used in medical imaging for example in:

• InExtremIS (weakly supervised learning) https://arxiv.org/abs/2107.00583
• "Going to Extremes" https://arxiv.org/abs/2009.11988

For improved user experience, it may anyway probably be good that when we implicitely compute a bbox, the result be shown to the user. It will probably make things much more interpretable / less cognitively frustrating. See e.g. https://doi.org/10.1007/s11548-020-02222-y for a study that analyses user experience with interactive segmention of 3D medical images.

[tvercaut]

A bit off topic in this isssue but for clarification @lassoan, is the grabcut plugin in Slicer this one? https://github.com/Slicer/SlicerGraphCutSegment

If so, @masadcv, you could try to compare your results with it (And maybe catch-up with @reubendo to see if he managed to use it in the end, see https://github.com/Slicer/SlicerGraphCutSegment/issues/1)

[masadcv]

Thanks @lassoan and @tvercaut !

I will check with Reuben about the SlicerGraphCut and try to compare it with the version we have in MONAILabel.

I had an initial go at this with vtkMRMLAnnotationROINode, it seems to work quite well with the coldstart scribbles flow. Perhaps we can also look at automatic ROI selection from @lassoan 's suggestion and also check how extreme point selection methods can be used here.

[masadcv]

I have an initial implementation that uses vtkMRMLAnnotationROINode to define a bounding box which is then used to improve the scribbles-based labelling process. With this, we dont need any background scribbles as everything outside the bbox is marked as background.

Here is a quick demo for this (still WIP): https://youtu.be/yhC27V3PvIQ

[SachidanandAlle]

This is pretty good.. very nice :)

[lassoan]

A bit off topic in this isssue but for clarification @lassoan, is the grabcut plugin in Slicer this one?

We had many graph cut algorithms variants for segmentation in Slicer over the years. "Fast grow cut" has beaten all of them, mostly due to its robustness, accuracy, quick initialization, and extremely fast interactive updates. The method has been recently ported into ITK as a remote module. So, now the algorithm is available in Slicer with a GUI ("Grow from seeds" effect in Segment Editor) and as a VTK filter; and in the ITK remote module as an ITK filter. Both implementations are Python-wrapped.

If you want to compare a new segmentation algorithm against some baseline algorithm, I would recommend to compare it to this fast grow cut method.

[tvercaut]

Thanks. Adding the relevant pointers here for the record: https://github.com/InsightSoftwareConsortium/ITKGrowCut https://github.com/Slicer/Slicer/blob/master/Modules/Loadable/Segmentations/Logic/vtkImageGrowCutSegment.cxx