search

NVIDIA / ai-assisted-annotation-client

Client side integration example source code and libraries for AI-Assisted Annotation SDK
Other
307 stars 64 forks source link

Slicer: Speeding Up Inference to Real Time? Eg Like Preview Mode From Slicer's Grow From Seeds #25

Closed SachidanandAlle closed 4 years ago

SachidanandAlle commented 5 years ago

For NVIDIA team, how fast could an annotation be updated when given additional points? Do you think it's possible to get a new segmentation update via AIAA after each point is added?

SachidanandAlle commented 4 years ago

We will support deep grow (annotation model) in future for AIAA. Once it's available we can work on the client action in the plugin

lassoan commented 4 years ago

Sounds great. Le me know if you need any help with Slicer integration.

SachidanandAlle commented 4 years ago

@lassoan need your help on few things.. please refer https://github.com/NVIDIA/ai-assisted-annotation-client/tree/deepgrow

NVIDIA is working on a deepgrow AI model (kind of region grow algo)... Currently its 2D version and 3D model is WIP

NOTE:: AIAA will support session mechanism to save the image (with an expire/close session api). This will help to avoid sending the image/volume repeatedly over n/w for segmentation/dextr3d/deepgrow actions.

How it works? User adds some foreground points, and model will return the mask.. basically user should be able to do segmentation with one or few foreground clicks..

And if result mask has any overlapping/incorrect area, user case add background points to correct them.

I am having following TODOs 1) On UI side, I am not able to create/set different colors for foreground/background points during the creation time (Currently both are red by default)

2) Reset current slice and then union/add (deepgrow provides the mask for current slice only) For now, what I am doing is slicer.qSlicerSegmentEditorAbstractEffect.ModificationModeSet for the slice (label mask from deepgrow) but it's not correct to handle all the cases...

I want to clear the current slice in the labelmap and then add/union the result from the deepgrow model. How to clear the current slice?

https://github.com/Slicer/Slicer/blob/master/Modules/Loadable/Segmentations/EditorEffects/Python/SegmentEditorLogicalEffect.py#L43-L51

Couldn't get enough ideas from here... because operations are for entire labelmap... how to get the current slice and clear it in the labelmap as the first step?

Attached video and snapshot for reference:

Video (Deepgrow in real action): https://drive.google.com/open?id=1C88SrVjbiyFyDwsQhYwTKPmlgrmlxO8I

image

lassoan commented 4 years ago

AIAA will support session mechanism to save the image (with an expire/close session api). This will help to avoid sending the image/volume repeatedly over n/w for segmentation/dextr3d/deepgrow actions.

This will be great!

User adds some foreground points, and model will return the mask.. basically user should be able to do segmentation with one or few foreground clicks..

We have a couple of filters that work like this, such as "Grow from seeds" and "Watershed" and they work quite well. However, they are completely generic and don't use any underlying shape or intensity model, they just learn the intensity distribution from the provided sample points. Does deepgrow uses a model trained separately for each organ or it is general-purpose (you click on any part of any structure and it will try to segment that)?

I want to clear the current slice in the labelmap and then add/union the result from the deepgrow model. How to clear the current slice?

You can use self.scriptedEffect.modifySegmentByLabelmap, see for example here: https://github.com/Slicer/Slicer/blob/master/Modules/Loadable/Segmentations/EditorEffects/Python/SegmentEditorIslandsEffect.py#L223

You provide a "modifier" labelmap and can tell how to modify the segmentation with it: add it to the segment, remove it from the segment. You could first use "remove" to clean the slice and then "add" to paint the new content in the slice.

SachidanandAlle commented 4 years ago

Does deepgrow uses a model trained separately for each organ or it is general-purpose (you click on any part of any structure and it will try to segment that)?

Yes.. it's a general purpose (not specific to any organ)

You can use self.scriptedEffect.modifySegmentByLabelmap

The issue is this updates entire labelmap (based on modified labelmap).. either it resets or does union... from AIAA, we get new label mask only for one slice... (background points could have corrected/deleted some portion in new mask for the same slice)..

Currently, I am doing this hack.. https://github.com/NVIDIA/ai-assisted-annotation-client/blob/deepgrow/slicer-plugin/NvidiaAIAA/SegmentEditorNvidiaAIAALib/SegmentEditorEffect.py#L253-L277

But may be there are better ways to get this done..

How about? Any idea?

On UI side, I am not able to create/set different colors for foreground/background points during the creation time (Currently both are red by default)

lassoan commented 4 years ago

The issue is this updates entire labelmap (based on modified labelmap).. either it resets or does union...

To modify a single slice:

On UI side, I am not able to create/set different colors for foreground/background points during the creation time (Currently both are red by default)

Each markups control point has a "selected" state and you can set different color for selected and non-selected state. So, if you strictly have two classes of points then this could work. However, probably users will want to place seeds for multiple objects at once, and not just inside/outside seeds for a single object. For this, you need to add a markups fiducial node for each segment.

SachidanandAlle commented 4 years ago

Following related PRs merged to master to support deepgrow action in Slicer plugin:

https://github.com/NVIDIA/ai-assisted-annotation-client/pull/54 https://github.com/NVIDIA/ai-assisted-annotation-client/pull/56