[Bug][PlanarFreehandContourSegmentationTool] A drawn SegmentationRepresentations.Contour on one view does not show across multiple orthographic viewports

prerakmody commented 1 week ago

Describe the Bug

Task

When using ViewportType.ORTHOGRAPHIC, making contours with PlanarFreehandContourSegmentationTool and using SegmentationRepresentations.Contour, I am unable to see the contours reflected across different views (i.e. axial, sagittal, coronal). Check this netlify example1.
- Use the ReferenceLines (in green) to align the views together
- Draw a contour on any one view (e.g. axial). It will NOT show up in the other view.

Alternative

However, when using the same viewports but drawing with BrushTool and using SegmentationRepresentations.Labelmap I am able to see the labelmap across different contours. Check this other netlify example2.
- Use the ReferenceLines (in green) to align the views together
- Draw a labelmap using brush on any one view (e.g. axial). It WILL show up in the other view.
- This behaviour is similar to the image in this documentation

Versions

"@cornerstonejs/core": "^1.78.1",
"@cornerstonejs/tools": "^1.77.13",

Why is this important

Many commercial contouring tools, use contours during the segmentation phase (check links here). Hence, clinicians get used to the smoothness of these contours compared to the "jaggedness" of labelmap contours. This is further exacerbated when the 3D volume spacing is low.

Steps to Reproduce

Full code

"Inspect" the netlify webpage --> go to "Sources" tab and refer to the .js file under visualiser/src
- index3DContours.js (which uses contours - example1)
- index3DBrush.js (which uses labelmap - example2)

Basic code for both examples

(not in order)

const viewportInputArray = [
        {viewportId: viewportIds[0],type: cornerstone3D.Enums.ViewportType.ORTHOGRAPHIC,element: element1,defaultOptions: {orientation: cornerstone3D.Enums.OrientationAxis.AXIAL,},},
        {viewportId: viewportIds[1],type: cornerstone3D.Enums.ViewportType.ORTHOGRAPHIC,element: element2,defaultOptions: {orientation: cornerstone3D.Enums.OrientationAxis.SAGITTAL,},},
        {viewportId: viewportIds[2],type: cornerstone3D.Enums.ViewportType.ORTHOGRAPHIC,element: element3,defaultOptions: {orientation: cornerstone3D.Enums.OrientationAxis.CORONAL,},},
    ];

const renderingEngine = new cornerstone3D.RenderingEngine(renderingEngineId);

renderingEngine.setViewports(viewportInputArray);

await cornerstone3D.setVolumesForViewports(renderingEngine, [{ volumeId }], viewportIds); 

renderingEngine.renderViewports(viewportIds);

const segmentationDisplayTool = cornerstone3DTools.SegmentationDisplayTool;
const toolGroup = cornerstone3DTools.ToolGroupManager.createToolGroup(toolGroupId);
toolGroup.addTool(segmentationDisplayTool.toolName);
toolGroup.setToolEnabled(segmentationDisplayTool.toolName);

await cornerstone3D.volumeLoader.createAndCacheDerivedSegmentationVolume(volumeId, {volumeId: segmentationId,});

Example 1 - with contours

const planarFreehandContourSegmentationTool = cornerstone3DTools.PlanarFreehandContourSegmentationTool;
toolGroup.setToolActive(planarFreehandContourSegmentationTool.toolName, {bindings: [{mouseButton: cornerstone3DTools.Enums.MouseBindings.Primary, },],});// Left Click

segmentation.addSegmentations([ { segmentationId, representation: { type: cornerstone3DTools.Enums.SegmentationRepresentations.Contour, data: { volumeId: segmentationId, },}, }, ]);
const segmentationRepresentationUIDs = await segmentation.addSegmentationRepresentations(toolGroupId, [ { segmentationId, type: cornerstone3DTools.Enums.SegmentationRepresentations.Contour, }, ]);
segmentation.activeSegmentation.setActiveSegmentationRepresentation(toolGroupId,segmentationRepresentationUIDs[0]);

Example 2 - with labelmap

const brushTool  = cornerstone3DTools.BrushTool;
toolGroup.addToolInstance(strBrushCircle, brushTool.toolName, { activeStrategy: 'FILL_INSIDE_CIRCLE', brushSize:5});
toolGroup.setToolActive(strBrushCircle, { bindings: [ { mouseButton: cornerstone3DTools.Enums.MouseBindings.Primary, }, ], });
toolGroup.setToolEnabled(segmentationDisplayTool.toolName);

segmentation.addSegmentations([ {segmentationId, representation: { type: cornerstone3DTools.Enums.SegmentationRepresentations.Labelmap, data: { volumeId: segmentationId, }, }, }, ]);
const segmentationRepresentationUIDs = await segmentation.addSegmentationRepresentations(toolGroupId, [ {segmentationId, type: cornerstone3DTools.Enums.SegmentationRepresentations.Labelmap,}, ]);
segmentation.activeSegmentation.setActiveSegmentationRepresentation(toolGroupId,segmentationRepresentationUIDs[0]);

The current behavior

Unable to view contours across orthographic views

But able to view labelmap across orthographic views

The expected behavior

Behaviour of contours should be similar to that of labelmap

OS

MacOS v14.2 (Sonoma)

Node version

20.8.0

Browser

Brave (Version 1.66.118)

sedghi commented 1 week ago

This is expected behavior. Contours are based in 2D space, while labelmaps are 3D.

prerakmody commented 1 week ago

Thanks for the response.

Are there any plans in the roadmap to ensure that the 2D contours (which have an internal (x,y,z) representation ?) are also made visible in 3D space? Given that this is a feature available in commercial tools (e.g. MIM, ReLU, Eclipse, Raystation etc), having such a feature in open-source tools makes them more ecologically valid.

Maybe I can change this to a feature request?

sedghi commented 3 days ago

Theoretically, you cannot display x, y, z coordinates and a plane without assuming some thickness for the point and the plane. It's impossible to make the plane pass through that exact Z coordinate due to floating-point errors. All these tools make certain assumptions.

We already have labelmaps, which are volumetric, and you can use brush tools and other editing tools to modify the segmentation mask. The ReLU example was interesting; if you watch closely, they have the labelmap (edged mask) and then generate a contour around it. This allows you to edit the contour, and the changes apply back to the labelmap.

I'm not sure about MIM, but the smooth edges on their segmentation suggest that the internal representation is labelmap-based (volume and voxel-based). However, rendering is likely done via contours. The same process occurs when the contour is updated: the voxel map is edited, and then the contour is regenerated, I assume

Alternatively, you can create surfaces from contours, cut through the surface, and display it as contours again. This is similar to what we have with our polyseg converters, although it's not yet mature enough.

Another approach is to generate or edit a contour, then run it through a contour-to-surface conversion. On other viewports, we could cut through that surface. This is certainly possible, although it requires at least two slices with the contour since we can't generate the surface from a single contour plane. This approach is most attractive to me in the short term, as we already have the infrastructure for it.

cornerstonejs / cornerstone3D