Wrong Segmentation Position when opening with CornerstoneTools

[AlexandrineRibeiro]

I'm writing a script to attach AI made lesion segmentations to existing DICOM files (It's my first time that I dive so deep into DICOM, so I might have missed something).

I took an existing DICOM file (of a CT scan) that contains both a scan and a segmentation, converted the segmentation to np.array, and used is as the mask input, while the scan folder is the source_images.

When I try to load the resulting segmentation with Cornerstone Segmentation Tools, my segmentation appears in the right order but always placed in the first slices. For example, if it is the liver segmentation, it will appear in the first slices and not in the liver area.

My env:

• python 3.6.2
• pydicom 2.3.1
• pydicom-seg 0.3.0
• highdicom 0.20.0

The code I use to store the segmentations:

# Read Image data sets
image_datasets = [dcmread(str(f)) for f in image_files]

segs_descriptions = []
for label in np.unique(y_pred_all):
    if label != 0:
        # Describe the segment
        description_segment = hd.seg.SegmentDescription(
            segment_number=int(label),
            segment_label=f"Segment {int(label)}",
            segmented_property_category=codes.cid7150.Tissue,
            segmented_property_type=codes.cid7166.ConnectiveTissue,
            algorithm_type=hd.seg.SegmentAlgorithmTypeValues.MANUAL,
        )
        segs_descriptions.append(description_segment)

# Create the Segmentation instance
seg_dataset = hd.seg.Segmentation(
    source_images=image_datasets,
    pixel_array=np.uint8(y_pred_all),
    segmentation_type=hd.seg.SegmentationTypeValues.FRACTIONAL,
    segment_descriptions=segs_descriptions,
    series_instance_uid=image_datasets[0].SeriesInstanceUID,
    series_number=99,
    sop_instance_uid=image_datasets[0].SOPInstanceUID,
    instance_number=1,
    max_fractional_value=255,
    transfer_syntax_uid="1.2.840.10008.1.2.5",
    manufacturer="Manufacturer",
    manufacturer_model_name="Model",
    software_versions="v1",
    device_serial_number="Device XYZ",
    omit_empty_frames=False,
)

seg_dataset.save_as(output_dcm)

[CPBridge]

HI @AlexandrineRibeiro thanks for using the library and reporting this issue!

Generally your code looks good but there are a couple of comments unrelated to the issue:

• The sop_instance_uid and series_instance_uid that you pass to the segmentation should not be the uids taken from the input datasets, since the segmentation is a new instance in a new series and therefore needs its own instance/series UIDs. In nearly all cases you just want to generate two new uids, one for each using hd.UID(). Early on we took the decision to make users pass the UIDs themselves in case they want more control over this, which I regret because a lot of people make the same mistake that you do... We should probably improve the docs here.
• If your segments are lesions, "Connective Tissue" is probably not the best code to use to describe them

But to your actual question: I suspect that this is neither an issue with highdicom or your code. There have been issues with the way cornerstone/ohif/dcmjs display segmentations for a while and many people have reported them. We believe that this is a problem in dcmjs, not highdicom. However, it is still worth checking a few things:

• Ensure that before you are creating the segmentation dataset, the order of the source images image_datasets matches the order of slices of the segmentation mask down the 0th dimension of the numpy array.
• Check (using some python visualization such as matplotlib) that if you reconstruct the numpy array using the following method, the mask looks like it should:

reconstructed_mask = seg_dataset.get_pixels_by_source_instance(
    source_sop_instance_uids=[ds.SOPInstanceUID for ds in image_datasets],
    combine_segments=True
)

If it does, then probably everything is fine and the issue is in the viewer, though there is a small chance the bug is in highdicom. Please report this issue to cornerstone or dcmjs, and even better if you can provide example segmentations and screenshots. You may wish to refer to this issue on OHIF (built on cornerstone): https://github.com/OHIF/Viewers/issues/2833

[CPBridge]

P.s. better docs for segs generally are coming soon: you may wish to read the docs on this branch and in particular this file

[pieper]

You can also check your segmentations with Slicer using the Quantitative Reporting extension. If you find that a source image and segmentation pair renders differently in OHIF vs Slicer please post the data and hopefully the difference can be reconciled.

[AlexandrineRibeiro]

@CPBridge @pieper Thank you for your prompt response. I agree with you, the problem appears to be with cornerstone or dcmjs.

[TuanDTr]

@CPBridge I have a more or less similar issue when overlaying the seg mask on OHIF viewer. It is a seg file from a multiframe source image that has annotation for only 20/121 slices. On OHIF, the segmentation seems to be rendered in a reversed order, i.e., annotation for slice 50 is shown for slice 70. On Slicer, the seg mask is correctly shown on the image. I know it could be the issue with OHIF, but I wonder if there is a known issue or if there is any workaround for it. Thanks!

[CPBridge]

Hi @TuanDTr

Unfortunately I get quite a lot of reports about segmentations not displaying as expected in OHIF, but with the viewers changing all the time it's hard to keep track and I've think I've ever fully grasped the problem (s). I believe it has always ended up being a bug in the viewer not highdicom, up until now anyway.

A few ideas though:

• Are you using the latest version of OHIF v3? I think many of these issues were fixed between v2 and v3.
• If you want to make sure that the file is encoded as you expect, try calling the get_pixels_by_source_frame and passing a single frame number, then check whether the returned segmentation array matches your expectation. Bear in mind that dicom, and hence this function, indexes frames starting at 1 (not 0). If you are able to share the segmentation and source images, I can also do some other sanity checks.
• A simple workaround may be to pass omit_empty_frames=False when constructing the segmentation. This is wasteful of space but i think has worked around similar problems for people in the past.

[fedorov]

@TuanDTr can you share a de-identified sample that does not work and the corresponding image? If you cannot share a sample, can you create a sample using public data available in Imaging Data Commons?

@CPBridge we definitely have segmentations with skipped empty frames that work fine in OHIF (both v2 and v3).

[TuanDTr]

@CPBridge: I'm using OHIF v3.8.3. I have tried calling the get_pixels_by_source_frame and the returned array is correct. The segmentation file is exported with empty slices. So it seems to me that the problem is not on the side of highdicom.

@fedorov: I am not able to share the image here but I use example file in highdicom's example for generating seg from multiframe source image. But this one has another issue in OHIF. Basically OHIF only visualizes the first frame and the second frame is completely white. Could you have a look and see if you encounter similar issue? I think this has something to do with the multiframe dicom image in OHIF.

example.zip

[fedorov]

I think this has something to do with the multiframe dicom image in OHIF

I missed this part earlier. Indeed, this may very likely be the problem. Unfortunately, multiframe DICOM is still a rarity for imaging modalities, and I am afraid it is just the reality that we have not ironed out many of the issues in the implementations.

Thank you for sharing the example. I cannot tell you if we will have resources to prioritize fixing this in OHIF, resources are spread very thin. But can you open the issue in OHIF related to the problems visualizing the image itself? It is best to separate issues with multiframe imaging modality support vs SEG.

Couple of observations looking at the CT part of your sample:

1. It is a derived image, not the one coming from the modality, and it is a perfusion map. I think a more suitable object for this kind of image is Parametric Map (which is also multiframe), not Enhanced CT. If you produced this image, you could consider experimenting with the parametric map converter from dcmqi: https://qiicr.gitbook.io/dcmqi-guide/opening/cmd_tools/dicom-parametric-map-support/itkimage2paramap.
2. I opened the image and segmentation in Slicer, and just wanted to confirm - is the below what you expect to see?

[CPBridge]

Andrey, this is a test image used in our test suite (but actually coming from the pydicom test files, not one we originally created), so it is not something @TuanDTr constructed. I believe it is this file: https://github.com/pydicom/pydicom-data/blob/master/data_store/data/eCT_Supplemental.dcm

[TuanDTr]

@fedorov Like @CPBridge said, the image I used is from the test files of highdicom, and I created a dummy segmentation for it. The overlaid segmentation you show is the correct one.

[fedorov]

Thanks for the clarification. I am not quite sure what to do about this. I do not have the cycles to debug this, and I cannot prioritize allocating resources to debug this in OHIF since this issue cannot be reproduced with any segmentation we have in Imaging Data Commons.... Unfortunately, there is a long backlog of issues in OHIF that are priority for IDC, and we don't have the luxury of spare cycles at this moment.

[CPBridge]

I'm not sure there's much we can do beyond reporting it to OHIF and letting them prioritize it. For others having this issue note that an issue has been reported on OHIF's tracker here: https://github.com/OHIF/Viewers/issues/4421

Perhaps one thing we could do would be reproduce the issue and let OHIF developers know, they will presumably put a reproduced issue at a higher priority. I will try to this at some point

[CPBridge]

I'm also reopening this issue to make it easier to find

[igorsimko]

I'm also facing same issue (see https://github.com/OHIF/Viewers/issues/4421#issuecomment-2447219881). There seems to be breaking commit in OHIF https://github.com/OHIF/Viewers/commit/3dd0666c0c090cbd66161f24bc9795f96abb3697

[igoroctaviano]

I'm also facing same issue (see OHIF/Viewers#4421 (comment)). There seems to be breaking commit in OHIF OHIF/Viewers@3dd0666

Replied to https://github.com/OHIF/Viewers/issues/4421#issuecomment-2447219881