Share IDC Demo Images with Annotations

[bnapora]

Is it possible to gain access to the sample images demonstrated here: https://imagingdatacommons.github.io/slim/studies/2.25.18199272949575141157802058345697568861/series/1.3.6.1.4.1.5962.99.1.3510881361.982628633.1635598486609.2.0

Maybe I missed it somewhere, but can't find a link for downloading those images. If your willing to share that would be fantastic. If not, do you know of any public datasets with DICOM SR annotations, and in particular the Microscopy Bulk Simple Annotations.

Thanks for your help

[fedorov]

@bnapora thank you for reaching out! We at Imaging Data Commons have been working for quite some time now on this issue, and it's been a moving target. We do have some samples that we created - rather, that Chris Bridge @CPBridge has created! - but I am not sure we are confident they are final/valid, and we are aware of only one viewer - Slim! - that can render those.

I think it should be fine to share a sample, but I want to make sure @CPBridge and David Clunie @dclunie are onboard with this.

Do you happen to have any alternative - even non-public - slide viewer that we could test to evaluate interoperability of those?

[bnapora]

@fedorov thanks for the quick reply. There are a few moving targets in this domain currently. =) For rendering SR annotations in a DICOM viewer I'm working exclusively with the SLIM codebase currently, so it would be very helpful to try out the images @CPBridge prepared.

We don't have an alternative viewer yet that we would test it out on, but my experience to date is that there is still a fair amount of challenge getting consistent wsi-dicom and dicomweb functionality from the different dicom-servers/stores. Just working with some real files with the annotations in them would be a big help as we wade through the specs.

If possible to share the images it would be fantastic. I'll certainly share any findings.

[dclunie]

@bnapora, you ask two questions, one about annotations, which @fedorov answered, and the other about a link to download the images, which he didn't.

The Slim viewer doesn't provide a direct download link (which one could argue might be a nice feature).

There are numerous other ways to download the images, which are stored not only in the DICOMweb server from which they are made available to the viewer, but also from buckets directly, at Google and AWS locations listed in the IDC portal (search by Collection Name aka Protocol ID then Patient ID aka Case ID, then use download icon at the Selected Studies level or below will generate a manifest to download from the cmd line, or you can build a Cohort using the left panel filters and download that) as well as available via BigQuery (described here).

@fedorov, we should probably update the downloading instructions to mention the recently added Studies/Series level Portal download icon, which isn't currently mentioned there (or described anywhere AFAIK). Also, we should consider adding it at the Case/Patient level too.

[dclunie]

@bnapora, I see that your SlideRunner tool has recently added DICOM WSI reading support, which is great, so we would be very interested in hearing how well it works on the DICOM WSI images in IDC, most of which are brightfield H&E. We have also worked with the OpenSlide team to add DICOM WSI reading support, which should come out soon, but I gather you used pydicom and read directly.

It would be great if SlideRunner was able to create and read Microscopy Bulk Simple Annotations, so let us know if you need any help testing or validating that (feel free to use my dicom3tools dciodvfy tool to check their conformance and let me know if you find any issues with the validator). As @fedorov mentioned, @CPBridge is working on getting highdicom to produce both Microscopy Bulk Simple Annotations as well as convert them to DICOM Segmentation (bitmask) representations.

I am particularly keen to test all permutations of the 2D image-relative and 3D physical-slide-position-relative coordinates in various integer and floating point encoded forms that are possible. I think the DICOM object already supports all the different graphic types of annotation for your tool (center point, rectangle, circle, polygon), but if not let me know if I missed anything. The circle is supported as a degenerate ellipse, which is a bit wasteful I suppose, but the rationale was that the plane of the circle would be unspecified without at least a 3rd point anyway. I would be interested to know if your tool "natively" uses 2D or 3D coordinates (and if 2D how you track the layer they are relative too). You use only integer coordinates, right, so no sub-pixel resolution?

Also, looking at your information model, I see that you can add multiple labels to each annotation, but specify a single class, and whereas DICOM really only supports one property category/type/label/description (other than for numeric measurements, and we don't record the person creator for each measurement, since it was really intended for machines)

For small numbers of human generated annotations, the DICOM SR mechanism works too, so if you are interested in pursuing that (and selecting which features of which SR template to use, or need to be added if there are gaps), I would be happy to discuss that too.

[dclunie]

See also #1639.

[CPBridge]

Hi @bnapora I am happy so share some annotations. But in order to prepare something to send it might be helpful to get a bit more context on what you wish to do with them.

Currently for several images in the TGCA collections (now also housed within IDC as DICOM as David mentions), we have nuclei segmentations. We are representing these as both Microscopy Bulk Simple Annotations (note that these are not really a type of DICOM SR but a different entity). But there are two choices we have to make when encoding these:

• Whether to encode them as image (pixel) coordinates within the total pixel matrix of the slide ("2D") or as frame-of-reference coordinates ("3D") within DICOM's slide coordinate system (which may be shared for example between different levels of a multiresolution pyramid). Currently slim can only display "3D" frame of reference coordinates.
• Whether the nuclei are encoded as POLYGONS representing the boundary, or POINTS representing just the centroid. Again slim can only display POINTs right now though there is an open pull request #170 to address this and allow POLYGONS.

So if your objective is to have annotations that you can visualize in the current version of slim, then I can send you "3D" POINT annotations. If you want other types for some other purpose let me know and I can try to accommodate, though they will not currently render in slim.

Note that the images displayed on slim's readme page are of "true" DICOM SRs, specifically Comprehensive3DSR objects and not Microscopy Bulk Simple Annotation objects. For true SRs we are able to render polygons right now (but they are not really usable for large numbers of annotations due to the way they are structured).

Also let me know if you require annotations for specific images, or whether any will do.

[dclunie]

Current limitations on what Slim can or cannot display should not discourage us from selecting the most natural representation in DICOM for whatever SlideRunner currently creates, edits and displays, which AFAICT is 2D image-relative integer geometric shapes, and may serve to modulate our priorities for Slim enhancements in this respect.

[bnapora]

@dclunie and @CPBridge thanks for the rapid and helpful responses. Its awesome to visualize real world application of the DICOM SR for wsi-dicom.

Currently our primary goal is to demonstrate a POC for storing image analysis output (primarily nuclei points and cell segmentation) and rendering that directly from the DICOM using DICOMweb, (which you've already demonstrated with Slim. (https://imagingdatacommons.github.io/slim/studies/2.25.18199272949575141157802058345697568861/series/1.3.6.1.4.1.5962.99.1.3510881361.982628633.1635598486609.2.0) We're also exploring utilizing the "rendered" method of DICOMweb to provides instances to an OSD based viewer, which will be interesting to test.

Regarding the sample image, we'll take the current 3D Point annotations, since that is currently working in SLIM. The polygons will be helpful when available, but points will work. If you can only provide for one image, then the image listed above would be great. If you can provide DICOM SR and Bulk Microscopy Annotations for the 4-5 images demonstrated on Slim demo site it would be helpful. Slim is a fantastic reference implementation, and we would just like to validate in our own environment with some of annotations you have, before we start extending. Does that answer question about what image and type of annotation @CPBridge?

@dclunie As far as testing in SlideRunner, that is a great idea. We've been so focused on DICOMweb I forgot about SlideRunner. I'm going to loop in @ChristianMarzahl (one of the contributors to SlideRunner) who's also working on this project as well. He and I will talk early next week on how to incorporate that into our current testing. It would be very helpful to get your perspective on how single and multiple human generated annotations for the same point of interest could be stored in DICOM SR. (This is one of the challenges that has kept us tethered to a separate data store .)

Thanks again for your help.

[bnapora]

@CPBridge Any update on annotations for image above? It would be very helpful to see an example of polygon annotations if you have ability to generate and provide those.

[dclunie]

@bnapora the most recently generated annotation samples from @CPBridge are at: https://console.cloud.google.com/storage/browser/idc-annotation-conversion-outputs/2023_10_03_tiled_full_example/read_polygon https://console.cloud.google.com/storage/browser/idc-annotation-conversion-outputs/2023_10_03_tiled_sparse_example/read_polygon

[CPBridge]

Thanks @dclunie, although note that those are 3D polygon annotations (as well as segmentation) that will not currently render in the latest release of slim

I will create some 3D POINT annotations of the nuclei centroids and send them at some point in the next few days

[CPBridge]

I also do not have any "real" annotations that I can feasibly encode in an SR but I am happy to concoct some contrived example for you

[CPBridge]

Here are a few examples of 3D "point" annotations of nuclei for some of the publicly available images in the IDC (the TCGA-READ collection):

https://console.cloud.google.com/storage/browser/idc-annotation-conversion-outputs/2023_10_11_annotations_3d_points/read_polygon?pageState=(%22StorageObjectListTable%22:(%22f%22:%22%255B%255D%22))&project=idc-etl-processing&prefix=&forceOnObjectsSortingFiltering=false

@bnapora

[bnapora]

@bnapora the most recently generated annotation samples from @CPBridge are at: https://console.cloud.google.com/storage/browser/idc-annotation-conversion-outputs/2023_10_03_tiled_full_example/read_polygon https://console.cloud.google.com/storage/browser/idc-annotation-conversion-outputs/2023_10_03_tiled_sparse_example/read_polygon

Thanks @dclunie @CPBridge It's much easier to implement the standard using examples. Thanks for providing. This helps.

[bnapora]

@CPBridge @dclunie If your interested, here's our research viewer, rendering pixel and nuclei annotation data from: https://imagingdatacommons.github.io/slim/studies/2.25.18199272949575141157802058345697568861 Very exciting to see interoperability bearing fruit!

[CPBridge]

Wow, very cool. Thanks for sharing!

[bnapora]

@CPBridge @dclunie Thanks for help on this. We got a great jump start thanks to your help. Closing issue for now.

[igoroctaviano]

@CPBridge @dclunie If your interested, here's our research viewer, rendering pixel and nuclei annotation data from: https://imagingdatacommons.github.io/slim/studies/2.25.18199272949575141157802058345697568861 Very exciting to see interoperability bearing fruit!

@bnapora have you experienced slowness interacting with the image after rendering all annotations? I have samples with a million annotations and rendering them (highest resolution) using Open Layers is taking some time and feels laggy after they are rendered. I'm looking for alternative ways e.g. cluster layers based on zoom level but I'm curious how your tool is performing.

[bnapora]

@igoroctaviano Sorry for the slow response, I missed your note. Our viewer is using OpenSeaDragon, with a PaperJS overlay. To answer your question, we haven't tried an image with "a million annotations", but have tried with about 300k, and yes there is noticeable sluggishness when navigating. I would be interested in testing out your image (with 1 mill annos) to attempt a general comparison of architecture performance. We're also very interested in alternative ways for rendering at lower levels of magnification. We're attempting a "down sampling" algorithm but obviously that gets pretty complex when increasing the number of classes visualized. Have you had much progress in your "clustering" technique?

[CPBridge]

Hi @bnapora a few months ago you were asking for examples of SRs and I didn't have any at that time. But now I do: https://console.cloud.google.com/storage/browser/idc-annotation-conversion-outputs/2023_12_07_rms_hand_annotations?project=idc-etl-processing&pageState=(%22StorageObjectListTable%22:(%22f%22:%22%255B%255D%22))&prefix=&forceOnObjectsSortingFiltering=false

I believe you should have access to these if you had access to the old ones. Again these refer to publicly available images in the IDC.

Hope this helps, let me know if you have any further questions

[igoroctaviano]

@igoroctaviano Sorry for the slow response, I missed your note. Our viewer is using OpenSeaDragon, with a PaperJS overlay. To answer your question, we haven't tried an image with "a million annotations", but have tried with about 300k, and yes there is noticeable sluggishness when navigating. I would be interested in testing out your image (with 1 mill annos) to attempt a general comparison of architecture performance. We're also very interested in alternative ways for rendering at lower levels of magnification. We're attempting a "down sampling" algorithm but obviously that gets pretty complex when increasing the number of classes visualized. Have you had much progress in your "clustering" technique?

@bnapora FYI (using cluster layer): https://drive.google.com/file/d/11Rd0486RhHESFlCcSciUEM8mHYBauBHY/view?usp=drive_link

[dclunie]

FYI, wrt. what other viewers do with rendering high resolution polygons at lower resolution zoom, see Figure 10 of this paper [1], described as:

"Figure 10 - Nuclear segmentation and Tumor Infiltrating Lymphocytes (TILs) result sets at four different scales. a) Full WSI with heatmap of TILs b) Closer view with panning sync’d c) Even closer view, but now with nuclear segmentation results superimposed d) maximum zoom."

The discussion of multi-resolution Hilbert space-filling curves, spatial queries and the use of RDF and corresponding databases is also interesting, but not immediately germane since we (DICOM WG26) selected a Cartesian representation rather than a Hilbert curve for the coordinates (for now).

1. Bremer E, DiPrima T, Balsamo J, Almeida J, Gupta R, Saltz J. Halcyon -- A Pathology Imaging and Feature analysis and Management System. arXiv; 2023. Available from: http://arxiv.org/abs/2304.10612 doi:10.48550/arXiv.2304.10612