Implement meaningful default visualization mode for multichannel images

[fedorov]

As discussed at the IDC team meeting, currently the defaults create a visualization that is not appealing and is mostly black. In absence of the presets available for the given series, some basic heuristic for more attractive default visualization should be added.

We would like to, by default: 1) pre-select first 3-4 channels 2) automatically calculate some window based on min/max for those channels 3) automatically assign pre-selected colors to those channels

This could be done just for the thumbnail image. We can have a callback that would update rendering once the needed thumbnail channels have been fetched.

Sample data mentioned in this issue https://github.com/ImagingDataCommons/IDC-ProjectManagement/issues/1186#issuecomment-1109931706 (internal to IDC team).

[hackermd]

The major difficulty is that we have to prefetch the image pixel data to compute statistics (e.g., min/max) on the pixel values and dynamically set the value of interest range (limits of the window) before we render/display the pixel data.

Another complication is that there may not always be a single THUMBNAIL image. In that case, we may have to load a potentially large number of tiles into memory first. This can take time and degrade the viewing experience by increasing the time to image display, in particular if we want to display multiple channels. The alternative of first rendering the images and then subsequently adjusting the display settings is also not great.

[pieper]

The alternative of first rendering the images and then subsequently adjusting the display settings is also not great.

Agreed, this is not great, but as I recall we discussed this during our call and agreed that it's likely to only take a fraction of a second and either delaying the first render, or rendering first in black and then flashing to a better display within a short time would be acceptable.

[hackermd]

We may be able to prevent loading the data twice by attaching an event listener to the loader and computes the statistics on the first n tiles on the fly and then update the default display settings before/while the data gets rendered. We would need to think about how we expose that by the library so that it can be performed by the application.

[hackermd]

Agreed, this is not great, but as I recall we discussed this during our call and agreed that it's likely to only take a fraction of a second and either delaying the first render, or rendering first in black and then flashing to a better display within a short time would be acceptable.

It will depend on the network bandwidth (which we throttle in IDC) and other factors such as the compression codec.

[hackermd]

If we don't communicate the rendering intent via the image metadata or a separate presentation state, then we will have to accept some rendering delays/complications. We will see whether those negatively impact the viewing experience in practice.

[fedorov]

I thought we discussed that @Punzo would develop a prototype, we would evaluate performance impact, and then decide on the next steps?

[hackermd]

I thought we discussed that @Punzo would develop a prototype, we would evaluate performance impact, and then decide on the next steps?

Yes, that's the plan. I am just documenting requirements and constraints that we need to consider for design and implementation. For example, we cannot assume that there will always be a THUMBNAIL.

[Punzo]

I thought we discussed that @Punzo would develop a prototype, we would evaluate performance impact, and then decide on the next steps?

yes, but I will wait the merge of the webWorker PR (https://github.com/herrmannlab/dicom-microscopy-viewer/pull/76), because the fetching logic changes a lot, and it will not make sense to develop a prototype twice

[fedorov]

I will wait the merge of the webWorker PR

Thanks for bringing this up Davide!

[Punzo]

Events Infrastructure for the task implemented for DMV in https://github.com/herrmannlab/dicom-microscopy-viewer/pull/78 togheter with the webWorkers. Markus will add the calculations directly in SLIM and create a new release with this and the webWorker infrastructure.

[hackermd]

We have "successfully" implemented the feature.

Here is an example:

The first three channels are selected and pseudo-colored: blue, green, and red. The limits of the value of interest (window) for each channel is set to the min/max values of the THUMBNAIL frame.

[fedorov]

@pieper @dclunie do you have any suggestions on improving the heuristic?

@hackermd Can you confirm that calculating the histogram is not feasible? How much time would you need to create presentation state objects that would store channel configuration presets?

The way things are evolving, the lack of meaningful visualization of those multichannel images may become the main roadblock for releasing HTAN images in IDC.

[dclunie]

Thanks for doing the experiment.

I assume that the black result in the screenshot was meant to imply that this experiment was not actually "successful" after all.

@hackermd (or @punzo) can you explain why?

Is this because you can't find sufficiently small images to run the statistics, or the default channel selection is unhelpful, or the statistics don't produce a good window?

• If it was due to the absence of a designated THUMBNAIL ImageType image for the selected channels (as you anticipated would be a possibility) (which we don't see in converted OME-TIFF source images, as distinct from SVS), i.e. the viewer does not actually have any statistics to work with, then perhaps you could select the VOLUME (i.e., pyramid) image with the lowest NumberOfFrames value (usually there is a DICOM image with a single frame), for each channel (optical path), based on the metadata?
• If selecting the first 3 channels does not give a good result, then when there is channel-specific metadata, e.g., OpticalPathDescription (or the corresponding code) has recognizable values, then picking some common ones that you recognize, or not using ones that say "control" or whatever, might be better logic than the first 3 (e.g., the first "dna" or "nucleus", anything with "keratin", or in it, the first "CD something", or whatever); or perhaps you can use some characteristic of the statistics to pick the channels
• If using a "window" of min to max, or 0 to max, does not give good result, then is there another choice based on the image statistics (e.g., mean and standard deviation, whether it be of pixel values or log pixel values) that is still fast enough to compute (on a single frame image per channel) that does better?

I am assuming that whatever algorithms you have been considering for automatic generation of presentation states "prospectively" are too computationally expensive to implement in the viewer "retrospectively" on the fly, otherwise you could just apply the same algorithm in the viewer and get the same result.

I did look at this VIM presentation Image rendering: perceptual and technical issues to get some sense of what the pattern of statistics of the pixel data looks like, which is why I mention looking at the statistics on the log of the pixel data values. I am not suggesting that you implement extraction of Gaussian mixture models in the viewer.

I see that you used an HTAN-HMS image for testing; did you have similar unsatisfactory results with the (multi-channel) images from the other atlases? Each will have different characteristics.

[Gibbsdavidl]

Just a thought: have you seen this repo Adam is working on?
miniature

I believe it's for creating thumbnails.

""" An approach using dimensionality reduction to create thumnails for high-dimensional imaging. Miniature enables rapid visual assesment of molecular heterogneity within highly multiplexed images of complex tissues.

Miniature images embeds each pixel into low dimensional space by corelation distance, and colours them by conversion of their position in low-D space to LAB colour. Therefore areas of similar colour can be expected to have comperable marker expression (within a single image). """

[hackermd]

As I mentioned previously, finding "optimal" or visually pleasing default window limit values can get complex.

We could try to further improve the display setting by computing different statistics (e.g., 0.1 and 0.9 percentiles instead of min and max). However, I am reluctant to add more complexity to the viewer for two reasons: First, because it may have a negative impact on viewing performance. Second, because I don't want Slim to become too smart. Slim should in my opinion make as little assumptions about the data as possible, and rather display data as instructed either via metadata directly included in the DICOM images or via separate DICOM Presentation State instances.

Here is screen recording of the features implemented in #103.

If Slim finds a DICOM Advanced Blending Presentation State instance, it will automatically apply the described grayscale transformation sequence, i.e., select the appropriate images, set the limit values for each selected image using the VOI LUT, and then pseudo-color the pixels using the Palette Color LUT. In case there is more than one DICOM Presentation State instance, the first one will be selected.

If there are no DICOM Presentation State instances, Slim will apply a default "presentation state". To this end, it will select the first three optical paths, set the limit values for each selected image by computing the min/max values for the THUMBNAIL image (the first image frame that gets loaded into the viewport), and then pseudo-color the pixels of the selected images using linear interpolation between black and blue/green/red. The default presentation state will also be visible, if the presentation state is reset.

https://user-images.githubusercontent.com/6855297/171314747-e69f307a-2f4a-4e11-a9ff-f85968b8df11.mov