search

FNNDSC / cube-design

All CUBE related designs
MIT License
5 stars 2 forks source link

Concepts / Ideas around visual diffs & layering in Radiology viewer #8

Open mairin opened 6 years ago

mairin commented 6 years ago

Problem statement

Researchers (maybe in some clinical cases, maybe not) want to be able to visually compare / contrast different images that are registered to one another to understand the differences, or compare / contrast an image with the output of an image plugin (e.g. a segmentation map) to inform their viewing of the image (or an image diff.)

We need to provide these users controls in order to understand the differences between images in an uncluttered, intuitive, and manageable way.

Low-level use cases

Some practical low-level examples of what the user might be seeking to do:

Design assumptions

(If any of these assumptions are wrong, please flag!)

  1. The viewer UI has a high-level and consistent concept of an image / image set that is displayed within one or more viewers within the interface. Transforms / controls (e.g. LUT) can be applied to the image / imageset object and simultaneously impact one or multiple viewing panes in the viewer interface based on those transforms applied to the high-level image / imageset object.
  2. Users will want to step through each image set (plane-by-plane along the selected axis / viewing angle) while viewing images in this visual diff mode.
  3. The selection of images available in the viewer for the user to drag onto the viewer interface / grid of viewing panes is already set. (We'll need to cover how in another ticket.)

Questions re: user base

  1. At what scale is a 'visual diff' workflow being done? E.g. might a researcher perform this manually on hundreds of images?

Relevant UI patterns

Below is a summary of some UI patterns @NicolasRannou and I talked through regarding visual diffs which may be good food for thought / models to follow or riff off of here.

  1. Two-up / Side-by-side
images-2up-view

(from https://help.github.com/articles/rendering-and-diffing-images/#2-up)

  1. Swipe

Github has a version of this too in their image diff tool. It's basically a vertical slider that divides the image into two panes, so you can 'swipe' between the two layers / images. The example below is from Gimp 2.10.

images-onion-view

  1. Onion-skin / Layer blending

This is the basic model most graphics applications follow for multi-layered images. It involves a stack ordering of images / layering, and each individual layer has an opacity property (and maybe other properties too.) Github's version uses difference blending between layers, based on another document (referenced below.)

images-onion-view

(from https://help.github.com/articles/rendering-and-diffing-images/#onion-skin)

  1. Diff Highlights

Basically there's a highlight layer that shows where the differences are between the two images. This would only be useful in comparing two registered images. This sort of pattern is used for example in highlighting out-of-gamut-colors for printing press preparation in a DTP app or highlighting areas unreadable by the color blind in an image creation app. The first two examples below are visual difference highlighting for photos, and the last one is an example of tracking differences over multiple photo frames in a surveillance context.

images-diff-highlight

https://youtu.be/1mOCk4QxPAo?t=12s

https://www.youtube.com/watch?v=yQBu8Bz22rQ

Those are pretty much on / off - there is a difference or there isn't. This doc has some interesting examples including one that shows a measurement of the variation of the difference based on how dark the diff marks are:

https://jeffkreeftmeijer.com/ruby-compare-images/

  1. Masking

Basic concept: basic-mask

Super-fancy version: fancy-mask

  1. Interactive Masking
rob-baron commented 6 years ago

If you are dealing with a before and after image, don't you need rotation and some means of stretching on image as patients do not necessarily get into the same position.

danmcp commented 6 years ago

@mairin @NicolasRannou This is a really great breakdown. The problem is I can imagine each of these being useful in different circumstances. Do we have any idea how feasible each of the different implementations might be?

NicolasRannou commented 6 years ago

@robbaronbu you are right, maybe at some point, we will need it. Right now the assumption is that both images are registered into the same position.

NicolasRannou commented 6 years ago

@danmcp we are missing 4) and 5) - all the rest is already supported by ami. -> play with "layer mix" at: https://fnndsc.github.io/ami/#viewers_compare

I can not foresee challenges for 4) and 5). I think those are not common comparison methods in the medical world but not necessarily because it is not useful, maybe also because it is not supported by the current tools.

mairin commented 6 years ago

I think maybe #4 is a requirement for #5, because you would use the shape(s) generated by #4 as the mask for #5.