Hide Images from Multi-Image Filesets

[will-moore]

See https://forum.image.sc/t/filter-only-main-image/42214

There are times when a multi-image file-set includes superfluous images that you don't want to see or use in the future. For example, 'thumbnail' or 'macro' images, often in WSI images but also found in other formats.

Options:

• Simply 'cut' the superfluous images from the Dataset and let them become "orphaned"

  • Disadvantage: You will soon end up with hundreds of orphan images - Then if you accidentally make any other image into an orphan (e.g. import not into a Dataset or accidentally 'cut') then you will struggle to find it to put into a Dataset

• Put the superfluous images into a different Dataset. Probably the best option currently.

  • Disadvantages (compared with delete option): You will still need to include these images in chgrp actions with the fileset. Images will still show up in search results etc.

• Filter/Hide within the same Dataset. e.g. add a rating or Tag to all the non-superfluous images, then filter by this to hide the superfluous ones.

  • You can't currently exclude particular images when filtering images in a Dataset, but this could be added.
  • Disadvantage: You'd have to apply the filter every time you opened the Dataset. We don't "save" filters.

• Delete the superfluous images: Needs some more investigation.

  • Disadvantages: You can't 'undo' except by download and re-import.
  • Advantage: Images are really gone and you won't have to manage them in future

EDIT: clear preference below for Filtering within the one Dataset to preserve the 'macro' images beside the originals, allowing users to view them if they wish.

[mtbc]

Delete the superfluous images

• If no underlying files are deleted then one could add an in-place import feature to resurrect the whole.
• It should be possible to delete underlying files, i.e. those not present among Bio-Formats' getSeriesUsedFiles for any remaining image.

[joshmoore]

• It should be possible to delete underlying files, i.e. those not present among Bio-Formats' getSeriesUsedFiles for any remaining image.

What files are you thinking of? Pyramids and attachments?

[mtbc]

The files originally uploaded into the managed repository too. E.g., HCS readers will tend to be able to point to individual TIFFs, so one could delete a run and leave the others.

[joshmoore]

I was under the impression that there's no XOR guarantee between what getSeriesUsedFiles returns.

[mtbc]

There doesn't need to be? It just needs to report all the files that are needed to show the given series (so those files are then excluded from the deletion).

[imagesc-bot]

This issue has been mentioned on Image.sc Forum. There might be relevant details there:

https://forum.image.sc/t/filter-only-main-image/42214/2

[imagesc-bot]

This issue has been mentioned on Image.sc Forum. There might be relevant details there:

https://forum.image.sc/t/filter-only-main-image/42214/6

[imagesc-bot]

This issue has been mentioned on Image.sc Forum. There might be relevant details there:

https://forum.image.sc/t/filter-only-main-image/42214/7

[mstritt]

I guess an important point is to differentiate between a filter series==0 vs imagetype!=macro

I can understand that an general filter "series==0 only" might not be suitable for other formats, but a "hide macro" filter would be fine - independent of the format. At least in the web viewer it seems OMERO is aware of a macro because the files have a [macro] at the end. So would a "hide ([macro] and [macro mask image])" filter be feasible?

Manuel

[mtbc]

If permitting "Delete the superfluous images" then it would be worth revisiting https://github.com/ome/omero-blitz/pull/100#issuecomment-687080165 as there is no point in restricting duplication of partial MIFs if deletion does not preserve their wholeness either.

[will-moore]

@mstritt I think it could be feasible. We need to get more feedback from the community, to judge whether that behaviour would disrupt the expectations or workflow of others. Also to assess the scope of that feature and to decide on demand/priorities. In the meantime, try putting the macro images in a different Dataset (either manually or via script, possibly as part of import). Regards, Will

[sbesson]

Another potential solution is that it should be possible to not import these superfluous images into the system in the first place.

While ignoring these ancillary images unconditionally is not feasible, the Bio-Formats API supports reader-specific options allowing the user to choose specific behavior at the level of the individual filesets. We are currently discussing an extension of this API allowing to serialize these options to disk - see https://github.com/ome/bioformats/pull/3605 for more details.

Concretely, in the example above, an option could be added to the NDPIReader to ignore the macro and the macro mask image during the reader initialization. Adding an option file with the correct naming pattern in the same directory as the data would lead OMERO to only import the WSI data into the system and remove the need for client-level filtering e.g.

ndpi.macro =true
ndpi.macromask =true

[jfkotw]

Just wanted to jump in on this discussion to clarify the thinking that @mstritt and I had on this issue. First and foremost, we see the root cause of the issue to be a user experience issue. Our users are mostly interested only in the whole slide image, therefore 99% of the time they only want to see the main image. However, there are cases where it would be useful to view the slide overview (e.g. to double check a barcode, or manually drawn label), or to check the mask area is as expected.

I think that means that deleting, or not importing the overview/mask images is not an option. Moving to another dataset also feels wrong (although we do lack some perspective on the Omero data model).

Trying to think from the user perspective, I could imagine adding a decorator icon to those images that shows that there are more related images. E.g. Photoshop elements organizer generates 'stacked images' for visually similar photos taken in succession (https://www.youtube.com/watch?v=lJJNDOYOCIE). An MVP would probably just allow a filter checkbox that shows/hides the extra images. Maybe there would be a global setting that allows that to be set default on/off.

I'm open to a quick call if any further input would be helpful.

[imagesc-bot]

This issue has been mentioned on Image.sc Forum. There might be relevant details there:

https://forum.image.sc/t/expected-behaviour-when-opening-an-image-in-iviewer/42571/2

[imagesc-bot]

This issue has been mentioned on Image.sc Forum. There might be relevant details there:

https://forum.image.sc/t/expected-behaviour-when-opening-an-image-in-iviewer/42571/5

[will-moore]

Thanks @jfkotw - I updated the description above to indicate that's the preferred solution. Is the current filtering sufficient? (except maybe missing an 'exclude' option)? If so, then from a TODO point of view, I see a couple of main parts to this feature:

• Storing of filter settings:
  • I imagine we could use a MapAnnotation on the Dataset, with a designated namespace e.g. omero.client.imagefilter to indicate this represented a filter. With Key-Values like: name: series[0], rating:5, tag: GFP etc. Maybe even need a version number in case we change how we want to store these.
  • You'd want to have a "Save" button alongside the Filtering controls to be able to save the current state.
  • When we load the Dataset in webclient, we'd have to always check for the presence of the filter annotation and apply the filter accordingly
  • We may want other clients such as iviewer (see above) - probably not Insight? to respect either the saved filter or the current filter.
  • It may also be useful to have the ability to apply the saved filters on the omero-web server. Then iviewer could just request images from Dataset with query ?filter=true and not have to worry about implementing filters itself. This could also help with pagination issues (discussed https://forum.image.sc/t/paging-behaviour-in-thumbnail-parade/42573) but not if we also do client-side filtering (as in webclient)
  • I can imagine users wanting to be able to copy a filter from one Dataset and apply/save it to another.
• Automatic creation of settings:
  • When images are imported, you want a particular filter applied immediately.
  • If you are scripting your own import, it could be added to that using the CLI or Python/Java API.
  • I'm not sure if there are other "hooks" in the import process where this could be implemented?
  • Or do we want OMERO to do something "out of the box".

[imagesc-bot]

This issue has been mentioned on Image.sc Forum. There might be relevant details there:

https://forum.image.sc/t/correct-way-to-convert-svs-slides-for-omero-import/50942/22