iviewer many ROIs

[will-moore]

We need to be able to handle & browse (and edit?) larger numbers of ROIs in iviewer.

Limitations on large numbers are:

• Loading from server (current limit 500 - see below)
• How many can the OpenLayers viewer handle?

Currently we don't support pagination so we are limited to the max for a single page specified by $ bin/omero config set omero.web.api.max_limit which is 500 by default (see https://docs.openmicroscopy.org/omero/5.4.9/developers/json-api.html).

We could add pagination support to load more ROIs, but we probably want to keep all loaded ROIs in hand as much as possible instead of a single page at a time. This is similar to thumbnails on left panel - we load 10 at a time but don't discard any so it's possible to load ALL eventually.

• How to request "next page"?

  • If the image is multi-T or multi-Z, we could request ROIs that have shapes on the current plane, but this may still be too large to be a single page. Do we load ALL shapes for an ROI at once (even if only 1 shape is on the current plane?)
  • Load by scrolling the ROI table (similar to loading thumbnails). But, if you pan around the image and don't see any shapes it could be confusing that you need to scroll the table to load more.
  • We just try to load ALL the ROIs at once, one page at a time, with a progress bar and Cancel option. This is probably the best option since we mostly want to work with all ROIs at once. But, what is the maximum number that we can support (OpenLayers and Aurelia can handle)?

• What to do when:

  • We have more ROIs than OpenLayers can support? (need to know numbers)
  • User doesn't want to wait for loading of ALL the ROIs and they only want to load & view ROIs on the last time-point or at a small region of a Big image?

What does GS do with Pathviewer?

cc @jburel @joshmoore

[will-moore]

Do we have any way to query ROIs / shapes and filter by coordinates? Easy enough for e.g. Points and Rectangles (unless they have a crazy transform on them) but what about Polygons & Polylines? Without that it's impossible to load shapes for a region.

Clustering (example: https://openlayers.org/en/latest/examples/cluster.html) is nice but we still need all the shapes loaded in the browser. It might be possible to load a clustered ROI summary from the server for a particular zoom level (like loading tiles for low resolutions) but this would still require a DB query based on coordinates.

[manics]

Polygons/polylines are stored as a single strings of coordinates so the only way to query them would be to parse them all... not very efficient. Masks have a bounding box (though that's defined by the user so can include empty space), maybe this could be added to the DB for polygons/polylines?

[pwalczysko]

Actually, there was an externally coming RFE, which requested that iviewer would show the ROIs whilst hovering with the mouse over the image. The person coming up with the RFE was struggling to find the ROI in the image (large image, small ROIs). Although it is not directly what is being discussed here, it gives a perspective about how the user wants to work with ROIs (and their loading) -> image centric, not table centric (see your second bullet point in "how to request next page" @will-moore )

[joshmoore]

maybe this could be added to the DB for polygons/polylines?

A previously discussed alternative was to make use of postgis for these look ups.

[will-moore]

See https://github.com/ome/omero-iviewer/pull/227 for an investigation of loading shapes by tile for big images.

[will-moore]

After discussing https://github.com/ome/omero-iviewer/pull/227 with @jburel, we considered various stages of support for lots of ROIs:

• Many ROIs not editable:
  • If image has "many" ROIs (e.g. > 500) then we load ROIs by tile instead of via Aurelia UI
  • Don't show a table of ROIs
  • Shapes not viewable on the image until we zoom to load 100% tile level.
  • Main limitation: no way to know where the shapes are until you find them at 100%, correct plane etc
  • Selectable Shapes:
  • Select Shape to show ID, label, coordinates, length/area and load intensities
  • Browsable ROIs:
  • Need to show users where the Shapes are: which Z/T planes and where on the plane
  • Shape counts for each Z/T index - show on right panel
  • At lower zoom levels, show where shapes are e.g. a heat-map (or cluster counts) to show which tiles/regions have most Shapes
  • Need server to have pre-calculated "pyramids" of shape counts per tile (similar to image pyramid).
  • Editable Tile-loaded ROIs
  • If possible, allow editing of Shapes
  • NB: When loading Shapes by tile, we don't load ROIs or have ROI links.

[will-moore]

After thinking about this a bit more, I think we can get a long way by:

• Enabling pagination of the ROI table. E.g. 100 ROIs per page.
• Clicking a Shape on the table takes you to that point in the image (at 100% zoom so you will see it) and selects the Shape on the image.
• Use Tiled-ROIs on the image - clicking a shape would load the correct page of the table.

This means that you can still edit ROIs and Shapes and you can still find Shapes on the image via loading them in the table first. Limitations with pagination:

• We currently sort ROIs by ID, but if you wanted to sort by Shape.theT or Shape.theZ then we'd need to be able to do a DB query "Give me the 3rd page of 100 ROIs sorted by Shape.theT" which could be tricky, especially for ROIs with many shapes (some shapes may have theT=null).
• We'd also need a "look-up-table" of ROI:Shape ID to page. E.g. user clicks on Shape:45235 in the image, we need to know which page of ROIs that is on to load that page into the table. For images in a paginated Dataset we do this in paths_to_object by getting all ImageIDs in Dataset.

[mtbc]

It's good you're pushing the client side: if need be we could even consider a bulk-ROI microservice that would maintain a consistent API for clients while we figure out server-side indexed storage like PostGIS.

[will-moore]

Following the demo of Tiled ROIs https://github.com/ome/omero-iviewer/pull/242 at the team meeting yesterday, I have added the current API query I'm making to the Trello card at https://trello.com/c/Tsv0yVe1/97-query-shapes-by-geometry

[will-moore]

From @chris-allan: see example at https://openlayers.org/en/master/examples/icon-sprite-webgl.html which loads 80k points. Could use this approach to show the centre-point for a similar number of shapes on an image.

[chris-allan]

See also:

• https://openlayers.org/en/latest/examples/cluster.html
• https://openlayers.org/en/latest/examples/earthquake-clusters.html

/cc @knabar

[jburel]

Harald looks at clustering a while back so it might be worth resurrecting some of the investigations work