Web Viewer - Brainstorm - Concepts on New Interaction Approaches - 1

[gusferguson]

Having talked over the problems with legacy and design in the existing Insight Measurement Tool with J-M, we had some new ideas on how to accommodate ROI workflows better in the upcoming Web Viewer.

This is a first blast at trying to formulate the sorts of UI interactions that might make some of these ideas possible.

This will be expanded as more of the functionality and interactions start coming together and I do mockups of them.

Please add comments and feedback.

Caveats:

• this is very conceptual and very much a first brain-dump so polish and finesse is a bit patchy
• I have not done an iteration with J-M yet - so these are largely my take on what he said
• with the multi-layered complexity of new unreleased features such as ROI folders, new technologies for the Web Viewer and so on, we are getting to the mockup upon mockups stage, so details can be missed/messed up - if any identified please indicate
• need to really careful not to disrupt existing user's mental model of ROIs and ROI drawing
• still need to thrash out what shapes and ROIs are

The key points raised and this brainstorming aims to address are:

• provide for ROIs not attached to Z or T (equivalent of “overlay” in ImageJ)
• easy way to take unattached ROI an attach or “stick” them to Zs and/or Ts
• feedback on intensities while drawing ROIs
• using palettes to free up real-estate and allow ROI drawing and access to settings etc
• running scripts from Viewer
• indication of ROI number on thumbnails
• indication of ROI distribution along Z and T sliders (still to come - time ran out)
• do all shape-property editing in pop-up i.e. current Inspector tab - fill, line thickness, tagging etc. (still to come)
• ROIs containing different kinds of shapes (still to come)
• ROIs across images (still to come)
• metadata on mouseover of thumbnails - needs a lot more thought

Requirements/Task analysis

This still needs to be expanded with lab users.

Taking an abstracted view, the diagram below gives an idea of the scope of the mockups.

Mockups:

Default Viewer on opening:

• thumbnails down LH side
• settings in RH pane - pane collapsable to give more real-estate (see following image)

Any of the RH pane tabs can be broken out as palettes allowing more flexibility

• this shows the minimal ROI palette (more detail later) being used with a zoomed in image

This is normal 512x512 image with RH pane open in ROIs tab

• Drawing Mode and Analysis accordion tabs will be explained later

A couple of new toolbar items

• open as palette button
• stamp - i.e. one-click drawing of selected existing ROI
• Save to All

Showing the drawing tools from the ROI tab as a palette

• expanded to show Drawing Mode options

The various propagation options can be selected and then the Drawing Mode can be collapsed to maximise real-estate. The idea is that you get into your mode of working and then add ROIs as efficiently as possible by:

• clicking on a particular Z and T
• just Zs
• just Ts
• or moving through Z and T using sliders and ROI is automatically added as the section is viewed
• specifying the Z and T (like selecting pages in Print dialog)
• or graphically as it is at the moment (manually or graphically using pop-ups)

Improving the dynamic interaction between the analysis and drawing

• Analysis accordion tab expanded in RHP
• start drawing a line
• feedback in the Line Profile and Histogram

• position end of line as desired

This could be augmented with dynamic Intensity results while drawing

• examples for line and point

These could both be dragged out as palettes

At any stage full results can be dumped out:

• in spreadsheet format
• or the graphs as a PNG via the results menu
• downloaded/saved as attachment or viewed

Another advantage of having both RH pane and palettes

• e.g. adjust rendering settings to allow more accurate drawing of ROIs

And last but not least

• mouseover the thumbnails to get metadata

To be continued ...

[will-moore]

Even before I read your "indication of ROI distribution along Z and T sliders (still to come)" I thought it would be a great feature - like Chrome does when you "find" a word and it's distribution is shown in the slider:

This could be enhanced to highlight shapes that are in the same ROI as the currently select ROI, as well as indicate ALL shapes with a less bright colour?

One thing to consider is how well we support multiple shapes on the same Z/T plane within an ROI. Insight doesn't support it at all (crashes). We need to at least allow it when we find this in the DB, but do we allow users to create multiple shapes on a plane for a single ROI?

I don't think we ever need to show a Line profile AND a Histogram. Line profile for lines, histogram for shapes.

Clone tool is nice idea. It needs to be beside other shapes in the toolbar, since it is a mutually exclusive / alternative tool to other shape creation buttons.

[gusferguson]

Reminder: also need some way to distinguish non-attached "overlay" shapes from attached ones.

[will-moore]

Some features of Google maps might be useful to consider - E.g. organisation of multiple shapes into "layers", editing shapes, floating palette & toolbar etc

[jrswedlow]

@gusferguson There's alot to like here. @will-moore 's slide idea would be GREAT!!! Finding ROIs in large stacks or timelapse sequences is sometimes quite hard. I suspect layers, pallettes, etc need to be considered in the context of large (>50) ROIs/image. GMaps sometimes frustrates me with its insistence on placing layers etc on the stuff I want to look at.

Keep 'em coming!

[waxenegger]

I like your ideas @gusferguson. Viewing it from the user's perspective I believe that lots of what you mocked up is quite useful.

@will-moore in general there are quite a few products out there that do one or the other thing well, particularly if one looks at the features. there are, however, a few things to be said:

1. google maps is inherently 'geo' in how it wants the data. Thinking of omero-marshal and api that means pretty much one or the other (unless there is some common exchange format that will work which would require more overhead of wrapping/converting most likely)
2. I'm not a lawyer but, in one of my previous jobs, they would not for the world let us use the maps api and google maps as base layers. They claimed that there was something that made it not entirely free. Now I don't know what nor whether that was 100% correct but I only remember them saying that. It might have applied to commercial use or so only...
3. many products tend to be especially good at some things, not so good at others. if it's for idea gathering only, nothing wrong to steal, but, technologically, using a lot of them to achieve exactly what one wants might be tempting as well as dangerous: Version conflicts (even in java script), unwanted interactions, perhaps different formats expected by each of them, the question of maintenance/responsibility.

Don't get me wrong I'm not trying to be exclusive or negative, I just wanted to point toward some pitfalls that's all.

I have one other nitpicky (technical) remark in regards to @gusferguson's z/t orphans: I'm sure it could be implemented easily enough in the front-end (I do have a context menu on the viewer that I'm working on so that one could assign regions/clusters of regions to a specific z/t easily, and putting a list of orphans on any layer is also not rocket science) but, in the end, a corresponding decision has to be made how orphaned shapes are to be persisted and dealt with in the data model. Does one need another table since there is now a n:m relationship? Or does one simply store them redundantly for each t/z anyhow? Bear in mind that that impacts on both the storage effort as well as the use thereafter. One is at least in the first instance a shared shape (therefore less storage) although the user could later decide to just edit that one shape in that z/t context so that it would have to be singled out and stored individually anyhow. Once again, no showstopper by far, just something to keep in mind.

ok, I have finished my comment of epic dimensions. ... ;-)

[waxenegger]

I have actually a question since I'm at the moment trying to find the best way to interact with regions and there are a few options available. In specific I'm talking about selection and selection if one has a clustered view, something that will be (at a minimum in the web) a necessity IF the goal is to be able to view a large number of rois, say more than 500 only actually, in what is a small viewing extent of the image. The view can get cluttered and overloaded quite quickly, but depending on the overall image size, even for larger images (beyond 10,000 pixels square let's say), anything over 10K shapes in an zoom level that reveals the entire extent can be a bit of a challenge and clustering seems to be a not-too-shabby solution.

But how does one best interact with the clusters. Does one reveal what's underneath on mouse over? I tested that and it works well in webkit based browsers but not well in Firefox or IE. Does one reveal them when clicking on them (via context menus rightclick or left simply)? Or should the user just zoom in onto an area before they get to see in which case one would have to have knowledge of what's there or otherwise it's a bit frustrating to find something I think...

What should one do with multiple selections when in a clustered view? If I reveal too much I'm sort of back in the unclustered view because all of the sudden I want to show lots of things again, so perhaps only one cluster and how many shapes should be the limit that are behind that cluster?

And, even worse, what would I do if I transition back/forth between a zoom level that needs the clustered view and one that shows the individual shapes (I'm zoomed in already). If I select something there, should it be still selected when I'm zoomed out and back in the clustered view. Should I make it sticky, and if so, what happens if it is sticky (selected) and I then work in another region and perform an action on the selected things and forgot that somewhere else I had selected a shape ...

Honestly, I don't know how to deal with those questions and they have technical implications as well.

[will-moore]

@waxenegger Apologies for the confusion. I wasn't suggesting we use google maps at-all, except for getting ideas/inspiration from their UI (hence the screenshot).

[will-moore]

@waxenegger In the current roidisplay.js in the webclient, to find what shapes to show for the current plane we simply iterate through all shapes and pick those with matching Z & T OR no Z / T set. There's no duplication of shapes so our js model matches the OMERO model.

[will-moore]

@waxenegger For clustering, it would be good to see your current progress on this. I can imagine problems when clustering shapes of different type/size. Clustering works well on maps for points, but if you try to cluster some points with lines / ellipses / rectangles that could be as large as the image itself then how does this work? I can see that clustering will help prevent the UI from becoming too crowded, but with a large number of shapes, we will still hit bottlenecks with retrieving and marshalling all those shapes into json. It may be that we need some pagination strategy, with UI widgets for loading / browsing an overview of the shapes. Possibly with the server providing a summary / cluster of the ROIs and Shapes rather than loading ALL the data to the browser. These issues are likely to have a big effect on any client-side data model so are important if we want to handle a lot of data. Q: do we have an idea of the numbers of ROIs/Shapes we are intending to support? cc @sbesson @manics. I feel that clustering of shapes on the image viewer itself may be a useful solution in some cases but is not going to be the main blocker to working with large amounts of ROIs and Shapes. It might be that there's some threshold number of Shapes where we no-longer allow editing in the client (E.g. created by analysis results) so we may not need to handle clustering and shape editing at the same time?

[waxenegger]

@will-moore

about google-maps: I wasn't quite sure. either way it was just a remark and by no means criticism

regarding the z/t shapes: this would be only a potential question for the implementation of the z/t "orphans" that were mentioned. Unless they are purely temporary in a sense that they are not persisted the question does not arise. If they are stored as not belonging to a partiular z/t, I suppose they can stay without designated z/t as well and there is no issue. Same is true for backend/data model and front end. But once they start belonging to something and are being treated unique they need to be stored as such. Yes, duplication, is therefore not a good term at all I agree, only if one were to store a db record for such an occasion to begin with.

[waxenegger]

@will-moore

thanks for input on the clustering. yes I aim at showing some next week, tonight I'll have to go in a minute, sorry. I'm not sure I understand the size of the shape comment because any shape by itself could potentially cover the image if drawn big enough and filled without transparency.

A common approach is to just place a circle in kind of the center of the group of shapes as a representative and, on demand, you show them. This does help a bit with performance and sort of hides the complexity of the scene.

What you say about retrieval of data is true and I did mention in one of the meetings with Chris that selective querying on demand in terms of z/t could help if one were to have a lot of shapes AND not all in the same plane. If they are all in the plane the transfer would take a while and one could roughly calculate the number of bytes by estimating the size of the average json definition for a region and then assess how big the content would be and what is feasible and what not. But say one region definition needed to render in the client is 1,000 bytes (I think that's generous, right) then 10,000 would mean 10MB which takes a while, no doubt, but it's doable.

Yes there is also the marshalling that needs to be done server side, I didn't forget about that, and I suppose that needs testing.

These questions remain, but strictly speaking, they are the same for say the mask. At the moment one could generate a mask about 1GB large (db column limit) and if one were to retrieve it, it would also be a bit of a strain on the server and the communication, and if one really did it, one would have to resort to other strategies.

I believe there is also some room for improvement to make the regions work up to a certain threshold and although I don't have a concrete number I'm pretty sure it's high enough to need something in the client because the number might overwhelm the viewing extent.

[will-moore]

@waxenegger Re clustering - using this as an example: http://openlayers.org/en/v3.15.1/examples/cluster.html All the features are points, but what if one of the features was a road that went from one side of Africa to the other? Would I be able to see the road when fully zoomed out or would it be clustered with other points? How much would I have to zoom in to see it? If I have to zoom in until every feature is not clustered then I wouldn't be able to see the whole road at once?

[gusferguson]

Thanks for all the feedback - I will press on with the remaining mockups and work on the suggestions. @waxenegger - re ROIs not attached to Z or T - that is supported in the model already, just not exposed in the clients.

[gusferguson]

Further mockups in - Web Viewer - Brainstorm - Concepts on New Interaction Approaches - 2 https://github.com/openmicroscopy/design/issues/34

[gusferguson]

Summary of points raised in this issue - Viewer Brainstorming 1:

ROI Distribution wrt Z and T

Will:

• Chrome does similar with location of search results in page indicated in scroll bar

Enhancements:

• highlight shapes in the same ROI as the currently selected shape with brighter colour
• currently Insight crashes with multiple shapes on the same Z/T plane within an ROI
• do we allow users to create multiple shapes on a plane for a single ROI?
• GF note: don’t know how it handles shapes of different types in same ROI (possible in model)

Jason:

• good idea to pursue

Graph Display:

Will:

• just need line profile for lines, histogram for shapes (not both for line)

Rendering Settings

• need histogram display in rendering settings tab - allow adjustment to shape of histogram (Icy has e.g.)

Clone tool:

Will:

• should be beside other shapes in the toolbar (it is a mutually exclusive/alternative tool to other shape creation buttons)

Overlay Shapes (shapes not attached to any Z or T)

Note: these are already supported in the model

Gus:

• need some way to distinguish non-attached "overlay" shapes from attached ones

Layers

Will:

• some features of Google maps useful to consider - e.g. organisation of multiple shapes into "layers", editing shapes, floating palette & toolbar etc. Jason:
• layers, pallettes, etc. need to be considered in the context of large numbers of ROIs on an image (>50) .
• GMaps can be frustrating with insistence on placing layers etc. over the area of the image I want to look at.

Harald: Some caveats:

• could not use Google API directly - clashes with proposed frameworks
• legal issues with use
• functionality does not map well into OMERO domain
• maintenance issues

Clustering

Harald: Will be needed if:

• 500 or so ROIs in a small area of the image

• 10K shapes in a 10Kx10K pixel image Questions with interaction with clustered images ? do you:

• reveal what is underneath on mouseover
• reveal when clicked on
• just reveal on zooming in
• how to handle multiple selections
• how to balance between revealing enough and defeating point of clustering
• when transitioning from clustered to non-clustered views or Vice versa - e.g. on zooming in and out - how are selections of individual shapes handled - are they lost, retained, indicated by flag on cluster etc.

Will:

• could be problems when clustering shapes of different type/size
• clustering works well on maps for points, but if you try to cluster some points with lines / ellipses / rectangles that could be as large as the image itself then how does this work?
• clustering will help prevent the UI from becoming too crowded, but with a large number of shapes, we will still hit bottlenecks with retrieving and marshalling all those shapes into json
• may need pagination strategy with UI widgets for loading/browsing an overview of the shapes
• server could provide a summary/cluster of the ROIs and Shapes - avoid loading ALL the data to the browser. These issues likely to have a big effect on any client-side data model so are important if we want to handle a lot of data. Q: do we have an idea of the numbers of ROIs/Shapes we are intending to support?
• clustering of shapes on the image viewer itself may be a useful solution in some cases but is not going to be the main blocker to working with large amounts of ROIs and Shapes.
• ? need a threshold number of Shapes above which editing in the client is not allowed (e.g. created by analysis results)
• this would mean clustering and shape editing would not need to be handled at the same time

Harald:

• re: clustering of large shapes - potentially covering the whole image - common approach is to just place a circle in the center of the group of shapes as a representative and, on demand, you show them
• this helps with performance and hides the complexity of the ROIs
• as masks could also be very large - will have to be dealt with as well

Will:

• using this as an example: http://openlayers.org/en/v3.15.1/examples/cluster.html
• all the features are points - but if one of the features was a road that went from one side of Africa to the other - would I be able to see the road when fully zoomed out or would it be clustered with other points?
• how much would I have to zoom in to see it?
• if I have to zoom in until every feature is not clustered then I wouldn't be able to see the whole road at once?

[jburel]

adding the following for ref https://trello.com/c/yuRPX18c/65-viewer-issues