search

napari / napari

napari: a fast, interactive, multi-dimensional image viewer for python
https://napari.org
BSD 3-Clause "New" or "Revised" License
2.2k stars 421 forks source link

Points & polygons position inconsistent in grid view mode (needs spatial coordinates) #860

Open GenevieveBuckley opened 4 years ago

GenevieveBuckley commented 4 years ago

🐛 Bug

Points and polygons do not share spatial coordinates with images they are selected from. This means when you switch grid view on/off points & polygons no longer overlay the image.

This is just one example of what we're trying to fix with the discussion around spatial coordinates at https://github.com/napari/napari/issues/763. I think the self-contained example in this issue could be useful as a test case.

To Reproduce

Steps to reproduce the behavior:

  1. Open napari with an image.

    $ ipython --gui=qt
Python 3.7.5 (default, Oct 31 2019, 15:18:51) [MSC v.1916 64 bit (AMD64)]
Type 'copyright', 'credits' or 'license' for more information
IPython 7.10.1 -- An enhanced Interactive Python. Type '?' for help.
In [1]: import napari
In [2]: napari.__version__
Out[2]: '0.2.3+104.g0d1c812'
In [3]: from skimage.data import astronaut, camera
In [4]: viewer = napari.Viewer()
In [5]: viewer.add_image(camera())

  2. Add a points layer, and select some points on the image.

    napari_points_image_1

  3. Click the "grid mode" view.

    napari_points_image_2

Expected behavior

Points and polygons should share spatial coordinates with the underlying image data, and always be displayed overlaid regardless of which image mode (grid-view/2D/3D) is selected.

Environment

GL version: 4.5.0 - Build 25.20.100.6472 MAX_TEXTURE_SIZE: 16384


 - Any other relevant information:

## Additional context

<!-- Add any other context about the problem here. -->
Relevant issues: https://github.com/napari/napari/issues/763
jni commented 4 years ago

@GenevieveBuckley actually this issue is somewhat orthogonal to the world coordinates. Grid mode does not modify world coordinates. This is what we would call world2canvas, and grid mode adds a translate transform to each layer so that they are separated on the canvas. But the world coordinates of each layer are preserved.

To overlay the points on the image, you need to add stride=2 to the grid mode call. (There is no way to do this in the GUI yet, but it could be done with a modal popup like the one @tlambert03 added to the play button.) Then things work as expected.

This will be necessary even after we switch to properly separating world2canvas and data2world transforms.

One idea I had is that layers could be in groups, ie you could have non-uniform strides in the grid mode, so you could have a group of three layers offset from another group of two layers. This might be useful e.g. if you want to see an image, its segmentation, and points, together with another image and points (but no segmentation), in order to register the segmentation.

Overall, this requires a bit of fiddling with our layer list interactions, but it could happen before or after the world coordinates.

GenevieveBuckley commented 4 years ago

One idea I had is that layers could be in groups

I think this could be very useful, especially if there was support for hierarchies within groups.

Eg: you will often have two or more treatment groups, with multiple samples in each group, and many cells in each sample. Being able to group them and toggle the visibility on/off for different subsets of the hierarchy would help you see if, say, all the cells from a single sample were messed up (maybe the slide was stored improperly or something - this is a terrible example because I'm a physicist not a biologist, but you can see how it would be useful!)

GenevieveBuckley commented 4 years ago

To overlay the points on the image, you need to add stride=2 to the grid mode call.

This is not a generally applicable solution, is it? It should work in the example I have here where there is one image and one points layer, but if there are an arbitrary number of images and an arbitrary number of points/polygon layers, all added to the viewer in a random order - am I right in thinking your suggestion may not work in a more general case?

jni commented 4 years ago

but if there are an arbitrary number of images and an arbitrary number of points/polygon layers, all added to the viewer in a random order

Correct: you need to have some sort of regular cadence. Layers can be rearranged so the random order is not a huge concern, but you can't mix and match image layers with corresponding points and others without, for example.

GenevieveBuckley commented 4 years ago

you can't mix and match image layers with corresponding points and others without, for example.

To me this suggests something probably needs to be re-thought about the data models/objects. I'm not sure how, but maybe @sofroniewn has some thoughts?

sofroniewn commented 4 years ago

I think this could be very useful, especially if there was support for hierarchies within groups.

I like this idea. This will also be relevant when thinking about the true multicanvas stuff in #760 - here I'm imagining more flexibility like being able to assign layers or groups to particular canvases. This will overcome the limitations of the current stride approach, but agreed will require some re-thought around models / objects.

I think when we work on #760 it will be a natural time to work on this too. Does that sound about right?

psobolewskiPhD commented 1 year ago

I wonder if linked layers could be used to adress this? They're available in the GUI via contextual menu (though would be nice to have a visual indicator...) Linked layers would be stacked in grid view?