Properties handling should be unified across layers

[jni]

🚀 Feature

We have a bunch of layers with properties now, including points, labels, tracks, and (I think?) shapes. Really any layer with unique IDs could have properties, with rows corresponding to IDs. (ie vectors should also have properties.)

Currently, each layer's properties were implemented independently and share almost no code/infrastructure. This has resulted in weird layer-specific bugs and also unexpected behaviour where things that work in one layer (e.g. points.face_color = 'property-name') don't work in another (labels.color = 'property-name' doesn't work, iirc). Long story short, we should make a big lift to unify all property handling for all layers.

Some random comments about what to do here. I've made them tick boxes as many of them can be separate tasks.

• [ ] We should have a unified selection model as well. Currently labels only allow the selection of one label at a time. This could be super nice in combination with linked layers — you could have e.g. object centroids (points) + labels (full segmentation), do a filter selection based on the labels and get a subset of points selected.
• [ ] Mesh "values" in the Surface layer could also have/be generated from properties.
• [ ] We should probably add a dependency on pandas: they are the library/API for tabular data in the Python ecosystem and we would get a lot of optimised operations such as row/column indexing and categorical data types for free.
• [ ] Having said that, using pandas might conflict with the API envisioned in https://github.com/napari/napari/issues/2592#issuecomment-824576248. I don't know how to do "multidimensional columns" in pandas nicely. But perhaps it can be done. I think probably this is a case of having multiple tables, or a pandas GroupBy object? Dunno, this is possibly the trickiest bit.
• [ ] When dealing with properties tables and visual properties (e.g. face_color, size), I would very much like to keep in mind principles of declarative data visualisation, visual channels, encodings, and transformations. Some good introductions to these concepts can be found in this UW/Altair tutorial, and this visual encoding design slide deck. If we stick to these design principles, clearly separating the data, the channels, and the transformation/encoding steps in between, then tools like @lukasz-migas and @brisvag's cross-visualisation and filtering can become much easier and more natural to develop, and to apply across all layers.

Related issues (list should be updated as we come across more): #2755 #2756.

[brisvag]

Other related issue: #2756.

On dataframes: xarray advertises its DataSet object as an N-dimensional alternative to pandas DataFrames, and heavily inspired by it. I never used them, but mayeb they are worth taking a look at, if this can help with point number 4. EDIT: http://xarray.pydata.org/en/stable/user-guide/data-structures.html#dataset

[jni]

On dataframes: xarray advertises its DataSet object as an N-dimensional alternative to pandas DataFrames, and heavily inspired by it. I never used them, but mayeb they are worth taking a look at, if this can help with point number 4.

Worth exploring! I don't know whether that analogy goes further than skin deep though — ie if I pass in a DataSet to functions expecting DataFrames, I expect bad things to happen. 😂 But we should investigate, absolutely. 👍

[alisterburt]

While there is some unification effort underway (@andy-sweet @brisvag) I'll add a discussion point here

Currently, if you add points onto a points layer which is coloured by property the new point will 'inherit' the properties of the last added point. Here's an example you can play with

import napari
import numpy as np

n = 200
points = np.random.normal(size=(n, 2))
properties = {k: np.random.normal(size=n) for k in ('prop1', 'prop2')}

v = napari.Viewer()
layer = v.add_points(points, properties=properties, face_color='prop1', size=0.1)

# add some points manually then look at layer.properties['prop1'] and layer.properties['prop2']

This is not always wanted and makes it hard to tell what was added and what is from the data - it would be nice to be able to define what happens for newly added points e.g. provide a NaN type value or a per-column default

[jni]

it would be nice to be able to define what happens for newly added points

I think that's controlled by layer.current_properties. Of course, there is no way to access those from the UI, because that opens up hard design issues, and who needs those? 😂 Also, if I remember correctly, even if you update them from the console, they will get updated back if you change the selection or the last point's properties. So that's definitely something to consider. We definitely went with "let's think more about this later" back when @kevinyamauchi was first implementing this. 😅

But, yes, absolutely hear you, and again, this would be made easier by unified handling across layers — currently, you would have to fix this issue for each layer type individually.

[kevinyamauchi]

Great discussion everyone!

I think that's controlled by layer.current_properties. Of course, there is no way to access those from the UI, because that opens up hard design issues, and who needs those? 😂 Also, if I remember correctly, even if you update them from the console, they will get updated back if you change the selection or the last point's properties. So that's definitely something to consider. We definitely went with "let's think more about this later" back when @kevinyamauchi was first implementing this. 😅

I agree that this behavior needs to be re-visited! As @jni said, the behavior was meant to be a proposal that was then revisited after some user testing. I have a listed a few thoughts below that might give some context for the decision:

• @jni is correct, new points are given the colors/properties from Points.current_*_color and Points.current_properties respectively.
• The Points.current_* values are set by either the API or by the selecting a point in the viewer. The rationale at the time for updating Points.current_* was that we wanted users to be able to add new points that have the same colors/properties as existing points.
• Over time, there has been some back and forth over whether newly added shapes/points should be: (a) selected and/or (b) set the new layer.current_* properties. I am not sure which is the best behavior, but whatever we go with, we should probably make sure it is uniform across all of the layers and well-documented.
• As noted above, there isn't currently a way to set the layer.current_properties via the UI except for selecting an existing point/shape. We had considered adding some sort of UI to the layer controls to do this, but it felt cluttered - especially considering that users may have a ton of columns in their property table. Maybe this is best handled as a dock widget?

We have a bunch of layers with properties now, including points, labels, tracks, and (I think?) shapes. Really any layer with unique IDs could have properties, with rows corresponding to IDs. (ie vectors should also have properties.)

@jni, I believe the Vectors layer has properties and colors can be set by those properties (see this example). Now, whether the behavior is perfectly consistent with the other layers is another question 😆

Documentation

Another thing I would like to add to the "task list" is improving the documentation around properties. We have some tutorials on napari.org and some examples in the repo that show the usage of properties, but I believe we are missing documentation that explains how properties work (i.e., the "Explanation" type of documentation from the "four types of documentation"). I would be happy to help with this!

[andy-sweet]

I've made a lot of notes on this topic so far, but they mostly unintelligible. One thing I can share now that might serve as a useful reference is the following table that tells us what property related values are stored in various layers.

The vectors layer does not have a current_properties attribute, but it does make them when needed for ColorManager.

This doesn't include more specific properties (e.g. Points.face_color), but maybe it should. I see Labels.label_index as a specific implementation/optimization detail of the Labels layer, but it does relate to some shared code already, so thought it was worth listing.

[jni]

Looks like I never added what I think should be the canonical reference here:

Tamara Munzner (2014) Visualization Analysis & Design

It's a reasonably thick book for a semester-long course, but everything is nicely summarised in this blog post:

https://maelfabien.github.io/machinelearning/Dataviz/

[andy-sweet]

Here's a doc I've been working on that is my understanding of the behavior, implementation, and API of properties and related concepts: https://docs.google.com/document/d/1ZSlUPmGop6wuVEPTVC_DCb49E0orhyplrEyV302-sdU/edit?usp=sharing

It doesn't include any proposals or plans, but feel free to comment on any misunderstandings or anything that's missing.

I'm also thinking about converting the doc to Markdown and hosting on hackmd instead.

[imagesc-bot]

This issue has been mentioned on Image.sc Forum. There might be relevant details there:

https://forum.image.sc/t/using-layer-properties-to-pass-variables/63342/2

[imagesc-bot]

This issue has been mentioned on Image.sc Forum. There might be relevant details there:

https://forum.image.sc/t/exloring-image-segment-features-in-napari/75222/12

[andy-sweet]

Posting here to make sure this work/progress isn't lost.

The latest hackmd doc is probably the best way to consume the altair/vega-style style encoding proposal: https://hackmd.io/IOuBuY11QtS56dyrdH85aQ?view

That is in the form of a NAP, but didn't actually go through the NAP process. So the Accepted status is just to indicate that there was some more informal consensus around this approach.

We did the work to define string and color encodings, but we only use them in TextManager right now. I did draft a PR for using them in Points, but that's the effort paused.

[brisvag]

That is in the form of a NAP, but didn't actually go through the NAP process. So the Accepted status is just to indicate that there was some more informal consensus around this approach.

Would be great to retroactively add it to the NAPs :)

[imagesc-bot]

This issue has been mentioned on Image.sc Forum. There might be relevant details there:

https://forum.image.sc/t/exloring-image-segment-features-in-napari/75222/21

[imagesc-bot]

This issue has been mentioned on Image.sc Forum. There might be relevant details there:

https://forum.image.sc/t/changing-color-of-labels-when-using-volume-rendering/75289/3

[imagesc-bot]

This issue has been mentioned on Image.sc Forum. There might be relevant details there:

https://forum.image.sc/t/error-while-adding-text-to-napari/76960/6