Accept scalar values as labels

[youngdjn]

Allow the user to provide scalar values as labels (for the mesh and raw images). This could enable image regression approaches (e.g. estimation of biomass) and other applications involving scalar values for pixels/faces. This also requires tolerating an arbitrary number of label "classes" instead of 10 or less, and likely using NaN, rather than a specific numeric value, to represent background.

[russelldj]

I think there are two separate issues here.

With respect to scalar labels, I completely agree that we should support this, and it is actually already mostly developed. In fact, the underlying representation of the texture is always a float, and is coerced into that format here. We can also render float labels properly with interpolation using the render_pytorch3d method with shade_by_indexing = False . The only issue is going through the logic to make sure that we use the appropriate style of rendering for the input data and making sure we don't request a list of unique labels for scalar data.

Dealing with a large number of classes is also already implemented. The internal representation is actually int values represented as a float so background/null can be represented as NaN. The issue is in two places, vis and export. With visualization, I often default to using the tab10 colorpallete, so any class ID more than 9 will be shown in the same color as class 9. With exporting, I'm currently saving out standard uint8 images, so we are limited to 256 classes. At the export stage, the background class, which is represented as NaN, is transformed into the value 255. A more flexible representation would be saving float-typed npy files. This would allow us to be more explicit by preserving NaN and also support an effectively unbounded number of classes. The downside here is this will require more disk space. Each value with be represented as a float instead of uint8 and we won't have any image compression, though I don't know if we currently get any benefit from that.

[youngdjn]

OK thanks, makes sense! Sounds like these are just changes for the demo notebooks, except saving rendered label images, is that right? I guess preserving the potential to compress images is worth sacrificing an explicit NaN. Here's what I've pulled out as to-do items; feel free to modify and/or split some out into a separate issue.

Notebooks:

• [x] Use appropriate label rendering style depending on input datatype (int or float)
• [x] Do not request unique labels for scalar data (perhaps just a comment for the user)

Module:

• [ ] When saving rendered images, use lowest-memory datatype that will contain the data (e.g. uint8 if < 256 classes; float if float, i.e. not casting to int here)

[russelldj]

I think I've addressed most of this issue. Your comment about saving in the lowest-memory datatype is a good one and I'm moving it to its own issue here.