[FR] Segmentation mask labels in the app

[ehofesmann]

Provide a color key or add labels on the mask visualization itself for segmentation tasks. Not sure if this feature works in specific settings, but applying the deeplab model from the zoo did not provide labels making it pretty difficult to understand what was happening.

---

I'm sure this is in the backlog but I didn't see an issue about it so I figure I'd open on. Not top priority but this would be required to write a blog post about segmentations.

Looking in the docs, it was not clear what to expect segmentations to even look like. We should add images here (probably for all label types)

[brimoor]

I like the idea of providing a color-coded legend that can optionally be shown in this case.

ETA provides a mask_index field on classes like ImageLabels which is a dictionary mapping pixel values to labels for the pixels. This is stored on a per-image basis, which is a bit wasteful as typically all images in a dataset will follow the same legend. Nonetheless, FiftyOne could provide analogous functionality by adding a new DictField key fiftyone.core.labels.Segementation.index that allows the user to define the index for a mask.

However, the proper way to support this is probably to provide a mechanism to let a user store mask indexes in dataset.info that the App would refer to as necessary. In principle, the data model would need to allow for different mask indexes for each Segmentation field.

If we do this, we should also consider providing the analogous behavior for any Label classes that have string label fields (Classification.label, Detection.label, Polyline.label). In this case, the user could store int labels in the label fields and then the App would lookup the corresponding string labels in dataset.info at display time.

One thing I considered here was providing an optional target field that is an int. That way the user could choose how they want to store their labels:

• target<int> + dataset.info classes list
• directly provide label<string>

[brimoor]

@benjaminpkane show label for current pixel on hover if a segmentation field is visible

[benjaminpkane]

I am going to add a new field on fiftyone.core.odm.DatasetDocument (not worrying about naming right now), which will be a MongoEngine MapField mapping field names to a list of strings.

MongoDB (...and MongoEngine) does not support dict's with integers as keys. I assume a list field is sufficient? Where a target value is the index of the label in the list of strings.

[brimoor]

Unfortunately a common pattern used by some folks is to use 255 to represent background. So they may have a label map like:

{
    1: 'road',
    2: 'sky',
    255: 'background'
}

A list would be a less than ideal for that.

We have adopted the convention that class labels are stored as a list, though, and if the model doesn't use a particular index, then the labels list might look like:

["cat", "dog", "-", "-", ..., "horse"]

I'm on board with doing something just to get the feature working, and then we can tweak later if desired.

One thought is to consider a custom Field that will convert str <-> int when committing to the DB.

[brimoor]

I bring up class labels here because they're analogous in usage to mask labels. So if we choose to store mask labels in a dedicated field in DatasetDocument, then we'd likely want to do the same with class labels.

Class labels are currently just stored (by convention) in dataset.info["classes"].

In terms of use cases, I'd say users are likely to either have a single label map or class labels list that applies to all fields (maybe we call this a global field), while other times they may have different sets of labels for individual fields.

So perhaps we should support optionally keying label maps by field names, and falling back to a reserved global field if no per-field map is found.

The same could apply to class labels, although this would merely be a data model convention, as the App doesn't currently do anything with this information.