New rendering settings specification

dominikl commented 2 years ago

The render plugin is a good starting point to play around with different ideas. Wonder if we should have an 'experimental' branch maybe? /cc @jburel @dgault @francesw

Here's a first attempt, kind of a hybrid of the current render plugin output and the current NGFF draft specification (code see draft PR #54 ):

{
    "channels": {
        "0": {
            "active": true,
            "color": "00FF00",
            "color-format": "hex",
            "color-type": "rgb",
            "label": "CEP192-M",
            "window": {
                "end": 16633.45703125,
                "max": 16633.45703125,
                "min": 0.0,
                "start": 0.0
            }
        },
        "1": {
            "active": true,
            "color": "FF0000",
            "color-format": "hex",
            "color-type": "rgb",
            "label": "CENT2",
            "window": {
                "end": 103353.1796875,
                "max": 103353.1796875,
                "min": 0.0,
                "start": 0.0
            }
        }
    },
    "color-model": "color",
    "default-dimension-0": 0,
    "default-dimension-1": 0,
    "dimensions": "zt",
    "plane": "xy",
    "version": 3
}

Instead of defaultZ/T it's more generic default-dimension- and what the dimensions actually mean is specified in dimensions, what a plane actually means likewise in plane. Also the color can be more generic. See above for the common html color specification, but it could also hold lookup tables (e.g. specified by a name or maybe url, etc.), e.g.

    "channels": {
        "0": {
            "active": true,
            "color": "glasbey",
            "color-format": "name",
            "color-type": "lut",
            "label": "CEP192-M",
            "window": {
                "end": 23900.765625,
                "max": 23900.765625,
                "min": 0.0,
                "start": 0.0
            }
        },
        ...
    }

Edit: Actually default-dimension- should better be a list, not several keys.

jburel commented 2 years ago

Discussion about RGB support, possible options:

"views": [
    {
        "@id": "view1",
        "channel": 0:2, or [0, 1, 2]
        "visible": true,
        "label": "RGB",
    },

"colorModel": "color",
use groups with 3 views corresponding to each "channel"
"defaultIndices": [null, 0:2, null, null, null], "views": [ { "@id": "view1", "visible": true, "label": "RGB", },

will-moore commented 2 years ago

napari has an rgb=True flag for an Image, in which case the Image name is used for labelling the rgb channel: https://napari.org/#simple-example

We could support this with "colorModel": "rgb", and then we wouldn't need views at all.

jburel commented 2 years ago

Comment from @joshmoore Some rnd thoughts:

(1) there was a suggestion somewhere to use “component” rather than “channel” would that help to cover angle?
(2) when angles are introduced, would a more complex (multidimensional) indexing than channel: 1 be useful?
(3) I wonder if some of the fields belong more “together” like family/noiseReduction/… in an object like “window”
(4) I’m unsure how the window plays together with the defaultIndices? (perhaps because I generally don’t follow defaultIndices).
(5) did “zoom” get discussed rather than “resolution”?
(6) something feels a bit off about the “defaultPlane” representation but I haven’t pinned it down, yet.

jburel commented 2 years ago

for (1)&(2) I think we need to review how we can extend when it comes into play. Initially it was channel and angle but that will not scale when we extend (3) yes (4) "defaultIndices": [5, null, 10:20, 0:300, 100:200] // e.g. shape is (t,c,z,y,x) will indicate to display the specified rectangle [100, 0, 100 (200-100), 300]
(5) zoom is for me a "preferences". Happy to rethink that if people think we should include it (6) this was previously "plane": "xy",. This matches what we currently have in OMERO (planeDef)

jburel commented 2 years ago

(3) something like:

"mapping": {
  "family": "polynomial",
  "coefficient": "1.5",
  "noiseReduction": "False",
}

will-moore commented 2 years ago

I've just noticed that "model": "greyscale" is ignored in vizarr, so you end up seeing actual channel colour (e.g. grey) https://hms-dbmi.github.io/vizarr/?source=https://uk1s3.embassy.ebi.ac.uk/0.3_samples/7751.zarr/B/11/0 I'm wondering if we want to request that all OME-NGFF clients handle this flag and support a greyscale viewing mode, when you can achieve the same rendering effect by saving a channel as white (and turning the other channels off, if there are more than 1). I don't know if any viewers (outside of OMERO) support this greyscale mode of toggling channels and if it's really an OMERO-specific viewing behaviour rather than a rendering setting?

jburel commented 2 years ago

@will-moore Thanks for checking. This could be a limiting factor for the rgb mode we discussed the other day. Could you check the strategy in Napari and Vizarr? i.e. passing the flag rgb=true in napari does it set a "mode" and what is the value when that flag is not. This will help driving forward

will-moore commented 2 years ago

In napari, the rgb flag is stored on the base image class and used in a bunch of methods: https://github.com/napari/napari/blob/main/napari/layers/image/image.py If no value for rgb is provided, it is guessed from the shape (True if last dimension is 3).

vizarr has no 'rgb' mode

will-moore commented 2 years ago

I think the rgb concept is a separate issue from greyscale. I think there is a case for supporting rgb (since we know that there are viewers that support it), but I'm proposing that we don't support greyscale. For images coming from OMERO (which are the only times we have colorModel: greyscale) we should just set the active channel to white. This is what I do for OMERO.figure. It's also what we do in napari-ome-zarr because napari doesn't support greyscale. But we force every client to adopt the same logic. It would be better to remove it from the spec and handle greyscale when we export from OMERO.

jburel commented 2 years ago

Following today's discussion

"renderingDefinition": {
   "@id": "renderingDef1"
   "groups": [
        {
            "views": ["view1", "view2"],
            "name": "Cycle1",
            "visible": "True"
        },
        {
            "views": ["view3"],
            "name": "Cycle2",
            "visible": "True"
        }
    ],
    "views": [
        {
            "@id": "view1",
            "dimension": [null, 0, null, null, null]  // (t,c,z,y,x)
            "visible": true,
            "mapping": {
               "family": "linear",
            }
            "color": {
                "format": "name",
                "type": "lut",
                "value": "brgbcmyw.lut"
            },
            "deviceSpaceMapping": {
                "reverseIntensity": "False"
            },
            "label": "GFP",
            "window": {
                "end": 188.89093017578125,
                "max": 188.89093017578125,
                "min": 0.007875712588429451,
                "start": 0.007875712588429451
            }
        },
        {
            "@id": "view2",
             "dimension": [null, 1, null, null, null]  // (t,c,z,y,x)
            "visible": "False",
            "color": {
                "format": "hex",
                "type": "rgb",
                "value": "00FF00"
            },
            "deviceSpaceMapping": {
                "reverseIntensity": "False"
            },
            "mapping": {
               "family": "polynomial",
               "coefficient": "1.5",
               "noiseReduction": "False",
            }
            "label": "DAPI",
            "window": {
                "end": 1542.5789794921875,
                "max": 1542.5789794921875,
                "min": 0.0,
                "start": 0.0
            }
        },
        {
            "@id": "view3",
             "dimension": [null, 2, null, null, null]  // (t,c,z,y,x)
            "visible": "True",
            "color": {
                "format": "hex",
                "type": "rgb",
                "value": "FFFFFF"
            },
            "deviceSpaceMapping": {
                "reverseIntensity": "False"
            },
            "mapping": {
               "family": "linear",
            }
            "label": "OOOO",
            "window": {
                "end": 1555.0537109375,
                "max": 1555.0537109375,
                "min": 0.0,
                "start": 0.0
            }
        }
    ],
    "colorModel": "color",
    // e.g. shape is (t,c,z,y,x)
    "defaultIndices": [5, null, 10:20, null, null],   // default view: t=5 and range 10-20 on z axis 
    // we want to view y,x plane by default
    "planarView": [3, 4],
    "zoom": 50 // percentage starting from highest resolution
}

RGB case could possibly be covered:

"renderingDefinition": {
   "@id": "renderingDef2"
    "views": [
        {
            "@id": "view1",
            "dimension": [null, 0:2, null, null, null]  // (t,c,z,y,x)
            "visible": true,
            "label": "RGB",
        },
        {
            "@id": "view2",
            "dimension": [null, 3, null, null, null]  // (t,c,z,y,x)
            "visible": "False",
            "color": {
                "format": "hex",
                "type": "rgb",
                "value": "00FF00"
            },
            "deviceSpaceMapping": {
                "reverseIntensity": "False"
            },
            "mapping": {
               "family": "polynomial",
               "coefficient": "1.5",
               "noiseReduction": "False",
            }
            "label": "DAPI",
            "window": {
                "end": 1542.5789794921875,
                "max": 1542.5789794921875,
                "min": 0.0,
                "start": 0.0
            }
        },
    ],
    "colorModel": "color",
    // e.g. shape is (t,c,z,y,x)
    "defaultIndices": [5, null, 10:20, null, null],   // default view: t=5 and range 10-20 on z axis 
    // we want to view y,x plane by default
    "planarView": [3, 4],
    "zoom": 100
}

jburel commented 2 years ago

Other option for RGB is to use a view per channel (as suggested previously) and combine the views in an RGB group

Last point: drop ColorModel

jburel commented 2 years ago

Latest example from today's discussion

"renderingDefinition": {
   "@id": "renderingDef1"
   "groups": [
        {
            "views": ["view1", "view2"],
            "name": "Cycle1",
            "visible": "True"
        },
        {
            "views": ["view3"],
            "name": "Cycle2",
            "visible": "True"
        }
    ],
    "views": [
        {
            "@id": "view1",
            "dimension": [null, 0, null, null, null]  // (t,c,z,y,x)
            "visible": true,
            "mapping": {
               "family": "linear",
            }
            "color": {
                "format": "name",
                "type": "lut",
                "value": "brgbcmyw.lut"
            },
            "deviceSpaceMapping": {
                "reverseIntensity": "False"
            },
            "label": "GFP",
            "window": {
                "end": 188.89093017578125,
                "max": 188.89093017578125,
                "min": 0.007875712588429451,
                "start": 0.007875712588429451
            }
        },
        {
            "@id": "view2",
             "dimension": [null, 1, null, null, null]  // (t,c,z,y,x)
            "visible": "False",
            "color": {
                "format": "hex",
                "type": "rgb",
                "value": "00FF00"
            },
            "deviceSpaceMapping": {
                "reverseIntensity": "False"
            },
            "mapping": {
               "family": "polynomial",
               "coefficient": "1.5",
               "noiseReduction": "False",
            }
            "label": "DAPI",
            "window": {
                "end": 1542.5789794921875,
                "max": 1542.5789794921875,
                "min": 0.0,
                "start": 0.0
            }
        },
        {
            "@id": "view3",
            "dimension": [null, 2, null, null, null]  // (t,c,z,y,x)
            "visible": "True",
            "color": {
                "format": "hex",
                "type": "rgb",
                "value": "FFFFFF"
            },
            "deviceSpaceMapping": {
                "reverseIntensity": "False"
            },
            "mapping": {
               "family": "linear",
            }
            "label": "OOOO",
            "window": {
                "end": 1555.0537109375,
                "max": 1555.0537109375,
                "min": 0.0,
                "start": 0.0
            }
        }
    ],
    // e.g. shape is (t,c,z,y,x)
    "defaultIndices": [5, null, 10:20, null, null],   // default view: t=5 and range 10-20 on z axis 
    "planarView": [3, 4], // we want to view y,x plane by default
    "zoom": 50 // percentage starting from highest resolution
}

RGB example

"renderingDefinition": {
   "@id": "renderingDef1"
   "groups": [
        {
            "views": ["view1", "view2", "view3"],
            "name": "RGB",
            "visible": "True"
        }
    ],
    "views": [
        {
            "@id": "view1",
            "dimension": [null, 0, null, null, null]  // (t,c,z,y,x)
            "visible": true,
            "color": {
                "format": "hex",
                "type": "rgb",
                "value": "FF0000"
            },
            "label": "Red",
        },
        {
            "@id": "view2",
            "dimension": [null, 2, null, null, null]  // (t,c,z,y,x)
            "visible": true,
            "color": {
                "format": "hex",
                "type": "rgb",
                "value": "00FF00"
            },
            "label": "Green",
        },
        {
            "@id": "view3",
            "dimension": [null, 1, null, null, null]  // (t,c,z,y,x)
            "visible": true,
            "color": {
                "format": "hex",
                "type": "rgb",
                "value": "0000FF"
            },
            "label": "Blue",
        },
    ],
}

Summary

drop colorModel
channel replaced by dimension (list) so it can be extended when for example angle is supported
combine mapping parameters from space to deviceSpace into a mapping field
rename defaultPlane by planarView
RGB case will be handled by groups and views
replace resolution by zoom percentage from the highest resolution

will-moore commented 2 years ago

I still think the rgb case needs a more explicit flag on the group. The "name" is not enough, and we likely want to allow a more descriptive name. So I think "rgb": true would be easiest.

"groups": [
        {
            "views": ["view1", "view2", "view3"],
            "name": "My histology image",
            "visible": true
            "rgb": true
        }
    ],

jburel commented 2 years ago

Thinking about it other use cases might need some "group" flexibility e.g. FLIM with "small t" so something like that will be extensible

"groups": [
        {
            "views": ["view1", "view2", "view3"],
            "name": "My histology image",
            "visible": true
            "metadata": {"rgb": true}    //similar to deviceSpaceMapping
        }
    ],

cc @will-moore

jburel commented 2 years ago

Final proposal before migrating to ngff issue.

"renderingDefinition": {
   "@id": "renderingDef1"
   "groups": [
        {
            "views": ["view1", "view2"],
            "name": "Cycle1",
            "visible": true
        },
        {
            "views": ["view3"],
            "name": "Cycle2",
            "visible": true
        }
    ],
    "views": [
        {
            "@id": "view1",
            "dimension": [null, 0, null, null, null]  // (t,c,z,y,x)
            "visible": true,
            "mapping": {
               "family": "linear",
            }
            "color": {
                "format": "name",
                "type": "lut",
                "value": "brgbcmyw.lut"
            },
            "deviceSpaceMapping": {
                "reverseIntensity": false
            },
            "label": "GFP",
            "window": {
                "end": 188.89093017578125,
                "max": 188.89093017578125,
                "min": 0.007875712588429451,
                "start": 0.007875712588429451
            }
        },
        {
            "@id": "view2",
             "dimension": [null, 1, null, null, null],  // (t,c,z,y,x)
            "visible": false,
            "color": {
                "format": "hex",
                "type": "rgb",
                "value": "00FF00"
            },
            "deviceSpaceMapping": {
                "reverseIntensity": false
            },
            "mapping": {
               "family": "polynomial",
               "coefficient": 1.5,
               "noiseReduction": false,
            }
            "label": "DAPI",
            "window": {
                "end": 1542.5789794921875,
                "max": 1542.5789794921875,
                "min": 0.0,
                "start": 0.0
            }
        },
        {
            "@id": "view3",
            "dimension": [null, 2, null, null, null]  // (t,c,z,y,x)
            "visible": true,
            "color": {
                "format": "hex",
                "type": "rgb",
                "value": "FFFFFF"
            },
            "deviceSpaceMapping": {
                "reverseIntensity": false
            },
            "mapping": {
               "family": "linear",
            }
            "label": "OOOO",
            "window": {
                "end": 1555.0537109375,
                "max": 1555.0537109375,
                "min": 0.0,
                "start": 0.0
            }
        }
    ],
    // e.g. shape is (t,c,z,y,x)
    "defaultIndices": [5, null, "10:20", null, null],   // default view: t=5 and range 10-20 on z axis 
    "planarView": [3, 4], // we want to view y,x plane by default
    "zoom": 50 // percentage starting from highest resolution
}

RGB example

"renderingDefinition": {
   "@id": "renderingDef1"
   "groups": [
        {
            "views": ["view1", "view2", "view3"],
            "name": "The name you want to use",
            "visible": true,
            "metadata": {"rgb": true}
        }
    ],
    "views": [
        {
            "@id": "view1",
            "dimension": [null, 0, null, null, null]  // (t,c,z,y,x)
            "visible": true,
            "color": {
                "format": "hex",
                "type": "rgb",
                "value": "FF0000"
            },
            "label": "Red",
        },
        {
            "@id": "view2",
            "dimension": [null, 2, null, null, null]  // (t,c,z,y,x)
            "visible": true,
            "color": {
                "format": "hex",
                "type": "rgb",
                "value": "00FF00"
            },
            "label": "Green",
        },
        {
            "@id": "view3",
            "dimension": [null, 1, null, null, null]  // (t,c,z,y,x)
            "visible": true,
            "color": {
                "format": "hex",
                "type": "rgb",
                "value": "0000FF"
            },
            "label": "Blue",
        },
    ],
}

imagesc-bot commented 1 year ago

This issue has been mentioned on Image.sc Forum. There might be relevant details there:

https://forum.image.sc/t/saving-volumetric-data-with-voxel-size-colormap-annotations/85537/28

dominikl commented 7 months ago

I suppose we can just close this now @jburel ? bioformats2raw seems to put the rendering settings already in the .zattrs anyway, guess that's the spec then.

dominikl commented 7 months ago

Indeed, it's already in the spec: https://ngff.openmicroscopy.org/latest/#omero-md .

joshmoore commented 7 months ago

I understood the purpose of this work was to replace the #omero-md block with something more formal. I would very much :+1: an RFC following the previous model. The one topic that it might be worth considering is that with the ongoing ome-types work, whether or not we don't also define an XSD for the rendering settings which would allow us to be backwards and forwards compatible.

ome / omero-cli-render

New rendering settings specification #55