Axes v0.4 - Githubissues

will-moore commented 2 years ago

Starting to implement Axes and Transformations (see https://github.com/ome/ngff/pull/57). cc @constantinpape

This requires https://pypi.org/project/ome-zarr/0.3a1/ for mask writing and new HCS metadata (row/column Indexes).

To test:

$ omero zarr export Image:6001240     # /6001240.zarr/.zattrs  should have new axes (see below)
$ omero zarr masks Image:6001240      # /6001240.zarr/labels/0/.zattrs should also have new axes

$ omero zarr export Plate:1234           # 1234.zarr/.zattrs each 'well' should have rowIndex and columnIndex

Some data exported with this PR can be viewed with vizarr: https://deploy-preview-139--vizarr.netlify.app/?source=https://minio-dev.openmicroscopy.org/idr/v0.4/2022-01-05/idr0062/6001240.zarr (with PR at https://github.com/hms-dbmi/vizarr/pull/139)

Plate data: https://minio-dev.openmicroscopy.org/idr/v0.4/2022-01-24/plates/idr0004/1752.zarr

Exported NGFF includes:

axes - eg. at https://minio-dev.openmicroscopy.org/idr/v0.4/2022-01-05/idr0062/6001240.zarr/.zattrs :

"axes": [
            {
                "name": "c",
                "type": "channel"
            },
            {
                "name": "z",
                "type": "space",
                "units": "micrometer"
            },
            {
                "name": "y",
                "type": "space",
                "units": "micrometer"
            },
            {
                "name": "x",
                "type": "space",
                "units": "micrometer"
            }
        ]

and each dataset has transformations. E.g. czyx image is scaled along axis z, y, x or [1, 2, 3]: NB: this format is still being discussed at https://github.com/ome/ngff/pull/57#pullrequestreview-836379663

              "transformations": [
                    {
                        "axisIndices": [
                            1,
                            2,
                            3
                        ],
                        "scale": [
                            0.5002025531914894,
                            0.3603981534640209,
                            0.3603981534640209
                        ],
                        "type": "scale"
                    }
                ]

constantinpape commented 2 years ago

For me installation of cfunits via conda install -c conda-forge cfunits went smooth and without any issues. What system did you try this on? Maybe we can raise this issue with the conda-forge maintainers of cfunits / uduints2.

will-moore commented 2 years ago

Hah! OK, it worked this time (on an aging Mac 10.14). But I still don't know if I want to make this a dependency. I might try to validate all the units that we get from OMERO for length (and if not valid then map to a valid unit).

It seems that a lot of things are valid (see below), so if the spec allows all of these, then it becomes very hard to read OME-NGFF data without having cfunits installed. If the spec were more restrictive, then we might not have to make cfunits a dependency of napari-ome-zarr, which would be nice. I'll add a comment to the spec PR...

>>> Units("micrometer").isvalid
True
>>> Units("micrometre").isvalid
True
>>> Units("micrometres").isvalid
True
>>> Units("micrometress").isvalid
False
>>> Units("micrometres")
<Units: micrometres>
>>> Units("micrometre")
<Units: micrometre>
>>> Units("nm")
<Units: nm>
>>> Units("nm").isvalid
True
>>> Units("nanom").isvalid
True
>>> Units("nanome").isvalid
True
>>> Units("nanomet").isvalid
False
>>> Units("nanometer").isvalid
True
>>> Units("nanometre").isvalid
True
>>> Units("nanometre").formatted()
'1e-09 m'
>>> Units("nanome").formatted()
'1.602176487e-31 C'
>>> Units("nanome").units
'nanome'
>>> Units("nanom").units
'nanom'
>>> Units("nano").units
'nano'
>>> Units("nano").formatted()
>>> Units("nanom").formatted()
'1e-09 m'
>>> Units("nanometer").formatted()
'1e-09 m'
>>> Units("nanometre").formatted()
'1e-09 m'
>>> Units("nanom").formatted()
'1e-09 m'

will-moore commented 2 years ago

Plate data exported today with current state of this branch and https://pypi.org/project/ome-zarr/0.3a1/ https://minio-dev.openmicroscopy.org/idr/v0.4/2022-01-24/plates/idr0004/1752.zarr Screenshot 2022-01-24 at 11 49 10

sbesson commented 2 years ago

In addition to the various comments above, following this morning's discussion, I have been thinking of ways to exercise the translation transformation that is being introduced in https://github.com/ome/ngff/pull/57.

In the OMERO -> OME-NGFF workflow, a potential candidate might be the WellSample.PositionX and WellSample.PositionY i.e. within the context of a well specification have the individual multiscales images registering the X/Y position relative to some well origin so that a client would be able to display a well grid using the absolute positions.

Unless this is something straightforward to implement, I would definitely consider this outside the scope of this PR and limit the transformations support to scale for now. The primary reason for bringing it up is that it might something to keep in mind alongside the API questions raised in https://github.com/ome/omero-cli-zarr/pull/93#discussion_r790864982. Since this translation metadata would be not be available from the ImageWrapper itself and would either need to be passed to add_image or be added via a follow-up call which would update the metadata.

sbesson commented 2 years ago

Trying to use the latest version of this PR, I get

(zarr) [sbesson@pilot-zarr1-dev ~]$ omero zarr export Plate:9512 --help

Error loading: /home/sbesson/miniconda3/envs/zarr/lib/python3.9/site-packages/omero/plugins/zarr.py
Traceback (most recent call last):
  File "/home/sbesson/miniconda3/envs/zarr/lib/python3.9/site-packages/omero/cli.py", line 1690, in loadpath
    execfile(str(pathobj), loc)
  File "/home/sbesson/miniconda3/envs/zarr/lib/python3.9/site-packages/past/builtins/misc.py", line 87, in execfile
    exec_(code, myglobals, mylocals)
  File "/home/sbesson/miniconda3/envs/zarr/lib/python3.9/site-packages/omero/plugins/zarr.py", line 1, in <module>
    from omero_zarr.cli import HELP, ZarrControl
  File "/home/sbesson/miniconda3/envs/zarr/lib/python3.9/site-packages/omero_zarr/cli.py", line 15, in <module>
    from .raw_pixels import (
  File "/home/sbesson/miniconda3/envs/zarr/lib/python3.9/site-packages/omero_zarr/raw_pixels.py", line 10, in <module>
    from ome_zarr.writer import (
ImportError: cannot import name 'write_multiscales_metadata' from 'ome_zarr.writer' (/home/sbesson/miniconda3/envs/zarr/lib/python3.9/site-packages/ome_zarr/writer.py)

Should the ome-zarr-py dependency be upgraded similarly to https://github.com/ome/napari-ome-zarr/pull/31/commits/dc4f0bd83d03df870b568977589883037fa95cb1?

will-moore commented 2 years ago

@sbesson Fixed ome-zarr requirement.

will-moore commented 2 years ago

That last commit de7703a uses PR: https://github.com/ome/ome-zarr-py/pull/162

will-moore commented 2 years ago

@joshmoore @sbesson I guess we probably don't want that last commit if this PR goes into a pre-release of omero-cli-zarr that works with ome-zarr>=0.3a1? So, I can revert that and then we can make those changes in a follow-up PR? I also have a commit locally that uses "scale": {"x": 0.5, "y":0.5: "z": 0.3} format and I guess that should go into a later PR too (so that the pre-releases are all aligned on the axisIndices format?

sbesson commented 2 years ago

Yes :+1: for getting a pre-release of omero-cli-zarr with ome-zarr-py 0.3.0a that matches the current proposal and focus on the new writing API and new variant for scale as a follow-up set of pre-releases.

will-moore commented 2 years ago

Reverted de7703a

will-moore commented 2 years ago

As discussed today, a screenshot illustrating the use of different names for Z/T axes and how it looks in Vizarr: cc @joshmoore @sbesson @constantinpape

constantinpape commented 2 years ago

Thanks @will-moore. Is this the dataset from https://minio-dev.openmicroscopy.org/idr/v0.4/2022-01-05/idr0062/6001240.zarr? I can open that one with Fiji/MoBIE now, but I am not sure if it's quite correct yet:

will-moore commented 2 years ago

Looking at https://deploy-preview-139--vizarr.netlify.app/?source=https://minio-dev.openmicroscopy.org/idr/v0.4/2022-01-05/idr0062/6001240.zarr (use Firefox - bugs on Chrome just now). The .zattrs has:

"datasets": [
                {
                    "path": "0",
                    "transformations": [
                        {
                            "axisIndices": [
                                1,
                                2,
                                3
                            ],
                            "scale": [
                                0.5002025531914894,
                                0.3603981534640209,
                                0.3603981534640209
                            ],
                            "type": "scale"
                        }
                    ]
                },

which should mean it applies to both channels? It looks like your example has different transforms applied to the 2 channels?

sbesson commented 2 years ago

As discussed this morning, merging and tagging this as 0.3.0a1 alongside all the other Python libraries pre-releases created ahead of the NGFF community call. Ongoing work is happening in #103 to support the latest proposed changes to the 0.4 specification and the conversation on how to validate transformations started above can happen there.

ome / omero-cli-zarr

Axes v0.4 #93