Use nested chunk store for labels

[will-moore]

This uses nested chunks when exporting labels from OMERO. Without this, the Image data is nested but labels are not - so they are ignored in napari.

Also, it updates labels to v0.3 (corresponding to https://github.com/ome/omero-cli-zarr/pull/70 for image data) where we discard any 'empty' dimensions. This means that the image data and labels data have same number of dimensions so that they display correctly in napari.

Testing setup

You'll need a python environment with: omero-cli-zarr, ome-zarr-py, napari-ome-zarr installed. If we just want the released versions of everything:

Create an environment, using this environment.yml:

channels:
  - ome
  - conda-forge
  - defaults
dependencies:
  - python
  - wheel
  - omero-py == 5.10.0
  - zarr == 2.9.5
  - napari[all]
  - pip
  - pip:
      - omero-cli-zarr
      - napari-ome-zarr

conda env create -f environment.yml -n ome_ngff
conda activate ome_ngff

To get a specific branch of https://github.com/ome/omero-cli-zarr or https://github.com/ome/ome-zarr-py or https://github.com/ome/napari-ome-zarr:

• clone the repo and checkout the branch you need
• cd into the repo
• pip uninstall omero-cli-zarr # or ome-zarr or napari-ome-zarr
• pip install -e . # reinstall from the current branch

To test with https://idr.openmicroscopy.org/webclient/?show=image-6001247:

With ome-zarr-py from https://github.com/ome/ome-zarr-py/pull/114 needed for down-sampling of labels with fewer than 5D.

$ omero zarr export Image:6001247
$ omero zarr masks Image:6001247

The shape of the labels should match the image data (same number of dimensions) although size_c for channels will be 1.

This also needs https://github.com/ome/napari-ome-zarr/pull/16 for viewing:

$ napari 6001247.zarr

[sbesson]

So that I understand the compatibility matrix here, is napari effectively only support version 0.2+ of the multiscales specification?

[will-moore]

No, latest napari + napari_ome_zarr still supports e.g.

https://s3.embassy.ebi.ac.uk/idr/zarr/v0.1/6001240.zarr

But before this PR, the labels didn't use nested chunks but the image data did. It looks like napari expects both to be read the same way, so the labels are all black (unless they are also nested).

[joshmoore]

NB: Newer versions of zarr should ignore the requested chunk and use the .zarray metadata correctly.

[sbesson]

But before this PR, the labels didn't use nested chunks but the image data did. It looks like napari expects both to be read the same way, so the labels are all black (unless they are also nested).

Cheers, that makes sense although to the best of my knowledge there is no current enforcement that the label image should have the same multiscales version as the image?

[sbesson]

Overall no objection from my side to unifying the writing behavior of this tool so that both images and label images use the same version of the spec. Happy to see this released as 0.1.1.

[sbesson]

In order to fix the napari bug, this PR migrates the open_store logic into a new public API. This has impact on other PRs like https://github.com/ome/omero-cli-zarr/pull/83. To minimize conflicts and reduce future pain, I would propose we make a decision on integrating these changes sooner rather than later. @will-moore can you comment on https://github.com/ome/omero-cli-zarr/pull/82#pullrequestreview-757484746 ?

EDIT: @will-moore pointed out that the commit associated to https://github.com/ome/omero-cli-zarr/pull/82#discussion_r711071657 got reverted in https://github.com/ome/omero-cli-zarr/pull/82#event-5319549664. This should be ready for a final round of review

[imagesc-bot]

This pull request has been mentioned on Image.sc Forum. There might be relevant details there:

https://forum.image.sc/t/prepping-and-including-roi-masks/48750/24

[pwalczysko]

Addition of

- conda-forge to the channels block of the environment.yml seems to be necessary, otherwise see https://openmicroscopy.slack.com/archives/C0K5EED3R/p1633099632195100 cc @khaledk2 @dominikl

[pwalczysko]

Built the environment as prescribed in the header. Included in it the present PR, the https://github.com/ome/napari-ome-zarr/pull/16 and the https://github.com/ome/ome-zarr-py/pull/114. Exported the image from IDR as per the PR header, with the labels. Opened the image in napari as per PR header. All works, viewing in napari with no probs, labels are there, as well as the two channels of the image. The labels perfectly match the intensities in the image, as verified by switching on and off the displays.

This looks like the workflow 1 in https://docs.google.com/document/d/1LOihtEZWTA7DKsA6zXG1OlQuGTuV7SxAa5OVFSQNeDc/edit#bookmark=id.aygdx8up5vho is tested ?

[dominikl]

Exported images look fine in napari. But I don't really understand the directory structure any more. Does that mean the image has 3 resolution (0,1,2) (for a 253 x 210 px image!?) and the labels only have one resolution (0)? What are the subdirectories 0,1,2,3 beneath labels/0 ?

[will-moore]

@dominikl I tend to use my editor (VSCode) to inspect the .zarr data since you can then see and read the hidden files. E.g. you can see that the smallest '2' resolution has a .zarray of:

The 0 is just the name of the first labels array, and it has 4 resolutions, the smallest being:

"shape": [
        1,
        257,
        26,
        31
    ],

[will-moore]

@pwalczysko With your test above (https://github.com/ome/omero-cli-zarr/pull/82#issuecomment-932342958) that is a successful test of https://github.com/ome/napari-ome-zarr/pull/16, so could you give a 👍 on that PR too? Thanks.

[will-moore]

$ ssh -A ome-zarr-dev1.openmicroscopy.org
$ conda activate omero_zarr_export
$ pip freeze | grep zarr
ome-zarr==0.1.0
omero-cli-zarr @ git+https://github.com/will-moore/omero-cli-zarr.git@8dc47076e5bd22f5ffc1244d968f140873e972f0
zarr==2.9.5

# update ome-zarr-py and omero-cli-zarr to install from correct branches...
$ pip uninstall ome-zarr
$ pip install git+https://github.com/will-moore/ome-zarr-py.git@write_scale_less_than_5D
$ pip uninstall omero-cli-zarr
$ pip install git+https://github.com/will-moore/omero-cli-zarr.git@labels_chunks_nested

# export... - Use commit in the dir name, so we know what created it...
$ cd /uod/idr/objectstore/minio/idr/v0.3/
$ mkdir idr0047
$ cd idr0047
$ mkdir 3dc087c
$ cd 3dc087c

$ omero zarr export Image:4496763
Exporting to 4496763.zarr (0.3)
Finished.

$ omero zarr masks Image:4496763
Export Masks on Image: Exp1_rep1_0min_im1.tif
Found 430 mask shapes in 430 ROIs
Unique dimensions: {'T': {None}, 'C': {None}, 'Z': {None}}
source_image 4496763.zarr
Ignoring dimensions {'T', 'Z', 'C'}
Created 4496763.zarr/labels/0

Data is available at https://minio-dev.openmicroscopy.org/idr/v0.3/idr0047/3dc087c/4496763.zarr and can be opened locally with https://github.com/ome/napari-ome-zarr/pull/16:

$ napari  https://minio-dev.openmicroscopy.org/idr/v0.3/idr0047/3dc087c/4496763.zarr

Turning on the labels:

[will-moore]

NB: In the previous example, the masks on Image https://idr.openmicroscopy.org/webclient/img_detail/4496763/ don't have any Z-index (or T or C) so in IDR they are shown across all Z sections.

These dimensions are ignored when exporting labels (see output above Ignoring dimensions {'T', 'Z', 'C'}) and we are left with a 4D array - NB: this probably could/should be 2D:

$ curl https://minio-dev.openmicroscopy.org/idr/v0.3/idr0047/3dc087c/4496763.zarr/labels/0/0/.zarray
{
    "chunks": [
        1,
        1,
        256,
        512
    ],
    "compressor": {
        "blocksize": 0,
        "clevel": 5,
        "cname": "lz4",
        "id": "blosc",
        "shuffle": 1
    },
    "dimension_separator": "/",
    "dtype": "<i8",
    "fill_value": 0,
    "filters": null,
    "order": "C",
    "shape": [
        1,
        1,
        2048,
        2048
    ],
    "zarr_format": 2

This explains why it is only visible for the first Z-index in napari above.

[will-moore]

I also exported https://minio-dev.openmicroscopy.org/idr/v0.3/idr0052/3dc087c/5514374.zarr from idr0052 but mask export fails since the masks overlap:

Exception: Mask 1369052 overlaps with existing labels

``` $ omero zarr masks Image:5514374 Using session for public@idr-next.openmicroscopy.org:4064. Idle timeout: 10 min. Current group: Public Export Masks on Image: cell0005_R0001.companion.ome Found 2183 mask shapes in 80 ROIs Unique dimensions: {'T': {0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39}, 'C': {None}, 'Z': {1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29}} source_image 5514374.zarr Ignoring dimensions {'C'} Traceback (most recent call last): File "/lifesci/groups/jrs/wmoore/miniconda3/envs/omero_zarr_export/bin/omero", line 11, in sys.exit(main()) File "/lifesci/groups/jrs/wmoore/miniconda3/envs/omero_zarr_export/lib/python3.9/site-packages/omero/main.py", line 125, in main rv = omero.cli.argv() File "/lifesci/groups/jrs/wmoore/miniconda3/envs/omero_zarr_export/lib/python3.9/site-packages/omero/cli.py", line 1784, in argv cli.invoke(args[1:]) File "/lifesci/groups/jrs/wmoore/miniconda3/envs/omero_zarr_export/lib/python3.9/site-packages/omero/cli.py", line 1222, in invoke stop = self.onecmd(line, previous_args) File "/lifesci/groups/jrs/wmoore/miniconda3/envs/omero_zarr_export/lib/python3.9/site-packages/omero/cli.py", line 1299, in onecmd self.execute(line, previous_args) File "/lifesci/groups/jrs/wmoore/miniconda3/envs/omero_zarr_export/lib/python3.9/site-packages/omero/cli.py", line 1381, in execute args.func(args) File "/lifesci/groups/jrs/wmoore/miniconda3/envs/omero_zarr_export/lib/python3.9/site-packages/omero_zarr/cli.py", line 65, in _wrapper return func(self, *args, **kwargs) File "/lifesci/groups/jrs/wmoore/miniconda3/envs/omero_zarr_export/lib/python3.9/site-packages/omero_zarr/cli.py", line 236, in masks image_shapes_to_zarr(image, ["Mask"], args) File "/lifesci/groups/jrs/wmoore/miniconda3/envs/omero_zarr_export/lib/python3.9/site-packages/omero_zarr/masks.py", line 167, in image_shapes_to_zarr saver.save(list(masks.values()), args.label_name) File "/lifesci/groups/jrs/wmoore/miniconda3/envs/omero_zarr_export/lib/python3.9/site-packages/omero_zarr/masks.py", line 300, in save labels, fill_colors, properties = self.masks_to_labels( File "/lifesci/groups/jrs/wmoore/miniconda3/envs/omero_zarr_export/lib/python3.9/site-packages/omero_zarr/masks.py", line 513, in masks_to_labels raise Exception( Exception: Mask 1369052 overlaps with existing labels ```

[will-moore]

$ omero zarr masks Image:11511036
Using session for public@idr-next.openmicroscopy.org:4064. Idle timeout: 10 min. Current group: Public
Export Masks on Image: 1X [2]
Found 52 mask shapes in 52 ROIs
Unique dimensions: {'T': {None}, 'C': {None}, 'Z': {None}}
source_image 11511036.zarr
Ignoring dimensions {'C', 'T', 'Z'}
Created 11511036.zarr/labels/0

napari https://minio-dev.openmicroscopy.org/idr/v0.3/idr0095/3dc087c/11511036.zarr

Zoomed-in to see labels:

[pwalczysko]

(ome_ngff) pwalczysko@ls31618~/Work/omero-cli-zarr$ omero zarr masks Image:5514374
Using session for public@idr.openmicroscopy.org:4064. Idle timeout: 10 min. Current group: Public
Export Masks on Image: cell0005_R0001.companion.ome
Found 2183 mask shapes in 80 ROIs
Unique dimensions: {'T': {0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39}, 'C': {None}, 'Z': {1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29}}
source_image 5514374.zarr
Ignoring dimensions {'C'}
Traceback (most recent call last):
  File "/Users/pwalczysko/miniconda3/envs/ome_ngff/bin/omero", line 11, in <module>
    sys.exit(main())
  File "/Users/pwalczysko/miniconda3/envs/ome_ngff/lib/python3.9/site-packages/omero/main.py", line 125, in main
    rv = omero.cli.argv()
  File "/Users/pwalczysko/miniconda3/envs/ome_ngff/lib/python3.9/site-packages/omero/cli.py", line 1784, in argv
    cli.invoke(args[1:])
  File "/Users/pwalczysko/miniconda3/envs/ome_ngff/lib/python3.9/site-packages/omero/cli.py", line 1222, in invoke
    stop = self.onecmd(line, previous_args)
  File "/Users/pwalczysko/miniconda3/envs/ome_ngff/lib/python3.9/site-packages/omero/cli.py", line 1299, in onecmd
    self.execute(line, previous_args)
  File "/Users/pwalczysko/miniconda3/envs/ome_ngff/lib/python3.9/site-packages/omero/cli.py", line 1381, in execute
    args.func(args)
  File "/Users/pwalczysko/Work/omero-cli-zarr/src/omero_zarr/cli.py", line 65, in _wrapper
    return func(self, *args, **kwargs)
  File "/Users/pwalczysko/Work/omero-cli-zarr/src/omero_zarr/cli.py", line 236, in masks
    image_shapes_to_zarr(image, ["Mask"], args)
  File "/Users/pwalczysko/Work/omero-cli-zarr/src/omero_zarr/masks.py", line 167, in image_shapes_to_zarr
    saver.save(list(masks.values()), args.label_name)
  File "/Users/pwalczysko/Work/omero-cli-zarr/src/omero_zarr/masks.py", line 300, in save
    labels, fill_colors, properties = self.masks_to_labels(
  File "/Users/pwalczysko/Work/omero-cli-zarr/src/omero_zarr/masks.py", line 513, in masks_to_labels
    raise Exception(
Exception: Mask 1369052 overlaps with existing labels

[will-moore]

@pwalczysko Yep, that's what I saw at https://github.com/ome/omero-cli-zarr/pull/82#issuecomment-940160778 We could handle that better, but it's not really related to this PR.

[pwalczysko]

napari 4496763.zarr - specific issue as follows: In napari, the labels seem to be associated with first z plane only. If the labels are put on display on any other plane, then they do not show. This is in contrast with napari /6001247.zarr where the labels show on every z plane. Checked the masks on the 4496763 image in IDR and they do not seem to be associated with any plane.

[pwalczysko]

In summary,

• 13417011 and 11511036 -> no issues
• 5514374 has https://github.com/ome/omero-cli-zarr/pull/82#issuecomment-940170259 which seems not to be caused by this PR
• 4496763 has https://github.com/ome/omero-cli-zarr/pull/82#issuecomment-940179664 which needs an explanation

[will-moore]

The strategy for omero zarr masks export when the masks in OMERO have no Z/T index is to only output a single plane, instead of outputting potentially hundreds of identical planes and increasing the size of the data many times. So, instead of a labels data that matches the dimensions of the Image, we have size-Z and/or size-T to be 1.

It would be nice if there was some way of getting napari to display the same labels across all Z/T indices without actually duplicating the data. But regardless, this isn't a new issue related to this PR.

[will-moore]

@pwalczysko I remembered that there is a way of handling overlapping masks as described at https://github.com/ome/omero-cli-zarr#masks (using an idr0052 image as an example). Using this approach, exported masks for this image:

napari https://minio-dev.openmicroscopy.org/idr/v0.3/idr0052/3dc087c/5514374.zarr

[pwalczysko]

For me, the workflow described in https://github.com/ome/omero-cli-zarr#masks for overlapping masks ended with error

omero zarr masks Image:5514374 --label-map=5514374.rois
Using session for public@idr.openmicroscopy.org:4064. Idle timeout: 10 min. Current group: Public
Export Masks on Image: cell0005_R0001.companion.ome
Found 2183 mask shapes in 80 ROIs
Error parsing 5514374.rois: 1369132
Label map: Cell (count: 0)
Unique dimensions: {'T': set(), 'C': set(), 'Z': set()}
source_image 5514374.zarr
Ignoring dimensions set()
Traceback (most recent call last):
  File "/Users/pwalczysko/miniconda3/envs/ome_ngff/bin/omero", line 11, in <module>
    sys.exit(main())
  File "/Users/pwalczysko/miniconda3/envs/ome_ngff/lib/python3.9/site-packages/omero/main.py", line 125, in main
    rv = omero.cli.argv()
  File "/Users/pwalczysko/miniconda3/envs/ome_ngff/lib/python3.9/site-packages/omero/cli.py", line 1784, in argv
    cli.invoke(args[1:])
  File "/Users/pwalczysko/miniconda3/envs/ome_ngff/lib/python3.9/site-packages/omero/cli.py", line 1222, in invoke
    stop = self.onecmd(line, previous_args)
  File "/Users/pwalczysko/miniconda3/envs/ome_ngff/lib/python3.9/site-packages/omero/cli.py", line 1299, in onecmd
    self.execute(line, previous_args)
  File "/Users/pwalczysko/miniconda3/envs/ome_ngff/lib/python3.9/site-packages/omero/cli.py", line 1381, in execute
    args.func(args)
  File "/Users/pwalczysko/Work/omero-cli-zarr/src/omero_zarr/cli.py", line 65, in _wrapper
    return func(self, *args, **kwargs)
  File "/Users/pwalczysko/Work/omero-cli-zarr/src/omero_zarr/cli.py", line 236, in masks
    image_shapes_to_zarr(image, ["Mask"], args)
  File "/Users/pwalczysko/Work/omero-cli-zarr/src/omero_zarr/masks.py", line 165, in image_shapes_to_zarr
    saver.save(values, name)
  File "/Users/pwalczysko/Work/omero-cli-zarr/src/omero_zarr/masks.py", line 302, in save
    mask_shape,
UnboundLocalError: local variable 'mask_shape' referenced before assignment

[will-moore]

@pwalczysko My output for that was:

$ omero zarr masks Image:5514374 --label-map=5514374.rois
Using session for public@idr-next.openmicroscopy.org:4064. Idle timeout: 10 min. Current group: Public
Export Masks on Image: cell0005_R0001.companion.ome
Found 2183 mask shapes in 80 ROIs
Label map: Cell (count: 40)
Unique dimensions: {'T': {0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39}, 'C': {None}, 'Z': {1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29}}
source_image 5514374.zarr
Ignoring dimensions {'C'}
Created 5514374.zarr/labels/Cell
Label map: Chromosomes (count: 40)
Unique dimensions: {'T': {0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39}, 'C': {None}, 'Z': {1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25}}
source_image 5514374.zarr
Ignoring dimensions {'C'}
Created 5514374.zarr/labels/Chromosomes

The first line to differ is where you have Label map: Cell (count: 0), so it looks like it's not finding or reading the 5514374.rois file correctly. Can you cat 5514374.rois to check it looks right?

[pwalczysko]

Sorry, my 5514374.rois was actually 5514375.rois as I did not amend the ID in the hql. Now, with correct 5514374.rois I get

omero zarr masks Image:5514374 --label-map=5514374.rois
Using session for public@idr.openmicroscopy.org:4064. Idle timeout: 10 min. Current group: Public
Export Masks on Image: cell0005_R0001.companion.ome
Found 2183 mask shapes in 80 ROIs
Label map: Cell (count: 40)
Unique dimensions: {'T': {0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39}, 'C': {None}, 'Z': {1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29}}
source_image 5514374.zarr
Ignoring dimensions {'C'}
Created 5514374.zarr/labels/Cell
Label map: Chromosomes (count: 40)
Unique dimensions: {'T': {0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39}, 'C': {None}, 'Z': {1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25}}
source_image 5514374.zarr
Ignoring dimensions {'C'}
Created 5514374.zarr/labels/Chromosomes

Also, it opens and looks fine in napari too.

[sbesson]

@will-moore https://pypi.org/project/ome-zarr/0.2.0/ has just been released with the new axes API for writing 2D-4D data. Can you update setup.py to indicate the minimal requirement?