search

ome / ZarrReader

Other
5 stars 8 forks source link

Multi-file images do not import correctly if nested in a .zarr folder #25

Open pwalczysko opened 2 years ago

pwalczysko commented 2 years ago

Workflow:

  1. Convert the ...downloads.openmicroscopy.org/images/OME-TIFF/2016-06/binaryonly images into zarr using bf2raw version 0.4.0, using the command
  2. bioformats2raw binaryonly/multifile-Z1.ome.tiff multifile-Z1.ome.zarr
  3. Nest the resulting multifile-Z1.ome.zarr into another folder which is also a result of an bf2raw output, such as
  4. /uod/idr-scratch/test.zarr/petr-test/multifile-Z1.ome.zarr
  5. Import the resulting multifile-Z1.ome.zarr folder to OMERO using the command (note the usage of the --depth 10)
  6. omero import --transfer=ln_s --skip=checksum --parallel-upload=10 --skip=minmax --depth 10 multifile-Z1.ome.zarr/
  7. Observe that the image in OMERO is consisting of 3 images with z=1 each https://merge-ci.openmicroscopy.org/web/webclient/?show=image-228865. This is not expected, as the result should be 1 image with 5 z planes, as shown in the ome.tiff import case https://merge-ci.openmicroscopy.org/web/webclient/?show=image-228823
  8. log in as user-8 to the merge-ci to see the images

This problem is specific to the nesting of the ome.zarr file into another .zarr file.

cc @joshmoore @sbesson

sbesson commented 2 years ago

The top-level Zarr generated by bioformats2raw 0.4.0 is a Zarr group containing one or more images as defined by the OME-NGFF specification as well as an OME group with the metadata. From my side, this issue raises the questions of what the behavior should be in ZarrReader when being pointed to this top-level group, currently unspecified in the OME-NGFF specification.

I think there is an overlap with the issue reported in https://github.com/ome/ZarrReader/issues/18 i.e. when the Zarr group deviates from the case of a single multiresolution image, the reader is trying to discover more and we need to define the contract in that case.

I can imagine a few options: 1- be strict and do nothing i.e. detect this as a Zarr group that is not specified in OME-NGFF and ignore it 2- discover and detect the image/series groups as separate single-image filesets 3- discover and detect the image/series groups as a multi-image fileset

I assume the ongoing work on https://github.com/ome/ngff/issues/104 is leading us towards 2 but with the existing specification, should we use a different intermediate representation?

joshmoore commented 2 years ago

https://github.com/ome/ngff/pull/112 was my first attempt at capturing the multi-image-ness. Not sure if it's yet specific enough for ZarrReader, @dgault. (It also doesn't try to capture the semantics when it's a non-bioformats2raw mix of groups)

pwalczysko commented 2 years ago

25 was my first attempt at capturing the multi-image-ness. Not sure if it's yet specific enough for ZarrReader, @dgault. (It also doesn't try to capture the semantics when it's a non-bioformats2raw mix of groups)

@joshmoore not sure why is your link above pointing to this very issue ? (#25)

joshmoore commented 2 years ago

Fixed: https://github.com/ome/ngff/pull/112

pwalczysko commented 2 years ago

I think that this issue is solved now. The .zarr folder was inside another .zarr folder which probably confused the importer. Now, with the adjustment described in https://github.com/ome/ZarrReader/issues/26#issuecomment-1091877301, the new import is indeed producing 1 image with 5 z planes as expected, see https://merge-ci.openmicroscopy.org/web/webclient/?show=image-228872

pwalczysko commented 2 years ago

Should I close this or is it still of relevance ?

joshmoore commented 2 years ago

I'd propose updating the title & description but leaving this open lest someone else get themselves into this same situation.