ZarrReader: Add support for OME-XML from bf2raw

dgault commented 2 years ago

This PR adds parsing for the METADATA.ome.xml file produced via bioformats2raw. Currently the XML is parsed first and any conflicting data in the Zarr attributes will override the XML values.

To test this the best approach would be to take existing OME-TIFF samples, convert them using bf2raw and then compare the OME-XML output from the original to that from the ZarrReader. The ZarrReader is still currently dumping the ngff metadata as annotations, so you will see extra annotations in the converted version.

joshmoore commented 2 years ago

A clarification:

Without this PR, you could only import each of the two directories "0" and "1" under out after bioformats2raw a&series=2.fake out (The OME/METADATA.ome.xml is ignored)
With this PR, we can finally use bioformats2raw a&series=2.fake a_real.ome.zarr and import the entire output and the OME/METADATA.ome.xml is taken into account, e.g. for image name, etc.

Background information is available in https://github.com/ome/ngff/pull/112

cc: @pwalczysko

will-moore commented 2 years ago

Testing with...

$ omero import --fetch-jars https://merge-ci.openmicroscopy.org/jenkins/job/OMERO-build/lastSuccessfulBuild/artifact/src/target/OMERO.java-5.6.3-228-6e1847c-ice36-b1178.zip 11511419.zarr

pwalczysko commented 2 years ago

A clarification:

* Without this PR, you could only import each of the two directories "0" and "1" under out after `bioformats2raw a&series=2.fake out` (The `OME/METADATA.ome.xml` is ignored)

* With this PR, we can finally use `bioformats2raw a&series=2.fake a_real.ome.zarr` and import the entire output and the `OME/METADATA.ome.xml` is taken into account, e.g. for image name, etc.

Background information is available in ome/ngff#112

cc: @pwalczysko

thank you @joshmoore . I think I undestand. The two directories "0" and "1" were the images in the two "Series" as defined by the bf2raw. But ZarrReader did not read the ome.xml file where the info which is BF usually using to decide whether this is a MIF or not and how to order it and name it. Thus, prior to this PR, the MIF OME-NGFFs were not imported as MIFs, you could only import single images ? And surely they were wrongly named, I saw that during testing.

joshmoore commented 2 years ago

@pwalczysko : that all sounds correct, yes.

sbesson commented 2 years ago

And surely they were wrongly named, I saw that during testing.

This is possibly a topic for a separate issue but I am raising here as I suspect this will become apparent with the testing of non-HCS multi-image OME-NGFF.

It would be good to write down our expectation for the naming of these images. The current historical logic for non-HCS imports to use the filename of the first file in the fileset (as detected by Bio-Formats), hence for OME-NGFF most images will be named 0 since this is the filename of the first chunk. With the OME metadata included, an alternative would be to start consuming the Image.Name metadata.

pwalczysko commented 2 years ago

start consuming the Image.Name metadata.

This seems natural to me. I remember a discussion where this issue was raised even in pre-ngff times (i.e. this should be what BF is doing, but is not.. (was one voice of the discussion)). This seems to me now like the best solution.

will-moore commented 2 years ago

Image converted to NGFF with bioformats2raw then imported at https://merge-ci.openmicroscopy.org/web/webclient/?show=image-230717 (user-3) has same metadata as original image imported into same Dataset. The only difference I can see is the name (known issue) and the thumbnail (seems to cover an area larger than the image itself, so that the image is in the top-left quarter of the thumbnail). @sbesson I think the rendering settings (default Z and T) have been changed / saved in webclient since import.

sbesson commented 2 years ago

@willmoore after resetting the rendering settings to Imported, the two images (original and converted) look identical although the thumbnail is not updated

will-moore commented 2 years ago

$ wget https://downloads.openmicroscopy.org/images/SVS/77928.svs
$ bioformats2raw 77928.svs 77928.zarr
$ find 77928.zarr/ -type f | wc
  14613   14613  387520
$ omero import -d 54570 --depth 10 77928.zarr
Using session for user-3@merge-ci-devspace.openmicroscopy.org:4064. Idle timeout: 10 min. Current group: read-only-1
2022-05-02 16:28:08,943 276        [      main] INFO          ome.formats.importer.ImportConfig - OMERO.blitz Version: 5.5.11-SNAPSHOT
2022-05-02 16:28:08,973 306        [      main] INFO          ome.formats.importer.ImportConfig - Bioformats version: 6.10.0-SNAPSHOT revision: 0b26396c63282885de616d6f1c71aedd9024b3ef date: 29 April 2022
2022-05-02 16:28:09,305 638        [      main] INFO   formats.importer.cli.CommandLineImporter - Log levels -- Bio-Formats: ERROR OMERO.importer: INFO
2022-05-02 16:28:09,868 1201       [      main] INFO      ome.formats.importer.ImportCandidates - Depth: 10 Metadata Level: MINIMUM
2022-05-02 16:28:15,175 6508       [      main] INFO      ome.formats.importer.ImportCandidates - 14613 file(s) parsed into 1 group(s) with 1 call(s) to setId in 2116ms. (5307ms total) [0 unknowns]
2022-05-02 16:28:15,275 6608       [      main] INFO       ome.formats.OMEROMetadataStoreClient - Attempting initial SSL connection to merge-ci-devspace.openmicroscopy.org:4064
2022-05-02 16:28:15,821 7154       [      main] INFO       ome.formats.OMEROMetadataStoreClient - Insecure connection requested, falling back
2022-05-02 16:28:16,300 7633       [      main] INFO       ome.formats.OMEROMetadataStoreClient - Pinging session every 300s.
2022-05-02 16:28:16,313 7646       [      main] INFO       ome.formats.OMEROMetadataStoreClient - Server: 5.6.3
2022-05-02 16:28:16,313 7646       [      main] INFO       ome.formats.OMEROMetadataStoreClient - Client: 5.5.11-SNAPSHOT
2022-05-02 16:28:16,314 7647       [      main] INFO       ome.formats.OMEROMetadataStoreClient - Java Version: 1.8.0_152-release
2022-05-02 16:28:16,314 7647       [      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Name: Linux
2022-05-02 16:28:16,314 7647       [      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Arch: amd64
2022-05-02 16:28:16,314 7647       [      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Version: 3.10.0-1160.15.2.el7.x86_64
2022-05-02 16:28:16,468 7801       [      main] WARN                    ome.system.UpgradeCheck - UPGRADE AVAILABLE:Please upgrade to 5.6.1. See http://downloads.openmicroscopy.org/latest/omero for the latest version.
2022-05-02 16:28:16,898 8231       [      main] INFO          ome.formats.importer.ImportConfig - Using import target: Dataset:54570
2022-05-02 16:28:16,928 8261       [2-thread-1] INFO         ome.formats.importer.ImportLibrary - Import target specifies container: Dataset:54570
2022-05-02 16:28:16,936 8269       [2-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - FILESET_UPLOAD_PREPARATION
2022-05-02 16:31:24,159 195492     [2-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - FILESET_UPLOAD_START
2022-05-02 16:31:24,187 195520     [3-thread-1] INFO   ts.importer.transfers.UploadFileTransfer - Transferring /uod/idr-scratch/will/svs/77928.zarr/2/3/0/2/0/0/0...
...
2022-05-02 16:51:21,383 1392716    [3-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_COMPLETE: /uod/idr-scratch/will/svs/77928.zarr/.zgroup
2022-05-02 16:52:17,362 1448695    [2-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - FILESET_UPLOAD_END
2022-05-02 16:52:18,079 1449412    [2-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - IMPORT_STARTED Logfile: 1344084
2022-05-02 16:52:29,340 1460673    [l.Client-1] INFO   ormats.importer.cli.LoggingImportMonitor - METADATA_IMPORTED Step: 1 of 5  Logfile: 1344084
2022-05-02 16:52:30,157 1461490    [l.Client-2] INFO   ormats.importer.cli.LoggingImportMonitor - PIXELDATA_PROCESSED Step: 2 of 5  Logfile: 1344084
2022-05-02 16:52:37,304 1468637    [l.Client-3] INFO   ormats.importer.cli.LoggingImportMonitor - THUMBNAILS_GENERATED Step: 3 of 5  Logfile: 1344084
2022-05-02 16:52:37,328 1468661    [l.Client-1] INFO   ormats.importer.cli.LoggingImportMonitor - METADATA_PROCESSED Step: 4 of 5  Logfile: 1344084
2022-05-02 16:52:37,348 1468681    [l.Client-2] INFO   ormats.importer.cli.LoggingImportMonitor - OBJECTS_RETURNED Step: 5 of 5  Logfile: 1344084
2022-05-02 16:52:37,950 1469283    [l.Client-3] INFO   ormats.importer.cli.LoggingImportMonitor - IMPORT_DONE Imported file: /uod/idr-scratch/will/svs/77928.zarr/2/3/0/2/0/0/0
Image:230866,230867,230868
Other imported objects:
Fileset:123608

https://merge-ci.openmicroscopy.org/web/webclient/?show=image-230866

But, thumbnails are broken (for full-sized image 230866) or incorrect as above for the other images in the FileSet. Also, the number of resolutions is more limited than the original (imported into same Dataset), meaning you can't zoom out to see the whole image in iviewer.

sbesson commented 2 years ago

From the logs when trying to save a thumbnail with the file above

2022-05-02 16:09:19,009 INFO  [                 org.perf4j.TimingLogger] (.Server-25) start[1651507758938] time[70] tag[omero.call.exception]
2022-05-02 16:09:19,009 WARN  [        ome.services.util.ServiceHandler] (.Server-25) IllegalArgumentException thrown.

java.lang.IllegalArgumentException: 62425×60101 plane too large, cannot exceed 2^31 pixels
    at omeis.providers.re.RGBIntBuffer.<init>(RGBIntBuffer.java:63)
    at omeis.providers.re.RenderingStrategy.getIntBuffer(RenderingStrategy.java:186)
    at omeis.providers.re.HSBStrategy.renderAsPackedInt(HSBStrategy.java:291)
    at omeis.providers.re.Renderer.renderAsPackedInt(Renderer.java:558)
    at ome.services.ThumbnailBean.createScaledImage(ThumbnailBean.java:648)
    at ome.services.ThumbnailBean._createThumbnail(ThumbnailBean.java:845)
    at ome.services.ThumbnailBean.retrieveThumbnail(ThumbnailBean.java:1189)
    at ome.services.ThumbnailBean.retrieveThumbnailAndUpdateMetadata(ThumbnailBean.java:1140)
    at ome.services.ThumbnailBean.getThumbnailByLongestSide(ThumbnailBean.java:1320)

It almost feels like the server only sees the full resolution of the WSI - which would match the slow user experience in the viewer.

sbesson commented 2 years ago

@dgault pointing showinf at the file generated by Will above, I get

Series count = 3
Series #0 :
    Resolutions = 9
        sizeX[0] = 62425
        sizeX[1] = 62425
        sizeX[2] = 62425
        sizeX[3] = 62425
        sizeX[4] = 62425
        sizeX[5] = 62425
        sizeX[6] = 62425
        sizeX[7] = 62425
        sizeX[8] = 62425
    Image count = 3
    RGB = false (1) 
    Interleaved = false
    Indexed = false (false color)
    Width = 62425
    Height = 60101
    SizeZ = 1
    SizeT = 1
    SizeC = 3
    Tile size = 1024 x 1024
    Thumbnail size = 128 x 123
    Endianness = intel (little)
    Dimension order = XYZCT (uncertain)
    Pixel type = uint8
    Valid bits per pixel = 8
    Metadata complete = false
    Thumbnail series = false
    -----
    Plane #0 <=> Z 0, C 0, T 0
    Plane #1 <=> Z 0, C 1, T 0
    Plane #2 <=> Z 0, C 2, T 0

Series #1 :
    Resolutions = 2
        sizeX[0] = 428
        sizeX[1] = 428
    Image count = 3
    RGB = false (1) 
    Interleaved = false
    Indexed = false (false color)
    Width = 428
    Height = 424
    SizeZ = 1
    SizeT = 1
    SizeC = 3
    Tile size = 428 x 424
    Thumbnail size = 128 x 126
    Endianness = intel (little)
    Dimension order = XYZCT (uncertain)
    Pixel type = uint8
    Valid bits per pixel = 8
    Metadata complete = false
    Thumbnail series = false
    -----
    Plane #0 <=> Z 0, C 0, T 0
    Plane #1 <=> Z 0, C 1, T 0
    Plane #2 <=> Z 0, C 2, T 0

Series #2 :
    Resolutions = 4
        sizeX[0] = 1280
        sizeX[1] = 1280
        sizeX[2] = 1280
        sizeX[3] = 1280
    Image count = 3
    RGB = false (1) 
    Interleaved = false
    Indexed = false (false color)
    Width = 1280
    Height = 431
    SizeZ = 1
    SizeT = 1
    SizeC = 3
    Tile size = 1024 x 431
    Thumbnail size = 128 x 43
    Endianness = intel (little)
    Dimension order = XYZCT (uncertain)
    Pixel type = uint8
    Valid bits per pixel = 8
    Metadata complete = false
    Thumbnail series = false
    -----
    Plane #0 <=> Z 0, C 0, T 0
    Plane #1 <=> Z 0, C 1, T 0
    Plane #2 <=> Z 0, C 2, T 0

so it looks like an issue with the detection of sub-resolutions

sbesson commented 2 years ago

This seems natural to me. I remember a discussion where this issue was raised even in pre-ngff times (i.e. this should be what BF is doing, but is not.. (was one voice of the discussion)). This seems to me now like the best solution.

Cross-linking to existing links

https://github.com/ome/design/issues/57 is the reference issue discussing how plate and image names are defined at import time
https://github.com/ome/openmicroscopy/pull/4992 is a corresponding PR which updates the HCS import logic to use the Plate.Name populated by Bio-Formats as the ground truth if no CLI option is specified to override it
within this PR, note in particular https://github.com/ome/openmicroscopy/pull/4992/commits/cc042d8861385d2cf393834a37831d0cec8a1055 which modifies the logic where the Image.Name is set (this change was reverted to focus on the HCS naming issue)

Overriding the target image name via the importer is always a possibility although this presents obvious usability and scalability issue.

It might be worth moving this discussion into a separate issue to limit the scope of the review for this particular PR. As a first question, what is the expectation for the default image name for these datasets under the various scenarios (single images, MIFs)?

pwalczysko commented 2 years ago

Pasting here a first start on the testing of comparisons of ome.xml files between pre-conversion ome.tiffs and post-conversion ome.zarrs. Testing done with

BF_CP=OMEZarrReader-0.1.6-SNAPSHOT-jar-with-dependencies.jar /Users/pwalczysko/Downloads/bftools/showinf -nopix -noflat -omexml tubhiswt-2D/tubhiswt_C0.ome.tif >> tubhiswt-2D/tubhiswt_C0.ome.xml
bioformats2raw tubhiswt-2D/tubhiswt_C0.ome.tif  tubhiswt-2D/tubhiswt_C0.ome.zarr
BF_CP=OMEZarrReader-0.1.6-SNAPSHOT-jar-with-dependencies.jar /Users/pwalczysko/Downloads/bftools/showinf -nopix -noflat -omexml tubhiswt-2D/tubhiswt_C0.ome.zarr/.zattrs >> tubhiswt-2D/tubhiswt_C0.ome.zarr.xml
diff -u tubhiswt-2D/tubhiswt_C0.ome.xml tubhiswt-2D/tubhiswt_C0.ome.zarr.xml > difference.txt

The content of the difference.txt is

``` --- tubhiswt-2D/tubhiswt_C0.ome.xml 2022-05-04 16:27:43.000000000 +0100 +++ tubhiswt-2D/tubhiswt_C0.ome.zarr.xml 2022-05-04 19:01:35.000000000 +0100 @@ -1,21 +1,31 @@ -Checking file format [OME-TIFF] +Checking file format [Zarr] Initializing reader -OMETiffReader initializing tubhiswt-2D/tubhiswt_C0.ome.tif -Reading IFDs -Populating metadata +ZarrReader initializing tubhiswt-2D/tubhiswt_C0.ome.zarr/.zattrs Parsing schema path http://www.openmicroscopy.org/Schemas/OME/2016-06/ome.xsd Validating OME-XML No validation errors found. -Initialization took 0.34s +Initialization took 0.538s Reading core metadata -filename = /Users/pwalczysko/ngff-testing/tubhiswt-2D/tubhiswt_C0.ome.tif +filename = tubhiswt-2D/tubhiswt_C0.ome.zarr/.zattrs Used files: - /Users/pwalczysko/ngff-testing/tubhiswt-2D/tubhiswt_C0.ome.tif - /Users/pwalczysko/ngff-testing/tubhiswt-2D/tubhiswt_C1.ome.tif + /Users/pwalczysko/ngff-testing/tubhiswt-2D/tubhiswt_C0.ome.zarr/.zattrs + /Users/pwalczysko/ngff-testing/tubhiswt-2D/tubhiswt_C0.ome.zarr/.zgroup + /Users/pwalczysko/ngff-testing/tubhiswt-2D/tubhiswt_C0.ome.zarr/0/.zattrs + /Users/pwalczysko/ngff-testing/tubhiswt-2D/tubhiswt_C0.ome.zarr/0/.zgroup + /Users/pwalczysko/ngff-testing/tubhiswt-2D/tubhiswt_C0.ome.zarr/0/0/.zarray + /Users/pwalczysko/ngff-testing/tubhiswt-2D/tubhiswt_C0.ome.zarr/0/0/0/0/0/0/0 + /Users/pwalczysko/ngff-testing/tubhiswt-2D/tubhiswt_C0.ome.zarr/0/0/0/1/0/0/0 + /Users/pwalczysko/ngff-testing/tubhiswt-2D/tubhiswt_C0.ome.zarr/0/1/.zarray + /Users/pwalczysko/ngff-testing/tubhiswt-2D/tubhiswt_C0.ome.zarr/0/1/0/0/0/0/0 + /Users/pwalczysko/ngff-testing/tubhiswt-2D/tubhiswt_C0.ome.zarr/0/1/0/1/0/0/0 + /Users/pwalczysko/ngff-testing/tubhiswt-2D/tubhiswt_C0.ome.zarr/OME/METADATA.ome.xml Series count = 1 -Series #0 -- tubhiswt: +Series #0 -- 0/0: + Resolutions = 2 + sizeX[0] = 512 + sizeX[1] = 256 Image count = 2 RGB = false (1) Interleaved = false @@ -28,10 +38,10 @@ Tile size = 512 x 512 Thumbnail size = 128 x 128 Endianness = intel (little) - Dimension order = XYZTC (certain) + Dimension order = XYZCT (uncertain) Pixel type = uint8 Valid bits per pixel = 8 - Metadata complete = true + Metadata complete = false Thumbnail series = false ----- Plane #0 <=> Z 0, C 0, T 0 @@ -57,10 +67,10 @@ - + 2013-01-15T17:02:40 - + @@ -81,5 +91,28 @@ + + + { + "bioformats2raw.layout" : 3 +} + + + { + "multiscales" : [ { + "metadata" : { + "method" : "loci.common.image.SimpleImageScaler", + "version" : "Bio-Formats 6.8.0" + }, + "datasets" : [ { + "path" : "0" + }, { + "path" : "1" + } ], + "version" : "0.2" + } ] +} + + ```

This does not look to me like the zarr reader is missing something, all differences seem logical ? cc @sbesson @dgault .

dgault commented 2 years ago

That looks rather promising to me. The differences there all seem to be expected or explainable

DimensionOrder: As this is a 2D image the unused dimensions are uncertain and reverting to a default dimensionOrder
Extra structuredAnnotations: To be expected as its currently dumping the Zarr attributes here to preserve them
ImageName: The only real difference but this is one that we are already aware of
MetadataComplete: The correct value here is probably up for discussion

pwalczysko commented 2 years ago

diff -u tubhiswt-3D/tubhiswt_C0.ome.xml tubhiswt-3D/tubhiswt_C0.ome.zarr.xml > tubhiswt-3D/C0.diff which produces the C0.diff below

``` --- tubhiswt-3D/tubhiswt_C0.ome.xml 2022-05-04 16:45:37.000000000 +0100 +++ tubhiswt-3D/tubhiswt_C0.ome.zarr.xml 2022-05-05 14:26:30.000000000 +0100 @@ -1,21 +1,107 @@ -Checking file format [OME-TIFF] +Checking file format [Zarr] Initializing reader -OMETiffReader initializing tubhiswt-3D/tubhiswt_C0.ome.tif -Reading IFDs -Populating metadata +ZarrReader initializing tubhiswt-3D/tubhiswt_C0.ome.zarr/.zattrs Parsing schema path http://www.openmicroscopy.org/Schemas/OME/2016-06/ome.xsd Validating OME-XML No validation errors found. -Initialization took 0.357s +Initialization took 0.819s Reading core metadata -filename = /Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.tif +filename = tubhiswt-3D/tubhiswt_C0.ome.zarr/.zattrs Used files: - /Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.tif - /Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C1.ome.tif + /Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/.zattrs + /Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/.zgroup + /Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/.zattrs + /Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/.zgroup + /Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/0/.zarray + /Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/0/18/0/0/0/0 + /Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/0/18/1/0/0/0 + /Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/0/9/0/0/0/0 + /Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/0/9/1/0/0/0 + /Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/0/0/0/0/0/0 + /Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/0/0/1/0/0/0 + /Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/0/11/0/0/0/0 + /Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/0/11/1/0/0/0 + /Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/0/7/0/0/0/0 + /Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/0/7/1/0/0/0 + /Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/0/16/0/0/0/0 + /Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/0/16/1/0/0/0 + /Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/0/6/0/0/0/0 + /Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/0/6/1/0/0/0 + /Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/0/17/0/0/0/0 + /Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/0/17/1/0/0/0 + /Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/0/1/0/0/0/0 + /Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/0/1/1/0/0/0 + /Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/0/10/0/0/0/0 + /Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/0/10/1/0/0/0 + /Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/0/19/0/0/0/0 + /Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/0/19/1/0/0/0 + /Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/0/8/0/0/0/0 + /Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/0/8/1/0/0/0 + /Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/0/4/0/0/0/0 + /Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/0/4/1/0/0/0 + /Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/0/15/0/0/0/0 + /Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/0/15/1/0/0/0 + /Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/0/3/0/0/0/0 + /Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/0/3/1/0/0/0 + /Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/0/12/0/0/0/0 + /Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/0/12/1/0/0/0 + /Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/0/2/0/0/0/0 + /Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/0/2/1/0/0/0 + /Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/0/13/0/0/0/0 + /Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/0/13/1/0/0/0 + /Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/0/5/0/0/0/0 + /Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/0/5/1/0/0/0 + /Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/0/14/0/0/0/0 + /Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/0/14/1/0/0/0 + /Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/1/.zarray + /Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/1/18/0/0/0/0 + /Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/1/18/1/0/0/0 + /Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/1/9/0/0/0/0 + /Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/1/9/1/0/0/0 + /Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/1/0/0/0/0/0 + /Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/1/0/1/0/0/0 + /Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/1/11/0/0/0/0 + /Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/1/11/1/0/0/0 + /Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/1/7/0/0/0/0 + /Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/1/7/1/0/0/0 + /Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/1/16/0/0/0/0 + /Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/1/16/1/0/0/0 + /Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/1/6/0/0/0/0 + /Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/1/6/1/0/0/0 + /Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/1/17/0/0/0/0 + /Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/1/17/1/0/0/0 + /Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/1/1/0/0/0/0 + /Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/1/1/1/0/0/0 + /Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/1/10/0/0/0/0 + /Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/1/10/1/0/0/0 + /Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/1/19/0/0/0/0 + /Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/1/19/1/0/0/0 + /Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/1/8/0/0/0/0 + /Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/1/8/1/0/0/0 + /Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/1/4/0/0/0/0 + /Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/1/4/1/0/0/0 + /Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/1/15/0/0/0/0 + /Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/1/15/1/0/0/0 + /Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/1/3/0/0/0/0 + /Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/1/3/1/0/0/0 + /Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/1/12/0/0/0/0 + /Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/1/12/1/0/0/0 + /Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/1/2/0/0/0/0 + /Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/1/2/1/0/0/0 + /Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/1/13/0/0/0/0 + /Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/1/13/1/0/0/0 + /Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/1/5/0/0/0/0 + /Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/1/5/1/0/0/0 + /Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/1/14/0/0/0/0 + /Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/0/1/14/1/0/0/0 + /Users/pwalczysko/ngff-testing/tubhiswt-3D/tubhiswt_C0.ome.zarr/OME/METADATA.ome.xml Series count = 1 -Series #0 -- tubhiswt: +Series #0 -- 0/0: + Resolutions = 2 + sizeX[0] = 512 + sizeX[1] = 256 Image count = 40 RGB = false (1) Interleaved = false @@ -28,18 +114,18 @@ Tile size = 512 x 512 Thumbnail size = 128 x 128 Endianness = intel (little) - Dimension order = XYZTC (certain) + Dimension order = XYZCT (uncertain) Pixel type = uint8 Valid bits per pixel = 8 - Metadata complete = true + Metadata complete = false Thumbnail series = false ----- Plane #0 <=> Z 0, C 0, T 0 - Plane #18 <=> Z 0, C 0, T 18 - Plane #19 <=> Z 0, C 0, T 19 - Plane #20 <=> Z 0, C 1, T 0 - Plane #21 <=> Z 0, C 1, T 1 - Plane #22 <=> Z 0, C 1, T 2 + Plane #18 <=> Z 0, C 0, T 9 + Plane #19 <=> Z 0, C 1, T 9 + Plane #20 <=> Z 0, C 0, T 10 + Plane #21 <=> Z 0, C 1, T 10 + Plane #22 <=> Z 0, C 0, T 11 Plane #39 <=> Z 0, C 1, T 19 @@ -62,10 +148,10 @@ - + 2013-01-15T17:02:41 - + @@ -197,46 +283,69 @@ urn:uuid:f801ea0a-e93e-4f4f-99b3-7ecc15048c12 - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + { + "bioformats2raw.layout" : 3 +} + + + { + "multiscales" : [ { + "metadata" : { + "method" : "loci.common.image.SimpleImageScaler", + "version" : "Bio-Formats 6.8.0" + }, + "datasets" : [ { + "path" : "0" + }, { + "path" : "1" + } ], + "version" : "0.2" + } ] +} + + ```

In the above output, I do not understand the Plane... T 18 bit, which seems to have vanished in the ome.zarr ?

The tubhiswt-3D/..C1.. seems to have analogous output/problem. (Edit)

pwalczysko commented 2 years ago

There is something I do not understand about the tubs.. samples. It seems that these are planes of a multi-z muti-t image, according to the names of the single ome.tiffs. But bf2raw does not accept folders as input (?) Hence I am converting the single planes. Is that a sound approach ? Am I missing some point here ? cc @sbesson @joshmoore

Maybe whatever ome.tiff inside the folder I pick, bf2raw will take all the other sections into account and produce the zarr out of all of them ? (eidt) Indeed, when importing the whole folder into OMERO, I have just one image.

pwalczysko commented 2 years ago

diff -u tubhiswt-4D/tubhiswt_C0_TP0.ome.xml tubhiswt-4D/tubhiswt_C0_TP0.ome.zarr.xml > tubhiswt-4D/C0.T0.diff produces C0.T0.diff below. This starts to be quite big.

So big in fact I cannot paste it here. I think this workflow needs a revision. @dgault I have put the C0.T0.diff on idr0-slot3:/tmp for you. We can study it in parallel

Studying the output more closely, I can see 4 questions

Resolutions motif added in zarr

+Series #0 -- 0/0:
+   Resolutions = 2
+       sizeX[0] = 512
+       sizeX[1] = 256

Similarly to the tubhiswt-3D/tubhiswt_C0.ome.xml described in the comment above, Planes seem to be taken away (from the xml of the original ome.tiff), such as some T 41 planes and they do not appear to me to be reported as being added in other places in the same numbers, e.g.

    Plane #0 <=> Z 0, C 0, T 0
-   Plane #428 <=> Z 8, C 0, T 42
-   Plane #429 <=> Z 9, C 0, T 42
-   Plane #430 <=> Z 0, C 1, T 0
-   Plane #431 <=> Z 1, C 1, T 0
-   Plane #432 <=> Z 2, C 1, T 0
+   Plane #428 <=> Z 8, C 0, T 21
+   Plane #429 <=> Z 9, C 0, T 21
+   Plane #430 <=> Z 0, C 1, T 21
+   Plane #431 <=> Z 1, C 1, T 21
+   Plane #432 <=> Z 2, C 1, T 21
    Plane #859 <=> Z 9, C 1, T 42

The <Plane... motives

 <Plane TheC="1" TheT="27" TheZ="0"/>
+         <Plane TheC="1" TheT="27" TheZ="1"/>
+         <Plane TheC="1" TheT="27" TheZ="2"/>
+         <Plane TheC="1" TheT="27" TheZ="3"/>
+         <Plane TheC="1" TheT="27" TheZ="4"/>
+         <Plane TheC="1" TheT="27" TheZ="5"/>
+         <Plane TheC="1" TheT="27" TheZ="6"/>
+         <Plane TheC="1" TheT="27" TheZ="7"/>
+         <Plane TheC="1" TheT="27" TheZ="8"/>
+         <Plane TheC="1" TheT="27" TheZ="9"/>

Hard to judge, there are too many of them. Does that maybe bind with the point Ad 2. in this comment ?

Extra metadata in zarr (possibly expected cf. @dgault 's comment re 2D case)

+      </Pixels>
+   </Image>
+   <StructuredAnnotations>
+      <XMLAnnotation ID="Annotation:0">
+         <Value>{
+  "bioformats2raw.layout" : 3
+}</Value>
+      </XMLAnnotation>
+      <XMLAnnotation ID="Annotation:1">
+         <Value>{
+  "multiscales" : [ {
+    "metadata" : {
+      "method" : "loci.common.image.SimpleImageScaler",
+      "version" : "Bio-Formats 6.8.0"
+    },
+    "datasets" : [ {
+      "path" : "0"
+    }, {
+      "path" : "1"
+    } ],
+    "version" : "0.2"
+  } ]
+}</Value>
+      </XMLAnnotation>
+   </StructuredAnnotations>
+</OME>

Imported the zarr I created during the workflow into merge-ci user-8 https://merge-ci.openmicroscopy.org/web/webclient/?show=image-231015 . It looks good, no difference spotted when comparing with the ome.tiff https://outreach.openmicroscopy.org/webclient/?show=image-95214

pwalczysko commented 2 years ago

Bug: When trying to create an ome.xml output from the BBBC plate, I got an error.

Command and error see below

BF_CP=OMEZarrReader-0.1.6-SNAPSHOT-jar-with-dependencies.jar /Users/pwalczysko/Downloads/bftools/showinf -nopix -noflat -omexml BBBC/NIRHTa-001.ome.zarr/.zattrs >> BBBC/NIRHTa-001.ome.zarr.xml
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/Users/pwalczysko/Downloads/bftools/bioformats_package.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/Users/pwalczysko/ngff-testing/OMEZarrReader-0.1.6-SNAPSHOT-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [ch.qos.logback.classic.util.ContextSelectorStaticBinder]
Exception in thread "main" java.lang.ClassCastException: class java.lang.Integer cannot be cast to class java.lang.String (java.lang.Integer and java.lang.String are in module java.base of loader 'bootstrap')
    at loci.formats.in.ZarrReader.parsePlate(ZarrReader.java:553)
    at loci.formats.in.ZarrReader.initFile(ZarrReader.java:281)
    at loci.formats.FormatReader.setId(FormatReader.java:1443)
    at loci.formats.ImageReader.setId(ImageReader.java:849)
    at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650)
    at loci.formats.tools.ImageInfo.testRead(ImageInfo.java:1035)
    at loci.formats.tools.ImageInfo.main(ImageInfo.java:1121)

joshmoore commented 2 years ago

Testing the zarr from https://github.com/spatial-image/spatial-image-multiscale/pull/32:

Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: -1
    at loci.formats.in.ZarrReader.getOptimalTileHeight(ZarrReader.java:130)
    at loci.formats.ImageReader.getOptimalTileHeight(ImageReader.java:753)
    at loci.formats.ReaderWrapper.getOptimalTileHeight(ReaderWrapper.java:585)
    at loci.formats.tools.ImageInfo.readCoreMetadata(ImageInfo.java:577)
    at loci.formats.tools.ImageInfo.testRead(ImageInfo.java:1048)

chunks are [64, 64, 64]. However, since this dataset follows https://github.com/ome/ngff/pull/114 we might should handle this separately from this PR.

sbesson commented 2 years ago

I was able to reproduce https://github.com/ome/ZarrReader/pull/31#issuecomment-1118978361 independently by converting a test plate in the context of https://github.com/broadinstitute/lincs-cell-painting/issues/54#issuecomment-1117842567.

I believe the issue is related to the handling of the column_index/row_index keys. I can see either trying to fix this as part of this PR or focus on the non-HCS modalities first and looking into another series of HCS fixes as as separate PR to facilitate the review. Happy to discuss more this morning

will-moore commented 2 years ago

Comparing the OME.XML from https://merge-ci.openmicroscopy.org/web/webclient/?show=image-230717 (converted to NGFF and imported) with that of the Original Image (in the same Dataset) I noticed that the <Plane> elements are ordered differently in the NGFF image. Looking at the images in webclient, I can see that in the Original Image, the Exposure values are 0.8 for 1 channel and 0.5 for the other channel, whereas in the NGFF image, they alternate 0.5 and 0.8 for both channnels, and the T indices look like they are sorted strangely: 1, 21, 2, 22, 3 ...

pwalczysko commented 2 years ago

@dgault Is the strange sorting of T indices reported by @will-moore something which was indicated already in the diff of https://github.com/ome/ZarrReader/pull/31#issuecomment-1118624516 maybe ? The T 42 in the diff seemed suspect to me.

pwalczysko commented 2 years ago

Here is the slightly truncated diff of xml files of ome.tiff vs ome.zarr of the Leica-1.ome.tiff from /uod/idr/repos/curated/ome-tiff/public/2016-06/sub-resolutions/Brightfield/Leica-1/Leica-1.ome.tiff

I am not sure about the handling of the RGB true or false difference between ome.tiff and ome.zarr, this feels wrong, the number of images is different as well.

Further, the number of Resolutions differ between the tiff and zarr.

< Series #0 -- macro:
<   Resolutions = 3
---
> Series #0 -- 0/0:
>   Resolutions = 6

``` cat sub-resolutions/Leica-1.diff | grep -v "/uod/idr" 1d0 < Checking file format [OME-TIFF] 3,9c2,4 < Reading IFDs < Populating metadata < Reading IFDs < Populating metadata < Reading IFDs < Populating metadata --- > Checking file format [Zarr] > Initializing reader > ZarrReader initializing sub-resolutions/Leica-1.ome.zarr/.zattrs 14c9 < Initialization took 0.799s --- > Initialization took 1.572s 17c12,5612 --- > filename = sub-resolutions/Leica-1.ome.zarr/.zattrs > Used files: 19,22c5614,5615 < Reading IFDs < Populating metadata < Series #0 -- macro: < Resolutions = 3 --- > Series #0 -- 0/0: > Resolutions = 6 24,27c5617,5623 < sizeX[1] = 404 < sizeX[2] = 101 < Image count = 1 < RGB = true (3) --- > sizeX[1] = 808 > sizeX[2] = 404 > sizeX[3] = 202 > sizeX[4] = 101 > sizeX[5] = 50 > Image count = 3 > RGB = false (1) 34,35c5630,5631 < SizeC = 3 (effectively 1) < Tile size = 1616 x 216 --- > SizeC = 3 > Tile size = 1024 x 1024 38c5634 < Dimension order = XYCZT (certain) --- > Dimension order = XYZCT (uncertain) 41c5637 < Metadata complete = true --- > Metadata complete = false 44a5641,5642 > Plane #1 <=> Z 0, C 1, T 0 > Plane #2 <=> Z 0, C 2, T 0 46,49c5644,5645 < Reading IFDs < Populating metadata < Series #1 -- : < Resolutions = 5 --- > Series #1 -- 1/0: > Resolutions = 9 51,56c5647,5656 < sizeX[1] = 9208 < sizeX[2] = 2302 < sizeX[3] = 576 < sizeX[4] = 144 < Image count = 1 < RGB = true (3) --- > sizeX[1] = 18416 > sizeX[2] = 9208 > sizeX[3] = 4604 > sizeX[4] = 2302 > sizeX[5] = 1151 > sizeX[6] = 575 > sizeX[7] = 287 > sizeX[8] = 143 > Image count = 3 > RGB = false (1) 63,64c5663,5664 < SizeC = 3 (effectively 1) < Tile size = 1616 x 216 --- > SizeC = 3 > Tile size = 1024 x 1024 67c5667 < Dimension order = XYCZT (certain) --- > Dimension order = XYZCT (uncertain) 70c5670 < Metadata complete = true --- > Metadata complete = false 73a5674,5675 > Plane #1 <=> Z 0, C 1, T 0 > Plane #2 <=> Z 0, C 2, T 0 77,105d5678 < BitsPerSample: 8 < Compression: JPEG < ImageLength: 4668 < ImageWidth: 1616 < MetaDataPhotometricInterpretation: RGB < NumberOfChannels: 3 < PhotometricInterpretation: YCbCr < PlanarConfiguration: Chunky < ReferenceBlackWhite: 0 < ResolutionUnit: Centimeter < SamplesPerPixel: 3 < TileLength: 512 < TileWidth: 512 < XResolution: 608.3301634770519 < YCbCrSubSampling: chroma image dimensions are half the luma image dimensions < YResolution: 608.3301634770519 < macro collection.name: ImageCollection_0000000128 < macro collection.uuid: urn:uuid:d0fa25ee-baa8-48cd-944d-5598a36b3f7b < macro creationDate: 2011-05-31T09:43:06.873Z < macro device.model for image: Leica SCN400;Leica SCN < macro device.version for image: 1.4.0.9691 2011/03/30 10:30:59;1.4.0.9708 < macro scanSettings.illuminationSettings.illuminationSource for image: brightfield < macro scanSettings.illuminationSettings.numericalAperture for image: 0.4 < macro scanSettings.objectiveSettings.objective for image: 20 < macro view.offsetX for image: 5389341 < macro view.offsetY for image: 17548313 < macro view.sizeX for image: 18416000 < macro view.sizeY for image: 19216000 < macro view.spacingZ for image: 400 116c5689 < --- > 121,122c5694,5701 < < --- > > > > > > > > 128a5708,5709 > > 131c5712 < --- > 136,137c5717,5724 < < --- > > > > > > > > 143a5731,5732 > > 148,153c5737,5739 < < < macro view.spacingZ for image < 400 < < --- > { > "bioformats2raw.layout" : 3 > } 156,161c5742,5763 < < < macro scanSettings.objectiveSettings.objective for image < 20 < < --- > { > "multiscales" : [ { > "metadata" : { > "method" : "loci.common.image.SimpleImageScaler", > "version" : "Bio-Formats 6.8.0" > }, > "datasets" : [ { > "path" : "0" > }, { > "path" : "1" > }, { > "path" : "2" > }, { > "path" : "3" > }, { > "path" : "4" > }, { > "path" : "5" > } ], > "version" : "0.2" > } ] > } 164,169c5766,5793 < < < macro scanSettings.illuminationSettings.illuminationSource for image < brightfield < < --- > { > "multiscales" : [ { > "metadata" : { > "method" : "loci.common.image.SimpleImageScaler", > "version" : "Bio-Formats 6.8.0" > }, > "datasets" : [ { > "path" : "0" > }, { > "path" : "1" > }, { > "path" : "2" > }, { > "path" : "3" > }, { > "path" : "4" > }, { > "path" : "5" > }, { > "path" : "6" > }, { > "path" : "7" > }, { > "path" : "8" > } ], > "version" : "0.2" > } ] > } ```

pwalczysko commented 2 years ago

Another example of ordering of planes problem is /uod/idr/repos/curated/ome-tiff/public/2016-06/sub-resolutions/EM/BGal_000438_frames.ome.tiff vs its zarr counterpart. Both imported into https://merge-ci.openmicroscopy.org/web/webclient/?show=dataset-54970

the output of @will-moore 's script see below:

``` Checking element -------- ** difference... ** -------------- CHILD1 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '22'} CHILD2 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '4'} ...---------------------- Checking element -------- ** difference... ** -------------- CHILD1 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '31'} CHILD2 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '1'} ...---------------------- Checking element -------- ** difference... ** -------------- CHILD1 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '9'} CHILD2 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '6'} ...---------------------- Checking element -------- ** difference... ** -------------- CHILD1 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '2'} CHILD2 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '36'} ...---------------------- Checking element -------- ** difference... ** -------------- CHILD1 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '37'} CHILD2 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '27'} ...---------------------- Checking element -------- ** difference... ** -------------- CHILD1 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '21'} CHILD2 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '24'} ...---------------------- Checking element -------- ** difference... ** -------------- CHILD1 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '17'} CHILD2 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '2'} ...---------------------- Checking element -------- ** difference... ** -------------- CHILD1 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '24'} CHILD2 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '34'} ...---------------------- Checking element -------- ** difference... ** -------------- CHILD1 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '32'} CHILD2 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '15'} ...---------------------- Checking element -------- ** difference... ** -------------- CHILD1 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '10'} CHILD2 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '16'} ...---------------------- Checking element -------- ** difference... ** -------------- CHILD1 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '3'} CHILD2 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '20'} ...---------------------- Checking element -------- ** difference... ** -------------- CHILD1 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '4'} CHILD2 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '11'} ...---------------------- Checking element -------- ** difference... ** -------------- CHILD1 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '23'} CHILD2 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '14'} ...---------------------- Checking element -------- ** difference... ** -------------- CHILD1 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '19'} CHILD2 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '28'} ...---------------------- Checking element -------- ** difference... ** -------------- CHILD1 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '5'} CHILD2 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '33'} ...---------------------- Checking element -------- ** difference... ** -------------- CHILD1 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '33'} CHILD2 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '7'} ...---------------------- Checking element -------- ** difference... ** -------------- CHILD1 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '18'} CHILD2 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '8'} ...---------------------- Checking element -------- ** difference... ** -------------- CHILD1 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '15'} CHILD2 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '37'} ...---------------------- Checking element -------- ** difference... ** -------------- CHILD1 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '14'} CHILD2 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '29'} ...---------------------- Checking element -------- ** difference... ** -------------- CHILD1 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '25'} CHILD2 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '9'} ...---------------------- Checking element -------- ** difference... ** -------------- CHILD1 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '27'} CHILD2 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '0'} ...---------------------- Checking element -------- ** difference... ** -------------- CHILD1 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '29'} CHILD2 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '5'} ...---------------------- Checking element -------- ** difference... ** -------------- CHILD1 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '12'} CHILD2 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '18'} ...---------------------- Checking element -------- ** difference... ** -------------- CHILD1 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '35'} CHILD2 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '30'} ...---------------------- Checking element -------- ** difference... ** -------------- CHILD1 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '16'} CHILD2 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '12'} ...---------------------- Checking element -------- ** difference... ** -------------- CHILD1 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '11'} CHILD2 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '25'} ...---------------------- Checking element -------- ** difference... ** -------------- CHILD1 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '7'} CHILD2 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '35'} ...---------------------- Checking element -------- ** difference... ** -------------- CHILD1 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '26'} CHILD2 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '13'} ...---------------------- Checking element -------- ** difference... ** -------------- CHILD1 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '36'} CHILD2 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '3'} ...---------------------- Checking element -------- ** difference... ** -------------- CHILD1 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '34'} CHILD2 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '10'} ...---------------------- Checking element -------- ** difference... ** -------------- CHILD1 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '1'} CHILD2 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '17'} ...---------------------- Checking element -------- ** difference... ** -------------- CHILD1 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '28'} CHILD2 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '26'} ...---------------------- Checking element -------- ** difference... ** -------------- CHILD1 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '0'} CHILD2 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '19'} ...---------------------- Checking element -------- ** difference... ** -------------- CHILD1 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '6'} CHILD2 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '22'} ...---------------------- Checking element -------- ** difference... ** -------------- CHILD1 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '13'} CHILD2 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '31'} ...---------------------- Checking element -------- ** difference... ** -------------- CHILD1 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '8'} CHILD2 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '21'} ...---------------------- Checking element -------- ** difference... ** -------------- CHILD1 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '30'} CHILD2 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '23'} ...---------------------- Checking element -------- ** difference... ** -------------- CHILD1 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '20'} CHILD2 {'ExposureTime': '0.0', 'ExposureTimeUnit': 's', 'TheC': '0', 'TheT': '0', 'TheZ': '32'} ...---------------------- ```

sbesson commented 2 years ago

Initial thought while reading https://github.com/ome/ZarrReader/pull/31#issuecomment-1123856430 is that the planes seem to be unordered in the XML output. This might be a feature of the exporter code which possible retrieves the plane metadata in no particular order.

The downside is that it makes the XML (at least the Plane section) not directly comparable unless the planes are sorted.

dgault commented 2 years ago

Ok, a number of updates with the new commits, namely in the areas below:

HCS data The reports of exceptions in https://github.com/ome/ZarrReader/pull/31#issuecomment-1118978361 and on issue https://github.com/ome/ZarrReader/issues/33 should be resolved. The parsing has been updated to handle 3 possible scenarios:

parsing rowIndex/columnIndex for NGFF v0.4 data
parsing column_index/row_index as per BF2raw 0.3 and 0.4
fallback to parsing the data out of the path

Original metadata The missing original metadata as reported on https://github.com/ome/ZarrReader/issues/34 should now be resolved. Note that the XML annotations were still present and correct beforehand, but now the original metadata will also be present in the UI.

Plane ordering This accounts for ordering of T and Z as mentioned in https://github.com/ome/ZarrReader/pull/31#issuecomment-1119467926, https://www.google.com/url?q=https://github.com/ome/ZarrReader/pull/31%23issuecomment-1118624516&sa=D&source=editors&ust=1652367499105103&usg=AOvVaw0yf2ZZWHAyQKCXodhL9I9C and https://github.com/ome/ZarrReader/pull/31#issuecomment-1123856430

The plane ordering issue is a bit a messy, the below are the steps that are occuring here:

original PFF is read in its original dimension order and planes are created in this order
the associated OME-XML for the format has a new dimensionOrder (note in the OME-XML the order of the planes differs from the dimension order)
The ZarrReader is initialized with the planes in the original order and the new dimensionOrder
ZarrReader attempts to update the Z, T, C indexes of the planes as per the data from the Zarr
The existing planes are in a different order which results in values being overwritten, so additional metadata fields such as exposure time are now incorrect

Although this is the most common scenario in which we would see problems, the list of planes could be in any given order and still be considered valid. This leads to a potentially messy attempt to reorder them or the hopefully simpler approach of trusting the original data. So with the last the commit the reader will now trust the original plane data to be correct and avoid any attempt to update those fields. As long as the associated OME-XML is valid for the dataset then this should be correct.

Number of resolutions The issues with number of resolutions as reported on https://github.com/ome/ZarrReader/pull/31#issuecomment-1119489986 and https://github.com/ome/ZarrReader/pull/31#issuecomment-1118624516 are due to the bioformats2raw conversion process which has created extra resolutions. So the behaviour here looks to be correct

RGB This one is probably open for discussion as it may not directly correlate to TIFF or other formats. For OME-TIFF for example we rely on samples per pixel or IFD tags for Photometric Interpretation.

Proposed 0.5 changes From Josh's comment https://github.com/ome/ZarrReader/pull/31#issuecomment-1119303509, I have only taken a quick look but this seem like a pretty major breaking change and certainly not for this PR. The metadata appears to have different values for shape and chunks. Currently we are reading the shape values via JZarr.

pwalczysko commented 2 years ago

Original metadata The missing original metadata as reported on https://github.com/ome/ZarrReader/issues/34 should now be resolved. Note that the XML annotations were still present and correct beforehand, but now the original metadata will also be present in the UI.

Retested and test passing. See user-8 https://merge-ci.openmicroscopy.org/web/webclient/?show=image-231365. Fresh reimport of the ome.zarr was necessary, but I guess this is expected. Good to go for my part of the tests.

sbesson commented 2 years ago

Also retested https://github.com/ome/ZarrReader/pull/31/commits/83843bc9b03b058defe424612a9018a0f42188b6 and confirmed that HCS OME-NGFF with plate.acquisitions without the maximumfieldcount can still import as expected.

sbesson commented 2 years ago

For the next steps, two questions:

are there any outstanding issues that needs to be fixed or minimally captured as issues ?
related, are we ready to cut a new release of the reader with the metadata changes included? If so would you call this rather a 0.1.6 or 0.2.0 @dgault ?

imagesc-bot commented 2 years ago

This pull request has been mentioned on Image.sc Forum. There might be relevant details there:

https://forum.image.sc/t/intermission-ome-ngff-0-4-1-bioformats2raw-0-5-0-et-al/72214/1

ome / ZarrReader

ZarrReader: Add support for OME-XML from bf2raw #31