Import of zarr is slower

[pwalczysko]

Comparing the import times of ome.zarr file with its original counterpart (an ome.tiff file).

multifile-Z1.ome.tiff from public ome-tiff samples. Both samples user-8, see https://latest-ci.openmicroscopy.org/web/webclient/?show=dataset-31869

ome.tiff:

(.venv3) bash-4.2$ omero fs importtime Fileset:32009
Using session for user-8@localhost:4064. Idle timeout: 10 min. Current group: private-1
   upload time of   0.79s for 5 files (0.158s/file)
    setId time of   0.35s
 metadata time of   0.12s
   pixels time of   0.31s for 5 planes (0.062s/plane)
    rdefs time of   0.12s for 1 rendering setting (0.118s/rdef)
thumbnail time of   0.14s for 1 thumbnail (0.141s/thumbnail)

ome.zarr omero fs importtime (Fileset:32008)

   upload time of   2.35s for 11 files (0.214s/file)
    setId time of   1.22s
 metadata time of   0.84s
   pixels time of   1.50s for 5 planes (0.300s/plane)
    rdefs time of   1.09s for 1 rendering setting (1.088s/rdef)
thumbnail time of   0.32s for 1 thumbnail (0.324s/thumbnail)

/uod/idr/repos/curated/ome-tiff/public/2016-06/sub-resolutions/Fluorescence/LuCa-7color_Scan1.ome.tiff, user-8, https://merge-ci.openmicroscopy.org/web/webclient/?show=image-231325:

omero fs importtime Fileset:124060
Using session for user-8@localhost:4064. Idle timeout: 10 min. Current group: private-1
   upload time of   0.37s for 1 file (0.369s/file)
    setId time of   0.47s
 metadata time of   0.28s
   pixels time of 171.62s for 14 planes (12.259s/plane)
    rdefs time of   0.40s for 4 rendering settings (0.100s/rdef)
thumbnail time of  14.91s for 4 thumbnails (3.728s/thumbnail)

LuCa-7color_Scan1.ome.zarr, user-8, https://merge-ci.openmicroscopy.org/web/webclient/?show=image-231328

omero fs importtime Fileset:124061
   upload time of 197.97s for 5927 files (0.033s/file)
    setId time of   5.05s
 metadata time of   0.17s
   pixels time of   1.25s for 14 planes (0.090s/plane)
    rdefs time of   3.17s for 4 rendering settings (0.792s/rdef)
thumbnail time of   0.41s for 4 thumbnails (0.102s/thumbnail)

/uod/idr/repos/curated/ome-tiff/public/2016-06/tubhiswt-4D/tubhiswt_C1_TP41.ome.tif user-8, https://latest-ci.openmicroscopy.org/web/webclient/?show=image-56469

omero fs importtime Fileset:32011
Using session for user-8@localhost:4064. Idle timeout: 10 min. Current group: private-1
   upload time of   2.53s for 86 files (0.029s/file)
    setId time of   1.36s
 metadata time of   0.21s
   pixels time of  21.91s for 860 planes (0.025s/plane)
    rdefs time of   0.85s for 1 rendering setting (0.847s/rdef)
thumbnail time of   0.46s for 1 thumbnail (0.463s/thumbnail)

tubhiswt_C0_TP0.ome.zarr user-8, https://latest-ci.openmicroscopy.org/web/webclient/?show=image-56468

omero fs importtime Fileset:32010
Using session for user-8@localhost:4064. Idle timeout: 10 min. Current group: private-1
   upload time of 144.26s for 1727 files (0.084s/file)
    setId time of   7.45s
 metadata time of   1.22s
   pixels time of  71.58s for 860 planes (0.083s/plane)
    rdefs time of   5.45s for 1 rendering setting (5.453s/rdef)
thumbnail time of   0.29s for 1 thumbnail (0.295s/thumbnail)

[joshmoore]

I wonder if https://github.com/ome/ZarrReader/pull/20 would have a significant impact. OMERO would likely need to be updated to make use of it though.