Closed sbesson closed 3 years ago
Initial set of sample files generated with and without --bf and uploaded to a temporary public bucket for comparison
From the performance perspective
(zarr) [sbesson@pilot-zarr1-dev data]$ time omero zarr --output /data/omero-cli-zarr_76/omero/ export Image:8343617
Previous session expired for public on idr.openmicroscopy.org:4064
Server: [idr.openmicroscopy.org:4064]
Username: [public]
Password:
Created session for public@idr.openmicroscopy.org:4064. Idle timeout: 10 min. Current group: Public
Exporting to /data/omero-cli-zarr_76/omero/8343617.zarr (0.3)
Finished.
real 58m52.531s
user 42m19.562s
sys 5m45.421s
(zarr) [sbesson@pilot-zarr1-dev data]$ time omero zarr --output /data/omero-cli-zarr_76/bf/ export --bf Image:8343617
Previous session expired for public on idr.openmicroscopy.org:4064
Server: [idr.openmicroscopy.org:4064]
Username: [public]
Password:
Created session for public@idr.openmicroscopy.org:4064. Idle timeout: 10 min. Current group: Public
OpenJDK 64-Bit Server VM warning: You have loaded library /tmp/opencv_openpnp5020382743495610090/nu/pattern/opencv/linux/x86_64/libopencv_java342.so which might have disabled stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.
Image exported to /data/omero-cli-zarr_76/bf/8343617.zarr
real 29m38.845s
user 55m35.191s
sys 2m17.729s
This worked once I'd updated to the 0.3.0
release of bioformats2raw.
One difference I noticed was in the number of pyramid levels exported is different. For a 512 x 512 image, bf exported 2 levels whereas omero exported 4 levels, down to 64 x 64.
Known difference: --bf
exports v0.2 whereas omero export uses v0.3 (with axes)
This worked once I'd updated to the 0.3.0 release of bioformats2raw.
Thanks. 0.3.0
is absolutely a requirement for this work as this PR uses some of the new options. I'll make this clear in the help.
One difference I noticed was in the number of pyramid levels exported is different. For a 512 x 512 image, bf exported 2 levels whereas omero exported 4 levels, down to 64 x 64.
Yes that's one of the implementation differences which comes from different defaults in the maximal size for the smallest resolution: 96 for omero-cli-zarr https://github.com/ome/omero-cli-zarr/blob/aab16a60444804076225d283804df4fceace16c6/src/omero_zarr/raw_pixels.py#L88 vs 256 fo bioformats2raw - https://github.com/glencoesoftware/bioformats2raw/blob/4114f1ef8340317df67d8940151f3b7a0159a5a3/src/main/java/com/glencoesoftware/bioformats2raw/Converter.java#L105.
--bf exports v0.2 whereas omero export uses v0.3 (with axes)
Yes that's captured as https://github.com/glencoesoftware/bioformats2raw/issues/113
Another large scale example of usage of this PR. Trying to convert the >1TB lightsheet dataset from McDole et al (https://doi.org/10.17867/10000116):
(zarr) [sbesson@pilot-zarr1-dev ~]$ time omero zarr --output /data/omero-cli-zarr_76/bf/ export --bf Image:4007801 --max_workers 16
Using session for public@idr.openmicroscopy.org:4064. Idle timeout: 10 min. Current group: Public
OpenJDK 64-Bit Server VM warning: You have loaded library /tmp/opencv_openpnp6140859124054326326/nu/pattern/opencv/linux/x86_64/libopencv_java342.so which might have disabled stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.
Image exported to /data/omero-cli-zarr_76/bf/4007801.zarr
real 1934m27.700s
user 22426m47.134s
sys 452m8.650s
(zarr) [sbesson@pilot-zarr1-dev ~]$ time aws --endpoint-url https://uk1s3.embassy.ebi.ac.uk s3 cp --recursive /data/omero-cli-zarr_76/bf/4007801.zarr/ s3://omero-cli-zarr_76/bf/4007801.zarr/
...
real 1510m25.233s
user 666m44.302s
So after 2.5 days of processing + S3 upload, the data can be viewed from
https://hms-dbmi.github.io/vizarr/?source=https://uk1s3.embassy.ebi.ac.uk/omero-cli-zarr_76/bf/4007801.zarr https://hms-dbmi.github.io/vizarr/?source=https://uk1s3.embassy.ebi.ac.uk/omero-cli-zarr_76/bf/4007802.zarr https://hms-dbmi.github.io/vizarr/?source=https://uk1s3.embassy.ebi.ac.uk/omero-cli-zarr_76/bf/4007803.zarr https://hms-dbmi.github.io/vizarr/?source=https://uk1s3.embassy.ebi.ac.uk/omero-cli-zarr_76/bf/4007804.zarr
From the outstanding points of this discussion, the command-line overhaul has been turned into an issue. And https://github.com/glencoesoftware/bioformats2raw/pull/114 should allow to align the number of resolutions generated with/without --bf
in the near future.
Any objections to getting this merged @joshmoore @will-moore ? I would propose a release of the plugin with bioformat2raw 0.3.0 support and start capturing the next items to review as issues.
SGTM :+1:
Follow-up of #75, this PR:
omero export Image:<id>
. The image series is passed to bioformats2raw and the unique image group (<fileset>/0
) is renamed as<id>.zarr
raw_pixels
The
add_omero_metadata
andadd_toplevel_metadata
are arguably outside the scope of theraw_pixels
module and could be moved elsewhere (utils
? new module?)