Closed will-moore closed 1 year ago
@joshmoore Updated as I think you suggested? Removed code duplication so we use tile sizes from rfs.getTileSize()
for ALL images now, even if not big images.
Maybe not quite as fast for non-big images as previously, but I think speed is not so critical for export, and it certainly means less code to support 1 way of exporting.
Also, this adds support for the --cache_numpy
behaviour for ALL images (including big images) now.
Looking to test this PR on the export of https://idr.openmicroscopy.org/webclient/?show=image-9846152, the preferred tile-size from that Image is tile_size_x: 19120, tile_size_y: 27
.
This seems like a sub-optimal tile size for viewing an OME-NGFF image, so I wonder if instead of using the preferred tile-size from rawPixelsStore, we simply choose a good default, e.g. 512 x 512 (which is what iviewer uses for all Images).
This might mean slightly slower export, which is OK, but then you have better NGFF exported.
I'll go ahead and make that change, unless there's any objection?
I'll go ahead and make that change, unless there's any objection?
I think the tile size given by the tile service is retrieved from the relevant Bio-Formats reader and effectively reflects how the data is stored on disk in its original file formats.
For comparison, bioformats2raw
completely ignores this value when converting the data and sets a chunk size of 1024x1024. This is largely in agreement with the proposal you are making here.
Since we already support --tile_width
and --tile_height
for bioformats2raw, I've used them for cli export too.
Running current version against https://idr.openmicroscopy.org/webclient/?show=image-9846152
Loading all tiles for resolution 0
completed after many re-runs, losing connection after ~12 Z planes each time.
Then downsampling failed at this point, during downsample from 1 -> 2:
$ ls -alh ./9846152.zarr
total 4.0K
drwxrwxr-x. 5 wmoore wmoore 68 Feb 1 20:18 .
drwxrwxr-x. 4 wmoore wmoore 53 Feb 1 06:04 ..
drwxrwxr-x. 5 wmoore wmoore 68 Feb 1 12:03 0
drwxrwxr-x. 5 wmoore wmoore 68 Feb 1 20:10 1
drwxrwxr-x. 5 wmoore wmoore 68 Feb 1 20:19 2
-rw-rw-r--. 1 wmoore wmoore 24 Feb 1 06:04 .zgroup
$ find ./9846152.zarr/0 -type f | wc -l
72619
$ find ./9846152.zarr/1 -type f | wc -l
19111
$ find ./9846152.zarr/2 -type f | wc -l
This pull request has been mentioned on Image.sc Forum. There might be relevant details there:
https://forum.image.sc/t/converting-other-idr-images-into-public-zarr/44025/20
The resize error above was fixed by https://github.com/ome/ome-zarr-py/pull/244/commits/e1fdd124fd3f518c5681e1b5a1aaf420e2e7800d
The branch at this point was used to generate the image at https://ome.github.io/ome-ngff-validator/?source=https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.4/idr0048A/9846152.zarr/
Also updated description with all changes in this PR.
The pre-commit.ci
check is failing with:
RuntimeError: The Poetry configuration is invalid:
- [extras.pipfile_deprecated_finder.2] 'pip-shims<=0.3.4' does not match '^[a-zA-Z-_.0-9]+$'
Same error at https://github.com/ome/ome-zarr-py/pull/244
https://pypi.org/project/poetry/ last release was 10th Jan 2023, but that doesn't appear to be the issue since this check has passed after that date above.
I don't see any mention of Poetry itself in this repo's config.
Googled for https://stackoverflow.com/questions/75269700/pre-commit-fails-to-install-isort-5-11-4-with-error-runtimeerror-the-poetry-co which suggested the fix in previous commit.
Re logging, I tried using logging but couldn't work out how to actually see the logging statements.
If I do:
import logging
LOGGER = logging.getLogger("omero_zarr.raw_pixels")
...
LOGGER.warn("t, c, z, chk_x, chk_y %s %s %s %s %s" % (t, c, z, chk_y, chk_x))
I see this e.g:
WARNING:omero_zarr.raw_pixels:t, c, z, chk_x, chk_y 0 2 0 0 0
which is not ideal (I really just want t, c, z, chk_x, chk_y 0 2 0 0 0
).
If I use LOGGER.info()
I see nothing at-all.
That comes down to the config. The idea is that someone else can also influence how it appears. If you are doing this in your own code, then use logging.basicConfig and you set the format and the log-level. Just make sure that that isn't in the released code so that the CLI, e.g., can set its preferences.
Testing the export of a Plate with this PR, I see an issue with the logic for testing whether a chunk has been previously written:
chunk_keys = list(zarray.chunk_store.keys())
separator = zarray.chunk_store.key_separator
chunk_key = separator.join([str(k) for k in key_dims])
print("chunk_key", chunk_key)
print("keys", chunk_keys)
if chunk_key not in chunk_keys:
... load from omero...
This is failing to detect existing chunks and prints out...
chunk_key 0/4/0/0
keys ['.zgroup', '1/.zarray', '1/0/0/0', '1/1/0/0', '1/2/0/0', '1/3/0/0', '1/4/0/0', 'L/.zgroup', 'L/9/.zattrs', 'L/9/.zgroup', 'L/9/0/.zattrs', 'L/9/0/.zgroup', 'L/9/0/0/.zarray', 'L/9/0/0/0/0/0', 'L/9/0/0/1/0/0', 'L/9/0/0/2/0/0', 'L/9/0/0/3/0/0', 'L/9/0/0/4/0/0', 'L/9/0/1/.zarray', 'L/9/0/1/0.0.0', 'L/9/0/1/1.0.0', 'L/9/0/1/2.0.0', 'L/9/0/1/3.0.0', 'L/9/0/1/4.0.0', 'L/9/0/2/.zarray', 'L/9/0/2/0.0.0', 'L/9/0/2/1.0.0', 'L/9/0/2/2.0.0', 'L/9/0/2/3.0.0', 'L/9/0/2/4.0.0', 'L/9/0/3/.zarray', 'L/9/0/3/0.0.0', 'L/9/0/3/1.0.0', 'L/9/0/3/2.0.0', 'L/9/0/3/3.0.0', 'L/9/0/3/4.0.0', 'L/9/1/.zattrs', 'L/9/1/.zgroup', 'L/9/1/0/.zarray', 'L/9/1/0/0/0/0', 'L/9/1/0/1/0/0', 'L/9/1/0/2/0/0', 'L/9/1/0/3/0/0', 'L/9/1/0/4/0/0', 'L/9/1/1/.zarray', 'L/9/1/1/0.0.0', 'L/9/1/1/1.0.0', 'L/9/1/1/2.0.0', 'L/9/1/1/3.0.0', 'L/9/1/1/4.0.0', 'L/9/1/2/.zarray', 'L/9/1/2/0.0.0', 'L/9/1/2/1.0.0', 'L/9/1/2/2.0.0', 'L/9/1/2/3.0.0', 'L/9/1/2/4.0.0', 'L/9/1/3/.zarray', 'L/9/1/3/0.0.0', 'L/9/1/3/1.0.0', 'L/9/1/3/2.0.0', 'L/9/1/3/3.0.0', 'L/9/1/3/4.0.0', 'L/9/2/.zattrs', 'L/9/2/.zgroup', 'L/9/2/0/.zarray', 'L/9/2/0/0/0/0', 'L/9/2/0/1/0/0', 'L/9/2/0/2/0/0', 'L/9/2/0/3/0/0', 'L/9/2/0/4/0/0', 'L/9/2/1/.zarray', 'L/9/2/1/0.0.0', 'L/9/2/1/1.0.0', 'L/9/2/1/2.0.0', 'L/9/2/1/3.0.0', 'L/9/2/1/4.0.0', 'L/9/2/2/.zarray', 'L/9/2/2/0.0.0', 'L/9/2/2/1.0.0', 'L/9/2/2/2.0.0', 'L/9/2/2/3.0.0', 'L/9/2/2/4.0.0', 'L/9/2/3/.zarray', 'L/9/2/3/0.0.0', 'L/9/2/3/1.0.0', 'L/9/2/3/2.0.0', 'L/9/2/3/3.0.0', 'L/9/2/3/4.0.0', 'L/9/3/.zattrs', 'L/9/3/.zgroup', 'L/9/3/0/.zarray', 'L/9/3/0/0/0/0', 'L/9/3/0/1/0/0', 'L/9/3/0/2/0/0', 'L/9/3/0/3/0/0', 'L/9/3/0/4/0/0', 'L/9/3/1/.zarray', 'L/9/3/1/0.0.0', 'L/9/3/1/1.0.0', 'L/9/3/1/2.0.0', 'L/9/3/1/3.0.0', 'L/9/3/1/4.0.0', 'L/9/3/2/.zarray', 'L/9/3/2/0.0.0', 'L/9/3/2/1.0.0', 'L/9/3/2/2.0.0', 'L/9/3/2/3.0.0', 'L/9/3/2/4.0.0', 'L/9/3/3/.zarray', 'L/9/3/3/0.0.0', 'L/9/3/3/1.0.0', 'L/9/3/3/2.0.0', 'L/9/3/3/3.0.0', 'L/9/3/3/4.0.0', 'L/9/4/.zgroup', 'L/9/4/0/.zarray']
It looks like a different path separator is being used for the path to the Image (/
) and for the Image dims (.
). Trying to match this behaviour - and have it behave differently for Images vv Plate - seems too fragile.
So I think I'll revert to my previous approach (changed in https://github.com/ome/omero-cli-zarr/pull/134/commits/6af4061fc87cde2df9f3b51581afd949a37f4f06) to check for existing chunks
:+1: Merging.
Anything else to roll into an 0.6.0, @will-moore? @dominikl?
Fixes #111
This adds support for exporting big images in a tile-based approach instead of a plane-based approach. As suggested by @joshmoore - this tile-based approach is used for ALL images (instead of using the whole plane for as a single chunk as before).
First we write a full-size array at
0
, a tile at a time. Then resize viaome_zarr.dask_utils.resize
(see bug-fix at https://github.com/ome/ome-zarr-py/pull/244) to create a pyramid.Other changes:
The
--numpy_cache
functionality has been replaced. Previously, if you anticipated connection issues with OMERO, you could use the--numpy_cache
option to cache chunks to disk at the same time as writing them to zarr. The problem with this was that if you lost connection and weren't expecting it, then you had nothing cached. Also, caching doubled the disk usage.Now, if connection fails, you can simply rerun the
omero zarr export Image:ID
command and we pick-up writing to the existing partially-generated array.This is nicer as you don't have to delete the partially-exported image as before and it's much faster as you do don't have to copy the cached chunks and re-write to zarr. Exporting the big image below required many re-runs of the export command - maybe 20 - 30 times before completion!
The
tile_width
andtile_height
options that previously only applied tobioformats2raw
usage, now also apply to API-based export.Image exported with this PR is at: https://ome.github.io/ome-ngff-validator/?source=https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.4/idr0048A/9846152.zarr/
I also fixed the Plate export to work with the new export logic and to support the interruption (partial export) of a Plate and the continued export from where you left off.