`zarr export` doesn't support big images

[joshmoore]

(omero) /opt/data $omero zarr export Image:9798462
Using session for public@idr.openmicroscopy.org:4064. Idle timeout: 10 min. Current group: Public
Exporting to 9798462.zarr (0.4)
WARNING:omero.gateway:UnknownLocalException on <class 'omero.gateway.OmeroGatewaySafeCallWrapper'> to <66327e22-3b85-4676-96a2-5d68a03e256comero.api.RawPixelsStore> getPlane((0, 0, 0), {})
Traceback (most recent call last):
  File "/opt/omero-py/target/omero/gateway/__init__.py", line 4853, in __call__
    return self.f(*args, **kwargs)
  File "/opt/omero-py/target/omero_api_RawPixelsStore_ice.py", line 1199, in getPlane
    return _M_omero.api.RawPixelsStore._op_getPlane.invoke(self, ((z, c, t), _ctx))
Ice.UnknownLocalException: exception ::Ice::UnknownLocalException
{
    unknown = ConnectionI.cpp:1573: Ice::MemoryLimitException:
protocol error: memory limit exceeded:
requested 346982825 bytes, maximum allowed is 256000000 bytes (see Ice.MessageSizeMax)
}
ERROR:omero.gateway:Failed to getPlane() or getTile() from rawPixelsStore
Traceback (most recent call last):
  File "/opt/omero-py/target/omero/gateway/__init__.py", line 7526, in getTiles
    rawPlane = rawPixelsStore.getPlane(z, c, t)
  File "/opt/omero-py/target/omero/gateway/__init__.py", line 4856, in __call__
    return self.handle_exception(e, *args, **kwargs)
  File "/opt/omero-py/target/omero/gateway/__init__.py", line 4853, in __call__
    return self.f(*args, **kwargs)
  File "/opt/omero-py/target/omero_api_RawPixelsStore_ice.py", line 1199, in getPlane
    return _M_omero.api.RawPixelsStore._op_getPlane.invoke(self, ((z, c, t), _ctx))
Ice.UnknownLocalException: exception ::Ice::UnknownLocalException
{
    unknown = ConnectionI.cpp:1573: Ice::MemoryLimitException:
protocol error: memory limit exceeded:
requested 346982825 bytes, maximum allowed is 256000000 bytes (see Ice.MessageSizeMax)
}
Traceback (most recent call last):
  File "/usr/local/anaconda3/envs/omero/bin/omero", line 11, in <module>
    load_entry_point('omero-py', 'console_scripts', 'omero')()
  File "/opt/omero-py/target/omero/main.py", line 125, in main
    rv = omero.cli.argv()
  File "/opt/omero-py/target/omero/cli.py", line 1784, in argv
    cli.invoke(args[1:])
  File "/opt/omero-py/target/omero/cli.py", line 1222, in invoke
    stop = self.onecmd(line, previous_args)
  File "/opt/omero-py/target/omero/cli.py", line 1299, in onecmd
    self.execute(line, previous_args)
  File "/opt/omero-py/target/omero/cli.py", line 1381, in execute
    args.func(args)
  File "/usr/local/anaconda3/envs/omero/lib/python3.7/site-packages/omero_zarr/cli.py", line 65, in _wrapper
    return func(self, *args, **kwargs)
  File "/usr/local/anaconda3/envs/omero/lib/python3.7/site-packages/omero_zarr/cli.py", line 265, in export
    image_to_zarr(image, args)
  File "/usr/local/anaconda3/envs/omero/lib/python3.7/site-packages/omero_zarr/raw_pixels.py", line 32, in image_to_zarr
    add_image(image, root, cache_dir=cache_dir)
  File "/usr/local/anaconda3/envs/omero/lib/python3.7/site-packages/omero_zarr/raw_pixels.py", line 95, in add_image
    cache_file_name_func=get_cache_filename,
  File "/usr/local/anaconda3/envs/omero/lib/python3.7/site-packages/omero_zarr/raw_pixels.py", line 155, in add_raw_image
    plane = next(planes)
  File "/usr/local/anaconda3/envs/omero/lib/python3.7/site-packages/omero_zarr/raw_pixels.py", line 74, in planeGen
    yield from planes
  File "/opt/omero-py/target/omero/gateway/__init__.py", line 7559, in getTiles
    raise exc
  File "/opt/omero-py/target/omero/gateway/__init__.py", line 7526, in getTiles
    rawPlane = rawPixelsStore.getPlane(z, c, t)
  File "/opt/omero-py/target/omero/gateway/__init__.py", line 4856, in __call__
    return self.handle_exception(e, *args, **kwargs)
  File "/opt/omero-py/target/omero/gateway/__init__.py", line 4853, in __call__
    return self.f(*args, **kwargs)
  File "/opt/omero-py/target/omero_api_RawPixelsStore_ice.py", line 1199, in getPlane
    return _M_omero.api.RawPixelsStore._op_getPlane.invoke(self, ((z, c, t), _ctx))
Ice.UnknownLocalException: exception ::Ice::UnknownLocalException
{
    unknown = ConnectionI.cpp:1573: Ice::MemoryLimitException:
protocol error: memory limit exceeded:
requested 346982825 bytes, maximum allowed is 256000000 bytes (see Ice.MessageSizeMax)
}

[joshmoore]

@will-moore : would any of the dask writer work help here?

[will-moore]

Possibly. I have an idea that once ome-zarr-py has good dask support, we could just hand it a virtual omero 5D stack (as we do with napari-omero at https://github.com/tlambert03/napari-omero/blob/cd71bb834fa974214ea9eca3efb8fccc23079afc/src/napari_omero/plugins/loaders.py#L114. Taking it to the next level, for Big images, the planes could be dask-concatenated tiles. But I'm not sure my dask skills are there yet!

[imagesc-bot]

This issue has been mentioned on Image.sc Forum. There might be relevant details there:

https://forum.image.sc/t/converting-other-idr-images-into-public-zarr/44025/19