Add support for zarr

[normanrz]

Zarr/N5 is becoming a popular chunked file format. I think it should be fairly easy to support it in wkconnect.

[hotzenklotz]

wkConnect should support the N5/Zarr(?) data streaming from the cloud (S3?).

We have a potential use case/clients/example available with the EMBL guys.

https://github.com/saalfeldlab/n5 https://zarr.readthedocs.io/en/stable/spec/v2.html

(I don't quite understand the relationship between N5 and Zarr. Although they are different data formats(?), they are interface compatible or something?

Related projects

• N5 (file system format spec and reference implementation in java)
• rust-n5 (implementation in rust, used in pyn5)
• pyn5 Python wrapper for Rust implementation
• zarr (similar chunked array storage library and format, supports some N5 features)
• z5 (C++ implementation of zarr and N5 with python bindings, depends on conda)
• h5py (hierarchical array storage)
• h5py_like (ABCs for APIs like h5py)

[hotzenklotz]

Comment's fron Constantine Pape:

- On the python side the best library for this would be zarr, which also supports N5 access. Note that S3 access to N5 currently has some issues, see https://github.com/zarr-developers/zarr-python/issues/540. I hope that this will be fixed soon, I think there is a PR pending.
- On the java side you can use the "native" implementation from the saalfeldlab: https://github.com/saalfeldlab/n5-aws-s3.