search

arttumiettinen / pi2

C++ library and command-line software for processing and analysis of terabyte-scale volume images locally or on a computing cluster.
GNU General Public License v3.0
48 stars 13 forks source link

OME-Zarr support #14

Open normanrz opened 10 months ago

normanrz commented 10 months ago

Hi!

I am looking for ways to make the interaction of NRStitcher easier with Webknossos because converting the input and output data has become quite a time sink. I was wondering if you were planning to add support for OME-Zarr. It is becoming a community standard for large images [1] and Webknossos already has good support for it.

I see that you have NN5 support. Is that compatible with this N5? If yes, the Zarr implementation should be straightforward as it is a similar chunked file format.

There is a C++ implementation for Zarr as well: https://google.github.io/tensorstore/

Happy to contribute a PR, if you're interested.

[1] https://link.springer.com/article/10.1007/s00418-023-02209-1

arttumiettinen commented 10 months ago

Hi,

Sure this is interesting, we should make file format conversions unnecessary as much as possible.

The NN5 format (Not N5) is a bit similar to N5, but not compatible. It is designed to support reading and writing of compressed data from multiple compute nodes in parallel, and removes some restrictions like 2 GB limit on chunk size, introduces differences in chunk file format etc. It should work well in a cluster environment, but it is still kind of work in progress and not well documented. Performance is not guaranteed.

I'm open to adding support for new file formats, but I must admit that such large changes might take significant time if I'm doing them myself. I would be happy to accept contributions, though!

Best, Arttu

normanrz commented 10 months ago

That sounds great. We are currently looking for funding that we could use to develop Zarr support in NRStitcher. I'll keep you updated.