Hierarchical chunk storage

ome / ngff

Next-generation file format (NGFF) specifications for storing bioimaging data in the cloud.

https://ngff.openmicroscopy.org

Other

117 stars 38 forks source link

Hierarchical chunk storage #29

Closed constantinpape closed 3 years ago

constantinpape commented 3 years ago

Currently, the ome zarr specification demands that the chunks are stored in a single directory. This causes issues on the file system for large images / volumes with many chunks.

N5 stores chunks in a hierarchy natively and zarr also supports this with the NestedDirectoryStore (although this is currently not working according to @joshmoore).

joshmoore commented 3 years ago

Here's what I've been working on:

n5 supports nested (of course) but zarr+s3 is missing: https://github.com/saalfeldlab/n5-zarr/pull/5
jzarr doesn't support nested: https://github.com/bcdev/jzarr/pull/17
NestedDirectoryStore supports zarr+nesting but not s3. The newer FSStore which does support s3 had a bug in the nesting: https://github.com/zarr-developers/zarr-python/pull/699

Conversations starting https://gitter.im/zarr-developers/community?at=601d471fc83ec358be27944f and during the 2021-02-10 community call suggest that even before the V3 spec (which defaults to nested storage) it might be possible for us to transition to using it more heavily with the V2 spec. A requirement would be a heuristic or metadata entry to determine if chunks are nested or flat. zarr-python is waiting on a PR from me.

imagesc-bot commented 3 years ago

This issue has been mentioned on Image.sc Forum. There might be relevant details there:

https://forum.image.sc/t/next-call-on-next-gen-bioimaging-data-tools-feb-23/48386/9

joshmoore commented 3 years ago

Status update:

bioformats2raw is moving towards 0.3.0. There will already be some breaking layout changes.
zarr-python has the FSStore fixe merged to the mainline. The 2.7.0 release will contain it.
The jzarr work will be taken over by @SabineEmbacher but is moving forward.

With all of this in mind, I'd propose that we go ahead and bump to OME-Zarr version to 0.2.0 stating in the spec that nested storage is a MUST. A client can then detect whether the fileset is 0.1.0 or 0.2.0 from the metadata and make a choice as to the chunk index separator.

cc: @manzt (since I've failed to engage on the JS front)

joshmoore commented 3 years ago

see: https://github.com/zarr-developers/zarr-python/pull/715

joshmoore commented 3 years ago

Feels like most implementations have now implemented this new strategy. Likely a solid next step would be to add v0.1 and v0.2 implementations to https://github.com/ome/ome_zarr_test_suite

imagesc-bot commented 3 years ago

This issue has been mentioned on Image.sc Forum. There might be relevant details there:

https://forum.image.sc/t/open-an-image-from-omero-in-imagej-js/47747/17