Zarr fork and blockd/big chunkus issues

[aschampion]

These were likely already tracked elsewhere, but TODOs from the Zarr V3 implementation hack in January. @clbarnes you may recall if it was.

• [ ] Establish traits for KV store

  • [x] Readable
  • [x] Writeable
  • [ ] Listable

• [x] Adapt N5Reader to be Zarr reader

  • [x] Factor listable from reader

• [x] Generic impl of high level API for KV store trait

• [x] Impl KV store for filesystem

• [ ] Factor out filesystem crate from core trait crate

• [ ] N5 shim

Note future renames

• [x] N5 -> Zarr
• [x] dataset -> array
• [x] dimensions -> shape
• [x] block -> chunk
• [ ] ambiguity grid
• [x] compression -> compressor
• [ ] attributes -> attributes
• [x] DatasetAttributes -> metadata/array metadata
• [x] metadata -> store meta
• [ ] version -> bootstrap entrypoint metadata
• [x] container (file in pyn5) -> hierarchy

Tests

Tests TODO:

• create_group
  • nested under array errs
  • existing array errs
  • existing group is ok
• create_dataset
  • nested under array errs
  • existing array errs
  • existing group errs

Random code block that may be discussing locking interface

set(&self, key: &str, write_value: FnOnce(Self::SetWriter) -> Result<()>)

self.set(key, |writer| GroupMetadata::default().to_writer(w)

[aschampion]

Backlog for our existing v3 implementation

• [ ] Identify unimplemented components
  • [ ] attribute listing
• [ ] Document deviations from spec
  • Separate vs same paths for metadata
• [ ] If possible, make a version that adheres more closely to spec
• [ ] Either fix or note code locations for locking
• [ ] Assemble list of our feedback in a location we won't lose
• [ ] Find and read an existing v3 test dataset

To proceed to forking out the implementation

• [ ] Rebase on upstream n5

[Farkal]

Hey ! As we start using multidimensional data COG doesn't seems to be a great format and i would like to help the support of zarr format in the rust community. Do you think we should focus on allowing n5 to be stored as zarr ? Or should we start a new lib dedicated to zarr ? I know you don't have lot of time so i can dev alone and just ask for some reviews. I also saw you already booked the zarr crate name :wink: