search

acquire-project / acquire-driver-zarr

Apache License 2.0
6 stars 5 forks source link

Blosc2 #21

Open mkitti opened 1 year ago

mkitti commented 1 year ago

I see that you are primarily using c-blosc 1. Have you considered c-blosc 2? It is being much more actively developed than c-blosc 1 and corrects some significant deficiencies. https://github.com/Blosc/c-blosc2

We'll probably need https://github.com/zarr-developers/numcodecs/issues/413 , but this is overdue.

aliddell commented 1 year ago

We had started out with C-Blosc 2 but quickly ran into exactly what you noted: zarr-python (actually numcodecs) couldn't read what we wrote, which is a non-starter. We're definitely interested in upgrading and we're following that issue.

mkitti commented 1 year ago

I just got c-blosc 1 to address a known corruption issue in ZSTD, so please use c-blosc version 1.21.4 or greater to avoid that. https://github.com/Blosc/c-blosc/issues/364

joshmoore commented 1 year ago

I'll add that in general there's also the consideration of whether or not the generated data can be accessed beyond zarr-python.

With Zarr v3, that will be covered by the individual ZEPs covering different codecs.

mkitti commented 1 year ago

A Blosc 2 PR is now pending as developed at the NumFocus Summit: https://github.com/zarr-developers/numcodecs/pull/463