search

pangeo-data / rechunker

Disk-to-disk chunk transformation for chunked arrays.
https://rechunker.readthedocs.io/
MIT License
163 stars 25 forks source link

Allow for single partial chunk #73

Open rsignell-usgs opened 3 years ago

rsignell-usgs commented 3 years ago

When I create a large rechunked dataset, I use rechunker on each time chunk, then append. The last time chunk currently fails because rechunker thinks the chunk plan is invalid (because only a single partial chunk is being written).

Here's a full example: https://nbviewer.jupyter.org/gist/rsignell-usgs/c0b87ed1fa5fc694e665fb789e8381bb

rabernat commented 3 years ago

This is what @davidbrochart was working on in #28. That stalled because we did some internal refactoring which broke his approach. But maybe we can convince him to pick it up again? 😊

davidbrochart commented 3 years ago

I'd be happy to work on this, but didn't have time so far :disappointed:

rabernat commented 3 years ago

I wish we could argue that this is within scope for the Quantstack Zarr EOSS work! 😆

davidbrochart commented 3 years ago

Me too! But not related to C++ or Zarr v3 unfortunately. If QuantStack could be involved in another Zarr project, I'd love to work on these things.