zarr-developers / zarr-specs

Zarr core protocol for storage and retrieval of N-dimensional typed arrays
https://zarr-specs.readthedocs.io/
Creative Commons Attribution 4.0 International
88 stars 28 forks source link

Tracking intermediate buffer sizes (in ZEP 2 or otherwise) #278

Open jakirkham opened 1 year ago

jakirkham commented 1 year ago

Currently if a pipeline of codecs is used, it would be useful to track the intermediate sizes of the buffers as they are transformed so that enough scratch space can be allocated to decode data efficiently

jbms commented 1 year ago

When it can be determined independent of the data, just based on e.g. shapes, implementations can already (and do already) do this. Some compression codecs also store the decoded size. In general, though, it might be necessary to store an additional size field along with the encoded data. Is that what you are proposing?

jakirkham commented 1 year ago

Indeed for the case of a single codec, this is unneeded

Think it comes up in cases when codecs are chained. For example using run-length encoding or bit packing followed by another compressor. It would be helpful to know what size the buffer between those steps is for decoding so as to allocate for it appropriately

Yeah this is what I'm thinking

It's possible the exact size of any intermediate buffers is not strictly necessary, but instead some largest intermediate buffer seen may be good enough. In this way it might be similar to Python's __length_hint__