zarr-developers / zarr-specs

Zarr core protocol for storage and retrieval of N-dimensional typed arrays
https://zarr-specs.readthedocs.io/
Creative Commons Attribution 4.0 International
87 stars 28 forks source link

Byte ranges need to permit reading from the end #266

Open LDeakin opened 1 year ago

LDeakin commented 1 year ago

The specification mentions that the decoded_regions parameter in c.partial_decode() specifies a list of byte ranges. With the sharding codec, to read the shard index without reading the whole file it must be possible to read some number of bytes from the end of the input_handle.

In the Abstract Store Interface section, a byte range is specified as

To read and write partial values, a range specifies two integers range_start and range_length, that specify a part of the value starting at byte range_start (inclusive) and having a length of range_length bytes. range_length may be none, indicating all available data until the end of the referenced value.

I think that paragraph should be updated to reflect that a byte range can be

Additionally, I think the details of how a byte range should be implemented (e.g. enum or negative offset/size) should be left up to the implementation.