Closed bendichter closed 6 months ago
Thanks @bendichter
This works well when there is a template-able sequence of offsets (e.g. regular intervals), and it doesn't save as much space in the usual case where the offsets and sizes vary without a pattern across all the chunks.
Oh, is it often the case that chunks are not along regular intervals? Is this because compression reduced each chunk by a different amount?
I suppose this would work well if data is chunked but not compressed, but that might not come up that often
Oh, is it often the case that chunks are not along regular intervals? Is this because compression reduced each chunk by a different amount?
Yes, that's right.
I see from https://github.com/NeurodataWithoutBorders/lindi/pull/51 that we now support using templates for URLs that appear many times.
The RFS template language v1 also supports using an iterator to generate dataset regions. This could be very helpful in minimizing JSON files that map to a large number of chunks.