search

NeurodataWithoutBorders / lindi

Linked Data Interface (LINDI) - cloud-friendly access to NWB data
BSD 3-Clause "New" or "Revised" License
2 stars 1 forks source link

expand use of templates #65

Closed bendichter closed 4 months ago

bendichter commented 4 months ago

I see from https://github.com/NeurodataWithoutBorders/lindi/pull/51 that we now support using templates for URLs that appear many times.

The RFS template language v1 also supports using an iterator to generate dataset regions. This could be very helpful in minimizing JSON files that map to a large number of chunks.

magland commented 4 months ago

Thanks @bendichter

This works well when there is a template-able sequence of offsets (e.g. regular intervals), and it doesn't save as much space in the usual case where the offsets and sizes vary without a pattern across all the chunks.

bendichter commented 4 months ago

Oh, is it often the case that chunks are not along regular intervals? Is this because compression reduced each chunk by a different amount?

bendichter commented 4 months ago

I suppose this would work well if data is chunked but not compressed, but that might not come up that often

magland commented 4 months ago

Oh, is it often the case that chunks are not along regular intervals? Is this because compression reduced each chunk by a different amount?

Yes, that's right.