search

NeurodataWithoutBorders / lindi

Linked Data Interface (LINDI) - cloud-friendly access to NWB data
BSD 3-Clause "New" or "Revised" License
2 stars 1 forks source link

WIP: add single_chunk_size_threshold opt #82

Closed magland closed 2 months ago

magland commented 2 months ago

@rly

I've been running into a problem where some of the large (raw ephys) datasets on DANDI are uncompressed and stored in a single very large chunk within the NWB file. For example 000935. In fact, as you know, the default in HDF5 is to create an uncompressed contiguous chunk. This is a problem for Zarr with remote files because (as far as I can see) there is no such thing as a partial read of a chunk. So if I try to randomly access a time segment with one of these arrays, the entire chunk/dataset needs to be downloaded.

This problem could potentially be addressed at various stages of the flow, but at this point I haven't found any that are satisfactory.

As a workaround, in this PR I have added another attribute to the LindiH5ZarrStoreOpts called single_uncompressed_chunk_size_threshold which defaults to 100 MB. It detects the situation where the dataset is a single uncompressed chunk of size greater than 100 MB. In this case, it is marked as an external array link. So in this case, when the client loads the array, it falls back to loading the data using h5py reader, which is capable of slicing the data efficiently.

codecov-commenter commented 2 months ago

Codecov Report

Attention: Patch coverage is 87.50000% with 2 lines in your changes missing coverage. Please review.

Project coverage is 79.51%. Comparing base (731fcb5) to head (bc03770). Report is 9 commits behind head on main.

Files Patch % Lines
lindi/LindiH5ZarrStore/LindiH5ZarrStore.py 86.66% 2 Missing :warning:
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #82 +/- ## ========================================== + Coverage 79.43% 79.51% +0.08% ========================================== Files 30 30 Lines 2256 2275 +19 ========================================== + Hits 1792 1809 +17 - Misses 464 466 +2 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

magland commented 2 months ago

Having second thoughts about this solution, so marking as WIP.

Maybe it would be better/cleaner to detect this situation and then auto split into a large number of chunks along the first dimension, so it is represented in the RFS as multiple chunks.