search

NeurodataWithoutBorders / lindi

Linked Data Interface (LINDI) - cloud-friendly access to NWB data
BSD 3-Clause "New" or "Revised" License
1 stars 1 forks source link

Copy chunks to staging area (2) #56

Closed magland closed 2 months ago

codecov-commenter commented 2 months ago

Codecov Report

Attention: Patch coverage is 44.44444% with 25 lines in your changes are missing coverage. Please review.

Project coverage is 80.45%. Comparing base (eefdcff) to head (833b3f3).

Files Patch % Lines
lindi/LindiStagingStore/LindiStagingStore.py 11.11% 24 Missing :warning:
lindi/LindiH5pyFile/LindiH5pyFile.py 75.00% 1 Missing :warning:
Additional details and impacted files ```diff @@ Coverage Diff @@ ## local-cache #56 +/- ## =============================================== - Coverage 82.34% 80.45% -1.90% =============================================== Files 28 29 +1 Lines 2101 2133 +32 =============================================== - Hits 1730 1716 -14 - Misses 371 417 +46 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

magland commented 2 months ago

Made the following changes:

Add a new function of the staging store called copy_chunks_to_staging_area. This is important in the case where we have created a LindiH5pyFile that refers to chunks of data in local files. Before uploading to a cloud bucket, we need to first copy those chunks into the staging area so that they can be properly consolidated prior to upload.

inline small numeric datasets in the .lindi.json file (size of <1000 entries in array)

Move LindiH5ZarrStoreOpts to its own file

rly commented 2 months ago

Looks good. Thanks for the change.