bcdev / zappend

Robustly creating and updating Zarr data cubes from smaller subsets
https://bcdev.github.io/zappend/
MIT License
40 stars 1 forks source link

Write multi-level datasets #102

Closed forman closed 1 month ago

forman commented 1 month ago

Added experimental function zappend.levels.write_levels() that generates datasets using the multi-level dataset format as specified by xcube.

It resembles the store.write_data(cube, "<name>.levels", ...) method provided by the xcube filesystem data stores ("file", "s3", "memory", etc.). The zappend version may be used for potentially very large datasets in terms of dimension sizes or for datasets with very large number of chunks. It is considerably slower than the xcube version (which basically uses xarray.to_zarr() for each resolution level), but should run robustly with stable memory consumption. The function requires xcube package to be installed.

Addresses #19.

Checklist (strike out non-applicable):

codecov[bot] commented 1 month ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 99.93%. Comparing base (c5270c5) to head (b6af3ec).

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #102 +/- ## ======================================= Coverage 99.93% 99.93% ======================================= Files 30 30 Lines 1458 1458 ======================================= Hits 1457 1457 Misses 1 1 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

AliceBalfanz commented 1 month ago

I am missing the possibility to :

another case, but currently not urgent:

forman commented 1 month ago

Do you think it is worth adding the above information?

Yes!

forman commented 1 month ago

Not sure where the right place would be, maybe the changelog?

The function's docstring.