search

pangeo-forge / pangeo-forge-recipes

Python library for building Pangeo Forge recipes.
https://pangeo-forge.readthedocs.io/
Apache License 2.0
123 stars 54 forks source link

Re-implement consolidated metadata in `StoreToZarr` #568

Open cisaacstern opened 1 year ago

cisaacstern commented 1 year ago

As noted here

https://github.com/pangeo-forge/pangeo-forge-recipes/blob/561897d003231ee3c6e97427015847adaef1714e/pangeo_forge_recipes/transforms.py#L386-L388

consolidated metadata is not implemented in StoreToZarr. Here's the prior implementation:

https://github.com/pangeo-forge/pangeo-forge-recipes/blob/3b3c13c358259a9b01367c91951d28826b6c252e/pangeo_forge_recipes/recipes/xarray_zarr.py#L694-L697

Like #556, this also depends on #564, so I'll address that first.

cisaacstern commented 1 year ago

xref https://github.com/leap-stc/ClimSim/issues/38#issuecomment-1687209517, the slow loading time there can probably be (entirely?) resolved by consolidated metadata.