google-research / arco-era5

Recipes for reproducing Analysis-Ready & Cloud Optimized (ARCO) ERA5 datasets.
https://cloud.google.com/storage/docs/public-datasets/era5
Apache License 2.0
305 stars 23 forks source link

Reduced Chunking size for model-levels-to-zarr. #29

Closed dabhicusp closed 1 year ago

dabhicusp commented 1 year ago

In response to Issue #28, where it was identified that the chunk size in the model-level Zarr data was excessively large, I have made significant improvements. The existing chunk size was in the order of gigabytes. To address this, I introduced a modification in the target_chunks method of the XarrayZarrRecipe.

By incorporating the hybrid: 1 parameter in the target_chunks method, I was able to split each chunk per level.