google-research / arco-era5

Recipes for reproducing Analysis-Ready & Cloud Optimized (ARCO) ERA5 datasets.
https://cloud.google.com/storage/docs/public-datasets/era5
Apache License 2.0
287 stars 22 forks source link

Clarification of differences between different `ar` datasets #77

Closed robbibt closed 1 month ago

robbibt commented 2 months ago

Hi all, thanks for the amazing work on this. I'm exploring the data, and it looks like there are many different variants of files available on Google Cloud, e.g.:

fs = gcsfs.GCSFileSystem()
fs.ls('gcp-public-data-arco-era5/ar')

image

Is there documentation available explaining what each of these files represents, and advice on the relative advantages/limitations of each or guidance on when one should be used over others?

For example, I can see that this file is suggested in the README (full_37-1h-0p25deg-chunk-1.zarr-v3) - is this file recommended as a good place to start for most purposes? (I don't have a specific application in mind yet, but want to start exploring a "generically useful" dataset rather than something customised for specific purposes).

I'd also love some info on the expected stability of these files or update process in the context of using them for operational processing or referencing them in external tools - are there plans to progressively update these files with new data, or were they generated as a once-off?

dabhicusp commented 2 months ago

Hello @robbibt, thank you for using this files.

The available documentation for this 2 two files(gs://gcp-public-data-arco-era5/ar/full_37-1h-0p25deg-chunk-1.zarr-v3, gs://gcp-public-data-arco-era5/ar/model-level-1h-0p25deg.zarr-v1) is here(https://github.com/google-research/arco-era5?tab=readme-ov-file#025-pressure-and-surface-level-data,https://github.com/google-research/arco-era5?tab=readme-ov-file#025-model-level-data).

Other than above files all other files in the gcp bucket(gcp-public-data-arco-era5/ar) are deprecated and we will remove it soon. Basically all this files contains data for the different different latitude, longitude & time-range.

Please let me know If I can help you in anyways.

P.S.: Close this issue if your doubts are resolved. 😄

dabhicusp commented 1 month ago

@shoyer Can you close this issue as your PR #79 is merged?