NVIDIA / modulus-launch

Repo of optimized training recipes for accelerating PyTorch workflows of AI driven surrogates for physical systems

Apache License 2.0

54 stars 27 forks source link

ERA5 Mirror #104

Closed loliverhennigh closed 1 year ago

loliverhennigh commented 1 year ago

Modulus Pull Request

Description

Download support from CDS api for training weather models. This allows any user to get the needed data to train ForcastNet for example.

closes issue https://github.com/NVIDIA/modulus-launch/issues/103

Checklist

[*] I am familiar with the Contributing Guidelines.
[*] New or existing tests cover these changes.
[*] The documentation is up to date with these changes.
[*] The CHANGELOG.md is up to date with these changes.

loliverhennigh commented 1 year ago

/blossom-ci

loliverhennigh commented 1 year ago

/blossom-ci

loliverhennigh commented 1 year ago

/blossom-ci

loliverhennigh commented 1 year ago

/blossom-ci

loliverhennigh commented 1 year ago

Hey @dallasfoster and @NickGeneva . Pushed the fsspec change. I can give a filesystem for our object storage and upload/download the zarr arrays. There is a detail about how the permissions are set up on our storage but this should work for average users. I think this is ready for merging unless there is anything else.

loliverhennigh commented 1 year ago

/blossom-ci

loliverhennigh commented 1 year ago

/blossom-ci