NVIDIA / earth2studio

Open-source deep-learning framework for exploring, building and deploying AI weather/climate workflows.
https://nvidia.github.io/earth2studio/
Apache License 2.0
73 stars 23 forks source link

Model Package Update to improve Model file caching #67

Closed NickGeneva closed 2 months ago

NickGeneva commented 2 months ago

Earth2Studio Pull Request

Description

Refactoring of how the earth2studio.models.px.auto.Package class works to lean more into Fsspec to use more of their existing caching system as well as provide better downloading feedback for users via tqdm. Also adds better ability to customize caching directly in model, namely default can not hash file names to make cache readable. I made the call to decouple this from Modulus file system, which operates a little differently, because it takes too long to upstream.

There is a slight "bug"/limitation in Fsspec which is the reason this took long to figure out related to the call back for caching. Had to extend the caching cache to fix.

Closes: https://github.com/NVIDIA/earth2studio/issues/46 Closes: https://github.com/NVIDIA/earth2studio/issues/59

Checklist

Dependencies

NickGeneva commented 2 months ago

/blossom-ci

NickGeneva commented 2 months ago

/blossom-ci

NickGeneva commented 2 months ago

/blossom-ci

NickGeneva commented 2 months ago

/blossom-ci

NickGeneva commented 2 months ago

/blossom-ci

NickGeneva commented 2 months ago

/blossom-ci

NickGeneva commented 2 months ago

/blossom-ci

NickGeneva commented 2 months ago

/blossom-ci