Refactoring of how the earth2studio.models.px.auto.Package class works to lean more into Fsspec to use more of their existing caching system as well as provide better downloading feedback for users via tqdm. Also adds better ability to customize caching directly in model, namely default can not hash file names to make cache readable. I made the call to decouple this from Modulus file system, which operates a little differently, because it takes too long to upstream.
There is a slight "bug"/limitation in Fsspec which is the reason this took long to figure out related to the call back for caching. Had to extend the caching cache to fix.
Earth2Studio Pull Request
Description
Refactoring of how the
earth2studio.models.px.auto.Package
class works to lean more into Fsspec to use more of their existing caching system as well as provide better downloading feedback for users via tqdm. Also adds better ability to customize caching directly in model, namely default can not hash file names to make cache readable. I made the call to decouple this from Modulus file system, which operates a little differently, because it takes too long to upstream.There is a slight "bug"/limitation in Fsspec which is the reason this took long to figure out related to the call back for caching. Had to extend the caching cache to fix.
Closes: https://github.com/NVIDIA/earth2studio/issues/46 Closes: https://github.com/NVIDIA/earth2studio/issues/59
Checklist
Dependencies