Closed ScottTodd closed 5 months ago
Oops, description from another PR:
As part of this, I added support for downloading models from Hugging Face (in addition to Azure Storage). The
huggingface_hub
package has its own caching strategy, which I adapted to work with our custom caching for our GitHub Actions runners.
- Hugging Face limits access to certain files based on authentication (e.g. the
HF_TOKEN
env var), which we already handle in thee2eshark/
subproject. If we add tests for gated models then we can set that up here somehow too.- Generic file downloading is still unimplemented, and as a reminder we can add other hosts like Google Cloud Storage, Amazon S3, etc. as needed.
As part of this, I added support for downloading models from Hugging Face (in addition to Azure Storage). The
huggingface_hub
package has its own caching strategy, which I adapted to work with our custom caching for our GitHub Actions runners.HF_TOKEN
env var), which we already handle in thee2eshark/
subproject. If we add tests for gated models then we can set that up here somehow too.