search

ApolloResearch / rib

Library for methods related to the Local Interaction Basis (LIB)
MIT License
2 stars 0 forks source link

Cache transformers datasets #320

Open danbraunai-apollo opened 5 months ago

danbraunai-apollo commented 5 months ago

As mentioned in https://github.com/ApolloResearch/rib/pull/316#issuecomment-1919287670:

Caching downloaded data (and models?) in some folder on the ssd drive. Best case we can set transformers to offline mode and shave a second off of imports (nice for fast tests!)

nix-apollo commented 5 months ago

We had transformers in offline mode at redwood. It was reasonably nice. Every once and a while you needed to get fresh data and figure out how to set it back to online mode. This was confusing to new people.

https://huggingface.co/docs/transformers/v4.15.0/installation#caching-models