dleemiller / WordLlama

Things you can do with the token embeddings of an LLM
MIT License
1.39k stars 47 forks source link

Use custom cache dir for tokenizer download, too #41

Closed erickpeirson closed 2 weeks ago

erickpeirson commented 2 weeks ago

Presently, passing cache_dir: Path to WordLlama.load() has no impact on the cache directory where tokenizer assets are stored. This makes it impossible to use WordLlama in an environment where the default cache path (the user's home directory) is not writable, which is often the case in production scenarios.

This PR does two things:

Note that this will effectively bust the cache on upgrade. But I'm hoping that's a small price to pay for the fix.

dleemiller commented 2 weeks ago

Nice - definitely a necessary change for deploying to places like lambda functions. Thanks!

dleemiller commented 2 weeks ago

https://github.com/dleemiller/WordLlama/pull/42

I have decided to clean everything up and simplify the API by removing the weights_dir as well. That feels legacy and over-complicated to me now to have both keyword arguments.