How do I use Huggingface tokenization to use a model on Huggingace in MODEL_PATH instead of my local machine?

rustformers / llm

[Unmaintained, see README] An ecosystem of Rust libraries for working with large language models

https://docs.rs/llm/latest/llm/

Apache License 2.0

6.07k stars 355 forks source link

How do I use Huggingface tokenization to use a model on Huggingace in MODEL_PATH instead of my local machine? #427

Open balqaasem opened 1 year ago

LLukas22 commented 1 year ago

I don't think i understand your question completely, but if you want to use an external tokenizer you can simply provide the path or model-name to the tokenizer you want to use. The tokenizers library will resolve it and if necessary download the tokenizer's config file.