mlc-ai / web-llm

High-performance In-browser LLM Inference Engine
https://webllm.mlc.ai
Apache License 2.0
13.84k stars 890 forks source link

Expose the tokenizer to clients #622

Open Bainainai opened 4 weeks ago

Bainainai commented 4 weeks ago

Some of the client exposed features of web-llm require tokenization and decoding of tokens to be used effectively. The tokenizer is already loaded for web-llm's internal functionality and can be made available to clients. When clients use the tokenizer that is already loaded, it avoids the need to load another copy and also provides immediate reliable access to a tokenizer for any model that is supported by web-llm, rather than requiring clients to manage different tokenizers for models themselves.