Some of the client exposed features of web-llm require tokenization and decoding of tokens to be used effectively. The tokenizer is already loaded for web-llm's internal functionality and can be made available to clients. When clients use the tokenizer that is already loaded, it avoids the need to load another copy and also provides immediate reliable access to a tokenizer for any model that is supported by web-llm, rather than requiring clients to manage different tokenizers for models themselves.
Some of the client exposed features of web-llm require tokenization and decoding of tokens to be used effectively. The tokenizer is already loaded for web-llm's internal functionality and can be made available to clients. When clients use the tokenizer that is already loaded, it avoids the need to load another copy and also provides immediate reliable access to a tokenizer for any model that is supported by web-llm, rather than requiring clients to manage different tokenizers for models themselves.