search

mlc-ai / mlc-llm

Universal LLM Deployment Engine with ML Compilation
https://llm.mlc.ai/
Apache License 2.0
19.23k stars 1.58k forks source link

[3rdparty] Bump tokenizers-cpp for HuggingFace tokenizer update #3029

Closed MasterJH5574 closed 6 days ago

MasterJH5574 commented 6 days ago

This PR bumps the 3rdparty tokenizers-cpp to update the HuggingFace tokenizers package dependency.