search

mlc-ai / tokenizers-cpp

Universal cross-platform tokenizers binding to HF and sentencepiece
Apache License 2.0
211 stars 47 forks source link

[Rust] Bump HuggingFace tokenizer version #36

Closed MasterJH5574 closed 1 month ago

MasterJH5574 commented 1 month ago

This PR bumps the HuggingFace tokenizers package version to work with some of the latest models (e.g., Mistral v0.3).