search

mlc-ai / tokenizers-cpp

Universal cross-platform tokenizers binding to HF and sentencepiece
Apache License 2.0
211 stars 47 forks source link

Update to reduce memory realloc #18

Closed tqchen closed 8 months ago