search

mlc-ai / tokenizers-cpp

Universal cross-platform tokenizers binding to HF and sentencepiece
Apache License 2.0
211 stars 47 forks source link

Allow parameters related to special tokens for HFTokenizer #27

Closed Abhishek8394 closed 3 months ago

Abhishek8394 commented 3 months ago

Fixes #26