[CMake] Support disable SentencePiece tokenizer

mlc-ai / tokenizers-cpp

Universal cross-platform tokenizers binding to HF and sentencepiece

Apache License 2.0

211 stars 47 forks source link

Closed MasterJH5574 closed 1 week ago

MasterJH5574 commented 1 week ago

This PR supports turning off SentencePiece tokenizer to reduce binary size.