search

openai / tiktoken

tiktoken is a fast BPE tokeniser for use with OpenAI's models.
MIT License
11.98k stars 816 forks source link

how to convert qwen.tiktoken to tokenzier.model #283

Open cloudyuyuyu opened 5 months ago

cloudyuyuyu commented 5 months ago

how to convert qwen.tiktoken to tokenzier.model