search

openai / tiktoken

tiktoken is a fast BPE tokeniser for use with OpenAI's models.
MIT License
11.06k stars 749 forks source link

gpt-4o tokenizer #295

Closed nxfi777 closed 1 month ago

nxfi777 commented 1 month ago

gpt-4o uses a new tokenizer o200k_base, already available in the original tiktoken library

hauntsaninja commented 1 month ago

Hello! Glad you're excited about gpt-4o. Closing, since I don't think there's anything to do here (tiktoken 0.7 will be released on PyPI once the emulated aarch64 builds complete)