search

aespinilla / Tiktoken

Openai's tiktoken implementation written in Swift
MIT License
49 stars 12 forks source link

Support o200k_base #6

Open twodayslate opened 1 month ago

twodayslate commented 1 month ago

Support o200k_base for gpt4o

twodayslate commented 1 month ago

Code from OpenAI's repo: https://github.com/openai/tiktoken/blob/c0ba74c238d18b4824c25f3c27fc8698055b9a76/tiktoken_ext/openai_public.py#L91-L117