search

dqbd / tiktokenizer

Online playground for OpenAPI tokenizers
https://tiktokenizer.vercel.app
MIT License
680 stars 81 forks source link

Add new `o200k_base` OpenAi encoding for `gpt-4o` #24

Closed kimwonj77 closed 4 months ago

kimwonj77 commented 4 months ago

It’s spring. And new encodings came out with new model.

https://platform.openai.com/docs/models

https://github.com/openai/tiktoken/commit/9d01e5670ff50eb74cdb96406c7f3d9add0ae2f8

kimwonj77 commented 4 months ago

d2350eb3f28e6ab5a80b451ffcc2cc088f2df149