search

openai / tiktoken

tiktoken is a fast BPE tokeniser for use with OpenAI's models.
MIT License
11.61k stars 785 forks source link

Deprecation of older models #166

Closed AlexanderMorou closed 1 year ago

AlexanderMorou commented 1 year ago

On OpenAI's website, they recently released information about the deprecation of older models in the Completions API.

Is there any word on when we can see the new model -> encodings available in the source?

Are all of the replacements going to map to the cl100k_base?

I'm trying to keep my implementation in C# up to date.

hauntsaninja commented 1 year ago

cc @ted-at-openai would you help me stay on top of this? :-)

ted-at-openai commented 1 year ago

Yes, when the new models come out, we'll update tiktoken. They'll all use cl100k_base, so you can add support ahead of time if you like.

AlexanderMorou commented 1 year ago

... They'll all use cl100k_base, so you can add support ahead of time if you like.

Thanks, I'll do just that in my own mappings.