search

openai / tiktoken

tiktoken is a fast BPE tokeniser for use with OpenAI's models.
MIT License
11.61k stars 785 forks source link

Support newer completion models in `encoding_for_model` #180

Closed rfriel closed 11 months ago

rfriel commented 1 year ago

Adds the new completion models davinci-002 and babbage-002 to the mapping used in encoding_for_model.

I believe "cl100k_base" is the correct encoding for these models, based on the results from this test I did. If that's not right, feel free to modify or close this.

hauntsaninja commented 11 months ago

Thank you! A change equivalent to this was released in 0.5.0