search

openai / tiktoken

tiktoken is a fast BPE tokeniser for use with OpenAI's models.
MIT License
11.17k stars 751 forks source link

Please add new mode gpt-3.5-turbo-instruct #223

Closed kensenjohn closed 5 months ago

kensenjohn commented 7 months ago

Please add the new model

gpt-3.5-turbo-instruct

https://platform.openai.com/docs/models/gpt-3-5

image
hauntsaninja commented 5 months ago 
>>> import tiktoken
>>> tiktoken.encoding_for_model("gpt-3.5-turbo-instruct")
<Encoding 'cl100k_base'>

This works fine on recent tiktoken