search

openai / tiktoken

tiktoken is a fast BPE tokeniser for use with OpenAI's models.
MIT License
12.31k stars 833 forks source link

gpt-3.5-turbo model support #45

Closed gobsimpacts0u closed 1 year ago

gobsimpacts0u commented 1 year ago

Are you considering adding gpt-3.5-turbo model support? Or should the gpt2 model be used (according to https://github.com/openai/openai-cookbook/blob/main/examples/How_to_count_tokens_with_tiktoken.ipynb)?

KineticCookie commented 1 year ago

I'm also interested in gpt-3.5-turbo support. Sometimes I need to calculate the number of tokens before sending it to the model.

hauntsaninja commented 1 year ago

Thanks for your interest in turbo and tiktoken! I've added support for turbo in encoding_for_model and released this in 0.3.0