search

openai / tiktoken

tiktoken is a fast BPE tokeniser for use with OpenAI's models.
MIT License
11.8k stars 803 forks source link

tiktoken should include function num_tokens_from_messages() #115

Closed rmilkowski closed 1 year ago

rmilkowski commented 1 year ago

Hi,

Instead of asking users top copy&paste num_tokens_from_messages() from cookbook examples to deal with ChatGPT models, the tiktoken library should provide one, so handling of any new tokens would be in one place without users having to update their code - instead they will need to update the tiktoken library). Also the function should be stripped from any print() messages.

Additionally there should be a more general function get_tokens(prompt, model) which would detect if promot is a list or a string and automatically do the right thing.

hauntsaninja commented 1 year ago

Thanks for the issue! I think this is out of scope for this repository, since I'm not familiar with what the prompts for the various models OpenAI offers look like and what is considered implementation detail by the product. Hopefully you get more of a reply on https://github.com/openai/openai-python/issues/412