Eagerly anticipating GPT-4 32k model support.

openai / tiktoken

tiktoken is a fast BPE tokeniser for use with OpenAI's models.

MIT License

11.76k stars 801 forks source link

Eagerly anticipating GPT-4 32k model support. #123

Closed harrison001 closed 1 year ago

harrison001 commented 1 year ago

NotImplementedError: num_tokens_from_messages() is not implemented for model gpt-4-32k. See https://github.com/openai/openai-python/blob/main/chatml.md for information on how messages are converted to tokens.

hauntsaninja commented 1 year ago

What exactly are you running? num_tokens_from_messages is not a function in tiktoken.

Type "help", "copyright", "credits" or "license" for more information.
>>> import tiktoken
>>> tiktoken.encoding_for_model("gpt-4-32k")
<Encoding 'cl100k_base'>

^ this works fine with tiktoken 0.4

hauntsaninja commented 1 year ago

Oh I see, that's from the OpenAI cookbook. Look like it got updated recently, see: https://github.com/openai/openai-cookbook/commits/main/examples/How_to_count_tokens_with_tiktoken.ipynb