search

openai / tiktoken

tiktoken is a fast BPE tokeniser for use with OpenAI's models.
MIT License
12.52k stars 856 forks source link

encoding_for_model("gpt-4") #220

Closed fanpeeps closed 12 months ago

fanpeeps commented 12 months ago

Readme suggestions selecting appropraite tokenizer with this request for gpt-4:

enc = tiktoken.encoding_for_model("gpt-4")

Yet, when doing so, this error is produced:

ValueError: Unknown encoding gpt-4. Plugins found: ['tiktoken_ext.openai_public']

hauntsaninja commented 12 months ago

That sounds like you ran tiktoken.get_encoding("gpt-4") instead of tiktoken.encoding_for_model("gpt-4")