Closed xhluca closed 6 months ago
tiktoken.encoding_for_model is a convenient function for automatically selecting models. However, the following will result in an error:
tiktoken.encoding_for_model
tiktoken.encoding_for_model("gpt-2") tiktoken.encoding_for_model("gpt-3.5")
This is because the only available options right now are:
tiktoken.encoding_for_model("gpt2") # Notice the lack of dash tiktoken.encoding_for_model("gpt-3.5-turbo") # gpt-3.5 is a common shorthand here
This PR will allow the use of "gpt-2" and "gpt-3.5"
Thank you!
tiktoken.encoding_for_model
is a convenient function for automatically selecting models. However, the following will result in an error:This is because the only available options right now are:
This PR will allow the use of "gpt-2" and "gpt-3.5"