search

pkoukk / tiktoken-go

go version of tiktoken
MIT License
601 stars 67 forks source link

Add 'gpt-3.5-turbo-16k' to NumTokensFromMessages example #32

Closed zachsmith1 closed 1 year ago

zachsmith1 commented 1 year ago

Adding 'gpt-3.5-turbo-16k' as another model for the NumTokensFromMessages example. This should avoid issues when users try this function for the first time while leveraging this model name from sashabaranov/go-openai

pkoukk commented 1 year ago

The current example doesn't seem to be equivalent to the original Python code. I will rewrite it later.