zurawiki / tiktoken-rs

Ready-made tokenizer library for working with GPT and tiktoken
MIT License
240 stars 47 forks source link

Add `gpt-4-0125` and `gpt-3.5-turbo-1106` context sizes #59

Closed mikaelsouza closed 7 months ago

mikaelsouza commented 7 months ago

There are a few new models released today from OpenAI

Links:

So, I added gpt-4-0125-preview and gpt-3.5-turbo-1106 context sizes to the repo, so the library can report their context size correctly.

Can you check if these changes make sense? Thanks!

zurawiki commented 7 months ago

Small nit otherwise LGTM