Add new models

[risu729]

Adds new models released on January 25th, 2024.

• gpt-3.5-turbo-0125
• gpt-4-turbo-preview (alias)
• gpt-4-0125-preview

The encodings are based on the definitions in model.py.

The following models are also added, but this PR does not include them. The encodings are still unknown since openai/tiktoken is not updated yet.

• text-embedding-3-large
• text-embedding-3-small
• text-moderation-007
• text-moderation-latest (alias)
• text-moderation-stable (alias)

OpenAI blog post: New embedding models and API updates OpenAI API: Models

[mystist]

@dqbd Please merge ASAP as we are blocking as this moment🙏

[nem035]

Should this PR also add gpt-4-turbo-preview, which was part of the recent release too?

[risu729]

The PR includes it.

The list in the comment is the models not included in this PR. I apologize for the confusion.

[mystist]

A forked version with @risu729 's code just in case you need it right away: https://github.com/mystist/tiktoken

npm install tiktoken-mystist

[risu729]

Thank you so much for the npm package! It's really helpful to me.

[mystist]

Thank you so much for the npm package! It's really helpful to me.

Thanks to yourself, actually your pr saved my life 😂

[dqbd]

Hey @risu729, thanks for the PR! LGTM