dmitry-brazhenko / SharpToken

SharpToken is a C# library for tokenizing natural language text. It's based on the tiktoken Python library and designed to be fast and accurate.
https://www.nuget.org/packages/SharpToken
MIT License
214 stars 14 forks source link

Add gpt-4o and variants of it to ModelToEncodingMapping #39

Closed splattne closed 6 months ago

splattne commented 6 months ago

The new GPT-4o model was introduced on 2024-05-13 at the OpenAI spring event

winzig commented 6 months ago

I'm working on a PR for this change also, I think you're going to need a lot more file changes than the one shown here...

winzig commented 6 months ago

My PR: https://github.com/dmitry-brazhenko/SharpToken/pull/41