Use new tokenizer from Microsoft to support newer gpt4o model. - Githubissues

dotnet / ResXResourceManager

Manage localization of all ResX-Based resources in one central place.

MIT License

1.33k stars 218 forks source link

Use new tokenizer from Microsoft to support newer gpt4o model. #664

Closed joakimriedel closed 1 month ago

joakimriedel commented 1 month ago

change tokenizer to new Microsoft.ML.Tokenizers
use latest REST API version 2024-06-01
now support latest gpt4o model