Closed realdavidvega closed 5 months ago
This PR brings the GPT4o o200k encoding for the KMP Tokenizer.
Test cases were generated using a small Python script and tiktoken library.
tiktoken
This PR brings the GPT4o o200k encoding for the KMP Tokenizer.
Test cases were generated using a small Python script and
tiktoken
library.