search

zurawiki / tiktoken-rs

Ready-made tokenizer library for working with GPT and tiktoken
MIT License
240 stars 47 forks source link

Add support: o200k_base tokenizer. #63

Closed CaffeeLake closed 4 months ago

CaffeeLake commented 4 months ago

This PR support o200k_base tokenizer for GPT-4o.