search

IAPark / tiktoken_ruby

Unofficial ruby binding for tiktoken by way of rust
MIT License
118 stars 26 forks source link

Add support: o200k_base tokenizer. #37

Closed CaffeeLake closed 4 months ago

CaffeeLake commented 4 months ago

This PR support o200k_base tokenizer for GPT-4o. I changed back from IAPark/tiktoken-rs to zurawiki/tiktoken-rs because async-openai became optional.