search

zurawiki / tiktoken-rs

Ready-made tokenizer library for working with GPT and tiktoken
MIT License
240 stars 46 forks source link

Bench #20

Closed zurawiki closed 1 year ago