search

zurawiki / tiktoken-rs

Ready-made tokenizer library for working with GPT and tiktoken
MIT License
240 stars 46 forks source link

Implement cl100k tokenizer #2

Closed zurawiki closed 1 year ago

zurawiki commented 1 year ago

Closes #1