search

zurawiki / tiktoken-rs

Ready-made tokenizer library for working with GPT and tiktoken
MIT License
240 stars 46 forks source link

test: Add unicode test cases #24

Closed zurawiki closed 1 year ago

zurawiki commented 1 year ago

Addresses and closes #23