search

zurawiki / tiktoken-rs

Ready-made tokenizer library for working with GPT and tiktoken
MIT License
240 stars 47 forks source link

feat: llama3 tokenizer #64

Open omarkilani opened 3 months ago

omarkilani commented 3 months ago

I'm not sure if the project wants to support non-OpenAI models, but just in case someone comes along and needs this, here you are. :)