search

huggingface / tokenizers

💥 Fast State-of-the-Art Tokenizers optimized for Research and Production
https://huggingface.co/docs/tokenizers
Apache License 2.0
8.93k stars 777 forks source link

Add quick doc to byte_level.rs #1420

Closed steventrouble closed 9 months ago

steventrouble commented 9 months ago

See https://github.com/huggingface/tokenizers/issues/1276

Sorry for the delay! Lost track of this.

ArthurZucker commented 9 months ago

Thanks 🤗