corrected typo in the documentations for pre-tokenizers - Githubissues

huggingface / tokenizers

💥 Fast State-of-the-Art Tokenizers optimized for Research and Production

https://huggingface.co/docs/tokenizers

Apache License 2.0

8.68k stars 745 forks source link

corrected typo in the documentations for pre-tokenizers #1503

Closed GorkaUrbizu closed 1 month ago

GorkaUrbizu commented 2 months ago

tokeize -> tokenize