search

koaning / tokenwiser

Bag of, not words, but tricks!
https://koaning.github.io/tokenwiser/
Apache License 2.0
68 stars 7 forks source link

Add support for pretrained huggingface tokenizer #39

Closed koaning closed 3 years ago

koaning commented 3 years ago

https://huggingface.co/docs/tokenizers/python/latest/quicktour.html