Updates for the pretrained tokenizers. - Githubissues

Hk669 / bpetokenizer

(py package) train your own tokenizer based on BPE algorithm for the LLMs (supports the regex pattern and special tokens)

https://pypi.org/project/bpetokenizer/

2 stars 1 forks source link

Updates for the pretrained tokenizers. #11

Closed Hk669 closed 3 months ago

Hk669 commented 3 months ago

Why are these changes needed?

added the docs for the pretrained tokenizer in the readme.
optimized the tokenizer class

Related issue number

Checks

[ ] I've included any doc changes needed for https://pypi.org/project/bpetokenizer/.
[ ] I've added tests (if relevant) corresponding to the changes introduced in this PR.
[ ] I've made sure all auto checks have passed.