search

Hk669 / bpetokenizer

(py package) train your own tokenizer based on BPE algorithm for the LLMs (supports the regex pattern and special tokens)
https://pypi.org/project/bpetokenizer/
2 stars 1 forks source link

feat: from_pretrained enabled with wi17k_base #6

Closed Hk669 closed 3 months ago

Hk669 commented 3 months ago

Fix Issue

this PR fixes the issue #5