deprecate the __version__ check when loading the tokenizer.

Hk669 / bpetokenizer

(py package) train your own tokenizer based on BPE algorithm for the LLMs (supports the regex pattern and special tokens)

https://pypi.org/project/bpetokenizer/

2 stars 1 forks source link

Closed Hk669 closed 4 weeks ago

Hk669 commented 4 weeks ago

i think this will cause issues to the user, when using a different version.

making it easier to the user should the 1st prior, so deprecating the __version__ check

Hk669 commented 4 weeks ago

addressed in #8