Can't load model with the latest version of `tokenizers`

QwenLM / CodeQwen1.5

CodeQwen1.5 is the code version of Qwen, the large language model series developed by Qwen team, Alibaba Cloud.

371 stars 22 forks source link

Can't load model with the latest version of `tokenizers` #66

Closed QwertyJack closed 1 month ago

QwertyJack commented 1 month ago

This issue is related to #18.

One quick fix is to downgrade tokenizers to a previous version, i.e. tokenizers==0.15.2, or more straightforward: pip uninstall transformers && pip install "tokenizers<0.19" transformers

See also:

https://github.com/huggingface/tokenizers/issues/1529
https://huggingface.co/Qwen/CodeQwen1.5-7B-Chat/discussions/8

QwertyJack commented 1 month ago

Appreciate your great work!

QwertyJack commented 1 month ago

Problem resolved after updating the tokenizer config file,