Can't load model with the latest version of `tokenizers`

QwenLM / Qwen2.5-Coder

Qwen2.5-Coder is the code version of Qwen2.5, the large language model series developed by Qwen team, Alibaba Cloud.

482 stars 30 forks source link

Can't load model with the latest version of `tokenizers` #66

Closed QwertyJack closed 4 months ago

QwertyJack commented 4 months ago

This issue is related to #18.

One quick fix is to downgrade tokenizers to a previous version, i.e. tokenizers==0.15.2, or more straightforward: pip uninstall transformers && pip install "tokenizers<0.19" transformers

See also:

https://github.com/huggingface/tokenizers/issues/1529
https://huggingface.co/Qwen/CodeQwen1.5-7B-Chat/discussions/8

QwertyJack commented 4 months ago

Appreciate your great work!

QwertyJack commented 4 months ago

Problem resolved after updating the tokenizer config file,