transformers报错

[liaoweiguo]

python CodeQwen1.5-Chat-stream.py Traceback (most recent call last): File "/home/xxxx/exgit/CodeQwen1.5/examples/CodeQwen1.5-Chat-stream.py", line 9, in tokenizer = AutoTokenizer.from_pretrained("CodeQwen1.5-7B-Chat") File "/home/xxxx/.local/lib/python3.10/site-packages/transformers/models/auto/tokenization_auto.py", line 862, in from_pretrained return tokenizer_class.from_pretrained(pretrained_model_name_or_path, *inputs, *kwargs) File "/home/xxxx/.local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 2089, in from_pretrained return cls._from_pretrained( File "/home/xxxx/.local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 2311, in _from_pretrained tokenizer = cls(init_inputs, **init_kwargs) File "/home/xxxx/.local/lib/python3.10/site-packages/transformers/tokenization_utils_fast.py", line 111, in init fast_tokenizer = TokenizerFast.from_file(fast_tokenizer_file) Exception: data did not match any variant of untagged enum PyPreTokenizerTypeWrapper at line 12564 column 3

版本信息

transformers 4.40.0 transformers-stream-generator 0.0.4 torch 2.1.2 torchdiffeq 0.2.3 torchmetrics 1.1.0 torchsde 0.2.5 torchvision 0.15.2

qwen1.5-14b没问题

[cyente]

您好，最近2天transfomers新的版本更新似乎带来了一些未知错误，我们会尽快查清楚原因，目前比较简单的解决方：

请尝试降低transformers的版本 pip install transformers==4.39.1