Closed qzheng527 closed 8 months ago
Hi @qzheng527
Fold name will not impact load, but if some files can not be found loader will raise exceptions. According to error message, loader cannot find tokenization_qwen.py
or related files from fold. Please check files in qwen folder.
BTW, this error maybe caused by Qwen's file out of data. Qwen just changed their files (not weights) last week. Please check if these files are the same with latest Qwen files.
https://huggingface.co/Qwen/Qwen-7B-Chat/blob/main/modeling_qwen.py https://huggingface.co/Qwen/Qwen-7B-Chat/blob/main/tokenizer_config.json https://huggingface.co/Qwen/Qwen-7B-Chat/blob/main/generation_config.json https://huggingface.co/Qwen/Qwen-7B-Chat/blob/main/config.json
To avoid loading exception on latest Qwen model, we updated bigdl qwen loader, but it may raise exception on legacy files.
@qiyuangong The tokenization_qwen.py
is in the model folder and the model can work fine on general chat test.
The model was released on Oct 2023.
I will update it and try again.
Updated bigdl, but it didn't work for simple qwen model chat test.
The demo could be found here
Hi @qzheng527 You can add trust_remote_code=True for local renamed model in bigdl_llm_model.py. https://github.com/intel-analytics/BigDL/blob/128198c6dbcce5c7e2a5edde1fe2304706a7a07a/python/llm/src/bigdl/llm/serving/bigdl_llm_model.py#L245
Here is the related PR https://github.com/intel-analytics/BigDL/pull/9762. You can refer to it, we will make some adjustments next week.
@Jasonzzt Still now worked. To narrow down, I just used the demo code in qwen. It reported the error as below.
BigDL version as below.
bigdl-llm 2.5.0b20231226
@Jasonzzt Still now worked. To narrow down, I just used the demo code in qwen. It reported the error as below.
BigDL version as below.
bigdl-llm 2.5.0b20231226
Please update Qwen model to the latest and try again.
It worked after updating the latest qwen model. Thanks.
Issue closed
Trying to use bigdl LLM serving for model qwen 7b following the doc.
Trying chatglm2 and vicuna 7b, both works fine.