[BUG] GLM4-Chat+Xinference：InternalServerError: Internal Server Error

[WinstonCHEN1]

问题描述 / Problem Description GLM4-Chat+Xinference，Follow了README的步骤，启动了模型之后，显存显示模型已经启动了，在xinference那边显示两个model也启动了。在chatchat的ui这边报错InternalServerError: Internal Server Error

环境信息 / Environment Information

• langchain-ChatGLM 版本/commit 号：0.3.0
• 是否使用 Docker 部署（是/否）：否
• 使用的模型（ChatGLM2-6B / Qwen-7B 等）：GLM4-Chat
• 使用的 Embedding 模型（moka-ai/m3e-base 等）：bge-large-zh-v1.5
• 使用的向量库类型 (faiss / milvus / pg_vector 等)： faiss
• 操作系统及版本 / Operating system and version:Linux
• Python 版本 / Python version:3.11

附加信息 / Additional Information 添加与问题相关的任何其他信息 / Add any other information related to the issue. 以下是ui界面的报错信息 InternalServerError: Internal Server Error Traceback: File "/home/chenguo/anaconda3/envs/chatchat/lib/python3.11/site-packages/streamlit/runtime/scriptrunner/script_runner.py", line 600, in _run_script exec(code, module.dict) File "/home/chenguo/anaconda3/envs/chatchat/lib/python3.11/site-packages/chatchat/webui.py", line 69, in dialogue_page(api=api, is_lite=is_lite) File "/home/chenguo/anaconda3/envs/chatchat/lib/python3.11/site-packages/chatchat/webui_pages/dialogue/dialogue.py", line 361, in dialogue_page for d in client.chat.completions.create( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/chenguo/anaconda3/envs/chatchat/lib/python3.11/site-packages/openai/_utils/_utils.py", line 277, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/chenguo/anaconda3/envs/chatchat/lib/python3.11/site-packages/openai/resources/chat/completions.py", line 643, in create return self._post( ^^^^^^^^^^^ File "/home/chenguo/anaconda3/envs/chatchat/lib/python3.11/site-packages/openai/_base_client.py", line 1261, in post return cast(ResponseT, self.request(cast_to, opts, stream=stream, stream_cls=stream_cls)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/chenguo/anaconda3/envs/chatchat/lib/python3.11/site-packages/openai/_base_client.py", line 942, in request return self._request( ^^^^^^^^^^^^^^ File "/home/chenguo/anaconda3/envs/chatchat/lib/python3.11/site-packages/openai/_base_client.py", line 1026, in _request return self._retry_request( ^^^^^^^^^^^^^^^^^^^^ File "/home/chenguo/anaconda3/envs/chatchat/lib/python3.11/site-packages/openai/_base_client.py", line 1074, in _retry_request return self._request( ^^^^^^^^^^^^^^ File "/home/chenguo/anaconda3/envs/chatchat/lib/python3.11/site-packages/openai/_base_client.py", line 1026, in _request return self._retry_request( ^^^^^^^^^^^^^^^^^^^^ File "/home/chenguo/anaconda3/envs/chatchat/lib/python3.11/site-packages/openai/_base_client.py", line 1074, in _retry_request return self._request( ^^^^^^^^^^^^^^ File "/home/chenguo/anaconda3/envs/chatchat/lib/python3.11/site-packages/openai/_base_client.py", line 1041, in _request raise self._make_status_error_from_response(err.response) from None

[EarZero]

可以检查一下xinference服务是否可以正常推理，尝试使用命令行请求一下xinference上的模型。 类似： curl -X 'POST' \ 'http://127.0.0.1:9997/v1/chat/completions' \ -H 'accept: application/json' \ -H 'Content-Type: application/json' \ -d '{ "model": "my-llama-2", "messages": [ { "role": "system", "content": "You are a helpful assistant." }, { "role": "user", "content": "What is the largest animal?" } ] }'

[tianxiajianghu]

可以检查一下xinference服务是否可以正常推理，尝试使用命令行请求一下xinference上的模型。 类似： curl -X 'POST' 'http://127.0.0.1:9997/v1/chat/completions' -H 'accept: application/json' -H 'Content-Type: application/json' -d '{ "model": "my-llama-2", "messages": [ { "role": "system", "content": "You are a helpful assistant." }, { "role": "user", "content": "What is the largest animal?" } ] }'

单独运行上面的命令是可以正确回复的

[zx1159652666]

有的模型不支持tool_choice=None的情况，如果为None，建议删掉，就可以正确回答了

[jinceon]

俺也一样

[liunux4odoo]

0.3.1 版已经发布，优化了配置方式，修改配置项无需重启服务器，可以更新尝试。

transformers 降级到 4.40 版本试试