qinxuye
commented
2 months ago 请贴下服务端的完整报错。
Open zhangjiekui opened 2 months ago
qinxuye
commented
2 months ago 请贴下服务端的完整报错。
ChengjieLi28
commented
2 months ago 把你怎么使用tool call的过程贴出来，只给个标题是什么意思？ issue请给出完整过程，怎么用tool call的？在哪出现这个报错的？
zhangjiekui
commented
2 months ago 错误信息1：
` Who won the super bowl in 2024? In what state is the winning team headquarters located?What is the GDP of that state? <|im_end|> <|im_start|>assistant , generate config: {'max_tokens': 32000, 'temperature': 0.01, 'stream': False, 'stop': ['<|endoftext|>', '<|im_start|>', '<|im_end|>'], 'stop_token_ids': [151643, 151644, 151645]} 2024-09-09 06:16:02,783 vllm.engine.async_llm_engine 3695534 INFO Added request fd5c46c5-6e72-11ef-b0dd-dbbd1831f9dd. 2024-09-09 06:16:02,783 vllm.engine.async_llm_engine 3695534 DEBUG Got new requests! 2024-09-09 06:16:07,780 vllm.engine.metrics 3695534 INFO Avg prompt throughput: 337.4 tokens/s, Avg generation throughput: 5.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.8%, CPU KV cache usage: 0.0%. 2024-09-09 06:16:08,535 vllm.engine.async_llm_engine 3695534 INFO Finished request fd5c46c5-6e72-11ef-b0dd-dbbd1831f9dd. 2024-09-09 06:16:08,536 xinference.model.llm.utils 3695534 ERROR Can't parse qwen tool call output: {"name": "WEB_SEARCH, "arguments": "{\"query\": \"2024 super bowl winner and team headquarters state and state GDP\"}""}. Error: Expecting ',' delimiter: line 1 column 24 (char 23)
`
错误信息2： ` [request f2e2edc4-6e72-11ef-b0dd-dbbd1831f9dd] Enter chat, args: <xinference.core.model.ModelActor object at 0x7f5a1e5636f0>,[{'content': "Solve/Answer user's questions as best as you can in the language the user asked, prefe...,{'max_tokens': 32000, 'temperature': 0.01, 'tools': <list_iterator object at 0x7f5bb3d0aef0>, 'strea..., kwargs: raw_params={'max_tokens': 32000, 'stream': False, 'temperature': 0.01, 'tools': [{'type': 'function', 'function... 2024-09-09 06:15:45,209 xinference.core.model 3695534 ERROR [request f2e2edc4-6e72-11ef-b0dd-dbbd1831f9dd] Leave chat, error: Tool response dicts require a 'name' key indicating the name of the called function!, elapsed time: 0 s Traceback (most recent call last): File "/root/.virtualenvs/xinfer/lib/python3.10/site-packages/xinference/core/utils.py", line 69, in wrapped ret = await func(*args, kwargs) File "/root/.virtualenvs/xinfer/lib/python3.10/site-packages/xinference/core/model.py", line 570, in chat response = await self._call_wrapper_json( File "/root/.virtualenvs/xinfer/lib/python3.10/site-packages/xinference/core/model.py", line 407, in _call_wrapper_json return await self._call_wrapper("json", fn, *args, *kwargs) File "/root/.virtualenvs/xinfer/lib/python3.10/site-packages/xinference/core/model.py", line 120, in _async_wrapper return await fn(args, kwargs) File "/root/.virtualenvs/xinfer/lib/python3.10/site-packages/xinference/core/model.py", line 416, in _call_wrapper ret = await fn(args, kwargs) File "/root/.virtualenvs/xinfer/lib/python3.10/site-packages/xinference/model/llm/vllm/core.py", line 667, in async_chat full_prompt = self.get_full_context( File "/root/.virtualenvs/xinfer/lib/python3.10/site-packages/xinference/model/llm/utils.py", line 113, in get_full_context return self._build_from_raw_template(messages, chat_template, kwargs) File "/root/.virtualenvs/xinfer/lib/python3.10/site-packages/xinference/model/llm/utils.py", line 87, in _build_from_raw_template rendered = compiled_template.render( File "/root/.virtualenvs/xinfer/lib/python3.10/site-packages/jinja2/environment.py", line 1304, in render self.environment.handle_exception() File "/root/.virtualenvs/xinfer/lib/python3.10/site-packages/jinja2/environment.py", line 939, in handle_exception raise rewrite_traceback_stack(source=source) File "", line 140, in top-level template code File "/root/.virtualenvs/xinfer/lib/python3.10/site-packages/jinja2/sandbox.py", line 394, in call return context.call(obj, args, **kwargs) File "/root/.virtualenvs/xinfer/lib/python3.10/site-packages/xinference/model/llm/utils.py", line 77, in raise_exception raise TemplateError(message) jinja2.exceptions.TemplateError: Tool response dicts require a 'name' key indicating the name of the called function!
`
ChengjieLi28
commented
2 months ago 错误信息1：
` Who won the super bowl in 2024? In what state is the winning team headquarters located?What is the GDP of that state? <|im_end|> <|im_start|>assistant , generate config: {'max_tokens': 32000, 'temperature': 0.01, 'stream': False, 'stop': ['<|endoftext|>', '<|im_start|>', '<|im_end|>'], 'stop_token_ids': [151643, 151644, 151645]} 2024-09-09 06:16:02,783 vllm.engine.async_llm_engine 3695534 INFO Added request fd5c46c5-6e72-11ef-b0dd-dbbd1831f9dd. 2024-09-09 06:16:02,783 vllm.engine.async_llm_engine 3695534 DEBUG Got new requests! 2024-09-09 06:16:07,780 vllm.engine.metrics 3695534 INFO Avg prompt throughput: 337.4 tokens/s, Avg generation throughput: 5.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.8%, CPU KV cache usage: 0.0%. 2024-09-09 06:16:08,535 vllm.engine.async_llm_engine 3695534 INFO Finished request fd5c46c5-6e72-11ef-b0dd-dbbd1831f9dd. 2024-09-09 06:16:08,536 xinference.model.llm.utils 3695534 ERROR Can't parse qwen tool call output: {"name": "WEB_SEARCH, "arguments": "{"query": "2024 super bowl winner and team headquarters state and state GDP"}""}. Error: Expecting ',' delimiter: line 1 column 24 (char 23)
`
错误信息2： ` [request f2e2edc4-6e72-11ef-b0dd-dbbd1831f9dd] Enter chat, args: <xinference.core.model.ModelActor object at 0x7f5a1e5636f0>,[{'content': "Solve/Answer user's questions as best as you can in the language the user asked, prefe...,{'max_tokens': 32000, 'temperature': 0.01, 'tools': <list_iterator object at 0x7f5bb3d0aef0>, 'strea..., kwargs: raw_params={'max_tokens': 32000, 'stream': False, 'temperature': 0.01, 'tools': [{'type': 'function', 'function... 2024-09-09 06:15:45,209 xinference.core.model 3695534 ERROR [request f2e2edc4-6e72-11ef-b0dd-dbbd1831f9dd] Leave chat, error: Tool response dicts require a 'name' key indicating the name of the called function!, elapsed time: 0 s Traceback (most recent call last): File "/root/.virtualenvs/xinfer/lib/python3.10/site-packages/xinference/core/utils.py", line 69, in wrapped ret = await func(*args, kwargs) File "/root/.virtualenvs/xinfer/lib/python3.10/site-packages/xinference/core/model.py", line 570, in chat response = await self._call_wrapper_json( File "/root/.virtualenvs/xinfer/lib/python3.10/site-packages/xinference/core/model.py", line 407, in _call_wrapper_json return await self._call_wrapper("json", fn, *args, *kwargs) File "/root/.virtualenvs/xinfer/lib/python3.10/site-packages/xinference/core/model.py", line 120, in _async_wrapper return await fn(args, kwargs) File "/root/.virtualenvs/xinfer/lib/python3.10/site-packages/xinference/core/model.py", line 416, in _call_wrapper ret = await fn(args, kwargs) File "/root/.virtualenvs/xinfer/lib/python3.10/site-packages/xinference/model/llm/vllm/core.py", line 667, in async_chat full_prompt = self.get_full_context( File "/root/.virtualenvs/xinfer/lib/python3.10/site-packages/xinference/model/llm/utils.py", line 113, in get_full_context return self._build_from_raw_template(messages, chat_template, kwargs) File "/root/.virtualenvs/xinfer/lib/python3.10/site-packages/xinference/model/llm/utils.py", line 87, in _build_from_raw_template rendered = compiled_template.render( File "/root/.virtualenvs/xinfer/lib/python3.10/site-packages/jinja2/environment.py", line 1304, in render self.environment.handle_exception() File "/root/.virtualenvs/xinfer/lib/python3.10/site-packages/jinja2/environment.py", line 939, in handle_exception raise rewrite_traceback_stack(source=source) File "", line 140, in top-level template code File "/root/.virtualenvs/xinfer/lib/python3.10/site-packages/jinja2/sandbox.py", line 394, in call return context.call(obj, args, **kwargs) File "/root/.virtualenvs/xinfer/lib/python3.10/site-packages/xinference/model/llm/utils.py", line 77, in raise_exception raise TemplateError(message) jinja2.exceptions.TemplateError: Tool response dicts require a 'name' key indicating the name of the called function!
`
怎么调用chat这个接口的？chat接口传入的参数messages是什么样的？
zhangjiekui
commented
2 months ago 2024-09-09 06:25:50,168 xinference.core.supervisor 3695328 DEBUG [request 5b78db04-6e74-11ef-b0dd-dbbd1831f9dd] Enter get_model, args: <xinference.core.supervisor.SupervisorActor object at 0x7fb2ab9bdc60>,qwen2-instruct, kwargs: 2024-09-09 06:25:50,168 xinference.core.worker 3695328 DEBUG Enter get_model, args: <xinference.core.worker.WorkerActor object at 0x7fb2ab9bd7b0>, kwargs: model_uid=qwen2-instruct-1-0 2024-09-09 06:25:50,169 xinference.core.worker 3695328 DEBUG Leave get_model, elapsed time: 0 s 2024-09-09 06:25:50,169 xinference.core.supervisor 3695328 DEBUG [request 5b78db04-6e74-11ef-b0dd-dbbd1831f9dd] Leave get_model, elapsed time: 0 s 2024-09-09 06:25:50,171 xinference.core.supervisor 3695328 DEBUG [request 5b78db05-6e74-11ef-b0dd-dbbd1831f9dd] Enter describe_model, args: <xinference.core.supervisor.SupervisorActor object at 0x7fb2ab9bdc60>,qwen2-instruct, kwargs: 2024-09-09 06:25:50,172 xinference.core.worker 3695328 DEBUG Enter describe_model, args: <xinference.core.worker.WorkerActor object at 0x7fb2ab9bd7b0>, kwargs: model_uid=qwen2-instruct-1-0 2024-09-09 06:25:50,172 xinference.core.worker 3695328 DEBUG Leave describe_model, elapsed time: 0 s 2024-09-09 06:25:50,172 xinference.core.supervisor 3695328 DEBUG [request 5b78db05-6e74-11ef-b0dd-dbbd1831f9dd] Leave describe_model, elapsed time: 0 s 2024-09-09 06:25:50,176 xinference.core.model 3695534 DEBUG Request chat, current serve request count: 0, request limit: None for the model qwen2-instruct 2024-09-09 06:25:50,177 xinference.core.model 3695534 DEBUG [request 5b7a363e-6e74-11ef-b0dd-dbbd1831f9dd] Enter chat, args: <xinference.core.model.ModelActor object at 0x7f5a1e5636f0>,[{'content': "Solve/Answer user's questions as best as you can in the language the user asked, prefe...,{'max_tokens': 32000, 'temperature': 0.01, 'tools': <list_iterator object at 0x7f5b4420b9a0>, 'strea..., kwargs: raw_params={'max_tokens': 32000, 'stream': False, 'temperature': 0.01, 'tools': [{'type': 'function', 'function... 2024-09-09 06:25:50,181 xinference.model.llm.vllm.core 3695534 DEBUG Enter generate, prompt: <|im_start|>system Solve/Answer user's questions as best as you can in the language the user asked, preferred/default to Simplified Chinese. You have access to the following tools:get_flight_number,python_repl_tool,WEB_SEARCH,CODE_GENERATION,CODE_EXECUTION,VECTORSTORE.You are allowed to make multiple calls (either together or in sequence).Choice Guidelines: 'CODE_GENERATION' is preferred rather than 'python_repl_tool' for drawing images, plotting graphs/charts, data file processing/analyzing/saving, Otherwise, please use 'python_repl_tool'.
You are a function calling AI model. You are provided with function signatures within
Args:
origin(str): destination(str): date(str): ", "parameters": {"properties": {"date": {"format": "date", "type": "string"}, "destination": {"type": "string"}, "origin": {"type": "string"}}, "required": ["origin", "destination", "date"], "type": "object"}}{"type": "function", "function": {"name": python_repl_tool", "description": "python_repl_tool(query: str) - A Python shell. Use this to execute Input(query:str) python commands, but only suitable for simple question. Waring: 'CODE_GENERATION' tool is much powerful, please use 'CODE_GENERATION' tool for drawing images, plotting graphs/charts, data file processing/analyzing/saving.
Input: query:str.Python code. You must use '' to seperate each code line
Output: you MUST using `print($code or value$)` to print out at least one value (mostly in the last line), or multiple values as needed to better answer/solve the user's problem.For examples:
`# print out at least one value: MUST.
import datetime
print(datetime.datetime.today() + datetime.timedelta(days=2))
`# print out multiple values: AS NEEDED.
print(1+1)
print(298*3)
print(8-1)
`
Args:
query(str): ", "parameters": {"properties": {"query": {"type": "string"}}, "required": ["query"], "type": "object"}}{"type": "function", "function": {"name": WEB_SEARCH", "description": "WEB_SEARCH(query: str) - The internet. Use web_search for questions that are related to anything else than agent memory , prompt engineering, and adversarial attacks, 特别是2024年及以后相关的问题及发生的事情.
Args:
query(str): The query to use when searching the internet.", "parameters": {"properties": {"query": {"description": "The query to use when searching the internet.", "type": "string"}}, "required": ["query"], "type": "object"}}{"type": "function", "function": {"name": CODE_GENERATION", "description": "CODE_GENERATION(query: str) - Code Assistant/Code generation . Use programming code (preferred Python language) to slove a user question, if the question related to [code generating, file processing/analyzing, data exploring/analyzing/visualizing].
Args:
query(str): The query to use when need to generate python code.", "parameters": {"properties": {"query": {"description": "The query to use when need to generate python code.", "type": "string"}}, "required": ["query"], "type": "object"}}{"type": "function", "function": {"name": CODE_EXECUTION", "description": "CODE_EXECUTION(query: str) - Code execution/Code intepreter. When user providing a code snippet and asking for running/executing, w or w/o returning the result.
Args:
query(str): The query to use when asking for running/executing a code snippet.", "parameters": {"properties": {"query": {"description": "The query to use when asking for running/executing a code snippet.", "type": "string"}}, "required": ["query"], "type": "object"}}{"type": "function", "function": {"name": VECTORSTORE", "description": "VECTORSTORE(query: str) - A vectorstore/knowledge base containing documents related to 晶奇公司、安徽晶奇网络科技股份有限公司、医疗卫生、医疗健康、医疗服务、医疗政策、medical treatment & health. 凡是与晶奇公司、安徽晶奇网络科技股份有限公司及医疗健康相关的问题，都必须使用vectorstore工具。MUST use the vectorstore for questions on these topics.
Args:
query(str): The query to use when searching the vectorstore.", "parameters": {"properties": {"query": {"description": "The query to use when searching the vectorstore.", "type": "string"}}, "required": ["query"], "type": "object"}} </tools>Use the following pydantic model json schema for each tool call you will make: {"properties": {"arguments": {"title": "Arguments", "type": "object"}, "name": {"title": "Name", "type": "string"}}, "required": ["arguments", "name"], "title": "FunctionCall", "type": "object"}For each function call return a json object with function name and arguments within
zhangjiekui
commented
2 months ago 以上是一次完整的调用日志，格式有点乱了，谢谢
ChengjieLi28
commented
2 months ago 以上是一次完整的调用日志，格式有点乱了，谢谢
这是服务端日志，你客户端怎么调用的？总要有个地方调用chat接口吧
ChengjieLi28
commented
2 months ago 以上是一次完整的调用日志，格式有点乱了，谢谢
如果是curl给我curl的完整命令，如果是python客户端，给我客户端调用代码。或者别的什么方式。
zhangjiekui
commented
2 months ago ChengjieLi28
commented
2 months ago 这个你们是自己封装了吗？invoke实际调用的就是chat？messages能具体贴出来啥样吗？
zhangjiekui
commented
2 months ago 客户端调用没问题呀，现在是服务端返回Tool response出现的问题： Tool response dicts
ChengjieLi28
commented
2 months ago 客户端调用没问题呀，现在是服务端返回Tool response出现的问题： Tool response dicts
有问题，服务端表明messages转化成模型真实的prompt的时候的报错。我需要你客户端传到chat接口那里的messages是什么样子。
zhangjiekui
commented
2 months ago 确实是封装的，所以贴出来意义不大，而且完整的模型输入在服务器日志里有呀
ChengjieLi28
commented
2 months ago 确实是封装的，所以贴出来意义不大，而且完整的模型输入在服务器日志里有呀
0.15.0改了，要贴出来，否则不知道问题在哪。0.15.0 qwen2的tool call我们保持跟openAI messages格式一致了。不是之前那个样子的。
ChengjieLi28
commented
2 months ago 确实是封装的，所以贴出来意义不大，而且完整的模型输入在服务器日志里有呀
如果你不方便贴出，请自行对照：https://platform.openai.com/docs/api-reference/chat/create
是否符合示例。messages和tools都传了。
zhangjiekui
commented
2 months ago zhangjiekui
commented
2 months ago 这个链接你等打开吗？
ChengjieLi28
commented
2 months ago 这个链接你等打开吗？
打不开，Forbidden。你加一下readme里面的企业微信吧
zhangjiekui
commented
2 months ago zhangjiekui
commented
2 months ago 工具已经正常执行，返回结果了。现在服务端要组装一个tool responce返回，但出现了错误
zhangjiekui
commented
2 months ago zhangjiekui
commented
2 months ago 我怀疑是content = json.loads(text) 这行里的 text本应是dict，但里面全部是双引号导致的，下面是一个服务器的输出： ` {"name": "WEB_SEARCH, "arguments": "{\"query\": \"What is the weather in SF and LA?\"}""}
`
ChengjieLi28
commented
2 months ago 工具已经正常执行，返回结果了。现在服务端要组装一个tool responce返回，但出现了错误
我看懂了，这个我只能说尽力而为，解析这个结果。qwen2的输出不是一个标准json。。。。所以json.loads出错了
zhangjiekui
commented
2 months ago zhangjiekui
commented
2 months ago 是否可以换成json_repair？我测试可以
ChengjieLi28
commented
2 months ago 是否可以换成json_repair？我测试可以
你有兴趣提PR吗？可以引入这个以来，当json.loads失败的时候用这个试试。
zhangjiekui
commented
2 months ago 我先测测看，是不是这个问题
zhangjiekui
commented
2 months ago 测试了，不是这个原因。 1）分析：这是服务端组装Tool Response阶段发生的，是要输出工具执行的结果，而不是把模型的输出内容解析成工具调用 2）实测：在调用到_eval_qwen_chat_arguments方法之前，Tool response dicts require a 'name' key indicating the name of the called function 这个错误已经发生了。 我现在也不知道怎么排查问题了，但可能是与“Compatible with openai >= 1.40” （https://github.com/xorbitsai/inference/pull/2231)有关
ChengjieLi28
commented
2 months ago File "/root/.virtualenvs/xinfer/lib/python3.10/site-packages/xinference/model/llm/vllm/core.py", line 667, in async_chat full_prompt = self.get_full_context( File "/root/.virtualenvs/xinfer/lib/python3.10/site-packages/xinference/model/llm/utils.py", line 113, in get_full_context return self._build_from_raw_template(messages, chat_template, *kwargs) File "/root/.virtualenvs/xinfer/lib/python3.10/site-packages/xinference/model/llm/utils.py", line 87, in _build_from_raw_template rendered = compiled_template.render( File "/root/.virtualenvs/xinfer/lib/python3.10/site-packages/jinja2/environment.py", line 1304, in render self.environment.handle_exception() File "/root/.virtualenvs/xinfer/lib/python3.10/site-packages/jinja2/environment.py", line 939, in handle_exception raise rewrite_traceback_stack(source=source) File "", line 140, in top-level template code File "/root/.virtualenvs/xinfer/lib/python3.10/site-packages/jinja2/sandbox.py", line 394, in call return context.call(obj, args, **kwargs) File "/root/.virtualenvs/xinfer/lib/python3.10/site-packages/xinference/model/llm/utils.py", line 77, in raise_exception raise TemplateError(message) jinja2.exceptions.TemplateError: Tool response dicts require a 'name' key indicating the name of the called function!
这是你的报错栈，从get_full_context这个函数进去，这是在调用之前。。。。生成完整prompt就出错了。所以我一开始必须要知道你是怎么传的参数。messages和tools怎么传的。 我给一个我这边测试完整的例子。 qwen2-instruct on vllm
model = client.get_model("qwen2-instruct")
tools = [{'type': 'function',
'function': {'name': 'find_sports_shoes',
'description': '查找商场内可用的运动鞋信息',
'parameters': {'type': 'object',
'properties': {'brand': {'type': 'string', 'description': '鞋子的品牌'},
'size': {'type': 'string', 'description': '鞋子的尺码'}},
'required': ['brand']},
'strict': False}}]
mode.chat(messages=[{"role": "user", "content": "如何查询耐克45码鞋？"}], tools=tools)我这样调用是完全没有问题的。我感觉还是你传参有问题。
zhangjiekui
commented
2 months ago 你再测试一下，把tool执行结果传给大模型，获取最终的回答。
ChengjieLi28
commented
2 months ago 你再测试一下，把tool执行结果传给大模型，获取最终的回答。
这是什么场景，chat接口只到此为止了。再传给大模型是什么意思？什么叫最终回答。现在不是react那个模式了。我们只兼容openai的function call。
zhangjiekui
commented
2 months ago 我加微信了，麻烦通过后直接语音聊
ChengjieLi28
commented
2 months ago 我加微信了，麻烦通过后直接语音聊
我知道你什么意思了，能复现出来这个问题，稍后我写个示例给你。
zhangjiekui
commented
2 months ago 谢谢。之前的版本都能正常运行。
下面是[v0.14.4.post1]版本运行结果：
09/09/2024 08:48:59 - [DEBUG] -jq_functions_and_tools.py->>> 'python_repl_tool'执行结果:2024-09-10
self.message_idx_in_a_question_epoch=2 [INFO:langgraph_in_class.py(715)] 09/09/2024 08:48:59 - [INFO] -langgraph_in_class.py->>> self.message_idx_in_a_question_epoch=2 工具调用执行正常，返回到大模型 [INFO:langgraph_in_class.py(879)] 09/09/2024 08:48:59 - [INFO] -langgraph_in_class.py->>> 工具调用执行正常，返回到大模型 call_llm:调用LLM时的消息数为:4 [INFO:langgraph_in_class.py(946)] 09/09/2024 08:48:59 - [INFO] -langgraph_in_class.py->>> call_llm:调用LLM时的消息数为:4 09/09/2024 08:49:04 - [INFO] -httpx->>> HTTP Request: POST http://10.1.150.105:9997/v1/chat/completions "HTTP/1.1 200 OK" self.message_idx_in_a_question_epoch=3 [INFO:langgraph_in_class.py(715)] 09/09/2024 08:49:04 - [INFO] -langgraph_in_class.py->>> self.message_idx_in_a_question_epoch=3 Returned 1 tool_calls [DEBUG:langgraph_in_class.py(911)] 09/09/2024 08:49:04 - [DEBUG] -langgraph_in_class.py->>> Returned 1 tool_calls LLM Returned tool_calling: [INFO:langgraph_in_class.py(842)] 09/09/2024 08:49:04 - [INFO] -langgraph_in_class.py->>> LLM Returned tool_calling: Tool_name:get_flight_number [DEBUG:langgraph_in_class.py(845)] 09/09/2024 08:49:04 - [DEBUG] -langgraph_in_class.py->>> Tool_name:get_flight_number Tool_args: {'origin': '合肥', 'destination': '上海', 'date': '2024-09-10'} [DEBUG:langgraph_in_class.py(846)] 09/09/2024 08:49:04 - [DEBUG] -langgraph_in_class.py->>> Tool_args: {'origin': '合肥', 'destination': '上海', 'date': '2024-09-10'} tool_call_id='call_6d552a7b-93c8-44e8-aefb-7e1ce5a4931d' 'get_flight_number'执行结果: 东方航空MU5646 【2024-09-10】 [INFO:jq_functions_and_tools.py(29)] 09/09/2024 08:49:04 - [INFO] -jq_functions_and_tools.py->>> 'get_flight_number'执行结果: 东方航空MU5646 【2024-09-10】 self.message_idx_in_a_question_epoch=4 [INFO:langgraph_in_class.py(715)] 09/09/2024 08:49:04 - [INFO] -langgraph_in_class.py->>> self.message_idx_in_a_question_epoch=4 工具调用执行正常，返回到大模型 [INFO:langgraph_in_class.py(879)] 09/09/2024 08:49:04 - [INFO] -langgraph_in_class.py->>> 工具调用执行正常，返回到大模型 call_llm:调用LLM时的消息数为:6 [INFO:langgraph_in_class.py(946)] 09/09/2024 08:49:04 - [INFO] -langgraph_in_class.py->>> call_llm:调用LLM时的消息数为:6 09/09/2024 08:49:09 - [INFO] -httpx->>> HTTP Request: POST http://10.1.150.105:9997/v1/chat/completions "HTTP/1.1 200 OK" self.message_idx_in_a_question_epoch=5 [INFO:langgraph_in_class.py(715)] 09/09/2024 08:49:09 - [INFO] -langgraph_in_class.py->>> self.message_idx_in_a_question_epoch=5 Returned 1 tool_calls [DEBUG:langgraph_in_class.py(911)] 09/09/2024 08:49:09 - [DEBUG] -langgraph_in_class.py->>> Returned 1 tool_calls LLM Returned tool_calling: [INFO:langgraph_in_class.py(842)] 09/09/2024 08:49:09 - [INFO] -langgraph_in_class.py->>> LLM Returned tool_calling: Tool_name:get_flight_number [DEBUG:langgraph_in_class.py(845)] 09/09/2024 08:49:09 - [DEBUG] -langgraph_in_class.py->>> Tool_name:get_flight_number Tool_args: {'date': '2024-09-10', 'destination': '北京', 'origin': '南昌'} [DEBUG:langgraph_in_class.py(846)] 09/09/2024 08:49:09 - [DEBUG] -langgraph_in_class.py->>> Tool_args: {'date': '2024-09-10', 'destination': '北京', 'origin': '南昌'} tool_call_id='call_45473e56-70d3-4d18-b3c9-43f36b3d4b7d' 'get_flight_number'执行结果: 东方航空MU5646 【2024-09-10】 [INFO:jq_functions_and_tools.py(29)] 09/09/2024 08:49:09 - [INFO] -jq_functions_and_tools.py->>> 'get_flight_number'执行结果: 东方航空MU5646 【2024-09-10】 self.message_idx_in_a_question_epoch=6 [INFO:langgraph_in_class.py(715)] 09/09/2024 08:49:09 - [INFO] -langgraph_in_class.py->>> self.message_idx_in_a_question_epoch=6 工具调用执行正常，返回到大模型 [INFO:langgraph_in_class.py(879)] 09/09/2024 08:49:09 - [INFO] -langgraph_in_class.py->>> 工具调用执行正常，返回到大模型 call_llm:调用LLM时的消息数为:8 [INFO:langgraph_in_class.py(946)] 09/09/2024 08:49:09 - [INFO] -langgraph_in_class.py->>> call_llm:调用LLM时的消息数为:8 09/09/2024 08:49:15 - [INFO] -httpx->>> HTTP Request: POST http://10.1.150.105:9997/v1/chat/completions "HTTP/1.1 200 OK" self.message_idx_in_a_question_epoch=7 [INFO:langgraph_in_class.py(715)] 09/09/2024 08:49:15 - [INFO] -langgraph_in_class.py->>> self.message_idx_in_a_question_epoch=7 No returned tool_calling and finish_reason='stop' [INFO:langgraph_in_class.py(921)] 09/09/2024 08:49:15 - [INFO] -langgraph_in_class.py->>> No returned tool_calling and finish_reason='stop' AI: 明天从合肥到上海的航班是东方航空MU5646，南昌到北京的航班也是东方航空MU5646。请注意，航班信息可能会有变动，请以航空公司官方信息为准。 [INFO:langgraph_in_class.py(922)] 09/09/2024 08:49:15 - [INFO] -langgraph_in_class.py->>> AI: 明天从合肥到上海的航班是东方航空MU5646，南昌到北京的航班也是东方航空MU5646。请注意，航班信息可能会有变动，请以航空公司官方信息为准。
`
ChengjieLi28
commented
2 months ago @zhangjiekui 完整复现你的流程：
from xinference.client import Client
client = Client("http://127.0.0.1:12306")
tools = [{'type': 'function',
'function': {'name': 'find_sports_shoes',
'description': '查找商场内可用的运动鞋信息',
'parameters': {'type': 'object',
'properties': {'brand': {'type': 'string', 'description': '鞋子的品牌'},
'size': {'type': 'string', 'description': '鞋子的尺码'}},
'required': ['brand']},
'strict': False}}]
model = client.get_model("qwen2-instruct")调用tools
model.chat([{"role": "user", "content": "如何查询鞋子？"}], tools=tools)由于信息不明确，LLM要求提供详细参数。LLM输出：
{'id': 'chatcmpl-d9fa64de-8d88-4229-92ee-a1c68100a9dc',
'model': 'qwen2-instruct',
'object': 'chat.completion',
'created': 1725870384,
'choices': [{'index': 0,
'message': {'role': 'assistant',
'content': '如果你想查询特定品牌和型号的运动鞋，你可以使用提供的`find_sports_shoes`函数。请提供你要查询的鞋子品牌和尺寸。\n\n示例：如果你想查询Nike的码数为43的运动鞋，你可以提问如下：\n\n查询Nike，43码的运动鞋',
'tool_calls': []},
'finish_reason': 'stop'}],
'usage': {'prompt_tokens': 369, 'completion_tokens': 62, 'total_tokens': 431}}把LLM的输出结果和user提供的信息组织成messages再次调用model chat
messages = [{"role": "user", "content": "如何查询鞋子？"}, {"role": "assistant", "content": "如果你想查询特定品牌和型号的运动鞋，你可以使用提供的`find_sports_shoes`函数。请提供你要查询的鞋子品牌和尺寸。\n\n示例：如果你想查询Nike的码数为43的运动鞋，你可以提问如下：\n\n查询Nike，43码的运动鞋"}, {"role": "user", "content": "品牌是安踏，尺码40码"}]
response = model.chat(messages, tools=tools)这次能输出tool call了，response为：
{'id': 'chatcmpl-be89e1f7-6067-4ff0-8ce2-46164c652e39',
'model': 'qwen2-instruct',
'object': 'chat.completion',
'created': 1725870504,
'choices': [{'index': 0,
'message': {'role': 'assistant',
'content': None,
'tool_calls': [{'id': 'call_be89e1f7-6067-4ff0-8ce2-46164c652e39',
'type': 'function',
'function': {'name': 'find_sports_shoes',
'arguments': '{"brand": "安踏", "size": "40码"}'}}]},
'finish_reason': 'tool_calls'}],
'usage': {'prompt_tokens': 450, 'completion_tokens': 37, 'total_tokens': 487}}模拟根据此调用函数的结果。假设调用find_sports_shoes的结果为{"num": 5, "color": "blue"}，意为有5双蓝色的鞋。 将结果继续组织一下继续送给LLM
import json
function_call_result_message = {
"role": "tool",
"content": json.dumps({
"brand": "安踏",
"size": "40码",
"sports_shoes": {"num": 5, "color": "blue"}
}),
"tool_call_id": response['choices'][0]['message']['tool_calls'][0]['id']
}
messages.append(response['choices'][0]['message'])
messages.append(function_call_result_message)model.chat(messages)
你这个报错就出现在这里。原因是：最后一个message没有加name字段。（OpenAI没要求，但是qwen2的template模版要求了，所以报错信息有这个name）。
你把function_call_result_message改为function_call_result_message = { "role": "tool", "content": json.dumps({ "brand": "安踏", "size": "40码", "sports_shoes": {"num": 5, "color": "blue"} }), "tool_call_id": response['choices'][0]['message']['tool_calls'][0]['id'], "name": "find_sports_shoes" }
即可。LLM正确输出：{'id': 'chat8250d354-6e87-11ef-832b-047c1643e4f5', 'object': 'chat.completion', 'created': 1725871376, 'model': 'qwen2-instruct', 'choices': [{'index': 0, 'message': {'role': 'assistant', 'content': '我查到的信息是，安踏品牌的40码运动鞋有5款，颜色为蓝色。'}, 'finish_reason': 'stop'}], 'usage': {'prompt_tokens': 237, 'completion_tokens': 23, 'total_tokens': 260}}
@zhangjiekui qwen2的tool call模版很复杂，来自于https://github.com/QwenLM/Qwen2/issues/805 这个PR。我们直接用的这个模版。 最后一个message加上name字段即可。name字段就是函数名。 整个这个过程符合https://platform.openai.com/docs/guides/function-calling OPENAI官方的完整流程。
zhangjiekui
commented
2 months ago 谢谢，我来试试！
zhangjiekui
commented
2 months ago 你说的是对的，单纯从正确运行的角度讲，可以跑过这一步。
但是我是应用langchain框架的，所以连带要修改langchain源码： langchain_openai/chat_models/base.py的第222、223行 message_dict["name"] = message.name supported_props = {"content", "role", "tool_call_id","name"}
也就是说：xinference v0.15.0已经无法兼容langchain
qinxuye
commented
2 months ago langchain的复现代码有吗？我明天来试试改jinja。
zhangjiekui
commented
2 months ago 但即使这样，碰到模型同时返回多个函数调用的情况，还是无法正确解析出tools/functions: 如我这有一个模型返回的原始text是：
`
text='
`
本想尝试这样修改，返回多个函数调用的list，但其他地方的代码又只接受形如 （None, content["name"], content["arguments"]），因这是一个链条，我对xinference的源码不熟，所以不知道要修改到什么时候，只好放弃了。
`
def _eval_qwen_chat_arguments(c):
text = c["choices"][0]["text"]
logger.error(f"原始文本:{text=}")
tool_calls_texts = text.split(f"{QWEN_TOOL_CALL_SYMBOLS[1]}\n\n{QWEN_TOOL_CALL_SYMBOLS[0]}")
for i,t in enumerate(tool_calls_texts):
logger.error(f"\n ==t{i}的内容:\n{t}")
tool_calls = []
for text in tool_calls_texts:
text: str = text.strip()
if text.startswith(QWEN_TOOL_CALL_SYMBOLS[0]):
text = text[len(QWEN_TOOL_CALL_SYMBOLS[0]) :]
if text.endswith(QWEN_TOOL_CALL_SYMBOLS[1]):
text = text[: -len(QWEN_TOOL_CALL_SYMBOLS[1])]
logger.error(f"处理后的文本:{text=}")
try:
content = json_repair.loads(text)
logger.error(f"json_repair.loads的结果:{content=}")
tool_calls.append([None, content["name"], content["arguments"]])
# return None, content["name"], content["arguments"]
except Exception as e:
logger.error("Can't parse qwen tool call output: %s. Error: %s", text, e)
return text, None, None
return tool_calls`
zhangjiekui
commented
2 months ago langchain的复现代码有吗？我明天来试试改jinja。
有，谢谢。我努力配合。
qinxuye
commented
2 months ago langchain的复现代码有吗？我明天来试试改jinja。
有，谢谢。我努力配合。
能贴一下最小复现代码吗？
zhangjiekui
commented
2 months ago 我们加个微信啥的，直接沟通好吗，然后只在这记录必要的内容。因为我的项目封装比较多（langchain\langgraph），所以要配合你的需要，我可能单写一个示例代码。
qinxuye
commented
2 months ago 企业微信加了以后备注联系我。
zhangjiekui
commented
2 months ago zhangjiekui
commented
2 months ago 上面的附件是完整运行代码，基于今天最新的langchain和langchai-openai版本，但目前需要修改langchain_openai/chat_models/base.py 第222、223行：
message_dict["name"] = message.name
supported_props = {"content", "role", "tool_call_id","name"}这是正确运行的结果，模型是qwen2-instruct 72B gptq Int4
zhangjiekui
commented
2 months ago zhangjiekui
commented
2 months ago xinference v0.15.0的源代码也需要做上图对应的修改，代码文件为： xinference/model/llm/utils.py
System Info / 系統信息
v0.15.0
Running Xinference with Docker? / 是否使用 Docker 运行 Xinfernece？
Version info / 版本信息
xinference v0.15.0
The command used to start Xinference / 用以启动 xinference 的命令
nohup xinference-local -H 0.0.0.0 --log-level debug >> /data/xinference/xinfer.log 2>&1 &
Reproduction / 复现过程
就是正常运行升级到v0.15.0前的代码
Expected behavior / 期待表现
期待正常运行