sigoden
commented
3 months ago The infinite loop is the bug of LLM. AIChat is also responsible for identifying dead loops and aborting.
Let me explain.
Let's first illustrate how a non-bug LLM works without parallel supporting.
# First Request: Question only
[{"role":"user","content":"what is the weather in Pairs and Berlin?"}]
# First Reponse: Returns the first function calling
{
"content": null,
"role": "assistant",
"tool_calls": [
{
"function": {
"arguments": "{\n \"location\": \"Paris\"\n}",
"name": "get_current_weather"
},
"id": "call_8e7VqAZILfH0CX4uPVW3lebx",
"type": "function"
}
]
}
# Second Request: Question + The result of first function calling
[
{
"role": "user",
"content": "what is the weather in Pairs and Berlin?"
},
{
"role": "assistant",
"content": "",
"tool_calls": [
{
"id": "call_8e7VqAZILfH0CX4uPVW3lebx",
"type": "function",
"function": {
"name": "get_current_weather",
"arguments": "{\n \"location\": \"Paris\"\n}"
}
}
]
},
{
"role": "tool",
"content": "{\"output\":\"Paris: ☀️ 🌡️+12°C 🌬️→1.1m/s\\n\"}",
"tool_call_id": "call_8e7VqAZILfH0CX4uPVW3lebx"
}
]
# Second Reponse: Return second function calling
{
"role": "assistant",
"content": null,
"tool_calls": [
{
"id": "call_clrUp5JmYoUr5WFGmf0t12Q8",
"type": "function",
"function": {
"name": "get_current_weather",
"arguments": "{\"location\":\"Berlin\"}"
}
}
]
}
# Third Reqeust: Question + The results of two function callings
[
{
"role": "user",
"content": "what is the weather in Pairs and Berlin?"
},
{
"role": "assistant",
"content": "",
"tool_calls": [
{
"id": "call_8e7VqAZILfH0CX4uPVW3lebx",
"type": "function",
"function": {
"name": "get_current_weather",
"arguments": "{\n \"location\": \"Paris\"\n}"
}
},
{
"id": "call_clrUp5JmYoUr5WFGmf0t12Q8",
"type": "function",
"function": {
"name": "get_current_weather",
"arguments": "{\"location\":\"Berlin\"}"
}
}
]
},
{
"role": "tool",
"content": "{\"output\":\"Paris: ☀️ 🌡️+12°C 🌬️→1.1m/s\\n\"}",
"tool_call_id": "call_8e7VqAZILfH0CX4uPVW3lebx"
},
{
"role": "tool",
"content": "{\"output\":\"Berlin: ☁️ 🌡️+11°C 🌬️→7.2m/s\\n\"}",
"tool_call_id": "call_clrUp5JmYoUr5WFGmf0t12Q8"
}
]
# Third Response: Final Result
{
"role": "assistant",
"content": "The current weather in Paris is sunny with a temperature of +12°C and a wind speed of 1.1 m/s.\n\nIn Berlin, the current weather is cloudy with a temperature of +11°C and a wind speed of 7.2 m/s."
}AIChat checks whether the response contains a function call. If so, it executes the function and sends the execution result together with the previous function call results to LLM for further processing until there is no function call.
The bug LLMs cannot handle function calling correctly, resulting in responses always containing function calls, eventually leading to an infinite loop.
blob42
Describe the bug
I am trying to use function calling using local LLM. With Ollama I could not find a way to do it yet.
With LocalAI however thay have a full support for function calling which I tested directly with API calls and works as expected.
Using the default readme example does not work and fails with:
The get_current_weather call has invalid arguments:I found out that this is due to the streaming call, the result is returned over multiple calls.
LocalAI side function call streaming API results
By disabling streaming, function calling works but enter into an infinite loop (see below).
To Reproduce
Debug log of a looping function call - non streaming
expected results
environment
I will try to track down the infinite loop issue and create a PR when I have some time