.Net: Bug: semantic kernel injecting user message between tool calls response messages

[franklinlindemberg]

Describe the bug Semantic Kernel is adding a user message between tool response messages, which causes the LLM to fail the request with 400 since its expecting to receive response from all tools first.

To Reproduce Steps to reproduce the behavior:

1. Create a flow which the LLM requests multiple tools to be called
2. one of the tools should fail (except the last one) by throwing an exception (which should be handled here)
3. Semantic kernel will handle the exception by adding a tool message with the exception and a following user message indicating tool failure
4. A following tool should succeed. We should have a chat history similar to:
   • tool_message
   • user_message
   • tool_message
   • tool_message
5. LLM will return the error below

   
   ---> Azure.RequestFailedException: An assistant message with 'tool_calls' must be followed by tool messages responding to each 'tool_call_id'. The following tool_call_ids did not have response messages: call_iE1vjQZ6NXoB3qJC0q2wtgAE, call_JZ7RjkHTMVjXrGw0hz5MjAHM
   Status: 400 (model_error)

Content: { "error": { "message": "An assistant message with 'tool_calls' must be followed by tool messages responding to each 'tool_call_id'. The following tool_call_ids did not have response messages: call_iE1vjQZ6NXoB3qJC0q2wtgAE, call_JZ7RjkHTMVjXrGw0hz5MjAHM", "type": "invalid_request_error", "param": "messages.[7].role", "code": null } }

**Expected behavior**
Semantic kernel should send all tool responses in sequence, and just after that add any specific user message related to tool failure. This could be achieved by initially aggregating all user messages related to tool failures, and add them to the history only after the function responses loop

**Platform**
 - OS: Windows
 - IDE: Visual Studio
 - Language: C#
 - Source: 1.8.0

**Additional context**
- We are using `FunctionCallingStepwisePlanner`
- I know we are not using the latest version, but I checked it and the problem seems to be there as well.

[dmytrostruk]

@franklinlindemberg Thanks for reporting this issue!

Could you please try to use automatic function calling instead of FunctionCallingStepwisePlanner and see if you can reproduce the problem? We recommend automatic function calling for planning capabilities since it's more efficient approach in terms of performance and cost.

Here is more information:

• https://devblogs.microsoft.com/semantic-kernel/planning-with-semantic-kernel-using-automatic-function-calling/
• https://github.com/microsoft/semantic-kernel/blob/main/dotnet/samples/Concepts/Planners/AutoFunctionCallingPlanning.cs
• https://github.com/microsoft/semantic-kernel/blob/main/dotnet/samples/Demos/StepwisePlannerMigration/README.md

[franklinlindemberg]

@dmytrostruk Thanks for the reply!

I'm gonna try using the automatic function calling and will let you know!

Also just wanted to clarify that this is not really a blocker for us (the try catch on each tool implementation kinda fixed it) but I decided to open the issue anyway since I had already debugged this and had all the details (it might be helpful in case other folks face the same issue)

[sergey-ko]

@franklinlindemberg thanks for identifying the cause!

@dmytrostruk i can confirm it for ToolCallBehavior = ToolCallBehavior.AutoInvokeKernelFunctions

[dmytrostruk]

@franklinlindemberg It would be great to learn about your experience using automatic function calling and if you can reproduce this issue with it. As for now, I'm going to close the issue, since we no longer recommend using FunctionCallingStepwisePlanner and suggest using automatic function calling instead. But in case you can still reproduce it with automatic function calling - feel free to re-open this issue or create a new one. Thanks a lot!