microsoft / teams-ai

SDK focused on building AI based applications and extensions for Microsoft Teams and other Bot Framework channels
MIT License
408 stars 177 forks source link

[Dev support]: Using StopCommandName with Tools augmentation #2104

Open Benjiiim opened 4 hours ago

Benjiiim commented 4 hours ago

When using tools augmentation (function calling), the bot crashes with the following error message when StopCommandName has been used in a function during the previous turn:

400 An assistant message with 'tool_calls' must be followed by tool messages responding to each 'tool_call_id'. The following tool_call_ids did not have response messages: call_xxxxxxxxx

call_xxxxxxxxx is the tool_call_id corresponding to the function call which used StopCommandName.

When analyzing the chat prompt in the logs, we can see that we are indeed missing the object with role "tool" and the corresponding action_call_id.

Is that expected? How can we workaround this?

Thanks

corinagum commented 4 hours ago

The StopCommand is intended to abort further calls to the LLM, so this behavior is expected. The workaround would be to not use StopCommand in the action that becomes the tool call.

From AI.ts

Image

If StopCommand is detected, it fully aborts the execution of the planner.

Benjiiim commented 4 hours ago

Thanks for confirming @corinagum With tools augmentation, is there an other way to abort further calls to the LLM in the current turn, without affecting the execution of the next turn?

To give more context, I'm using the StopCommand when I'm crafting a card inside a function to display the expected result with sendActivity(). I don't want the LLM to send an other response to the user after that.

corinagum commented 3 hours ago

Based off of how server-side tools work, to me the use of the stop command and tools are in direct contradiction to each other. Executing tools is part of the current turn when the tool_call message is sent by the LLM.

Take a look at the cardGazer sample. I ran into a similar issue where AI was responding despite having manually sent the card. Adding a link to the Adaptive Card schema and improving the differentiation between commands (json sent) fixed the issue for me without ever needing stop command