Open YagnaDeepika opened 4 months ago
I think the custom speaker selection does not automatically choose the tool executor agent to speaker next when the previous message is a tool call. cc @yiranwu0 @qingyun-wu
For now you can encode the tool call logic in the custom speaker selection function. But a longer term goal should be to perform the tool call of each agent in a nested chat.
Yes, you need to determine which one to call when write your function. For example, check for tool calls in the messages and make sure you always select 'admin' if a tool call is found.
Btw, is this error message from 'admin' or other agents? ***** Response from calling tool (call_azvlwve54bGiqDDNjYn4nuUI) ***** Error: Function planner_function not found.
If is from another agent that you are not registered as executor, you should revise your customize function. If it is from admin, can you double check the tool call is in admin's. _function_map
?
Hi @yiranwu0 - thank you very much for your response. As you can see from the "steps to reproduce" section above, I have a planner_function
tool that I register with planner
agent as caller and admin
agent as executor with the following code:
autogen.agentchat.register_function(
planner_function,
caller=planner,
executor=admin,
name="planner_function",
description="Planner function that takes a user query and generates a plan of action to solve the task",
)
Similarly, I register the coder_function
tool with coder
agent as caller and admin
agent as executor. The admin.function_map
returns the following:
{'planner_function': <function main.planner_function(user_query: typing.Annotated[str, "user's natural language query"]) -> dict>, 'coder_function': <function main.coder_function(plan: typing.Annotated[dict, "Planner output. This is the output of the Planner Agent's 'planner_function' tool"]) -> dict>}
It seems that the tool calls are all correct.
@ekzhu - thank you very much for your response. Am I understanding correctly that tool use with custom speaker selection in group chat is currently not supported? As you can see from the example above, I have registered the tools with the appropriate agents so I'm trying to understand why this does not work. Incidentally, if I set speaker_selection_method = auto
, the exact same code works perfectly and it is able to find all the registered tools. So I don't think there is any problem with the way the tools are registered. The issue seems to be something to do specifically with the speaker_selection_method=simplified_custom_speaker_selection_func
as defined above in the "steps to reproduce" section.
Thanks again for your time and responses. I look forward to understanding what the reason for this issue is
@YagnaDeepika yes you are correct that currently the logic of choosing the next agent to execute tools must be encoded inside your custom speaker selection function. Otherwise, please use auto
. You can also use constrained speaker selections which is compatible with auto
.
@ekzhu - thank you for your response. This is helpful. Can this "group chat with custom speaker selection with each agent using tools" support be submitted as a feature request?
This could be a useful feature. @joshkyh @marklysze @yiranwu0 @sonichi @freedeaths what do you think? I think the feature would be to bypass the custom speaker selection method when a tool call message is received.
Hi @ekzhu - in the use case that I have, I would like more control over the speaker selection through the custom speaker selection function option, and I want each agent to use tools available to them in a group chat conversation pattern. So for this use case, it's key to retain the custom speaker selection option but add tool support to it (so I would not want to bypass the custom speaker selection method). I hope that makes sense. You can take a look at the "Steps to reproduce" section above for a simplified example of this pattern. Thanks!
@YagnaDeepika You can write
if "tool_calls" in messages[-1]:
return 'auto' # to go back to the "auto" selection
or
if "tool_calls" in messages[-1]:
return admin
The first one is to use "auto" if needed. The second one is to return the admin directly.
Basically, if you return a class "Agent" object, that will be taken as the next speaker. But you can still choose the default selection methods by returning a string.
This could be a useful feature. @joshkyh @marklysze @yiranwu0 @sonichi @freedeaths what do you think? I think the feature would be to bypass the custom speaker selection method when a tool call message is received.
Hmm, currently when a tool call message is received, fallback to auto is possible, but the user needs to specify it in the function (Yiran's reply). Making the fallback automatic sounds like a tradeoff between precise control (from the user specified function) and improved user experience (from fallback). Another perspective is that User specified function currently overrides graph constraints, and this is considering tool call fallback to override user specified function. Could be fine, just raising awareness.
So for this use case, it's key to retain the custom speaker selection option but add tool support to it (so I would not want to bypass the custom speaker selection method). I hope that makes sense.
@YagnaDeepika I am not suggesting bypassing the custom speaker selection method all the time. I meant to bypass custom speaker selection only when there is a tool call message -- effectively what @yiranwu0 suggested previously and move that code from your custom speaker selection method into the GroupChat itself.
Making the fallback automatic sounds like a tradeoff between precise control (from the user specified function) and improved user experience (from fallback).
@joshkyh @yiranwu0 Thanks for the inputs. Is there a case when we don't want to trigger tool execution when a tool call message is suggested?
I guess we can first document this better mentioning specifically the case of tool call messages.
@ekzhu I guess it's more of when the automatic fallback kicks in and overrides the custom speaker selection function, is it what the user wanted...? Suppose there are 3 agents each holding a different but overlapping set of tools, the custom selection function precisely returned agent A, but if we fallback automatically to auto triggered by the detection of tool_call
, there is a risk that the llm picks agent b and c. Given the OP's initial question, perhaps if the tool is found in Agent A (returned by custom selection function) then disable automatic fallback?
On the other hand, I can also see the benefit of another use case where the user forgot to specify the tool call fallback, and the automatic fallback is useful. I guess this is a trade off.
This could be a useful feature. @joshkyh @marklysze @yiranwu0 @sonichi @freedeaths what do you think? I think the feature would be to bypass the custom speaker selection method when a tool call message is received.
I'm not overly familiar with the exact steps taken when a tool call message is received, however this suggested feature sounds useful. I am, myself, not clear how to best control the agent to a received tool call message and having a simple way to do that would help. I'd be happy to test with non-Open AI models any changes here as there are challenges in mixing tool calling and non-tool calling agents in a group chat with them.
Great points. One potential way to do this is to remove the concept of tool call agents and replace it with agents that self-executing tools so that we don't have these problems.
Describe the issue
I am implementing a multi-agent framework for Planner-Coder-CoderCritic-Executor-Admin agents using GroupChat. Each of these agents calls a tool (custom function) that I define for a specific code generation scenario. To have more control over the orchestration, I tried implementing the custom speaker selection method as outlined in this sample notebook. The difference between this sample notebook and my case is that I have to call custom functions with each of the agents.
I was able to get GroupChat with these agents (and tools) working by using the default speaker selection method. Since
llm_config
parameter in GroupChatManager cannot take tools or functions, I had to add the following code snippet to get the tools to work with GroupChatManager:However, when I add
speaker_selection_method=custom_speaker_selection_func
, I get the following error:***** Response from calling tool (call_azvlwve54bGiqDDNjYn4nuUI) ***** Error: Function planner_function not found.
Note thatplanner_function
is the name of the function that I registered with the Planner agent as caller and Admin agent as executor (user proxy). It is not able to find any of the tools registered with the agents, when I introduce thecustom_speaker_selection_func
.Steps to reproduce
I created a simple version of this issue below, with dummy Planner, Coder agents and an Admin agent.