Langchain multi-agents are not allowed to use return_direct=True

[TengHoo3]

Checked other resources

• [X] I added a very descriptive title to this issue.
• [X] I searched the LangChain documentation with the integrated search.
• [X] I used the GitHub search to find a similar question and didn't find it.
• [X] I am sure that this is a bug in LangChain rather than my code.
• [X] The bug is not resolved by updating to the latest stable version of LangChain (or the specific integration package).

Example Code

class TestInput(BaseModel):
    query: str = Field(description="description about user input query")
    test: str = Field(description="test")

class CustomTool(StructuredTool):
    name = "test_tool"
    description = """
        Testing
    """
    args_schema: Type[BaseModel] = TestInput
    return_direct: bool = True

Error Message and Stack Trace (if applicable)

14:24:03
[2024-04-01 06:24:03,102] [ERROR] [app.routers.query] [User ID: -] [Session ID: -] - Failed to get response UI 
01/04/2024
14:24:03
1 validation error for AgentExecutor
01/04/2024
14:24:03
__root__
01/04/2024
14:24:03
  Tools that have `return_direct=True` are not allowed in multi-action agents (type=value_error) - Traceback : 

Description

Currently trying to use the return_direct=True argument in Agents with multi-input tool. But it seems I am get the error above.

It seems like it is coming from this code line: langchain/libs/langchain/langchain/agents/agent.py

from langchain.agents import AgentExecutor

    @root_validator()
    def validate_return_direct_tool(cls, values: Dict) -> Dict:
        """Validate that tools are compatible with agent."""
        agent = values["agent"]
        tools = values["tools"]
        if isinstance(agent, BaseMultiActionAgent):
            for tool in tools:
                if tool.return_direct:
                    raise ValueError(
                        "Tools that have `return_direct=True` are not allowed "
                        "in multi-action agents"
                    )
        return values

Wondering why there is a validation for this? Cheers!

System Info

python=python:3.9.18 langchain=0.1.14

[owais-globalnorthstar]

@jacoblee93 @TengHoo3 This issue was resolved by langchain js community. Can we please get the solution of the above issue in langchain python too? Here is the URL: https://github.com/langchain-ai/langchainjs/discussions/4631

[owais-globalnorthstar]

@jacoblee93 Any update on this would be much appreciated. Thanks

[jacoblee93]

CC @baskaryan

[amittimalsina-kniru]

Really looking for a solution to this. cc: @jacoblee93 @TengHoo3 @baskaryan

[TengHoo3]

Should we just remove the validation? @baskaryan @jacoblee93

My logic would be if return_direct=True for that particular tool, we force the multi action agent to return the response directly - if not, then it is able to use multiple tools and continue its though process when tools where return_direct=False.

From my experience this will give us more control in reaching a particular tool and stopping the thought process there. Not sure if the the js implementation does the same and there is a better solution?

[amittimalsina-kniru]

@TengHoo3 I am using create_openai_tools_agent. Turns out this creates BaseMultiActionAgent. I have a tool, which asks questions to user. This tool has return_direct=True.

class AskClarifyingQuestion(BaseTool):
    name = "AskClarifyingQuestion"
    description = """Use this tool to ask clarifying questions needed for the user question."""
    args_schema: Type[BaseModel] = AskCarifyingQuestionInput

    def __init__(self):
        super().__init__()
        # self.return_direct returns the response of the tool directly to the user without sending to the agent.
        self.return_direct = True

    def _run(self, questions: list[Question]):
        return "Please answer the following questions:", [question.dict() for question in questions]

    async def _arun(self, questions: list[Question]):
        return "Please answer the following questions:", [question.dict() for question in questions]

All the other tools, i want to use tool calling as it can return a list of tools unlike function calling which is 1 at a time.

I was wondering if the problem will be solved by just removing the type check:

    @root_validator()
    def validate_return_direct_tool(cls, values: Dict) -> Dict:
        """Validate that tools are compatible with agent."""
        agent = values["agent"]
        tools = values["tools"]
        if isinstance(agent, BaseMultiActionAgent):
            for tool in tools:
                if tool.return_direct:
                    raise ValueError(
                        "Tools that have `return_direct=True` are not allowed "
                        "in multi-action agents"
                    )
        return values

Why was this added in the first place?

Does that make sense?

[omgseven]

Try chain with a function could avoid this.

def _do_nothing(data):
    return data

[TengHoo3]

@amittimalsina-kniru I have the same issue as you when using openai tools agent, my workaround was using another agent (I needed an agent that could handle multi-input tools so switching to structured chat agent did the trick for me) - potentially the same could be done for your use case. There was a tool_calling_agent that came out recently that could help - https://python.langchain.com/docs/modules/agents/agent_types/tool_calling/

You can try just removing the type check, that could work (I tried for a bit but realised switching agent was the better way for me).

[CRSiegfried]

Another vote to fix this. Sometimes I just need the results returned.

[piotr-piatkowski]

The problem is that AgentExecutor validator converts Runnable to RunnableMultiActionAgent - and by default it's the latter:

    @root_validator(pre=True)
    def validate_runnable_agent(cls, values: Dict) -> Dict:
        """Convert runnable to agent if passed in."""
        agent = values["agent"]
        if isinstance(agent, Runnable):
            try:
                output_type = agent.OutputType
            except Exception as _:
                multi_action = False
            else:
                multi_action = output_type == Union[List[AgentAction], AgentFinish]

            stream_runnable = values.pop("stream_runnable", True)
            if multi_action:
                values["agent"] = RunnableMultiActionAgent(
                    runnable=agent, stream_runnable=stream_runnable
                )
            else:
                values["agent"] = RunnableAgent(
                    runnable=agent, stream_runnable=stream_runnable
                )
        return values

Also create_openai_tools_agent returns just Runnable - and then it's converted to MultiActionAgent by the above validator. To avoid this (as a workaround) you can force SingleActionAgent by this:

        from langchain.agents.agent import RunnableAgent
        runnable = create_openai_tools_agent(llm, tools, prompt)
        agent = RunnableAgent(runnable=runnable)
        executor = AgentExecutor(agent=agent, tools=tools)

But that's still a workaround, and enforces using SingleActionAgent - I'm not sure if return_direct could really be used by multi-action agent.

[araujobma]

The answer from @piotr-piatkowski worked but the following error occurs Error in RootListenersTracer.on_chain_end callback: ValueError(). And no chat history is saved when the tool returns a result. here is the code I implemented:

def create_chat_agent():
    """
    Create a chat agent.
    """

    runnable = create_tool_calling_agent(chat_model, tools, chat_prompt)
    agent = RunnableAgent(runnable=runnable)
    agent_executor = AgentExecutor(
        agent=agent,
        tools=tools,
        verbose=True,
        max_iterations=5,
        max_execution_time=20,
    )

    agent_with_chat_history = RunnableWithMessageHistory(
        agent_executor,
        lambda session_id: TruncatedRedisHistory(session_id),
        input_messages_key="input",
        history_messages_key="chat_history",
    )

    return agent_with_chat_history