AgentExecutor.fromAgentAndTools can never resolved

wulucxy commented 3 months ago

Checked other resources

[X] I added a very descriptive title to this issue.
[X] I searched the LangChain.js documentation with the integrated search.
[X] I used the GitHub search to find a similar question and didn't find it.
[X] I am sure that this is a bug in LangChain.js rather than my code.
[X] The bug is not resolved by updating to the latest stable version of LangChain (or the specific integration package).

Example Code

const prompt = await pull<ChatPromptTemplate>('hwchase17/openai-tools-agent');
  const agent = await createOpenAIToolsAgent({
    llm,
    tools,
    prompt,
  });
  const agentExecutor = AgentExecutor.fromAgentAndTools({
    agent,
    tools,
    verbose: true,
    // maxIterations: 2,
  });

  const result = await agentExecutor.invoke({
    input: task,
  });

Error Message and Stack Trace (if applicable)

No response

Description

https://github.com/langchain-ai/langchainjs/blob/db9e35242abbb77a1dbf5f8f84ad80d6ab55f819/langchain/src/agents/openai_tools/output_parser.ts#L75

 if (message.additional_kwargs.tool_calls) {
// ...

should be:

 if (message.additional_kwargs.tool_calls && message.additional_kwargs.tool_calls.length > 0) {

System Info

langchainjs: v0.2.6

dosubot[bot] commented 3 months ago

The issue with AgentExecutor.fromAgentAndTools never resolving could indeed be related to the condition at line 75 of the output_parser.ts file. Specifically, if the output from the LLM contains both a final answer and a parse-able action, the code throws an error and does not return a result, which could cause the method to hang indefinitely.

Here is the relevant condition:

if (actionMatch) {
  if (includesAnswer) {
    throw new Error(
      `${FINAL_ANSWER_AND_PARSABLE_ACTION_ERROR_MESSAGE}: ${text}`
    );
  }
  // ... rest of the code
}

If the LLM output includes both an action and a final answer, the error is thrown, and no result is returned, which could lead to the AgentExecutor never resolving ^[1]^[2].

Additionally, the ToolCallingAgentOutputParser class in output_parser.ts is designed to parse the output text from the agent and return either an AgentAction or AgentFinish object. If the text includes both </tool> and <final_answer>, it might not be handled correctly, leading to the unresolved state ^[3].

Moreover, the AgentExecutor class has a mechanism to handle errors raised by the agent's output parser. The handleParsingErrors property can be set to handle these errors in different ways, such as sending the error back to the LLM as an observation or using a custom function to handle the error. If this property is not set correctly, it could also contribute to the method never resolving ^[4].

To address the specific issue you mentioned, modifying the condition as follows could help:

if (message.additional_kwargs.tool_calls && message.additional_kwargs.tool_calls.length > 0) {
// ...
}

This ensures that the condition only checks for tool calls if they exist and have a length greater than zero, which might prevent the method from hanging indefinitely.

To continue talking to Dosu, mention @dosu.

bracesproul commented 2 months ago

For now, you should just use:

  const agentExecutor = new AgentExecutor({
    agent,
    tools,
    verbose: true,
    // maxIterations: 2,
  });

wulucxy commented 2 months ago

@bracesproul

https://github.com/langchain-ai/langchainjs/blob/db9e35242abbb77a1dbf5f8f84ad80d6ab55f819/langchain/src/agents/openai_tools/output_parser.ts#L75

I think the assertion of this line is not correct

if (message.additional_kwargs.tool_calls) {
 // ....
}

it should be

if (message.additional_kwargs.tool_calls && message.additional_kwargs.tool_calls.length > 0) {
// ...
}

wulucxy commented 2 months ago

For now, you should just use:

  const agentExecutor = new AgentExecutor({
    agent,
    tools,
    verbose: true,
    // maxIterations: 2,
  });

AgentExecutor

https://github.com/langchain-ai/langchainjs/blob/ef201d0ee85ee4049078270a0cfd7a1767e624f8/langchain/src/agents/executor.ts#L448

these two methods call just the same

djaffer commented 2 months ago

This agent never triggers the _call for the tool.

langchain-ai / langchainjs