'openai-functions' Agent is randomly stuck after a tool has completed its function

[emmanuelvisage]

We have an usual bug that is happening randomly. Sometimes our 'openai-functions' agent get stuck after a tool has finished to run. The next stage is never properly started and executed so eventually the agent times out. Every phase of the agent process seems to be working very fast but after a tool is called, the agent never takes over and it stays stuck until the timeout is reached.

    const executor = await this.initializeAgentExecutorWithOptions(
      [
        new Tool1({
          foo
        }),
        new Tool2({
          bar,
        })
      ],
      new ChatOpenAI({
        openAIApiKey: <apiKey>,
        temperature: 0,
        streaming: true,
        modelName: 'gpt-4-1106-preview',
      }),
      {
        agentType: 'openai-functions',
        memory: new BufferWindowMemory({
          memoryKey: 'chat_history',
          inputKey: 'input',
          chatHistory: new RedisChatMessageHistory({
            sessionId: sessionId, 
            sessionTTL: 3000000,
            config: config.redis,
          }),
          returnMessages: true,
          k: 3,
        }),
        verbose: false,
        agentArgs: {
          prefix: prompt,
        },
      },
    );

    const response = await executor.call(
      {
        input: `${this.delimiter}\n${message}\n${this.delimiter}`,
        delimiter: this.delimiter,
        timeout: ONE_MINUTE,
        additional_context: JSON.stringify(additional_context_object),
      },
      [
        {
          handleLLMNewToken: onResponseToken, //send token to websocket
        },
      ],
    );

Do you know what could be causing this ?

[dosubot[bot]]

🤖

Hello @emmanuelvisage, good to see you again!

I'm diving into the issue you've reported with the 'openai-functions' agent. I'll be back with a full answer shortly. Thanks for your patience.

[dosubot[bot]]

🤖

Based on the code and the issue you've described, it seems like the 'openai-functions' agent is getting stuck after a tool has finished executing its function. This could be due to a variety of reasons, and without more specific information or the ability to debug the code, it's hard to provide a definitive answer. However, here are a few possibilities:

1. Network Issues: If the agent is making network requests (such as calling the OpenAI API), network issues could cause the agent to hang and eventually time out.

2. Tool Execution Time: If the tools (like Tool1 or Tool2 in your code) take a long time to execute their functions, it could cause the agent to time out.

3. OpenAI API Limitations: If you're hitting the rate limits of the OpenAI API, it could cause the agent to time out.

4. Code Errors: If there's an error in the code that's not being properly handled, it could cause the agent to hang and eventually time out.

5. Memory Limitations: The 'openai-functions' agent uses different types of memory based on the 'rememberIntermediateSteps' option. If the memory used (either OpenAIAgentTokenBufferMemory or ConversationSummaryBufferMemory) is not sufficient or is getting filled up, it could cause the agent to hang or time out.

6. Timeout Value: In your code, the timeout value is set to ONE_MINUTE. If the execution of the tools or the OpenAI model takes longer than this, it could cause the agent to time out. You might want to consider increasing this value if the operations being performed are expected to take longer.

To help identify where the agent is getting stuck, you might want to add more logging to your code. For example, you could log the start and end of each tool execution, as well as the start and end of the agent execution. This could help you identify if the issue is with a specific tool or with the agent itself.

I hope this helps! If you have more information or if there's anything else you'd like to know, please let me know.

Sources

• langchain/src/agents/tests/openai_tools.int.test.ts
• examples/src/agents/agent_timeout.ts
• langchain/src/util/tests/openai-stream.test.ts
• langchain/src/agents/toolkits/conversational_retrieval/openai_functions.ts
• langchain/src/llms/openai.ts
• langchain/src/chat_models/openai.ts
• langchain/src/agents/openai/index.ts
• langchain/src/llms/openai-chat.ts
• langchain/src/agents/initialize.ts

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[emmanuelvisage]

@dosu-bot Irrelevant, like I said, when I look at the logs, the tools are executed just fine, and they aren't taking long. Whatever is stuck happens between when the tool has completed and send the output of the function to the agent.

[dosubot[bot]]

🤖

Based on the code snippet and the issue you've described, it seems like the 'openai-functions' agent in your LangChain application is not transitioning properly between the completion of one tool's function and the initiation of the next stage. This could be due to a variety of reasons, including a bug in the executor's code or a specific configuration of your tools and chat model.

One common cause for such issues is if the tools or functions passed to the ChatOpenAI model are not implemented correctly or if they're not returning the expected output. Please ensure that these tools and functions are correctly implemented and that they're returning the expected output.

To troubleshoot this issue, you can add more detailed logging to your executor. This can help you identify exactly where the process is getting stuck. You can do this by modifying the initializeAgentExecutorWithOptions function to include more detailed logging:

const executor = await initializeAgentExecutorWithOptions(tools, chat, {
  agentType: "openai-functions",
  verbose: true,
  logLevel: 'debug', // Add this line
});

If the issue persists, you might need to look into the implementation of your tools. Ensure that each tool's function is properly returning a value and not causing any errors. For example, you can add error handling to your tools' functions:

class SerpAPI {
  // ...
  search() {
    try {
      // Your search logic here
    } catch (error) {
      console.error('Error in SerpAPI:', error);
    }
  }
}

If none of these solutions work, it might be a bug in the LangChain framework itself. In this case, you should report the issue to the LangChain team for further assistance.

For more information, you can refer to the following sources:

• examples/src/agents/openai.ts
• examples/src/chains/openai_functions_structured_format.ts
• examples/src/models/chat/integration_openai.ts
• examples/src/models/chat/integration_openai_tool_calls.ts

I hope this helps! If you have any more questions or need further clarification, feel free to ask.

Sources

• examples/src/agents/openai.ts
• examples/src/chains/openai_functions_structured_format.ts
• examples/src/models/chat/integration_openai.ts
• examples/src/models/chat/integration_openai_tool_calls.ts
• langchain/src/chat_models/openai.ts

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[dosubot[bot]]

🤖

Based on the code snippet and the issue you've described, it seems like the 'openai-functions' agent is not transitioning properly between the completion of one tool's function and the initiation of the next stage. This could be due to a variety of reasons, including a bug in the executor's code or a specific configuration of your tools and chat model.

One common cause of this issue is if the tools or functions you're passing to the ChatOpenAI model are not implemented correctly or if they're not returning the expected output. Please ensure that these tools and functions are correctly implemented and that they're returning the expected output.

To troubleshoot this issue, you can add more detailed logging to your executor. This can help you identify exactly where the process is getting stuck. You can do this by modifying the initializeAgentExecutorWithOptions function to include more detailed logging:

const executor = await initializeAgentExecutorWithOptions(tools, chat, {
  agentType: "openai-functions",
  verbose: true,
  logLevel: 'debug', // Add this line
});

If the issue persists, you might need to look into the implementation of your tools. Ensure that each tool's function is properly returning a value and not causing any errors. For example, you can add error handling to your tools' functions:

class SerpAPI {
  // ...
  search() {
    try {
      // Your search logic here
    } catch (error) {
      console.error('Error in SerpAPI:', error);
    }
  }
}

If none of these solutions work, it might be a bug in the LangChain framework itself. In this case, you should report the issue to the LangChain team for further assistance. You can refer to the following sources for more information:

• examples/src/agents/openai.ts
• examples/src/chains/openai_functions_structured_format.ts
• examples/src/models/chat/integration_openai.ts
• examples/src/models/chat/integration_openai_tool_calls.ts

I hope this helps! Please let me know if you have any other questions or if there's anything else I can assist you with.

Sources

• examples/src/agents/openai.ts
• examples/src/chains/openai_functions_structured_format.ts
• examples/src/models/chat/integration_openai.ts
• examples/src/models/chat/integration_openai_tool_calls.ts
• langchain/src/chat_models/openai.ts

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[emmanuelvisage]

For the langchain team, here is a proof of that behavior shown in langsmith

[emmanuelvisage]

CPU seems to be struggling as a result of this occuring.

[emmanuelvisage]

Still no solution

[albcm-simetrik]

Same here

[jsfaber]

Same here

[dosubot[bot]]

🤖

Hi, @emmanuelvisage

I'm helping the langchainjs team manage their backlog and am marking this issue as stale. From what I understand, the 'openai-functions' agent is getting stuck after a tool has completed its function, causing subsequent stages to not execute and eventually leading to a timeout. Despite the agent process working quickly, the issue persists. There have been discussions around potential reasons for the issue, including network issues, tool execution time, OpenAI API limitations, code errors, memory limitations, and timeout value. You mentioned that the tools are executed fine and the issue occurs between the completion of the tool and sending the output to the agent. Additionally, you shared screenshots showing CPU struggles due to the issue, and other users confirmed experiencing the same issue.

Could you please confirm if this issue is still relevant to the latest version of the langchainjs repository? If it is, please let the langchainjs team know by commenting on the issue. Otherwise, feel free to close the issue yourself, or the issue will be automatically closed in 7 days. Thank you!

[tomaszstoryful]

same issue, agent getting stuck after retriever action complete + output returned