handleToolEnd not called when using a tool calling agent with ChatAnthropic and Sonnet 3.5

zacharyfmarion commented 2 months ago

Checked other resources

[X] I added a very descriptive title to this issue.
[X] I searched the LangChain.js documentation with the integrated search.
[X] I used the GitHub search to find a similar question and didn't find it.
[X] I am sure that this is a bug in LangChain.js rather than my code.
[X] The bug is not resolved by updating to the latest stable version of LangChain (or the specific integration package).

Example Code

/* eslint-disable no-console */
import { ChatAnthropic } from '@langchain/anthropic';
import { Calculator } from '@langchain/community/tools/calculator';
import { HumanMessage } from '@langchain/core/messages';
import { ChatPromptTemplate, MessagesPlaceholder } from '@langchain/core/prompts';
import { AgentExecutor, createToolCallingAgent } from 'langchain/agents';

const model = new ChatAnthropic({
  model: 'claude-3-5-sonnet-20240620',
  temperature: 0,
});
const tools = [new Calculator()];

async function main() {
  const prompt = ChatPromptTemplate.fromMessages([
    ['system', 'You are a math phd student'],
    new MessagesPlaceholder('messages'),
    new MessagesPlaceholder('agent_scratchpad'),
  ]);

  const agent = createToolCallingAgent({
    llm: model,
    tools,
    prompt,
  });

  const executor = new AgentExecutor({
    agent,
    tools,
    callbacks: [
      {
        handleAgentAction() {
          console.log('handleAgentAction');
        },
        handleAgentEnd() {
          console.log('handleAgentEnd');
        },
        handleToolEnd() {
          console.log('handleToolEnd');
        },
      },
    ],
  });

  await executor.invoke({
    messages: [new HumanMessage({ content: 'What is 2 to the power of 4' })],
  });
}

main();

Error Message and Stack Trace (if applicable)

No response

Description

When using createToolCallingAgent with ChatAnthropic and passing in callbacks through an AgentExecutor the handleToolEnd callback does not get executed. Because of this I am not able to track token usage for the created agent.

Current Behavior

When run only two log statements appear:

handleAgentAction
handleAgentEnd

Expected Behavior

I would expect handleToolEnd to also be logged.

System Info

└─ langchain@npm:0.2.8::__archiveUrl=https%3A%2F%2F<redacted>%3A8081%2Fartifactory%2Fapi%2Fnpm%2F<redacted>%2Flangchain%2F-%2Flangchain-0.2.8.tgz
   ├─ Instances: 1
   ├─ Version: 0.2.8
   │ 
   └─ Dependencies
      ├─ @langchain/core@npm:>=0.2.9 <0.3.0 → npm:0.2.14::__archiveUrl=https%3A%2F%2F<redacted>%3A8081%2Fartifactory%2Fapi%2Fnpm%2F<redacted>%2F%40langchain%2Fcore%2F-%2Fcore-0.2.14.tgz
      ├─ @langchain/openai@npm:>=0.1.0 <0.3.0 → npm:0.2.1::__archiveUrl=https%3A%2F%2F<redacted>%3A8081%2Fartifactory%2Fapi%2Fnpm%2F<redacted>%2F%40langchain%2Fopenai%2F-%2Fopenai-0.2.1.tgz
      ├─ @langchain/textsplitters@npm:~0.0.0 → npm:0.0.3::__archiveUrl=https%3A%2F%2F<redacted>%3A8081%2Fartifactory%2Fapi%2Fnpm%2F<redacted>%2F%40langchain%2Ftextsplitters%2F-%2Ftextsplitters-0.0.3.tgz
      ├─ binary-extensions@npm:^2.2.0 → npm:2.3.0::__archiveUrl=https%3A%2F%2F<redacted>%3A8081%2Fartifactory%2Fapi%2Fnpm%2F<redacted>%2Fbinary-extensions%2F-%2Fbinary-extensions-2.3.0.tgz
      ├─ js-tiktoken@npm:^1.0.12 → npm:1.0.12::__archiveUrl=https%3A%2F%2F<redacted>%3A8081%2Fartifactory%2Fapi%2Fnpm%2F<redacted>%2Fjs-tiktoken%2F-%2Fjs-tiktoken-1.0.12.tgz
      ├─ js-yaml@npm:^4.1.0 → npm:4.1.0::__archiveUrl=https%3A%2F%2F<redacted>%3A8081%2Fartifactory%2Fapi%2Fnpm%2F<redacted>%2Fjs-yaml%2F-%2Fjs-yaml-4.1.0.tgz
      ├─ jsonpointer@npm:^5.0.1 → npm:5.0.1::__archiveUrl=https%3A%2F%2F<redacted>%3A8081%2Fartifactory%2Fapi%2Fnpm%2F<redacted>%2Fjsonpointer%2F-%2Fjsonpointer-5.0.1.tgz
      ├─ langchainhub@npm:~0.0.8 → npm:0.0.11::__archiveUrl=https%3A%2F%2F<redacted>%3A8081%2Fartifactory%2Fapi%2Fnpm%2F<redacted>%2Flangchainhub%2F-%2Flangchainhub-0.0.11.tgz
      ├─ langsmith@npm:~0.1.30 → npm:0.1.36::__archiveUrl=https%3A%2F%2F<redacted>%3A8081%2Fartifactory%2Fapi%2Fnpm%2F<redacted>%2Flangsmith%2F-%2Flangsmith-0.1.36.tgz
      ├─ ml-distance@npm:^4.0.0 → npm:4.0.1::__archiveUrl=https%3A%2F%2F<redacted>%3A8081%2Fartifactory%2Fapi%2Fnpm%2F<redacted>%2Fml-distance%2F-%2Fml-distance-4.0.1.tgz
      ├─ openapi-types@npm:^12.1.3 → npm:12.1.3::__archiveUrl=https%3A%2F%2F<redacted>%3A8081%2Fartifactory%2Fapi%2Fnpm%2F<redacted>%2Fopenapi-types%2F-%2Fopenapi-types-12.1.3.tgz
      ├─ p-retry@npm:4 → npm:4.6.2::__archiveUrl=https%3A%2F%2F<redacted>%3A8081%2Fartifactory%2Fapi%2Fnpm%2F<redacted>%2Fp-retry%2F-%2Fp-retry-4.6.2.tgz
      ├─ uuid@npm:^9.0.0 → npm:9.0.1::__archiveUrl=https%3A%2F%2F<redacted>%3A8081%2Fartifactory%2Fapi%2Fnpm%2F<redacted>%2Fuuid%2F-%2Fuuid-9.0.1.tgz
      ├─ yaml@npm:^2.2.1 → npm:2.4.5::__archiveUrl=https%3A%2F%2F<redacted>%3A8081%2Fartifactory%2Fapi%2Fnpm%2F<redacted>%2Fyaml%2F-%2Fyaml-2.4.5.tgz
      ├─ zod-to-json-schema@npm:^3.22.3 → npm:3.23.1::__archiveUrl=https%3A%2F%2F<redacted>%3A8081%2Fartifactory%2Fapi%2Fnpm%2F<redacted>%2Fzod-to-json-schema%2F-%2Fzod-to-json-schema-3.23.1.tgz
      └─ zod@npm:^3.22.4 → npm:3.23.8::__archiveUrl=https%3A%2F%2F<redacted>%3A8081%2Fartifactory%2Fapi%2Fnpm%2F<redacted>%2Fzod%2F-%2Fzod-3.23.8.tgz

Plaform: mac Node version: v20.15.0 Yarn version: 4.0.2

dosubot[bot] commented 2 months ago

Hi @zacharyfmarion! I'm here to help you with any issues you have. Let me know how I can assist you!

The handleToolEnd callback is not being called when using createToolCallingAgent with ChatAnthropic and AgentExecutor because the handleAgentAction method is incorrectly incrementing toolStarts instead of agentStarts. This causes a mismatch in the expected counts for tool starts and ends.

In your code, the handleAgentAction method should be implemented as follows:

async handleAgentAction(_action: AgentAction): Promise<void> {
    this.starts += 1;
    this.agentStarts += 1;
}

This discrepancy leads to the toolEnds count not being incremented correctly, which is why the handleToolEnd callback is not called as expected ^[1].

To continue talking to Dosu, mention @dosu.

zacharyfmarion commented 2 months ago

@dosu that is a test file, the issue must be somewhere in the code where handleToolEnd is called for an agent

dosubot[bot] commented 2 months ago

The handleToolEnd function is called within the executor.invoke method when using createToolCallingAgent with ChatAnthropic and AgentExecutor. This is evident from the provided code snippet where executor.invoke is called with an input and a set of callbacks, including handleToolEnd ^[1].

Additionally, the handleToolEnd method is called within the _call method of the AgentExecutor class. Here is the relevant code snippet:

while (this.shouldContinue(iterations)) {
  let output;
  try {
    output = await this.agent.plan(
      steps,
      inputs,
      runManager?.getChild(),
      config
    );
  } catch (e) {
    // eslint-disable-next-line no-instanceof/no-instanceof
    if (e instanceof OutputParserException) {
      let observation;
      let text = e.message;
      if (this.handleParsingErrors === true) {
        if (e.sendToLLM) {
          observation = e.observation;
          text = e.llmOutput ?? "";
        } else {
          observation = "Invalid or incomplete response";
        }
      } else if (typeof this.handleParsingErrors === "string") {
        observation = this.handleParsingErrors;
      } else if (typeof this.handleParsingErrors === "function") {
        observation = this.handleParsingErrors(e);
      } else {
        throw e;
      }
      output = {
        tool: "_Exception",
        toolInput: observation,
        log: text,
      } as AgentAction;
    } else {
      throw e;
    }
  }
  // Check if the agent has finished
  if ("returnValues" in output) {
    return getOutput(output);
  }

  let actions: AgentAction[];
  if (Array.isArray(output)) {
    actions = output as AgentAction[];
  } else {
    actions = [output as AgentAction];
  }
  // Call handleToolEnd here
  await this.handleToolEnd(actions, runManager);
}

The handleToolEnd method is called within the while loop in the _call method, specifically after determining the actions array ^[2].

To continue talking to Dosu, mention @dosu.

langchain-ai / langchainjs