streamEvents() with v2 schema and ChatAnthropic

[lionkeng]

I am trying out the streaming example provided in your documentation here with v2 schema. I got it to work based on suggestion provided by @jacoblee93 here

The code works when I use ChatOpenAI as the model. But it is not working when I swap the model with ChatAnthropic. What am I missing? Thank you.

import {
  HumanMessage,
  AIMessage,
  AIMessageChunk,
} from '@langchain/core/messages'
import { DynamicStructuredTool } from '@langchain/core/tools'
import { z } from 'zod'
import { ChatAnthropic } from '@langchain/anthropic'
import { ChatOpenAI } from '@langchain/openai'
import { END, START, StateGraph, StateGraphArgs } from '@langchain/langgraph'
import { MemorySaver } from '@langchain/langgraph'
import { ToolNode } from '@langchain/langgraph/prebuilt'
import { TavilySearchResults } from '@langchain/community/tools/tavily_search'

// Define the state interface
interface AgentState {
  messages: HumanMessage[]
}

// Define the graph state
const graphState: StateGraphArgs<AgentState>['channels'] = {
  messages: {
    value: (x: HumanMessage[], y: HumanMessage[]) => x.concat(y),
    default: () => [],
  },
}

// Define the tools for the agent to use

const searchTool = new DynamicStructuredTool({
  name: 'search',
  description: 'Call to surf the web.',
  schema: z.object({
    query: z.string().describe('The query to use in your search.'),
  }),
  func: async ({ query }: { query: string }) => {
    // This is a placeholder for the actual implementation
    if (
      query.toLowerCase().includes('sf') ||
      query.toLowerCase().includes('san francisco')
    ) {
      return "It's 60 degrees and foggy."
    }
    return "It's 90 degrees and sunny."
  },
})

const tools = [new TavilySearchResults({ maxResults: 1 })]
const toolNode = new ToolNode<AgentState>(tools)

const model = new ChatAnthropic({
  model: 'claude-3-sonnet-20240229',
  temperature: 0,
  streaming: true,
}).bindTools(tools)
// const model = new ChatOpenAI({
//   model: 'gpt-4o',
//   temperature: 0,
//   streaming: true,
// }).bindTools(tools)

// Define the function that determines whether to continue or not
function shouldContinue(state: AgentState): 'tools' | typeof END {
  const messages = state.messages
  const lastMessage = messages[messages.length - 1] as AIMessage

  // If the LLM makes a tool call, then we route to the "tools" node
  // if (lastMessage.additional_kwargs.tool_calls) {
  if (lastMessage.tool_calls?.length) {
    return 'tools'
  }
  // Otherwise, we stop (reply to the user)
  return END
}

// Define the function that calls the model
async function callModel(state: AgentState) {
  const messages = state.messages
  const response = await model.invoke(messages)

  // We return a list, because this will get added to the existing list
  return { messages: [response] }
}

// Define a new graph
const workflow = new StateGraph<AgentState>({ channels: graphState })
  .addNode('agent', callModel)
  .addNode('tools', toolNode)
  .addEdge(START, 'agent')
  .addConditionalEdges('agent', shouldContinue)
  .addEdge('tools', 'agent')

// Initialize memory to persist state between graph runs
const checkpointer = new MemorySaver()

// Finally, we compile it!
// This compiles it into a LangChain Runnable.
// Note that we're (optionally) passing the memory when compiling the graph
const app = workflow.compile({ checkpointer })

async function main() {
  let config = { configurable: { thread_id: 'conversation-num-1' } }
  let inputs = { messages: [new HumanMessage('Hello. My name is Joe')] }
  for await (const event of await app.streamEvents(inputs, {
    ...config,
    streamMode: 'values',
    version: 'v2',
  })) {
    console.log('event', event.event)
    if (event.event === 'on_chat_model_stream') {
      let msg = event.data?.chunk as AIMessageChunk
      if (msg.tool_call_chunks && msg.tool_call_chunks.length > 0) {
        console.log(msg.tool_call_chunks)
      } else {
        console.log(msg.content)
      }
    }
  }
}

main()

[hwchase17]

when you say "not working" - what exactly is happening?

[bracesproul]

+1 to Harrison's comment.

If you're referencing not being able to access token counts when streaming tool calls, I just pushed a fix here. I also see some other issues with your code. For extracting tool calls from messages, you should be looking inside message.tool_calls not message.additional_kwargs.tool_calls. If there's another bug, please let me know what the issue is and I can work it out. Thanks!

[lionkeng]

when you say "not working" - what exactly is happening? @hwchase17 When I run it with ChatAnthropic, streamEvents never seem to generate any event.event with on_chat_model_stream.

@bracesproul thank you! I see that too. But I seem to be running into something more fundamental.

[bracesproul]

@lionkeng what exactly is the issue? If you could provide a stack trace and an in depth description of the issue I can help fix it.

[lionkeng]

@lionkeng what exactly is the issue? If you could provide a stack trace and an in depth description of the issue I can help fix it.

@bracesproul There is no stack trace per-se as the program/script just runs but doesn't encounter any event named on_chat_model_stream. So if I run the script as shown above, the console.log will output all of the following but missing on_chat_model_stream:

event on_chain_start
event on_chain_start
event on_chain_end
event on_chain_stream
event on_chain_start
event on_chain_start
event on_chat_model_start
event on_chat_model_end
event on_chain_end
event on_chain_start
event on_chain_end
event on_chain_start
event on_chain_end
event on_chain_end
event on_chain_stream
event on_chain_end

[bracesproul]

Okay just dug into this, we don't support streaming tool calls with anthropic in the way you'd think. So yes you can call model.stream and it'll work, but under the hood of the ChatAnthropic class we're calling the non streaming method if there are tool calls. So because of this, streamEvents won't yield that event.

This is a top priority for us though, and we're aiming to get this implemented by midway through next week.

[lionkeng]

Thanks for the response @bracesproul ! Looking forwarding to testing out the changes.

[danny-avila]

Still having streaming issues with ChatAnthropic using streamEvents and v2 in particular.

Here are my langchain-specific dependencies from my project package.json:

"dependencies": {
  "@langchain/anthropic": "^0.2.8",
  "@langchain/aws": "^0.0.5",
  "@langchain/community": "^0.2.20",
  "@langchain/core": "^0.2.18",
  "@langchain/google-vertexai": "^0.0.20",
  "@langchain/langgraph": "^0.0.31",
  "@langchain/mistralai": "^0.0.26",
  "langchain": "^0.2.10",
},
"resolutions": {
  "@langchain/core": "0.2.18",
},

Code to Reproduce:

Note: this correctly streams tokens when we don't bind tools, but if we do, it fails.

import { BaseMessage, HumanMessage } from "@langchain/core/messages";
import { ChatAnthropic } from "@langchain/anthropic";
import { ChatPromptTemplate } from "@langchain/core/prompts";
import { TavilySearchResults } from '@langchain/community/tools/tavily_search';
import { RunnableConfig } from "@langchain/core/runnables";
import {
  END,
  // MemorySaver,
  START,
  StateGraph,
  StateGraphArgs,
} from "@langchain/langgraph";

interface IState {
  messages: BaseMessage[];
  userInfo: string;
}

async function main() {

const graphState: StateGraphArgs<IState>["channels"] = {
  messages: {
    value: (x: BaseMessage[], y: BaseMessage[]) => x.concat(y),
    default: () => [],
  },
  userInfo: {
    value: (x?: string, y?: string) => {
      return y ? y : x ? x : "N/A";
    },
    default: () => "N/A",
  },
};

const promptTemplate = ChatPromptTemplate.fromMessages([
  ["system", "You are a helpful assistant.\n\n## User Info:\n{userInfo}"],
  ["placeholder", "{messages}"],
]);

const initializeModel = (bindTools = false) => {
  const model = new ChatAnthropic({
    // model: "claude-3-haiku-20240307",
    model: 'claude-3-5-sonnet-20240620'
  })
  if (bindTools) {
    return model.bindTools([new TavilySearchResults()]);
  }
  return model;
}

const callModel = async (
  state: { messages: BaseMessage[]; userInfo: string },
  config?: RunnableConfig,
) => {
  const { messages, userInfo } = state;

  /*

  This correctly streams tokens when 
  we don't bind tools, but if we do, it fails.

  */
  const model = initializeModel(true);
  // const model = initializeModel();
  const chain = promptTemplate.pipe(model);
  const response = await chain.invoke(
    {
      messages,
      userInfo,
    },
    config,
  );
  return { messages: [response] };
};

const fetchUserInformation = async (
  _: { messages: BaseMessage[] },
  config?: RunnableConfig,
) => {
  const userDB = {
    user1: {
      name: "John Doe",
      email: "jod@langchain.ai",
      phone: "+1234567890",
    },
    user2: {
      name: "Jane Doe",
      email: "jad@langchain.ai",
      phone: "+0987654321",
    },
  };
  const userId = config?.configurable?.user;
  if (userId) {
    const user = userDB[userId as keyof typeof userDB];
    if (user) {
      return {
        userInfo:
          `Name: ${user.name}\nEmail: ${user.email}\nPhone: ${user.phone}`,
      };
    }
  }
  return { userInfo: "N/A" };
};

const workflow = new StateGraph({
  channels: graphState,
})
  .addNode("fetchUserInfo", fetchUserInformation)
  .addNode("agent", callModel)
  .addEdge(START, "fetchUserInfo")
  .addEdge("fetchUserInfo", "agent")
  .addEdge("agent", END);

// Here we only save in-memory
// let memory = new MemorySaver();
// const graph = workflow.compile({ checkpointer: memory });
const graph = workflow.compile();

const config = {
  configurable: {
    user: "user1",
  },
};
const inputs = {
  messages: [new HumanMessage("Could you remind me of my email?")],
};

  const stream = graph.streamEvents(inputs, {
    ...config,
    version: "v2",
    streamMode: "values",
  });
for await (const { event, data } of stream) {

  if (event === "on_chat_model_start") {
    console.log('=======CHAT_MODEL_START=======');
    console.dir(data, { depth: null });
  } else if (event === "on_chat_model_stream") {
    console.dir(data, { depth: null });
  } else if (event === "on_chat_model_end") {
    console.log('=======CHAT_MODEL_END=======');
    console.dir(data, { depth: null });
  } else {
    console.log(event);
  }
}
}

main().catch((err) => {
  console.error(err);
  process.exit(1);
});

---

Output:

on_chain_start
on_chain_start
on_chain_end
on_chain_stream
on_chain_start
on_chain_start
on_chain_end
on_chain_start
on_chain_end
on_chain_end
on_chain_stream
on_chain_start
on_chain_start
on_chain_start
on_prompt_start
on_prompt_end
=======CHAT_MODEL_START=======
{
  input: {
    messages: [
      [
        SystemMessage {
          lc_serializable: true,
          lc_kwargs: {
            content: 'You are a helpful assistant.\n' +
              '\n' +
              '## User Info:\n' +
              'Name: John Doe\n' +
              'Email: jod@langchain.ai\n' +
              'Phone: +1234567890',
            additional_kwargs: {},
            response_metadata: {}
          },
          lc_namespace: [ 'langchain_core', 'messages' ],
          content: 'You are a helpful assistant.\n' +
            '\n' +
            '## User Info:\n' +
            'Name: John Doe\n' +
            'Email: jod@langchain.ai\n' +
            'Phone: +1234567890',
          name: undefined,
          additional_kwargs: {},
          response_metadata: {},
          id: undefined
        },
        HumanMessage {
          lc_serializable: true,
          lc_kwargs: {
            content: 'Could you remind me of my email?',
            additional_kwargs: {},
            response_metadata: {}
          },
          lc_namespace: [ 'langchain_core', 'messages' ],
          content: 'Could you remind me of my email?',
          name: undefined,
          additional_kwargs: {},
          response_metadata: {},
          id: undefined
        }
      ]
    ]
  }
}
=======CHAT_MODEL_END=======
{
  output: AIMessageChunk {
    lc_serializable: true,
    lc_kwargs: {
      content: [
        {
          index: 0,
          type: 'text_delta',
          text: "Certainly! I'd be happy to remind you of your email address. Based on the user information provided, your email address is:\n" +
            '\n' +
            'jod@langchain.ai\n' +
            '\n' +
            "Is there anything else you'd like to know about your contact information?"
        }
      ],
      additional_kwargs: {
        id: 'msg_01TZ28cmm6nPk7FsmRHjEDet',
        type: 'message',
        role: 'assistant',
        model: 'claude-3-5-sonnet-20240620',
        stop_reason: 'end_turn',
        stop_sequence: null,
        usage: { input_tokens: 452, output_tokens: 55 }
      },
      response_metadata: {},
      tool_call_chunks: [],
      id: 'msg_01TZ28cmm6nPk7FsmRHjEDet',
      usage_metadata: { input_tokens: 956, output_tokens: 58, total_tokens: 1014 },
      tool_calls: [],
      invalid_tool_calls: []
    },
    lc_namespace: [ 'langchain_core', 'messages' ],
    content: [
      {
        index: 0,
        type: 'text_delta',
        text: "Certainly! I'd be happy to remind you of your email address. Based on the user information provided, your email address is:\n" +
          '\n' +
          'jod@langchain.ai\n' +
          '\n' +
          "Is there anything else you'd like to know about your contact information?"
      }
    ],
    name: undefined,
    additional_kwargs: {
      id: 'msg_01TZ28cmm6nPk7FsmRHjEDet',
      type: 'message',
      role: 'assistant',
      model: 'claude-3-5-sonnet-20240620',
      stop_reason: 'end_turn',
      stop_sequence: null,
      usage: { input_tokens: 452, output_tokens: 55 }
    },
    response_metadata: {},
    id: 'msg_01TZ28cmm6nPk7FsmRHjEDet',
    tool_calls: [],
    invalid_tool_calls: [],
    tool_call_chunks: [],
    usage_metadata: { input_tokens: 956, output_tokens: 58, total_tokens: 1014 }
  },
  input: {
    messages: [
      [
        SystemMessage {
          lc_serializable: true,
          lc_kwargs: {
            content: 'You are a helpful assistant.\n' +
              '\n' +
              '## User Info:\n' +
              'Name: John Doe\n' +
              'Email: jod@langchain.ai\n' +
              'Phone: +1234567890',
            additional_kwargs: {},
            response_metadata: {}
          },
          lc_namespace: [ 'langchain_core', 'messages' ],
          content: 'You are a helpful assistant.\n' +
            '\n' +
            '## User Info:\n' +
            'Name: John Doe\n' +
            'Email: jod@langchain.ai\n' +
            'Phone: +1234567890',
          name: undefined,
          additional_kwargs: {},
          response_metadata: {},
          id: undefined
        },
        HumanMessage {
          lc_serializable: true,
          lc_kwargs: {
            content: 'Could you remind me of my email?',
            additional_kwargs: {},
            response_metadata: {}
          },
          lc_namespace: [ 'langchain_core', 'messages' ],
          content: 'Could you remind me of my email?',
          name: undefined,
          additional_kwargs: {},
          response_metadata: {},
          id: undefined
        }
      ]
    ]
  }
}
on_chain_end
on_chain_end
on_chain_start
on_chain_end
on_chain_end
on_chain_stream
on_chain_end

Output if model is not bound

on_chain_start
on_chain_start
on_chain_end
on_chain_stream
on_chain_start
on_chain_start
on_chain_end
on_chain_start
on_chain_end
on_chain_end
on_chain_stream
on_chain_start
on_chain_start
on_chain_start
on_prompt_start
on_prompt_end
=======CHAT_MODEL_START=======
{
  input: {
    messages: [
      [
        SystemMessage {
          lc_serializable: true,
          lc_kwargs: {
            content: 'You are a helpful assistant.\n' +
              '\n' +
              '## User Info:\n' +
              'Name: John Doe\n' +
              'Email: jod@langchain.ai\n' +
              'Phone: +1234567890',
            additional_kwargs: {},
            response_metadata: {}
          },
          lc_namespace: [ 'langchain_core', 'messages' ],
          content: 'You are a helpful assistant.\n' +
            '\n' +
            '## User Info:\n' +
            'Name: John Doe\n' +
            'Email: jod@langchain.ai\n' +
            'Phone: +1234567890',
          name: undefined,
          additional_kwargs: {},
          response_metadata: {},
          id: undefined
        },
        HumanMessage {
          lc_serializable: true,
          lc_kwargs: {
            content: 'Could you remind me of my email?',
            additional_kwargs: {},
            response_metadata: {}
          },
          lc_namespace: [ 'langchain_core', 'messages' ],
          content: 'Could you remind me of my email?',
          name: undefined,
          additional_kwargs: {},
          response_metadata: {},
          id: undefined
        }
      ]
    ]
  }
}
{
  chunk: AIMessageChunk {
    lc_serializable: true,
    lc_kwargs: {
      content: 'Certainly! Your email',
      tool_calls: [],
      invalid_tool_calls: [],
      tool_call_chunks: [],
      additional_kwargs: {},
      response_metadata: {}
    },
    lc_namespace: [ 'langchain_core', 'messages' ],
    content: 'Certainly! Your email',
    name: undefined,
    additional_kwargs: {},
    response_metadata: {},
    id: undefined,
    tool_calls: [],
    invalid_tool_calls: [],
    tool_call_chunks: [],
    usage_metadata: undefined
  }
}
{
  chunk: AIMessageChunk {
    lc_serializable: true,
    lc_kwargs: {
      content: ' address is jo',
      tool_calls: [],
      invalid_tool_calls: [],
      tool_call_chunks: [],
      additional_kwargs: {},
      response_metadata: {}
    },
    lc_namespace: [ 'langchain_core', 'messages' ],
    content: ' address is jo',
    name: undefined,
    additional_kwargs: {},
    response_metadata: {},
    id: undefined,
    tool_calls: [],
    invalid_tool_calls: [],
    tool_call_chunks: [],
    usage_metadata: undefined
  }
}
{
  chunk: AIMessageChunk {
    lc_serializable: true,
    lc_kwargs: {
      content: 'd@langchain.',
      tool_calls: [],
      invalid_tool_calls: [],
      tool_call_chunks: [],
      additional_kwargs: {},
      response_metadata: {}
    },
    lc_namespace: [ 'langchain_core', 'messages' ],
    content: 'd@langchain.',
    name: undefined,
    additional_kwargs: {},
    response_metadata: {},
    id: undefined,
    tool_calls: [],
    invalid_tool_calls: [],
    tool_call_chunks: [],
    usage_metadata: undefined
  }
}
{
  chunk: AIMessageChunk {
    lc_serializable: true,
    lc_kwargs: {
      content: 'ai.',
      tool_calls: [],
      invalid_tool_calls: [],
      tool_call_chunks: [],
      additional_kwargs: {},
      response_metadata: {}
    },
    lc_namespace: [ 'langchain_core', 'messages' ],
    content: 'ai.',
    name: undefined,
    additional_kwargs: {},
    response_metadata: {},
    id: undefined,
    tool_calls: [],
    invalid_tool_calls: [],
    tool_call_chunks: [],
    usage_metadata: undefined
  }
}
=======CHAT_MODEL_END=======
{
  output: AIMessageChunk {
    lc_serializable: true,
    lc_kwargs: {
      content: 'Certainly! Your email address is jod@langchain.ai.',
      additional_kwargs: {
        id: 'msg_01Li1Z6ByM93sVQVLTJbaUdR',
        type: 'message',
        role: 'assistant',
        model: 'claude-3-5-sonnet-20240620',
        stop_reason: 'end_turn',
        stop_sequence: null,
        usage: { input_tokens: 50, output_tokens: 21 }
      },
      response_metadata: {},
      tool_call_chunks: [],
      id: 'msg_01Li1Z6ByM93sVQVLTJbaUdR',
      usage_metadata: { input_tokens: 117, output_tokens: 25, total_tokens: 142 },
      tool_calls: [],
      invalid_tool_calls: []
    },
    lc_namespace: [ 'langchain_core', 'messages' ],
    content: 'Certainly! Your email address is jod@langchain.ai.',
    name: undefined,
    additional_kwargs: {
      id: 'msg_01Li1Z6ByM93sVQVLTJbaUdR',
      type: 'message',
      role: 'assistant',
      model: 'claude-3-5-sonnet-20240620',
      stop_reason: 'end_turn',
      stop_sequence: null,
      usage: { input_tokens: 50, output_tokens: 21 }
    },
    response_metadata: {},
    id: 'msg_01Li1Z6ByM93sVQVLTJbaUdR',
    tool_calls: [],
    invalid_tool_calls: [],
    tool_call_chunks: [],
    usage_metadata: { input_tokens: 117, output_tokens: 25, total_tokens: 142 }
  },
  input: {
    messages: [
      [
        SystemMessage {
          lc_serializable: true,
          lc_kwargs: {
            content: 'You are a helpful assistant.\n' +
              '\n' +
              '## User Info:\n' +
              'Name: John Doe\n' +
              'Email: jod@langchain.ai\n' +
              'Phone: +1234567890',
            additional_kwargs: {},
            response_metadata: {}
          },
          lc_namespace: [ 'langchain_core', 'messages' ],
          content: 'You are a helpful assistant.\n' +
            '\n' +
            '## User Info:\n' +
            'Name: John Doe\n' +
            'Email: jod@langchain.ai\n' +
            'Phone: +1234567890',
          name: undefined,
          additional_kwargs: {},
          response_metadata: {},
          id: undefined
        },
        HumanMessage {
          lc_serializable: true,
          lc_kwargs: {
            content: 'Could you remind me of my email?',
            additional_kwargs: {},
            response_metadata: {}
          },
          lc_namespace: [ 'langchain_core', 'messages' ],
          content: 'Could you remind me of my email?',
          name: undefined,
          additional_kwargs: {},
          response_metadata: {},
          id: undefined
        }
      ]
    ]
  }
}
on_chain_end
on_chain_end
on_chain_start
on_chain_end
on_chain_end
on_chain_stream
on_chain_end

[jacoblee93]

CC @bracesproul can you have a look?

[wh1pp3rz]

There's an issue when using ChatAnthropic with tools bound and trying to stream, whether regular stream or streamEvents. There seem to be some logic expecting the message content to be a string. The console warning The "tool_calls" field on a message is only respected if content is a string. is being shown. When I dump the content, it's not a string but an object that looks like this:

"content": [
    {
      "index": 0,
      "type": "text_delta",
      "text": "To answer your question about what 4 times 6 is, I can use the \"multiply\" function that's available to me. Let me calculate that for you."
    },
    {
      "index": 1,
      "input": "{\"a\": 4, \"b\": 6}",
      "type": "input_json_delta"
    }
  ]

[jacoblee93]

Thanks for the heads up, will have a look!

[danny-avila]

There's an issue when using ChatAnthropic with tools bound and trying to stream, whether regular stream or streamEvents. There seem to be some logic expecting the message content to be a string. The console warning The "tool_calls" field on a message is only respected if content is a string. is being shown. When I dump the content, it's not a string but an object that looks like this:

"content": [
    {
      "index": 0,
      "type": "text_delta",
      "text": "To answer your question about what 4 times 6 is, I can use the \"multiply\" function that's available to me. Let me calculate that for you."
    },
    {
      "index": 1,
      "input": "{\"a\": 4, \"b\": 6}",
      "type": "input_json_delta"
    }
  ]

Same issue here thanks for sharing

[jacoblee93]

Really sorry about this 😕

[bracesproul]

@danny-avila @wh1pp3rz I just released version 0.2.9 with a fix. Please ping me if you experience any other issues.

[wh1pp3rz]

Thanks @bracesproul , I'm no longer getting the error, however ChatAnthropic doesn't seem to work as expected. Take this sample code for example:

const graphState = {
    messages: {
        value: (x, y) => x.concat(y),
        default: () => [],
    },
};

const multiplyTool = new DynamicStructuredTool({
    name: "multiply",
    description: "multiply two numbers together",
    schema: z.object({
        a: z.number().describe("the first number to multiply"),
        b: z.number().describe("the second number to multiply"),
    }),
    func: async ({ a, b }) => {
        return (a * b).toString();
    },
});

const tools = [multiplyTool];
const toolNode = new ToolNode(tools);

const model = new ChatOpenAI({
    model: 'gpt-4o-mini',
    temperature: 0,
    streaming: true
}).bindTools(tools);

function shouldContinue(state) {
    const messages = state.messages;
    const lastMessage = messages[messages.length - 1];

    if (lastMessage.tool_calls?.length) {
        return "tools";
    }

    return END;
}

const callModel = async (state, config) => {
    const { messages } = state;
    const prompt = ChatPromptTemplate.fromMessages([
        new SystemMessage('You are a helpful assistant'),
        new MessagesPlaceholder('messages'),
    ]);

    const response = await prompt.pipe(model).invoke({messages}, config);

    return { messages: [ response ] };
}

const workflow = new StateGraph({ channels: graphState })
    .addNode("agent", callModel)
    .addNode("tools", toolNode)
    .addEdge(START, "agent")
    .addConditionalEdges("agent", shouldContinue)
    .addEdge("tools", "agent");

const checkpointer = new MemorySaver();
const app = workflow.compile({ checkpointer });

const inputs = {
    messages: [new HumanMessage("what is 4 times 6?")],
};

const config = {
    configurable: {
       thread_id: "1",
    },
};

const eventStream = await app.streamEvents(inputs, {
    ...config,
    streamMode: "values",
    version: "v2",
});
for await (const { event, data } of eventStream) {
    if (event === "on_chat_model_stream") {
        const msg = data.chunk;
        if (!msg.tool_call_chunks?.length) {
            console.log(msg.content);
        }
    }
}

With ChatOpenAI the proper output is streamed, and looks like this:

4
 times

6
 is

24
.

However when I replace the model with ChatAnthropic and without changing anything else, the output now looks like this:

{"a": 4
, "
b"
: 6}

It seems ChatAnthropic is streaming the tool call, but not the final response.

[bracesproul]

Thanks @bracesproul , I'm no longer getting the error, however ChatAnthropic doesn't seem to work as expected. Take this sample code for example:

const graphState = {

    messages: {

        value: (x, y) => x.concat(y),

        default: () => [],

    },

};

const multiplyTool = new DynamicStructuredTool({

    name: "multiply",

    description: "multiply two numbers together",

    schema: z.object({

        a: z.number().describe("the first number to multiply"),

        b: z.number().describe("the second number to multiply"),

    }),

    func: async ({ a, b }) => {

        return (a * b).toString();

    },

});

const tools = [multiplyTool];

const toolNode = new ToolNode(tools);

const model = new ChatOpenAI({

    model: 'gpt-4o-mini',

    temperature: 0,

    streaming: true

}).bindTools(tools);

function shouldContinue(state) {

    const messages = state.messages;

    const lastMessage = messages[messages.length - 1];

    if (lastMessage.tool_calls?.length) {

        return "tools";

    }

    return END;

}

const callModel = async (state, config) => {

    const { messages } = state;

    const prompt = ChatPromptTemplate.fromMessages([

        new SystemMessage('You are a helpful assistant'),

        new MessagesPlaceholder('messages'),

    ]);

    const response = await prompt.pipe(model).invoke({messages}, config);

    return { messages: [ response ] };

}

const workflow = new StateGraph({ channels: graphState })

    .addNode("agent", callModel)

    .addNode("tools", toolNode)

    .addEdge(START, "agent")

    .addConditionalEdges("agent", shouldContinue)

    .addEdge("tools", "agent");

const checkpointer = new MemorySaver();

const app = workflow.compile({ checkpointer });

const inputs = {

    messages: [new HumanMessage("what is 4 times 6?")],

};

const config = {

    configurable: {

       thread_id: "1",

    },

};

const eventStream = await app.streamEvents(inputs, {

    ...config,

    streamMode: "values",

    version: "v2",

});

for await (const { event, data } of eventStream) {

    if (event === "on_chat_model_stream") {

        const msg = data.chunk;

        if (!msg.tool_call_chunks?.length) {

            console.log(msg.content);

        }

    }

}

With ChatOpenAI the proper output is streamed, and looks like this:

4

 times

6

 is

24

.

However when I replace the model with ChatAnthropic and without changing anything else, the output now looks like this:

{"a": 4

, "

b"

: 6}

It seems ChatAnthropic is streaming the tool call, but not the final response.

It looks like Anthropic is calling a tool here and OpenAI isn't because the LLM "thinks" it knows the answer? If you send me a LangSmith trace of both I could see what's going on

[wh1pp3rz]

OpenAI is definitely calling a tool. Here's the LangSmith trace

And. here's the trace for ChatAnthropic

[lionkeng]

@bracesproul I have updated to 0.2.9 but it is still not working as expected. I am looking for some guidance on whether this is a bug in my code or in the Langgraph/LangchainJS implementation.

My code example lives here, nothing fancy and actually looks a lot like the code example in the Langgraph documentation.

Here is a langsmith trace with ChatAnthropic.

Here is the langsmith tracing with the same code but I swapped out ChatAnthropic with ChatOpenAI - and it appears to work there.

If I look at the runs view on the langsmith dashboard, the ChatAnthropic call seems to be running outside of the Langgraph/RunnableLambda nesting.

With ChatOpenAI, the call is nested in the RunnableLambda.

[jacoblee93]

Ah @wh1pp3rz I think Anthropic is streaming back a chunk where content is an array instead of a string, and LangSmith is not rendering it properly. I'll flag that with the team, but this is within our spec - I can see how it is confusing but Anthropic could theoretically call another tool in a block after that call, so we can't just stream back string chunks here. See the Raw Output of your trace below.

@lionkeng can you try setting resolutions as described here?

https://langchain-ai.github.io/langgraphjs/how-tos/manage-ecosystem-dependencies/

[wh1pp3rz]

@jacoblee93 what would be the proper way to stream the final result of a chain when using anthropic with tools bound and streamEvents v2?

[lionkeng]

@jacoblee93 I can try setting resolutions. But what should be the right combination of versions to use for the different packages? Here are the relevant versions of packages from the langchain eco that I am using as shown by a pnpm list.

dependencies:
@langchain/anthropic 0.2.9
@langchain/community 0.2.20
@langchain/core 0.2.18
@langchain/langgraph 0.0.26
@langchain/openai 0.0.34
better-sqlite3 9.6.0
langchain 0.2.11
langsmith 0.1.39

[danny-avila]

After this update (0.2.10), anthropic will stream with bound tools but it will not stream tool chunks:

https://github.com/langchain-ai/langchainjs/pull/6206

[bracesproul]

After this update (0.2.10), anthropic will stream with bound tools but it will not stream tool chunks:

Could you expand on what you mean by this? Streams will contain tool chunks, but binding tools to the model doesn't do anything to streaming.

[bracesproul]

@jacoblee93 what would be the proper way to stream the final result of a chain when using anthropic with tools bound and streamEvents v2?

Could you open a new issue for "How to stream the final result with anthropic" in the langchain-ai/langchainjs repo

[wh1pp3rz]

After this update (0.2.10), anthropic will stream with bound tools but it will not stream tool chunks:

Could you expand on what you mean by this? Streams will contain tool chunks, but binding tools to the model doesn't do anything to streaming.

Yeah this is working sort of now, the graph is streaming when tools are bound, but the streamed output contains the tool input parameters appended at the end and it looks weird for an end user. As an example, a simple multiply tool called with "what is 2 * 3?" streamed the following:

To
 answer
 this question, I
'll
 use
 the multiplyer function
 to calculate
 2 *
 3.
{"a"
: 2
, "
b":
3}

The
 result of 2
 * 3
 is 6
.

Using streamEvents, the tool inputs are streamed under the on_chat_model_stream event, and I don't see a way to distinguish or filter them to not be in the final output to the user.

Here's the trace if that helps.