Closed lionkeng closed 1 month ago
when you say "not working" - what exactly is happening?
+1 to Harrison's comment.
If you're referencing not being able to access token counts when streaming tool calls, I just pushed a fix here. I also see some other issues with your code. For extracting tool calls from messages, you should be looking inside message.tool_calls
not message.additional_kwargs.tool_calls
. If there's another bug, please let me know what the issue is and I can work it out. Thanks!
when you say "not working" - what exactly is happening? @hwchase17 When I run it with ChatAnthropic, streamEvents never seem to generate any event.event with
on_chat_model_stream
.
@bracesproul thank you! I see that too. But I seem to be running into something more fundamental.
@lionkeng what exactly is the issue? If you could provide a stack trace and an in depth description of the issue I can help fix it.
@lionkeng what exactly is the issue? If you could provide a stack trace and an in depth description of the issue I can help fix it.
@bracesproul
There is no stack trace per-se as the program/script just runs but doesn't encounter any event named on_chat_model_stream
. So if I run the script as shown above, the console.log will output all of the following but missing on_chat_model_stream
:
event on_chain_start
event on_chain_start
event on_chain_end
event on_chain_stream
event on_chain_start
event on_chain_start
event on_chat_model_start
event on_chat_model_end
event on_chain_end
event on_chain_start
event on_chain_end
event on_chain_start
event on_chain_end
event on_chain_end
event on_chain_stream
event on_chain_end
Okay just dug into this, we don't support streaming tool calls with anthropic in the way you'd think. So yes you can call model.stream and it'll work, but under the hood of the ChatAnthropic
class we're calling the non streaming method if there are tool calls. So because of this, streamEvents
won't yield that event.
This is a top priority for us though, and we're aiming to get this implemented by midway through next week.
Thanks for the response @bracesproul ! Looking forwarding to testing out the changes.
Still having streaming issues with ChatAnthropic using streamEvents
and v2
in particular.
Here are my langchain-specific dependencies from my project package.json:
"dependencies": {
"@langchain/anthropic": "^0.2.8",
"@langchain/aws": "^0.0.5",
"@langchain/community": "^0.2.20",
"@langchain/core": "^0.2.18",
"@langchain/google-vertexai": "^0.0.20",
"@langchain/langgraph": "^0.0.31",
"@langchain/mistralai": "^0.0.26",
"langchain": "^0.2.10",
},
"resolutions": {
"@langchain/core": "0.2.18",
},
Note: this correctly streams tokens when we don't bind tools, but if we do, it fails.
import { BaseMessage, HumanMessage } from "@langchain/core/messages";
import { ChatAnthropic } from "@langchain/anthropic";
import { ChatPromptTemplate } from "@langchain/core/prompts";
import { TavilySearchResults } from '@langchain/community/tools/tavily_search';
import { RunnableConfig } from "@langchain/core/runnables";
import {
END,
// MemorySaver,
START,
StateGraph,
StateGraphArgs,
} from "@langchain/langgraph";
interface IState {
messages: BaseMessage[];
userInfo: string;
}
async function main() {
const graphState: StateGraphArgs<IState>["channels"] = {
messages: {
value: (x: BaseMessage[], y: BaseMessage[]) => x.concat(y),
default: () => [],
},
userInfo: {
value: (x?: string, y?: string) => {
return y ? y : x ? x : "N/A";
},
default: () => "N/A",
},
};
const promptTemplate = ChatPromptTemplate.fromMessages([
["system", "You are a helpful assistant.\n\n## User Info:\n{userInfo}"],
["placeholder", "{messages}"],
]);
const initializeModel = (bindTools = false) => {
const model = new ChatAnthropic({
// model: "claude-3-haiku-20240307",
model: 'claude-3-5-sonnet-20240620'
})
if (bindTools) {
return model.bindTools([new TavilySearchResults()]);
}
return model;
}
const callModel = async (
state: { messages: BaseMessage[]; userInfo: string },
config?: RunnableConfig,
) => {
const { messages, userInfo } = state;
/*
This correctly streams tokens when
we don't bind tools, but if we do, it fails.
*/
const model = initializeModel(true);
// const model = initializeModel();
const chain = promptTemplate.pipe(model);
const response = await chain.invoke(
{
messages,
userInfo,
},
config,
);
return { messages: [response] };
};
const fetchUserInformation = async (
_: { messages: BaseMessage[] },
config?: RunnableConfig,
) => {
const userDB = {
user1: {
name: "John Doe",
email: "jod@langchain.ai",
phone: "+1234567890",
},
user2: {
name: "Jane Doe",
email: "jad@langchain.ai",
phone: "+0987654321",
},
};
const userId = config?.configurable?.user;
if (userId) {
const user = userDB[userId as keyof typeof userDB];
if (user) {
return {
userInfo:
`Name: ${user.name}\nEmail: ${user.email}\nPhone: ${user.phone}`,
};
}
}
return { userInfo: "N/A" };
};
const workflow = new StateGraph({
channels: graphState,
})
.addNode("fetchUserInfo", fetchUserInformation)
.addNode("agent", callModel)
.addEdge(START, "fetchUserInfo")
.addEdge("fetchUserInfo", "agent")
.addEdge("agent", END);
// Here we only save in-memory
// let memory = new MemorySaver();
// const graph = workflow.compile({ checkpointer: memory });
const graph = workflow.compile();
const config = {
configurable: {
user: "user1",
},
};
const inputs = {
messages: [new HumanMessage("Could you remind me of my email?")],
};
const stream = graph.streamEvents(inputs, {
...config,
version: "v2",
streamMode: "values",
});
for await (const { event, data } of stream) {
if (event === "on_chat_model_start") {
console.log('=======CHAT_MODEL_START=======');
console.dir(data, { depth: null });
} else if (event === "on_chat_model_stream") {
console.dir(data, { depth: null });
} else if (event === "on_chat_model_end") {
console.log('=======CHAT_MODEL_END=======');
console.dir(data, { depth: null });
} else {
console.log(event);
}
}
}
main().catch((err) => {
console.error(err);
process.exit(1);
});
on_chain_start
on_chain_start
on_chain_end
on_chain_stream
on_chain_start
on_chain_start
on_chain_end
on_chain_start
on_chain_end
on_chain_end
on_chain_stream
on_chain_start
on_chain_start
on_chain_start
on_prompt_start
on_prompt_end
=======CHAT_MODEL_START=======
{
input: {
messages: [
[
SystemMessage {
lc_serializable: true,
lc_kwargs: {
content: 'You are a helpful assistant.\n' +
'\n' +
'## User Info:\n' +
'Name: John Doe\n' +
'Email: jod@langchain.ai\n' +
'Phone: +1234567890',
additional_kwargs: {},
response_metadata: {}
},
lc_namespace: [ 'langchain_core', 'messages' ],
content: 'You are a helpful assistant.\n' +
'\n' +
'## User Info:\n' +
'Name: John Doe\n' +
'Email: jod@langchain.ai\n' +
'Phone: +1234567890',
name: undefined,
additional_kwargs: {},
response_metadata: {},
id: undefined
},
HumanMessage {
lc_serializable: true,
lc_kwargs: {
content: 'Could you remind me of my email?',
additional_kwargs: {},
response_metadata: {}
},
lc_namespace: [ 'langchain_core', 'messages' ],
content: 'Could you remind me of my email?',
name: undefined,
additional_kwargs: {},
response_metadata: {},
id: undefined
}
]
]
}
}
=======CHAT_MODEL_END=======
{
output: AIMessageChunk {
lc_serializable: true,
lc_kwargs: {
content: [
{
index: 0,
type: 'text_delta',
text: "Certainly! I'd be happy to remind you of your email address. Based on the user information provided, your email address is:\n" +
'\n' +
'jod@langchain.ai\n' +
'\n' +
"Is there anything else you'd like to know about your contact information?"
}
],
additional_kwargs: {
id: 'msg_01TZ28cmm6nPk7FsmRHjEDet',
type: 'message',
role: 'assistant',
model: 'claude-3-5-sonnet-20240620',
stop_reason: 'end_turn',
stop_sequence: null,
usage: { input_tokens: 452, output_tokens: 55 }
},
response_metadata: {},
tool_call_chunks: [],
id: 'msg_01TZ28cmm6nPk7FsmRHjEDet',
usage_metadata: { input_tokens: 956, output_tokens: 58, total_tokens: 1014 },
tool_calls: [],
invalid_tool_calls: []
},
lc_namespace: [ 'langchain_core', 'messages' ],
content: [
{
index: 0,
type: 'text_delta',
text: "Certainly! I'd be happy to remind you of your email address. Based on the user information provided, your email address is:\n" +
'\n' +
'jod@langchain.ai\n' +
'\n' +
"Is there anything else you'd like to know about your contact information?"
}
],
name: undefined,
additional_kwargs: {
id: 'msg_01TZ28cmm6nPk7FsmRHjEDet',
type: 'message',
role: 'assistant',
model: 'claude-3-5-sonnet-20240620',
stop_reason: 'end_turn',
stop_sequence: null,
usage: { input_tokens: 452, output_tokens: 55 }
},
response_metadata: {},
id: 'msg_01TZ28cmm6nPk7FsmRHjEDet',
tool_calls: [],
invalid_tool_calls: [],
tool_call_chunks: [],
usage_metadata: { input_tokens: 956, output_tokens: 58, total_tokens: 1014 }
},
input: {
messages: [
[
SystemMessage {
lc_serializable: true,
lc_kwargs: {
content: 'You are a helpful assistant.\n' +
'\n' +
'## User Info:\n' +
'Name: John Doe\n' +
'Email: jod@langchain.ai\n' +
'Phone: +1234567890',
additional_kwargs: {},
response_metadata: {}
},
lc_namespace: [ 'langchain_core', 'messages' ],
content: 'You are a helpful assistant.\n' +
'\n' +
'## User Info:\n' +
'Name: John Doe\n' +
'Email: jod@langchain.ai\n' +
'Phone: +1234567890',
name: undefined,
additional_kwargs: {},
response_metadata: {},
id: undefined
},
HumanMessage {
lc_serializable: true,
lc_kwargs: {
content: 'Could you remind me of my email?',
additional_kwargs: {},
response_metadata: {}
},
lc_namespace: [ 'langchain_core', 'messages' ],
content: 'Could you remind me of my email?',
name: undefined,
additional_kwargs: {},
response_metadata: {},
id: undefined
}
]
]
}
}
on_chain_end
on_chain_end
on_chain_start
on_chain_end
on_chain_end
on_chain_stream
on_chain_end
on_chain_start
on_chain_start
on_chain_end
on_chain_stream
on_chain_start
on_chain_start
on_chain_end
on_chain_start
on_chain_end
on_chain_end
on_chain_stream
on_chain_start
on_chain_start
on_chain_start
on_prompt_start
on_prompt_end
=======CHAT_MODEL_START=======
{
input: {
messages: [
[
SystemMessage {
lc_serializable: true,
lc_kwargs: {
content: 'You are a helpful assistant.\n' +
'\n' +
'## User Info:\n' +
'Name: John Doe\n' +
'Email: jod@langchain.ai\n' +
'Phone: +1234567890',
additional_kwargs: {},
response_metadata: {}
},
lc_namespace: [ 'langchain_core', 'messages' ],
content: 'You are a helpful assistant.\n' +
'\n' +
'## User Info:\n' +
'Name: John Doe\n' +
'Email: jod@langchain.ai\n' +
'Phone: +1234567890',
name: undefined,
additional_kwargs: {},
response_metadata: {},
id: undefined
},
HumanMessage {
lc_serializable: true,
lc_kwargs: {
content: 'Could you remind me of my email?',
additional_kwargs: {},
response_metadata: {}
},
lc_namespace: [ 'langchain_core', 'messages' ],
content: 'Could you remind me of my email?',
name: undefined,
additional_kwargs: {},
response_metadata: {},
id: undefined
}
]
]
}
}
{
chunk: AIMessageChunk {
lc_serializable: true,
lc_kwargs: {
content: 'Certainly! Your email',
tool_calls: [],
invalid_tool_calls: [],
tool_call_chunks: [],
additional_kwargs: {},
response_metadata: {}
},
lc_namespace: [ 'langchain_core', 'messages' ],
content: 'Certainly! Your email',
name: undefined,
additional_kwargs: {},
response_metadata: {},
id: undefined,
tool_calls: [],
invalid_tool_calls: [],
tool_call_chunks: [],
usage_metadata: undefined
}
}
{
chunk: AIMessageChunk {
lc_serializable: true,
lc_kwargs: {
content: ' address is jo',
tool_calls: [],
invalid_tool_calls: [],
tool_call_chunks: [],
additional_kwargs: {},
response_metadata: {}
},
lc_namespace: [ 'langchain_core', 'messages' ],
content: ' address is jo',
name: undefined,
additional_kwargs: {},
response_metadata: {},
id: undefined,
tool_calls: [],
invalid_tool_calls: [],
tool_call_chunks: [],
usage_metadata: undefined
}
}
{
chunk: AIMessageChunk {
lc_serializable: true,
lc_kwargs: {
content: 'd@langchain.',
tool_calls: [],
invalid_tool_calls: [],
tool_call_chunks: [],
additional_kwargs: {},
response_metadata: {}
},
lc_namespace: [ 'langchain_core', 'messages' ],
content: 'd@langchain.',
name: undefined,
additional_kwargs: {},
response_metadata: {},
id: undefined,
tool_calls: [],
invalid_tool_calls: [],
tool_call_chunks: [],
usage_metadata: undefined
}
}
{
chunk: AIMessageChunk {
lc_serializable: true,
lc_kwargs: {
content: 'ai.',
tool_calls: [],
invalid_tool_calls: [],
tool_call_chunks: [],
additional_kwargs: {},
response_metadata: {}
},
lc_namespace: [ 'langchain_core', 'messages' ],
content: 'ai.',
name: undefined,
additional_kwargs: {},
response_metadata: {},
id: undefined,
tool_calls: [],
invalid_tool_calls: [],
tool_call_chunks: [],
usage_metadata: undefined
}
}
=======CHAT_MODEL_END=======
{
output: AIMessageChunk {
lc_serializable: true,
lc_kwargs: {
content: 'Certainly! Your email address is jod@langchain.ai.',
additional_kwargs: {
id: 'msg_01Li1Z6ByM93sVQVLTJbaUdR',
type: 'message',
role: 'assistant',
model: 'claude-3-5-sonnet-20240620',
stop_reason: 'end_turn',
stop_sequence: null,
usage: { input_tokens: 50, output_tokens: 21 }
},
response_metadata: {},
tool_call_chunks: [],
id: 'msg_01Li1Z6ByM93sVQVLTJbaUdR',
usage_metadata: { input_tokens: 117, output_tokens: 25, total_tokens: 142 },
tool_calls: [],
invalid_tool_calls: []
},
lc_namespace: [ 'langchain_core', 'messages' ],
content: 'Certainly! Your email address is jod@langchain.ai.',
name: undefined,
additional_kwargs: {
id: 'msg_01Li1Z6ByM93sVQVLTJbaUdR',
type: 'message',
role: 'assistant',
model: 'claude-3-5-sonnet-20240620',
stop_reason: 'end_turn',
stop_sequence: null,
usage: { input_tokens: 50, output_tokens: 21 }
},
response_metadata: {},
id: 'msg_01Li1Z6ByM93sVQVLTJbaUdR',
tool_calls: [],
invalid_tool_calls: [],
tool_call_chunks: [],
usage_metadata: { input_tokens: 117, output_tokens: 25, total_tokens: 142 }
},
input: {
messages: [
[
SystemMessage {
lc_serializable: true,
lc_kwargs: {
content: 'You are a helpful assistant.\n' +
'\n' +
'## User Info:\n' +
'Name: John Doe\n' +
'Email: jod@langchain.ai\n' +
'Phone: +1234567890',
additional_kwargs: {},
response_metadata: {}
},
lc_namespace: [ 'langchain_core', 'messages' ],
content: 'You are a helpful assistant.\n' +
'\n' +
'## User Info:\n' +
'Name: John Doe\n' +
'Email: jod@langchain.ai\n' +
'Phone: +1234567890',
name: undefined,
additional_kwargs: {},
response_metadata: {},
id: undefined
},
HumanMessage {
lc_serializable: true,
lc_kwargs: {
content: 'Could you remind me of my email?',
additional_kwargs: {},
response_metadata: {}
},
lc_namespace: [ 'langchain_core', 'messages' ],
content: 'Could you remind me of my email?',
name: undefined,
additional_kwargs: {},
response_metadata: {},
id: undefined
}
]
]
}
}
on_chain_end
on_chain_end
on_chain_start
on_chain_end
on_chain_end
on_chain_stream
on_chain_end
CC @bracesproul can you have a look?
There's an issue when using ChatAnthropic with tools bound and trying to stream, whether regular stream or streamEvents. There seem to be some logic expecting the message content to be a string. The console warning The "tool_calls" field on a message is only respected if content is a string.
is being shown. When I dump the content, it's not a string but an object that looks like this:
"content": [
{
"index": 0,
"type": "text_delta",
"text": "To answer your question about what 4 times 6 is, I can use the \"multiply\" function that's available to me. Let me calculate that for you."
},
{
"index": 1,
"input": "{\"a\": 4, \"b\": 6}",
"type": "input_json_delta"
}
]
Thanks for the heads up, will have a look!
There's an issue when using ChatAnthropic with tools bound and trying to stream, whether regular stream or streamEvents. There seem to be some logic expecting the message content to be a string. The console warning
The "tool_calls" field on a message is only respected if content is a string.
is being shown. When I dump the content, it's not a string but an object that looks like this:"content": [ { "index": 0, "type": "text_delta", "text": "To answer your question about what 4 times 6 is, I can use the \"multiply\" function that's available to me. Let me calculate that for you." }, { "index": 1, "input": "{\"a\": 4, \"b\": 6}", "type": "input_json_delta" } ]
Same issue here thanks for sharing
Really sorry about this 😕
@danny-avila @wh1pp3rz I just released version 0.2.9
with a fix. Please ping me if you experience any other issues.
Thanks @bracesproul , I'm no longer getting the error, however ChatAnthropic
doesn't seem to work as expected. Take this sample code for example:
const graphState = {
messages: {
value: (x, y) => x.concat(y),
default: () => [],
},
};
const multiplyTool = new DynamicStructuredTool({
name: "multiply",
description: "multiply two numbers together",
schema: z.object({
a: z.number().describe("the first number to multiply"),
b: z.number().describe("the second number to multiply"),
}),
func: async ({ a, b }) => {
return (a * b).toString();
},
});
const tools = [multiplyTool];
const toolNode = new ToolNode(tools);
const model = new ChatOpenAI({
model: 'gpt-4o-mini',
temperature: 0,
streaming: true
}).bindTools(tools);
function shouldContinue(state) {
const messages = state.messages;
const lastMessage = messages[messages.length - 1];
if (lastMessage.tool_calls?.length) {
return "tools";
}
return END;
}
const callModel = async (state, config) => {
const { messages } = state;
const prompt = ChatPromptTemplate.fromMessages([
new SystemMessage('You are a helpful assistant'),
new MessagesPlaceholder('messages'),
]);
const response = await prompt.pipe(model).invoke({messages}, config);
return { messages: [ response ] };
}
const workflow = new StateGraph({ channels: graphState })
.addNode("agent", callModel)
.addNode("tools", toolNode)
.addEdge(START, "agent")
.addConditionalEdges("agent", shouldContinue)
.addEdge("tools", "agent");
const checkpointer = new MemorySaver();
const app = workflow.compile({ checkpointer });
const inputs = {
messages: [new HumanMessage("what is 4 times 6?")],
};
const config = {
configurable: {
thread_id: "1",
},
};
const eventStream = await app.streamEvents(inputs, {
...config,
streamMode: "values",
version: "v2",
});
for await (const { event, data } of eventStream) {
if (event === "on_chat_model_stream") {
const msg = data.chunk;
if (!msg.tool_call_chunks?.length) {
console.log(msg.content);
}
}
}
With ChatOpenAI the proper output is streamed, and looks like this:
4
times
6
is
24
.
However when I replace the model with ChatAnthropic
and without changing anything else, the output now looks like this:
{"a": 4
, "
b"
: 6}
It seems ChatAnthropic is streaming the tool call, but not the final response.
Thanks @bracesproul , I'm no longer getting the error, however
ChatAnthropic
doesn't seem to work as expected. Take this sample code for example:const graphState = { messages: { value: (x, y) => x.concat(y), default: () => [], }, }; const multiplyTool = new DynamicStructuredTool({ name: "multiply", description: "multiply two numbers together", schema: z.object({ a: z.number().describe("the first number to multiply"), b: z.number().describe("the second number to multiply"), }), func: async ({ a, b }) => { return (a * b).toString(); }, }); const tools = [multiplyTool]; const toolNode = new ToolNode(tools); const model = new ChatOpenAI({ model: 'gpt-4o-mini', temperature: 0, streaming: true }).bindTools(tools); function shouldContinue(state) { const messages = state.messages; const lastMessage = messages[messages.length - 1]; if (lastMessage.tool_calls?.length) { return "tools"; } return END; } const callModel = async (state, config) => { const { messages } = state; const prompt = ChatPromptTemplate.fromMessages([ new SystemMessage('You are a helpful assistant'), new MessagesPlaceholder('messages'), ]); const response = await prompt.pipe(model).invoke({messages}, config); return { messages: [ response ] }; } const workflow = new StateGraph({ channels: graphState }) .addNode("agent", callModel) .addNode("tools", toolNode) .addEdge(START, "agent") .addConditionalEdges("agent", shouldContinue) .addEdge("tools", "agent"); const checkpointer = new MemorySaver(); const app = workflow.compile({ checkpointer }); const inputs = { messages: [new HumanMessage("what is 4 times 6?")], }; const config = { configurable: { thread_id: "1", }, }; const eventStream = await app.streamEvents(inputs, { ...config, streamMode: "values", version: "v2", }); for await (const { event, data } of eventStream) { if (event === "on_chat_model_stream") { const msg = data.chunk; if (!msg.tool_call_chunks?.length) { console.log(msg.content); } } }
With ChatOpenAI the proper output is streamed, and looks like this:
4 times 6 is 24 .
However when I replace the model with
ChatAnthropic
and without changing anything else, the output now looks like this:{"a": 4 , " b" : 6}
It seems ChatAnthropic is streaming the tool call, but not the final response.
It looks like Anthropic is calling a tool here and OpenAI isn't because the LLM "thinks" it knows the answer? If you send me a LangSmith trace of both I could see what's going on
OpenAI is definitely calling a tool. Here's the LangSmith trace
And. here's the trace for ChatAnthropic
@bracesproul I have updated to 0.2.9 but it is still not working as expected. I am looking for some guidance on whether this is a bug in my code or in the Langgraph/LangchainJS implementation.
My code example lives here, nothing fancy and actually looks a lot like the code example in the Langgraph documentation.
Here is a langsmith trace with ChatAnthropic.
Here is the langsmith tracing with the same code but I swapped out ChatAnthropic with ChatOpenAI - and it appears to work there.
If I look at the runs
view on the langsmith dashboard, the ChatAnthropic call seems to be running outside of the Langgraph/RunnableLambda nesting.
With ChatOpenAI, the call is nested in the RunnableLambda.
Ah @wh1pp3rz I think Anthropic is streaming back a chunk where content
is an array instead of a string, and LangSmith is not rendering it properly. I'll flag that with the team, but this is within our spec - I can see how it is confusing but Anthropic could theoretically call another tool in a block after that call, so we can't just stream back string chunks here. See the Raw Output
of your trace below.
@lionkeng can you try setting resolutions
as described here?
https://langchain-ai.github.io/langgraphjs/how-tos/manage-ecosystem-dependencies/
@jacoblee93 what would be the proper way to stream the final result of a chain when using anthropic with tools bound and streamEvents v2?
@jacoblee93
I can try setting resolutions. But what should be the right combination of versions to use for the different packages?
Here are the relevant versions of packages from the langchain eco that I am using as shown by a pnpm list
.
dependencies:
@langchain/anthropic 0.2.9
@langchain/community 0.2.20
@langchain/core 0.2.18
@langchain/langgraph 0.0.26
@langchain/openai 0.0.34
better-sqlite3 9.6.0
langchain 0.2.11
langsmith 0.1.39
After this update (0.2.10), anthropic will stream with bound tools but it will not stream tool chunks:
After this update (0.2.10), anthropic will stream with bound tools but it will not stream tool chunks:
Could you expand on what you mean by this? Streams will contain tool chunks, but binding tools to the model doesn't do anything to streaming.
@jacoblee93 what would be the proper way to stream the final result of a chain when using anthropic with tools bound and streamEvents v2?
Could you open a new issue for "How to stream the final result with anthropic" in the langchain-ai/langchainjs repo
After this update (0.2.10), anthropic will stream with bound tools but it will not stream tool chunks:
Could you expand on what you mean by this? Streams will contain tool chunks, but binding tools to the model doesn't do anything to streaming.
Yeah this is working sort of now, the graph is streaming when tools are bound, but the streamed output contains the tool input parameters appended at the end and it looks weird for an end user. As an example, a simple multiply tool called with "what is 2 * 3?" streamed the following:
To
answer
this question, I
'll
use
the multiplyer function
to calculate
2 *
3.
{"a"
: 2
, "
b":
3}
The
result of 2
* 3
is 6
.
Using streamEvents
, the tool inputs are streamed under the on_chat_model_stream
event, and I don't see a way to distinguish or filter them to not be in the final output to the user.
Here's the trace if that helps.
I am trying out the streaming example provided in your documentation here with v2 schema. I got it to work based on suggestion provided by @jacoblee93 here
The code works when I use ChatOpenAI as the model. But it is not working when I swap the model with ChatAnthropic. What am I missing? Thank you.