Open DevDeepakBhattarai opened 1 month ago
Streaming support for models like ChatVertexAI
, ChatGoogleGenerativeAI
, and ChatGroq
is indeed a work in progress in LangChain JS. Here are some details and workarounds you can consider:
For ChatVertexAI
, streaming support is being implemented. You can use the _streamResponseChunks
method to handle streaming responses. Here is an example:
import { ChatVertexAI } from "@langchain/google-vertexai";
const model = new ChatVertexAI({
temperature: 0.7,
model: "gemini-1.5-flash-001",
});
const stream = await model.stream([
["system", "You are a funny assistant that answers in pirate language."],
["human", "What is your favorite food?"],
]);
for await (const chunk of stream) {
console.log(chunk);
}
For ChatGoogleGenerativeAI
, you can use a similar approach as ChatVertexAI
. The streaming support is being added, and you can use the .stream()
method to handle streaming responses.
For ChatGroq
, you can enable streaming by setting the streaming
parameter to true
in the model configuration. Here is an example:
import { ChatGroq } from "@langchain/groq";
import { ChatPromptTemplate } from "@langchain/core/prompts";
import { StringOutputParser } from "@langchain/core/output_parsers";
const model = new ChatGroq({
apiKey: process.env.GROQ_API_KEY,
streaming: true,
model: "llama3-70b-8192",
temperature: 0.7,
});
const prompt = ChatPromptTemplate.fromMessages([
["system", "You are a helpful assistant"],
["human", "{input}"],
]);
const outputParser = new StringOutputParser();
const chain = prompt.pipe(model).pipe(outputParser);
const response = await chain.stream({
input: "Hello",
});
let res = "";
for await (const item of response) {
res += item;
console.log("stream:", res);
}
For models that do not natively support streaming, you can use the .stream()
method, which will yield a single chunk. This allows you to use the same general pattern when calling them.
import { ChatAnthropic } from "@langchain/anthropic";
import { concat } from "@langchain/core/utils/stream";
import type { AIMessageChunk } from "@langchain/core/messages";
const model = new ChatAnthropic({ model: "claude-3-sonnet-20240229" });
const stream = await model.stream("what color is the sky?");
let gathered: AIMessageChunk | undefined = undefined;
for await (const chunk of stream) {
console.log(chunk);
if (gathered === undefined) {
gathered = chunk;
} else {
gathered = concat(gathered, chunk);
}
}
console.log(gathered);
The on_llm_start
and on_llm_end
events are fired for all models, but the on_llm_stream
event is only fired for models that support token-by-token streaming. For models that do not support streaming, you will only see on_llm_start
and on_llm_end
events.
While streaming support is being added for various models, you can use the .stream()
method and handle the responses in a consistent manner. For models that do not support streaming natively, the entire response will be returned as a single chunk.
For more details, you can refer to the LangChain JS documentation and examples:
To continue talking to Dosu, mention @dosu.
Checked other resources
Example Code
When ever we pass any tools to model except ChatOpenAI, the streaming functionality does not work.
The model are being invoked here, but the langgraph agent is being streamed. OpenAI works as I expect , giving me
on_llm_stream
event. But when ever other model are used onlyon_llm_start
andon_llm_end
event are fired.Also while I am at this ,
ChatVertexAI
does not have any option to passstreaming
/stream
when I initiate it.Error Message and Stack Trace (if applicable)
No response
Description
I am trying to build a LLM app with multiple model support. ChatOpenAI works well for good user experience with streaming but other provider (mainly ChatVertexAI, ChatGoogleGenerativeAI and ChatGroq) do not stream data.
System Info
pnpm version 9.4 windows 10