Streaming tokens does not work

[rednebmas]

The documentation on how to do streaming is here https://js.langchain.com/docs/modules/agents/how_to/streaming

If I follow this, streaming still does not work. I've created an example repo https://github.com/rednebmas/langchain-streaming-bug

[dosubot[bot]]

🤖

Hey there, @rednebmas! Long time no bug, huh? 🐛 Hope you've been coding away without too much trouble until now!

Based on the information you've provided, it seems like the streaming feature is functioning as expected. The streamLog function is correctly called and the output is being logged. However, the streamed output string is empty, which might be causing confusion.

This could be due to the fact that the language model is generating the response in one go, rather than in chunks. The streamed_output_str is meant to capture the output as it is being generated, but if the response is generated all at once, this field will be empty.

The final output is correctly logged under final_output. So, the streaming feature is working, but the response is generated all at once, which is why the streamed_output_str is empty.

The streaming feature in LangChainJS is designed to process and return chunks of data as they become available, rather than waiting for all data to be processed. This is particularly useful for applications that need to handle large amounts of data or need to display results as they become available.

In the provided context, the streamLog method is used in three test cases. In all cases, the streamLog method is used in conjunction with a for await...of loop to process each chunk of data as it becomes available. The final state of the stream is determined by concatenating each chunk of data to the previous state.

You can find more details in the LangChainJS repository.

If you have any other questions or need further clarification, feel free to ask.

Sources

• examples/src/agents/stream_log.ts
• langchain-core/src/runnables/tests/runnable_stream_log.test.ts

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[rednebmas]

@dosubot seems unlikely the model is not actually streaming as requested, the "tell me a story" chunk is very long.

[dosubot[bot]]

🤖

Based on the context provided, it seems like you're trying to use the "tell me a story" chunk with the stream method of the ChatOpenAI model. However, in the provided code, you're passing "Hello there!" as the argument to the stream method.

To use the "tell me a story" chunk, you should pass it as the argument to the stream method. Here's how you can modify your code:

// ...
const stream = await model.pipe(parser).stream("tell me a story");
// ...

Please replace "Hello there!" with "tell me a story" and try again. If you're still facing issues, there might be other factors at play that aren't visible from the provided context.

The stream method in the ChatOpenAI model is used to generate a stream of responses from the AI model. When used with different chunks of text, it will return an asynchronous iterable of AIMessageChunk objects, each containing a piece of the AI's response. Here is an example of how it works:

import { ChatOpenAI } from "langchain/chat_models/openai";

const chat = new ChatOpenAI({
  maxTokens: 25,
});

const stream = await chat.stream([["human", "Tell me a joke about bears."]]);

for await (const chunk of stream) {
  console.log(chunk);
}

In this code, the stream method is called with a single human message "Tell me a joke about bears.". The method then generates a stream of AIMessageChunk objects, each containing a part of the AI's response. These chunks are logged to the console in the order they are received.

If you need further assistance, please provide more details about the issue you're facing.

Sources

• examples/src/prompts/http_response_output_parser_event_stream.ts
• examples/src/models/chat/chat_streaming_stream_method.ts

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[jacoblee93]

Thank you for the reproducible example - I think this is a regression. Will dive in and fix 👍

[jacoblee93]

Ok - I think not a regression but it seems to occur if no tools are used. Still digging in.

[jacoblee93]

Actually, it was a regression :( caused by #4083

Fix will be live soon.

[jacoblee93]

Upgrading @langchain/core to 0.1.21 should fix this!

[bkamapantula]

I was able to reproduce the original reported bug.

Thanks for the quick fix, Jacob!