using JsonOutputParser with ChatOpenAI seems to be breaking streaming and returning only one chunk

praveenkvr commented 1 month ago

Checked other resources

[X] I added a very descriptive title to this issue.
[X] I searched the LangChain.js documentation with the integrated search.
[X] I used the GitHub search to find a similar question and didn't find it.
[X] I am sure that this is a bug in LangChain.js rather than my code.
[X] The bug is not resolved by updating to the latest stable version of LangChain (or the specific integration package).

Example Code

When i switch from JsonOutputParser to StringOutputParser string works fine. Let me know if i am using the API wrong thanks!.

Error Message and Stack Trace (if applicable)

No error but just get a single chunk when i stream with JsonOutputParser

Description

I have the following code

const prompt = new PromptTemplate({
  template,
  inputVariables: ["text"],
  partialVariables: { format_instructions: parser.getFormatInstructions() },
});

const model = new ChatOpenAI({
  model: "gpt-4o",
  temperature: 0,
  apiKey: process.env.OPENAI_API_KEY,
});

const chain = prompt.pipe(model).pipe(new JsonOutputParser());

const stR = await chain.stream({ idea });
for await (const item of stR) {
  console.log(item);
}

I tried the exact same in python and streaming works there where i got multiple chunks of partial JSON responses back

parser = JsonOutputParser(pydantic_object=InputFormat)

prompt = PromptTemplate(
    template=template,
    input_variables=['text'],
    partial_variables={"format_instructions": parser.get_format_instructions() }
)

model = ChatOpenAI(model='gpt-4o', temperature=0, api_key=os.getenv("OPENAI_API_KEY"))

chain = prompt | model | parser

for chunk in chain.stream({ "idea": idea}):
    print(chunk, end='', flush=True)
    print('\n')

When i switch from JsonOutputParser to StringOutputParser string works fine. Let me know if i am using the API wrong thanks!.

System Info

Using the latest version of langchain code (0.3.13) and openAi

dosubot[bot] commented 1 month ago

Hey @praveenkvr! I'm here to help you with any bugs, questions, or contributions you have. Let's tackle this together!

The issue you're experiencing with the JsonOutputParser returning only one chunk in streaming mode might be related to how the parser handles streaming data. The JsonOutputParser is designed to process streamed JSON by incrementally building up the JSON object as tokens are received. However, any steps in the chain that operate on finalized inputs rather than on input streams can break streaming functionality ^[1].

Here are a few things you can check or try:

Ensure Streaming Compatibility: Make sure that all components in your chain are compatible with streaming. If any part of the chain processes data in a way that requires the complete input before producing output, it can disrupt the streaming process.
Check for Finalized Input Operations: Avoid any operations in the chain that require finalized inputs. For example, if you have any functions or methods that expect the entire JSON object to be available before processing, it will prevent streaming from working correctly.
Use the Correct Parser: If the StringOutputParser works fine, it indicates that the issue might be specific to how the JsonOutputParser is handling the data. Ensure that the JsonOutputParser is correctly configured to handle streaming JSON data.

If these suggestions don't resolve the issue, it might be worth reviewing the specific implementation details of your chain to ensure that all components are set up to handle streaming data correctly.

To continue talking to Dosu, mention @dosu.