bug: Conversational Retrieval Agent streaming does not work

[faileon]

Hello, I am trying to create an Agent based on the following documentation .

No matter what I try, I get the following error:

const inputValues = promptMessage.inputVariables.reduce((acc, inputVariable) => { Error: Missing value for input variable chat_history

Even with the simplest example:

    const llm = new ChatOpenAI({
        openAIApiKey: process.env['OPENAI_API_KEY'],
        temperature: 0.4,
        modelName: 'gpt-3.5-turbo-1106',
    });

    const tools = [];

    const agent = await createConversationalRetrievalAgent(llm, tools);

    const stream = await agent.stream({ input: "Hello, my name is Tom!" });

    for await (const chunk of stream) {
        console.log('chunk...', chunk);
    }

I tried setting the agent with memoryKey, but that did not help.

const agent = await createConversationalRetrievalAgent(llm, tools, {
        memoryKey: 'chat_history',
        outputKey: 'output',
    });

I also tried creating the more customizable agent with initializeAgentExecutorWithOptions(), passing it my own instance of OpenAIAgentTokenBufferMemory etc., but it throws the same error.

EDIT: I am running the latest langchain version 0.0.203

EDIT2: stackblitz https://stackblitz.com/edit/stackblitz-starters-ajoxqd?file=index.js

[dosubot[bot]]

🤖

Hey @faileon, good to see you back here!

Based on the error message and the context you've provided, it seems like the issue is related to the format of the chat_history field. In the LangChainJS framework, the chat_history field is expected to be an array of BaseMessage instances. However, in your code, it appears that you're passing an array of objects that are not instances of BaseMessage.

Here's how you can modify your code to use BaseMessage instances:

import { BaseMessage, HumanMessage, AIMessage } from 'langchain/schema';

// ...

const json = [
  new HumanMessage({ content: "hi my name is Mario", additional_kwargs: {} }),
  new AIMessage({ content: "Hello, Mario! How can I assist you today?", additional_kwargs: {} }),
];
const messages = json.map((x) => mapStoredMessageToChatMessage(x));
const ok = messages.every((x) => x instanceof BaseMessage);
console.log("ok", ok); // prints true
const memory = new BufferMemory({
  chatHistory: new ChatMessageHistory(messages),
  memoryKey: "chat_history",
});
const executor = await initializeAgentExecutorWithOptions(tools, model, {
  agentType: "chat-conversational-react-description",
  memory,
});
await executor.call({ input: "what is my name?" })

In this code, new HumanMessage({ content: "hi my name is Mario", additional_kwargs: {} }) and new AIMessage({ content: "Hello, Mario! How can I assist you today?", additional_kwargs: {} }) create instances of HumanMessage and AIMessage, respectively, which are subclasses of BaseMessage.

Please note that the HumanMessage and AIMessage classes require the message content and additional kwargs as parameters. If your chat history includes additional information, you'll need to adjust the code accordingly.

This solution was suggested in a similar issue: Field "chat_history" in prompt uses a MessagesPlaceholder, which expects an array of BaseMessages as an input value.

I hope this helps! If you have any other questions, feel free to ask.

Best, Dosu

Sources

• Field "chat_history" in prompt uses a MessagesPlaceholder, which expects an array of BaseMessages as an input value.
• Crash with ChatAgent and BufferMemory
• chat-conversational-react-description is not working
• examples/src/guides/conversational_retrieval/agent.ts
• langchain/src/agents/chat_convo/index.ts

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[faileon]

Dosu, I am not passing any messages to a memory object. I am not creating memory myself, I am letting the createConversationalRetrievalAgent create one.

Also this is the very first call to the agent, so history would be empty.

However I also tried creating new memory and agent like so :

const memory = new OpenAIAgentTokenBufferMemory({
            llm,
            memoryKey: 'chat_history',
            outputKey: 'output',
            chatHistory: new ChatMessageHistory([])
        })

        const executor = initializeAgentExecutorWithOptions(tools, llm, {
            agentType: 'openai-functions',
            memory,
            returnIntermediateSteps: true,
            agentArgs: {
                prefix: `Do your best to answer the questions. Feel free to use any tools available to look up relevant information, only if necessary.`,
            },
        });

But the same error is thrown.

[dosubot[bot]]

🤖

Based on the error message and the context you've provided, it seems like the issue is related to the format of the chat_history field. In the LangChainJS framework, the chat_history field is expected to be an array of BaseMessage instances. However, in your code, it appears that you're passing an array of objects that are not instances of BaseMessage.

Here's how you can modify your code to use BaseMessage instances:

import { BaseMessage, HumanMessage, AIMessage } from 'langchain/schema';

// ...

const json = [
  new HumanMessage({ content: "hi my name is Mario", additional_kwargs: {} }),
  new AIMessage({ content: "Hello, Mario! How can I assist you today?", additional_kwargs: {} }),
];
const messages = json.map((x) => mapStoredMessageToChatMessage(x));
const ok = messages.every((x) => x instanceof BaseMessage);
console.log("ok", ok); // prints true
const memory = new BufferMemory({
  chatHistory: new ChatMessageHistory(messages),
  memoryKey: "chat_history",
});
const executor = await initializeAgentExecutorWithOptions(tools, model, {
  agentType: "chat-conversational-react-description",
  memory,
});
await executor.call({ input: "what is my name?" })

In this code, new HumanMessage({ content: "hi my name is Mario", additional_kwargs: {} }) and new AIMessage({ content: "Hello, Mario! How can I assist you today?", additional_kwargs: {} }) create instances of HumanMessage and AIMessage, respectively, which are subclasses of BaseMessage.

Please note that the HumanMessage and AIMessage classes require the message content and additional kwargs as parameters. If your chat history includes additional information, you'll need to adjust the code accordingly.

This solution was suggested in a similar issue: Field "chat_history" in prompt uses a MessagesPlaceholder, which expects an array of BaseMessages as an input value.

Additionally, you need to set the memoryKey property on your custom memory as chat_history. Here is how you can do it:

const memory = new BufferMemory({
    chatHistory: new ChatMessageHistory(history),
    memoryKey: 'chat_history'
  })

This was suggested in another similar issue: chat-conversational-react-description is not working

I hope this helps! If you have any other questions, feel free to ask.

Sources

• Field "chat_history" in prompt uses a MessagesPlaceholder, which expects an array of BaseMessages as an input value.
• chat-conversational-react-description is not working
• examples/src/chains/conversational_qa_built_in_memory.ts
• examples/src/chains/conversational_qa.ts

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[faileon]

The difference I am seeing in the code is that during AgentExecutor._call() the inputs are properly assigned:

{
        input: "Some query",
        chat_history: []
}

whereas the streaming call inputs does not contain the chat_history key:

{
        input: "Some query"
}

[faileon]

For anyone having the same issue the current workaround thanks to @bracesproul is passing the chat_history yourself and using custom memory.

const memoryModel = new ChatOpenAI({
            openAIApiKey: '',
            temperature: 0,
            modelName: 'gpt-3.5-turbo',
        })

const chatModel = new ChatOpenAI({
            openAIApiKey: '',
            temperature: 0,
            modelName: 'gpt-4-1106-preview' // or gpt-3.5-turbo-1106
        })

const memory = new OpenAIAgentTokenBufferMemory({
            llm: memoryModel,
            chatHistory: new ChatMessageHistory([]),
            memoryKey: 'chat_history',
            outputKey: 'output',
            inputKey: 'input'
        })

const tools = [/*...your toolkit*/];
const agent = await initializeAgentExecutorWithOptions(tools, chatModel, {
            agentType: 'openai-functions',
            memory,
            returnIntermediateSteps: true
        });

const chat_history = await agent.memory.loadMemoryVariables({});
const stream = await agent.stream({input: 'My name is (chka-chka, Slim Shady)', chat_history});
for await (const chunk of stream){
    console.log(chunk)
}

[dosubot[bot]]

Hi, @faileon,

I'm helping the langchainjs team manage their backlog and am marking this issue as stale. From what I understand, you encountered an error with the Conversational Retrieval Agent streaming not working as expected, and despite suggestions and workarounds provided, the issue persisted. It seems that you were able to resolve the issue with a workaround provided by user "bracesproul" involving passing the chat_history and using custom memory.

Could you please confirm if this issue is still relevant to the latest version of the langchainjs repository? If it is, please let the langchainjs team know by commenting on the issue. Otherwise, feel free to close the issue yourself, or it will be automatically closed in 7 days. Thank you for your understanding and contribution to the langchainjs community!