Cannot return source documents within AgentExecutors

[leerobert]

I'm trying to write an agent executor that can use multiple tools and return direct from VectorDBQAChain with source documents.

Here is my setup:

      const chat = new ChatOpenAI({
        modelName: 'gpt-4',
        temperature: 0,
        streaming: false,
        openAIApiKey: OPENAI_API_KEY,
        callbackManager: callbacks
    });

    let chain = VectorDBQAChain.fromLLM(chat, vectorStore, { returnSourceDocuments: false });
    const qaTool = new ChainTool({
        name: '...',
        description: '...',
        chain: chain,
        returnDirect: true
    });

    const tools = [new Serper(SERPER_API_KEY), qaTool];

    const prompt = ChatAgent.createPrompt(tools, {
        prefix: `You are a helpful assistant. Use the following tools:`,
        suffix:  `Begin! Remember to either answer the prompt or use the tools to help you answer the prompt.`
    });

    const chatPrompt = ChatPromptTemplate.fromPromptMessages([
        new SystemMessagePromptTemplate(prompt),
        HumanMessagePromptTemplate.fromTemplate(`{input}

This was your previous work (but I haven't seen any of it! I only see what you return as final answer):
{agent_scratchpad}`)
    ]);

    const llmChain = new LLMChain({
        prompt: chatPrompt,
        llm: chat
    });

    const agent = new ChatAgent({
        llmChain,
        allowedTools: tools.map((tool) => tool.name)
    });
    let verbose = true;

    return AgentExecutor.fromAgentAndTools({
        agent,
        tools,
        returnIntermediateSteps: true,
        verbose
    });

Getting the following error when I set returnSourceDocuments to true:

Entering new agent_executor chain...
Thought: I need to check the organization's budget information for the project.

Action: 

{ "action": "qa", "action_input": "What is the amount we budgeted for our project?" }

Error: return values have multiple keys, `run` only supported when one key currently
    at VectorDBQAChain.run (file:///node_modules/.pnpm/langchain@0.0.45_ku2vntnma2imfrsyohnkxzette/node_modules/langchain/dist/chains/base.js:43:15)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async ChainTool.call (file:////node_modules/.pnpm/langchain@0.0.45_ku2vntnma2imfrsyohnkxzette/node_modules/langchain/dist/agents/tools/base.js:31:22)
    at async AgentExecutor._call (file:////node_modules/.pnpm/langchain@0.0.45_ku2vntnma2imfrsyohnkxzette/node_modules/langchain/dist/agents/executor.js:79:19)
    at async AgentExecutor.call (file:///node_modules/.pnpm/langchain@0.0.45_ku2vntnma2imfrsyohnkxzette/node_modules/langchain/dist/chains/base.js:61:28)

[nsantos16]

Same problem here!

[mangoplane]

I'm getting this same issue, in a similar context. Yet to identify the cause, something to do with this part of the code base, where it throws an error when the number of "keys" isn't exactly one:

https://github.com/hwchase17/langchainjs/blob/ace0be9877e38c80292764344d6223c8c85116e2/langchain/src/chains/base.ts#L97

If I figure out a fix, I'll PR it or raise an issue with the fix.

I'd really appreciate it if someone knows how I go about fixing this issue.

Thanks.

[ommeirelles]

As mentioned by @mangoplane , the BaseChain is only accepting single key result.

To make it work, using retrievalQaChain as example, instead of using the run method you can use the call directly.

example:

const model = new ChatOpenAI({
  openAIApiKey: OPENAI_KEY,
  temperature: 0.5,
  modelName: "gpt-3.5-turbo",
});
const retrieval = new RetrievalQAChain({
  combineDocumentsChain: loadQAStuffChain(model),
  retriever: store.asRetriever(),
  returnSourceDocuments: true,
});
const res = await retrieval.call({ ["query"]: queryInput });

and it will produce a result like object:

{
    "text": "De acordo com o texto, não há informações sobre quem criou o deus responsável pela criação dos dragões.",
    "sourceDocuments": [
        {
            "pageContent": "# Genesis\n\n## Chapter I (The Awakening of the Gods)\n\nIn the beginning there was only a big, gaping void. It was everywhere and nowhere.\n\nIn this emptiness two powerful entities appeared who were eventually known as the elder gods: Fardos the Creator and Uman Zathroth who combined in himself two unequal halves. One of these halves was Uman the Wise, a benign god who was gifted with divine intellect, while Zathroth the Destroyer was the other, darker half. These were the two halves of one single enigmatical entity, and even though either of them was perfectly capable of acting on his own just as if he had been fully independent, independent they were not. They were bound together by an eternal bond that could not be broken, and their destiny was one.\n\nNobody knows where the elder gods came from, or whether they had always existed and eventually awoke from the slumber of infinity. But somewhere along the way they decided to create a universe. Surely Fardos was the initiator, for he was driven by the need to create and give life. He was overflowing with creative power and impatient to release it, so he stepped into existence and began to unleash his powers. However, none of his attempts to create were successful. All of his creations were swallowed by the void before they were completed, and none survived.\nUman Zathroth regarded Fardos's undertakings thoughtfully. Uman was sagacious",
            "metadata": {
                "loc.lines.from": 1,
                "loc.lines.to": 10,
                "source": " Genesis 845f8cda606d4ba3a5d6ba1d63066f83.md"
            }
        },
        {
            "pageContent": " the course of centuries were burnt down to the ground, and their renowned forges were lost forever.\n\nThus the dragons had taken over the rule of the land, but the war was by no means over. Their bitter enemies, cyclopes and orcs, resented what they felt was an imprisonment in the bowels of the earth, and they continued the fight from their subterranean hideouts. And in fact the dragons, who had already been weakened in the course of the previous battles, suffered serious losses. But now war also erupted among the former allies, as cyclopes and orcs competed for food and space in their subterranean abodes. And even though no side was strong enough to overcome the others the war went on with undiminished force, and all of the races suffered greatly in the epic struggle. The land was scattered with bodies, and while it seemed that life itself would be wiped from the face of Tibia the losses of all races that were involved daily grew in number. It was as if the living would drown in the bodies of the slain.\n\nThe elder gods watched as the cataclysmic battle went on. They felt no pity for those that were slain because they cared little for Zathroth's creatures, but they knew that something was missing, that somebody was needed to take care of the bodies and souls of those who ceased to live. They began to look for a solution, and finally Uman proposed that a new god should be created, a god who should see to it that the dead would be taken care of. They decided that earth, which in a way was the giver of life, should have a part in taking it back, and that Uman should be the newly created god's father. But alas! The elder gods were not as cautious as they should have been, and so Zathroth the Destroyer learnt about their plans all too soon. He was fascinated by the idea of death from the start, because he saw in it a new chance to bring further havoc and destruction into the world. Soon he had devised a vicious plan. He posed as his good half Uman to fool earth, and with it he sired another god: Urgith the Master of the Undead",
            "metadata": {
                "loc.lines.from": 68,
                "loc.lines.to": 72,
                "source": "genesis.txt"
            }
        },
        {
            "pageContent": "# Genesis\n\n## Chapter I (The Awakening of the Gods)\n\nIn the beginning there was only a big, gaping void. It was everywhere and nowhere.\n\nIn this emptiness two powerful entities appeared who were eventually known as the elder gods: Fardos the Creator and Uman Zathroth who combined in himself two unequal halves. One of these halves was Uman the Wise, a benign god who was gifted with divine intellect, while Zathroth the Destroyer was the other, darker half. These were the two halves of one single enigmatical entity, and even though either of them was perfectly capable of acting on his own just as if he had been fully independent, independent they were not. They were bound together by an eternal bond that could not be broken, and their destiny was one.\n\nNobody knows where the elder gods came from, or whether they had always existed and eventually awoke from the slumber of infinity. But somewhere along the way they decided to create a universe. Surely Fardos was the initiator, for he was driven by the need to create and give life. He was overflowing with creative power and impatient to release it, so he stepped into existence and began to unleash his powers. However, none of his attempts to create were successful. All of his creations were swallowed by the void before they were completed, and none survived.\nUman Zathroth regarded Fardos's undertakings thoughtfully. Uman was sagacious and held awesome magical powers. Most importantly, however, he was driven by an insatiable hunger for knowledge and enlightenment. In his essence he resembled Fardos, but where Fardos worked openly and logically, Uman's domain was the realm of mystery. Still, he shared Fardos's interest in creation, whereas his dark half Zathroth was essentially corruptive. Zathroth was a vain god who was painfully aware that his own creative powers were poor. Because of this he looked at Fardos's work of creation with jealousy, and from the very beginning he was determined to prevent or at least corrupt it in any way he could. Fardos, who did not suspect this, asked him for assistance because he",
            "metadata": {
                "loc.lines.from": 1,
                "loc.lines.to": 10,
                "source": " genesis.txt"
            }
        },
        {
            "pageContent": " the course of centuries were burnt down to the ground, and their renowned forges were lost forever.\n\nThus the dragons had taken over the rule of the land, but the war was by no means over. Their bitter enemies, cyclopes and orcs, resented what they felt was an imprisonment in the bowels of the earth, and they continued the fight from their subterranean hideouts. And in fact the dragons, who had already been weakened in the course of the previous battles, suffered serious losses. But now war also erupted among the former allies, as cyclopes and orcs competed for food and space in their subterranean abodes. And even though no side was strong enough to overcome the others the war went on with undiminished force, and all of the races suffered greatly in the epic struggle. The land was scattered with bodies, and while it seemed that life itself would be wiped from the face of Tibia the losses of all races that were involved daily grew in number. It was as if the living would drown in the bodies of the slain.\n\nThe elder gods watched as the cataclysmic battle went on. They felt no pity for those that were slain because they cared little for Zathroth's creatures, but they knew that something was missing, that somebody was needed to take care of the bodies and souls of those who ceased to live. They began to look for a solution, and finally Uman proposed that a new god should be created, a god",
            "metadata": {
                "loc.lines.from": 68,
                "loc.lines.to": 72,
                "source": "Genesis 845f8cda606d4ba3a5d6ba1d63066f83.md"
            }
        }
    ]
}

[basyonic]

The issue is at this line where it mandates having only one argument returned from the chain https://github.com/hwchase17/langchainjs/blob/542c2f1cf35243c2967a3a338652acd1e46f65d2/langchain/src/chains/base.ts#L107

I made a workaround based on the info provided by @ommeirelles to use call instead of run while still using tools and agent.

by creating a tool and implement its call method to call RetrievalQAChain.call method.

here is my code for the tool

import { BaseLanguageModel } from "langchain/base_language";
import { BaseChain,RetrievalQAChain } from "langchain/chains";
import { PineconeStore } from 'langchain/vectorstores/pinecone';

interface VectorStoreTool {
    vectorStore: PineconeStore;
    llm: BaseLanguageModel;
  }

export class KBSearchTool implements VectorStoreTool{
    vectorStore: PineconeStore;

    llm: BaseLanguageModel;

    name: string;

    description: string;

    chain: BaseChain;
    returnDirect:boolean;

    constructor(name: string, description: string, fields: VectorStoreTool ) {
      this.name = name;
      this.description = description;
      this.vectorStore = fields.vectorStore;
      this.llm = fields.llm;
      this.returnDirect=true;
      this.chain = RetrievalQAChain.fromLLM(
        this.llm,
        this.vectorStore.asRetriever(),
        {
          returnSourceDocuments: true, //The number of source documents returned is 4 by default
        }
      );

    }

    static getDescription(name: string, description: string): string {
      return `${name} - useful for when you need to ask questions about ${description}.`
    }

    async call(input: string) {
         //call instead of run
        //return this.chain.run(input);
        return this.chain.call({query:input});
      }
  }

then you can add it normally in the list of tools provided to your agent

const SearchKBTool = new   KBSearchTool('My KB','Organization policies',{vectorStore:vectorStore,llm:model})
 const tools = [
    SearchKBTool,
  ];

  const memory = new BufferMemory({ returnMessages: true, memoryKey: "chat_history",inputKey:'input', outputKey:'output' });
  const executor = await initializeAgentExecutorWithOptions(tools, model, {
    agentType: "chat-conversational-react-description",
    maxIterations: 3,
    earlyStoppingMethod: 'generate',
    memory: memory,
    verbose: true,
    returnIntermediateSteps:false,
  });

[dosubot[bot]]

Hi, @leerobert! I'm Dosu, and I'm helping the langchainjs team manage their backlog. I wanted to let you know that we are marking this issue as stale.

Based on my understanding, the issue you reported was related to an error occurring with multiple return values when setting returnSourceDocuments to true in the agent executor setup. Other users, such as nsantos16 and mangoplane, have also experienced the same issue. A workaround suggested by ommeirelles was to use the call method instead of run in the code, and basyonic even provided a code snippet for a tool that implements this workaround. It seems that this solution has been successful, as it has received positive reactions from ommeirelles, 13point5, luandro, and nstankov-bg.

Before we close this issue, we wanted to check with you if it is still relevant to the latest version of the langchainjs repository. If it is, please let us know by commenting on this issue. Otherwise, feel free to close the issue yourself, or it will be automatically closed in 7 days.

Thank you for your contribution to langchainjs!

[marius-ciocoiu]

Still not resolved in the latest version.

[kinivi]

Still a problem