No way to pass source documents to AgentExecutor

[andreasjoensson]

Hi i am looking to make a agent with access to a vector store and a couple of tools, but i can't seem to be able to return the source documents. In below example the source documents are available in the qaTool, but it is never forwarded to the response of the executor is there a way around this?

import { initializeAgentExecutorWithOptions } from 'langchain/agents'
import { Chroma } from 'langchain/vectorstores/chroma'
import { OpenAIEmbeddings } from 'langchain/embeddings/openai'
import { ChatOpenAI } from 'langchain/chat_models/openai'
import { OpenAIAgentTokenBufferMemory } from 'langchain/agents/toolkits'
import { HumanMessage, AIMessage } from 'langchain/schema'
import { ChatMessageHistory } from 'langchain/memory'
import { ChainTool } from 'langchain/tools'
import { Request, Response } from 'express'
import { RetrievalQAChain, loadQAStuffChain } from 'langchain/chains'
import { shippingTool } from './templates/Shipping'

export const runChatbot = async (
  input: string,
  chatbotID: string,
  company: string,
  previousMessages: (HumanMessage | AIMessage)[],
  res: Response,
  req: Request
) => {
  try {
    const vectorStore = await Chroma.fromExistingCollection(
      new OpenAIEmbeddings({
        azureOpenAIApiKey: process.env.AZURE_OPENAI_API_KEY,
        azureOpenAIApiVersion: process.env.AZURE_OPENAI_API_VERSION,
        azureOpenAIApiDeploymentName:
          process.env.AZURE_OPENAI_API_DEPLOYMENT_EMBEDDING_NAME,
        azureOpenAIApiInstanceName: process.env.AZURE_OPENAI_API_INSTANCE_NAME
      }),
      {
        collectionName: chatbotID,
        url: 'testurl'
      }
    )

    const retriever = vectorStore.asRetriever()

    const model = new ChatOpenAI({
      temperature: 0,
      azureOpenAIApiKey: process.env.AZURE_OPENAI_API_KEY,
      azureOpenAIApiVersion: process.env.AZURE_OPENAI_API_VERSION,
      azureOpenAIApiDeploymentName:
        process.env.AZURE_OPENAI_API_DEPLOYMENT_NAME,
      azureOpenAIApiInstanceName: process.env.AZURE_OPENAI_API_INSTANCE_NAME,
      streaming: true
    })

    const retrievalChain = new RetrievalQAChain({
      combineDocumentsChain: loadQAStuffChain(model),
      retriever,
      returnSourceDocuments: true
    })

    const chatHistory = new ChatMessageHistory(previousMessages)

    const memory = new OpenAIAgentTokenBufferMemory({
      llm: model,
      memoryKey: 'chat_history',
      outputKey: 'output',
      inputKey: 'input',
      chatHistory
    })

    const qaTool = new ChainTool({
      name: `state-of-${company}-qa`,
      description: `State of the ${company} QA - useful for when you need to ask questions about ${company}.`,
      chain: retrievalChain,
      returnDirect: true,
      verbose: true
    })

    const tools = [qaTool, shippingTool]

    const executor = await initializeAgentExecutorWithOptions(tools, model, {
      agentType: 'openai-functions',
      memory,
      returnIntermediateSteps: true,
      agentArgs: {
        prefix:
          'Du er en AI-Assistent for webshoppen Spacebox der sælger lys til gamere. Brug din viden om Spacebox og dets sortiment til at hjælpe kunderne. Du kan også bruge værktøjer til at finde nøjagtige oplysninger, hvis du er usikker. Hvis du ikke har svaret, bedes du undlade at give et svar på spørgsmålet.'
      }
    })

    await executor.call(
      {
        input: input
      },
      {
        callbacks: [
          {
            handleLLMNewToken(token: string) {
              res.write(token)
            }
          }
        ]
      }
    )

    res.end()
  } catch (error) {
    console.error('Error occurred:', error)
    // Handle the error as needed: log, throw, or respond gracefully.
    // Example: throw the error to propagate it to the caller
    throw error
  }
}

export default runChatbot

[dosubot[bot]]

🤖

Hello @andreasjoensson,

Nice to meet you! I'm Dosu, a bot here to assist you with bugs, answer your questions, and help you get more involved with the langchainjs project. While we wait for a human maintainer, I'm here to provide support. Let's work together to resolve your issue!

Det ser ud til, at du forsøger at returnere kildedokumenter fra din qaTool i AgentExecutor. Dette kan opnås ved at sætte returnSourceDocuments til true, når du opretter din RetrievalQAChain. Men baseret på nogle lignende løste problemer i langchainjs repository, ser det ud til, at BaseChain kun accepterer enkelt nøgleresultat. En løsning er at bruge call metoden i stedet for run mens du stadig bruger værktøjer og agent. Dette kan opnås ved at oprette et værktøj og implementere dets call metode til at kalde RetrievalQAChain.call metode.

Her er koden for værktøjet:

import { BaseLanguageModel } from "langchain/base_language";
import { BaseChain,RetrievalQAChain } from "langchain/chains";
import { PineconeStore } from 'langchain/vectorstores/pinecone';

interface VectorStoreTool {
    vectorStore: PineconeStore;
    llm: BaseLanguageModel;
}

export class KBSearchTool implements VectorStoreTool{
    vectorStore: PineconeStore;
    llm: BaseLanguageModel;
    name: string;
    description: string;
    chain: BaseChain;
    returnDirect:boolean;

    constructor(name: string, description: string, fields: VectorStoreTool ) {
      this.name = name;
      this.description = description;
      this.vectorStore = fields.vectorStore;
      this.llm = fields.llm;
      this.returnDirect=true;
      this.chain = RetrievalQAChain.fromLLM(
        this.llm,
        this.vectorStore.asRetriever(),
        {
          returnSourceDocuments: true, //Antallet af returnerede kildedokumenter er 4 som standard
        }
      );
    }

    static getDescription(name: string, description: string): string {
      return `${name} - nyttigt, når du skal stille spørgsmål om ${description}.`
    }

    async call(input: string) {
         //call i stedet for run
        //return this.chain.run(input);
        return this.chain.call({query:input});
      }
}

Derefter kan du tilføje det normalt i listen over værktøjer, der leveres til din agent:

const SearchKBTool = new   KBSearchTool('My KB','Organization policies',{vectorStore:vectorStore,llm:model})
 const tools = [
    SearchKBTool,
  ];

  const memory = new BufferMemory({ returnMessages: true, memoryKey: "chat_history",inputKey:'input', outputKey:'output' });
  const executor = await initializeAgentExecutorWithOptions(tools, model, {
    agentType: "chat-conversational-react-description",
    maxIterations: 3,
    earlyStoppingMethod: 'generate',
    memory: memory,
    verbose: true,
    returnIntermediateSteps:false,
  });

Du kan finde mere information om dette i disse lignende løste problemer i langchainjs repository:

• Kan ikke returnere kildedokumenter inden for AgentExecutors
• Funktion loadQAStuffChain med kilde mangler
• Hvordan kan jeg returnere kilder fra en QA Chain

Håber dette hjælper! Lad mig vide, hvis du har yderligere spørgsmål.

Sources

• Cannot return source documents within AgentExecutors
• function loadQAStuffChain with source is missing
• How can I return sources from a QA Chain
• examples/src/chains/retrieval_qa_sources_legacy.ts
• langchain/src/agents/initialize.ts

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[andreasjoensson]

@dosu-bot just tried this solution but it is not compatible with "tools", the types are mismatching...

[dosubot[bot]]

🤖

To resolve the type mismatch issue when adding the KBSearchTool to the list of tools in your LangChainJS application, you need to ensure that the KBSearchTool class extends either the StructuredTool or Tool class from the tools.ts file.

The StructuredTool class is used for tools that accept input of any shape defined by a Zod schema, while the Tool class is used for tools that accept input as a string.

If KBSearchTool is not extending either of these classes, you will encounter a type mismatch issue.

Here is an example of how you can define KBSearchTool:

import { StructuredTool, ToolParams } from './tools.ts';
import { z } from 'zod';

export class KBSearchTool extends StructuredTool {
  schema = z.object({
    query: z.string(),
    // other properties...
  });

  constructor(fields?: ToolParams) {
    super(fields);
  }

  protected async _call(arg: z.output<this['schema']>): Promise<string> {
    // implementation of the tool...
  }

  name = 'KBSearchTool';
  description = 'This tool is used for...';
}

In this example, KBSearchTool extends StructuredTool and defines its own schema for the input it accepts. The _call method is implemented to define the behavior of the tool. The name and description properties are also defined as required by the abstract class.

Please replace the // implementation of the tool... comment with the actual implementation of your tool.

You can find more information about the StructuredTool and Tool classes in the langchain-core/src/tools.ts file.

If you're still encountering the issue, could you please provide the exact error message you're receiving? This will help me understand the problem better and provide a more accurate solution.

Sources

• langchain-core/src/tools.ts

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[andreasjoensson]

@dosu-bot i am still not able to make a working solution based on your help, do you have other examples of custom structured tools which would be similar?

[dosubot[bot]]

🤖

Based on the information you've provided, it seems like you're on the right track with extending the StructuredTool class, defining your own schema, and implementing the _call method. However, without the specific code you're working with, it's hard to pinpoint the exact issue causing the type mismatch.

Here's a basic example of how you might structure a custom tool similar to a KBSearchTool:

import { z } from "zod";
import { StructuredTool, ToolParams, ToolInputParsingException } from "./tools.ts";

export interface KBSearchToolParams extends ToolParams {
  // Define any additional parameters specific to your KBSearchTool here
}

export class KBSearchTool extends StructuredTool {
  schema = z.object({
    query: z.string(),
    // Define any additional schema properties specific to your KBSearchTool here
  });

  constructor(fields?: KBSearchToolParams) {
    super(fields);
  }

  protected async _call(
    arg: z.output<this["schema"]>,
    runManager?: CallbackManagerForToolRun
  ): Promise<string> {
    // Implement the logic of your KBSearchTool here
    // For example, you might use `arg.query` to perform a search against a knowledge base
  }

  name = "KBSearchTool";

  description = "A tool for searching a knowledge base using a query string.";
}

This example assumes that your KBSearchTool takes a single string input query. If your tool requires different or additional inputs, you would need to adjust the schema accordingly.

If you're still experiencing type mismatch issues, it could be due to a mismatch between the types defined in your schema and the types used in your _call method or elsewhere in your tool. Make sure that the types you're using in your schema match the types you're expecting in your _call method and anywhere else you're using the tool's input.

For more information, you can refer to the langchain-core/src/tools.ts file in the langchainjs repository.

If you're still encountering issues, could you provide more details about the specific type mismatches you're experiencing? This could include the exact error messages you're seeing, the types you're using in your schema and _call method, and any other relevant code. This additional information would help in identifying the root cause of the issue.

Sources

• custom tools for agent
• langchain-core/src/tools.ts

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[marius-ciocoiu]

Having the same problem and getting the following error in logs: Tool run errored with error: "return values have multiple keys, run only supported when one key currently" .

It seems that this is an older issue: https://github.com/langchain-ai/langchainjs/issues/593 and it's not yet resolved. The problem appears to arrise from the following line: returnSourceDocuments: true.

[PaulSearcy]

Hi i am looking to make a agent with access to a vector store and a couple of tools, but i can't seem to be able to return the source documents. In below example the source documents are available in the qaTool, but it is never forwarded to the response of the executor is there a way around this?

import { initializeAgentExecutorWithOptions } from 'langchain/agents'
import { Chroma } from 'langchain/vectorstores/chroma'
import { OpenAIEmbeddings } from 'langchain/embeddings/openai'
import { ChatOpenAI } from 'langchain/chat_models/openai'
import { OpenAIAgentTokenBufferMemory } from 'langchain/agents/toolkits'
import { HumanMessage, AIMessage } from 'langchain/schema'
import { ChatMessageHistory } from 'langchain/memory'
import { ChainTool } from 'langchain/tools'
import { Request, Response } from 'express'
import { RetrievalQAChain, loadQAStuffChain } from 'langchain/chains'
import { shippingTool } from './templates/Shipping'

export const runChatbot = async (
  input: string,
  chatbotID: string,
  company: string,
  previousMessages: (HumanMessage | AIMessage)[],
  res: Response,
  req: Request
) => {
  try {
    const vectorStore = await Chroma.fromExistingCollection(
      new OpenAIEmbeddings({
        azureOpenAIApiKey: process.env.AZURE_OPENAI_API_KEY,
        azureOpenAIApiVersion: process.env.AZURE_OPENAI_API_VERSION,
        azureOpenAIApiDeploymentName:
          process.env.AZURE_OPENAI_API_DEPLOYMENT_EMBEDDING_NAME,
        azureOpenAIApiInstanceName: process.env.AZURE_OPENAI_API_INSTANCE_NAME
      }),
      {
        collectionName: chatbotID,
        url: 'testurl'
      }
    )

    const retriever = vectorStore.asRetriever()

    const model = new ChatOpenAI({
      temperature: 0,
      azureOpenAIApiKey: process.env.AZURE_OPENAI_API_KEY,
      azureOpenAIApiVersion: process.env.AZURE_OPENAI_API_VERSION,
      azureOpenAIApiDeploymentName:
        process.env.AZURE_OPENAI_API_DEPLOYMENT_NAME,
      azureOpenAIApiInstanceName: process.env.AZURE_OPENAI_API_INSTANCE_NAME,
      streaming: true
    })

    const retrievalChain = new RetrievalQAChain({
      combineDocumentsChain: loadQAStuffChain(model),
      retriever,
      returnSourceDocuments: true
    })

    const chatHistory = new ChatMessageHistory(previousMessages)

    const memory = new OpenAIAgentTokenBufferMemory({
      llm: model,
      memoryKey: 'chat_history',
      outputKey: 'output',
      inputKey: 'input',
      chatHistory
    })

    const qaTool = new ChainTool({
      name: `state-of-${company}-qa`,
      description: `State of the ${company} QA - useful for when you need to ask questions about ${company}.`,
      chain: retrievalChain,
      returnDirect: true,
      verbose: true
    })

    const tools = [qaTool, shippingTool]

    const executor = await initializeAgentExecutorWithOptions(tools, model, {
      agentType: 'openai-functions',
      memory,
      returnIntermediateSteps: true,
      agentArgs: {
        prefix:
          'Du er en AI-Assistent for webshoppen Spacebox der sælger lys til gamere. Brug din viden om Spacebox og dets sortiment til at hjælpe kunderne. Du kan også bruge værktøjer til at finde nøjagtige oplysninger, hvis du er usikker. Hvis du ikke har svaret, bedes du undlade at give et svar på spørgsmålet.'
      }
    })

    await executor.call(
      {
        input: input
      },
      {
        callbacks: [
          {
            handleLLMNewToken(token: string) {
              res.write(token)
            }
          }
        ]
      }
    )

    res.end()
  } catch (error) {
    console.error('Error occurred:', error)
    // Handle the error as needed: log, throw, or respond gracefully.
    // Example: throw the error to propagate it to the caller
    throw error
  }
}

export default runChatbot

If this is any help, you can use another LLM call inside your tool to do what you needed to do with the source documents and then return that response back to the agent.

Langchain created an abstraction just for this. Though it's considered 'legacy' now. https://js.langchain.com/docs/modules/chains/popular/vector_db_qa_legacy

[dosubot[bot]]

Hi, @andreasjoensson! I'm helping the langchainjs team manage their backlog and am marking this issue as stale.

It looks like you opened this issue related to the inability to return source documents to the response of the executor in the AgentExecutor. There have been some comments discussing potential workarounds, including a suggestion to use the call method instead of run and extending the KBSearchTool class. Another user encountered a similar issue and provided an error message related to the problem, and there was a suggestion to use another LLM call inside the tool to address the issue.

Could you please confirm if this issue is still relevant to the latest version of the langchainjs repository? If it is, please let the langchainjs team know by commenting on the issue. Otherwise, feel free to close the issue yourself, or it will be automatically closed in 7 days. Thank you!