System Template and ChatPromptTemplate never makes it into the ChatVectorDB

[OmriNach]

I noticed this as i created identical templates in python and javascript and they were giving different behaviors. Followed the python guide for creating ChatVectorDB, then did the same format for javascript. Digging into the definitions of my chain I dont ever see the system message template

[OmriNach]

[OmriNach]

above is javascript, below is python. The 'messages' variable is not even included in the javascript version

[OmriNach]

i think the issue is here, based on limited knowledge of javascript packages: LoadQAStuffChain in python and javascript. I don't see the prompt argument logic in the javascript version

[OmriNach]

FIXES: in chat_vector_db_chain.js: changed qa_prompt line static fromLLM(llm, vectorstore, options = {}) { const { questionGeneratorTemplate, qaTemplate, ...rest } = options; const question_generator_prompt = PromptTemplate.fromTemplate(questionGeneratorTemplate || question_generator_template); const qa_prompt = qaTemplate || PromptTemplate.fromTemplate(qa_template); const qaChain = loadQAStuffChain(llm, { prompt: qa_prompt }); const questionGeneratorChain = new LLMChain({ prompt: question_generator_prompt, llm, }); const instance = new this({ vectorstore, combineDocumentsChain: qaChain, questionGeneratorChain, ...rest, }); return instance; }

in qa_answering load.js: changed prompt definition line:

export const loadQAStuffChain = (llm, params = {}) => { const { prompt } = params; const actualPrompt = prompt || QA_PROMPT_SELECTOR.getPrompt(llm); const llmChain = new LLMChain({ prompt: actualPrompt, llm }); const chain = new StuffDocumentsChain({ llmChain }); return chain; };

[toonverbeek]

Working on the same thing (trying to port over the python example openChatAI example) and running into a similar issue, except in my case TypeScript is throwing an error when I pass in prompt to the fromLLM() method.

Complete code snippet:

const SYSTEM_TEMPLATE = `Use the following pieces of context to answer the users question. 
If you don't know the answer, just say that you don't know, don't try to make up an answer.
----------------
{context}`;

  const messages = [
    SystemMessagePromptTemplate.fromTemplate(SYSTEM_TEMPLATE),
    new MessagesPlaceholder("history"),
    HumanMessagePromptTemplate.fromTemplate("{question}"),
  ];

  const prompt = ChatPromptTemplate.fromPromptMessages(messages);

  const chatChain = ChatVectorDBQAChain.fromLLM(chatModel, vectorstore, prompt);

  const conversationChain = new ConversationChain({
    llm: chatModel,
    prompt,
  });

Typescript STOPS throwing an error whenever I pass it an empty object after the prompt variable, suggesting that there is some sort of overloading going on, but in fact I can't find any overloaded function definition:

  const chatChain = ChatVectorDBQAChain.fromLLM(chatModel, vectorstore, prompt, {}); // TS does not complain 

[hwchase17]

the signature of fromLLM does not currently allow for this. see the function defintion below:

static fromLLM(
    llm: BaseLanguageModel,
    vectorstore: VectorStore,
    options: {
      inputKey?: string;
      outputKey?: string;
      k?: number;
      returnSourceDocuments?: boolean;
      questionGeneratorTemplate?: string;
      qaTemplate?: string;
    } = {}
  ): ChatVectorDBQAChain

however, the fromLLM method is just a classmethod. you can initialize the chain more explicitly to have more control. see the code thats inside the fromLLM method

const { questionGeneratorTemplate, qaTemplate, ...rest } = options;
    const question_generator_prompt = PromptTemplate.fromTemplate(
      questionGeneratorTemplate || question_generator_template
    );
    const qa_prompt = PromptTemplate.fromTemplate(qaTemplate || qa_template);

    const qaChain = loadQAStuffChain(llm, { prompt: qa_prompt });
    const questionGeneratorChain = new LLMChain({
      prompt: question_generator_prompt,
      llm,
    });
    const instance = new this({
      vectorstore,
      combineDocumentsChain: qaChain,
      questionGeneratorChain,
      ...rest,
    });
    return instance;
  }

if you wanted to use your own CUSTOM_PROMPT as the question-answering prompt, you could use that code but using CUSTOM_PROMPT, ala:

const { questionGeneratorTemplate, qaTemplate, ...rest } = options;
    const question_generator_prompt = PromptTemplate.fromTemplate(
      questionGeneratorTemplate || question_generator_template
    );
    const qa_prompt = PromptTemplate.fromTemplate(qaTemplate || qa_template);

    const qaChain = loadQAStuffChain(llm, { prompt: qa_prompt });
    const questionGeneratorChain = new LLMChain({
      prompt: CUSTOM_PROMPT,
      llm,
    });
    const instance = new this({
      vectorstore,
      combineDocumentsChain: qaChain,
      questionGeneratorChain,
      ...rest,
    });
    return instance;
  }

[toonverbeek]

@hwchase17 thanks for explaining, that makes sense!

[TimPietrusky]

@hwchase17 are there any plans to change the logic here so that it's more easy (like in Python) to have a custom prompt? Or is it as strict like this for a specific reason?

In my use case I want to set a specific SystemMessage and a specific HumanMessage. And as far as I understand, it's not working out of the box, I have to "overwrite" fromLLM, right?

[dosubot[bot]]

Hi, @OmriNach! I'm here to help the LangChain team manage their backlog and I wanted to let you know that we are marking this issue as stale.

From what I understand, you reported an issue regarding the system template and chat prompt template not being included in the ChatVectorDB, resulting in different behaviors between the Python and JavaScript versions. You provided a fix by changing the code in chat_vector_db_chain.js and qa_answering load.js. Another user, toonverbeek, also reported a similar issue when porting over the Python example to TypeScript.

It seems that the issue has been resolved, as you provided a fix and hwchase17 explained that the fromLLM method does not currently allow for custom prompts, but suggested an alternative approach. TimPietrusky also asked if there are plans to make it easier to have a custom prompt.

Before we close this issue, could you please confirm if it is still relevant to the latest version of the LangChain repository? If it is, please let the LangChain team know by commenting on the issue. Otherwise, feel free to close the issue yourself or it will be automatically closed in 7 days.

Thank you for your contribution to the LangChain project!