Issue: Not answering questions out of context using RetrievalQA Chain and ConversationalChatAgent

[pelyhe]

Issue you'd like to raise.

Hi!

I implemented a chatbot with gpt-4 and a docx file which is provided as context. If I ask questions according to this context, it is returning relevant answers, but if I want to ask a question which is out of this context, it responses 'Based on the provided context I cannot answer this question' or something like that. How can I implement it in such a way, where it uses the context for every question, but if it cant find relevant answer for it in the context provided, it should take a look in its own language model.

My AgentExecutor instance looks like this:

    def _create_chat_agent(self):

        self.llm = OpenAI(temperature=0, model_name="gpt-4", top_p=0.2, presence_penalty=0.4, frequency_penalty=0.2)

        # Data Ingestion
        word_loader = DirectoryLoader(DOCUMENTS_DIRECTORY, glob="*.docx")
        documents = []
        documents.extend(word_loader.load())
        # Chunk and Embeddings
        text_splitter = CharacterTextSplitter(chunk_size=768, chunk_overlap=200)
        documents = text_splitter.split_documents(documents)
        embeddings = OpenAIEmbeddings()
        vectorstore = FAISS.from_documents(documents, embeddings)

        # Initialise Langchain - QA chain
        qa = RetrievalQA.from_chain_type(llm=self.llm, chain_type="stuff", retriever=vectorstore.as_retriever())

        tools = [
            Tool(
                name="...",
                func=qa.run,
                description="..."
            ),
        ]

        system_msg = "You are a helpful assistant."

        agent = ConversationalChatAgent.from_llm_and_tools(
            llm=self.llm,
            tools=tools,
            system_message=system_msg
        )

        self.chat_agent = AgentExecutor.from_agent_and_tools(
            agent=agent, tools=tools, verbose=True, memory=ConversationBufferMemory(memory_key="chat_history", return_messages=True)
        )

Suggestion:

No response

[praneetreddy017]

Could you show me your prompt please?

[pelyhe]

I call the executor like this: response = self.chat_agent.run(input=prompt). My prompt was 'Say something about Eliza Douglas'.

[praneetreddy017]

Could you look at the below example and maybe change the LLM initialization portion like in the bottom half of the image and get back to me

In the image, my data is actually clinical note taken by a doc I'm asking an irrelevant question and it returns "I don't know" and I am thinking this is because of the way Langchain constructs the final query.

We can modify the query using a custom prompt template to suit our needs and as you can see, we get some answer from the LLM where it constructs a response from its parametric knowledge. Can you have a look at my example in the image and make those changes and get back to me

[praneetreddy017]

Include In[7] and In[8] from my code after the line where you initialize the vector store So it should be something like this:

from langchain.prompts import PromptTemplate
.......
.......
vectorstore = FAISS.from_documents(documents, embeddings)

prompt_template = """Use the following pieces of context to answer the question at the end. 
If you don't know the answer, please think rationally and answer from your own knowledge base 

{context}

Question: {question}
"""
PROMPT = PromptTemplate(
    template=prompt_template, input_variables=["context", "question"]
)

chain_type_kwargs = {"prompt": PROMPT}
qa = RetrievalQA.from_chain_type(llm=OpenAI(), 
                                 chain_type="stuff", 
                                 retriever=vectordb.as_retriever(), 
                                 chain_type_kwargs=chain_type_kwargs)

[pelyhe]

First of all, thank you for being so helpful! Unfortunately, the response for this looks like this: 'I apologize for the confusion. There is no information about Eliza Douglas in the context of this conversation.'

[pelyhe]

As you can see here, it always want to use the document tool, even if the prompt is not fitting the tool's description. (I asked it to say something about fifa here, which is irrelevant according to the context)

[praneetreddy017]

Can you share your code with me. I'm curious and wanna work on this on my own end.

[pelyhe]

Sure, thank you for your help!

from config.db import get_db
from langchain.memory import ConversationBufferMemory
from langchain import PromptTemplate
from langchain.agents import ConversationalChatAgent, Tool, AgentExecutor
from fastapi import HTTPException
from bson import ObjectId
import pickle
import os
import datetime
import logging
from controllers.user_controller import UserController
from langchain.llms import OpenAI
from langchain.document_loaders import DirectoryLoader
from langchain.text_splitter import CharacterTextSplitter
from langchain.vectorstores import FAISS
from langchain.embeddings.openai import OpenAIEmbeddings
from langchain.chains import RetrievalQA

DOCUMENTS_DIRECTORY = 'data'

db = get_db()
userModel = db['users']

# THE MODEL WHICH USES GENERAL GPT KNOWLEDGE, BUT DONT ALWAYS RESPONSE WITH THE RIGHT ANSWER

class ChatController(object):
    def __init__(self):
        self._create_chat_agent()

    def _create_chat_agent(self):

        self.llm = OpenAI(temperature=0, model_name="gpt-4", top_p=0.2, presence_penalty=0.4, frequency_penalty=0.2)

        # Data Ingestion
        word_loader = DirectoryLoader(DOCUMENTS_DIRECTORY, glob="*.docx")
        documents = []
        documents.extend(word_loader.load())
        # Chunk and Embeddings
        text_splitter = CharacterTextSplitter(chunk_size=768, chunk_overlap=200)
        documents = text_splitter.split_documents(documents)
        embeddings = OpenAIEmbeddings()
        vectorstore = FAISS.from_documents(documents, embeddings)

        prompt_template = """Use the following pieces of context to answer the question at the end.
        If you don't know the answer, please think rationally answer from your own knowledge base

        {context}

        Question: {question}
        """

        PROMPT = PromptTemplate(
            template=prompt_template, input_variables=["context", "question"]
        )

        chain_type_kwargs = {"prompt": PROMPT}
        # Initialise Langchain - QA chain
        qa = RetrievalQA.from_chain_type(llm=self.llm, chain_type="stuff", retriever=vectorstore.as_retriever(), chain_type_kwargs=chain_type_kwargs)

        tools = [
            Tool(
                name="Document tool",
                func=qa.run,
                description="useful for when you need to answer questions about artworks."
            ),
        ]

        system_msg = "You are a helpful assistant."

        agent = ConversationalChatAgent.from_llm_and_tools(
            llm=self.llm,
            tools=tools,
            system_message=system_msg
        )

        self.chat_agent = AgentExecutor.from_agent_and_tools(
            agent=agent, tools=tools, verbose=True, memory=ConversationBufferMemory(memory_key="chat_history", return_messages=True)
        )

    def askAI(self, prompt: str, id: str):
        try:
            objId = ObjectId(id)
        except:
            raise HTTPException(status_code=400, detail="Not valid id.")

        # create a conversation memory and save it if it not exists 
        if not os.path.isfile('conv_memory/'+id+'.pickle'):
            mem = ConversationBufferMemory(memory_key="chat_history", return_messages=True)
            with open('conv_memory/' + id + '.pickle', 'wb') as handle:
                pickle.dump(mem, handle, protocol=pickle.HIGHEST_PROTOCOL)
        else:
            # load the memory according to the user id
            with open('conv_memory/'+id+'.pickle', 'rb') as handle:
                mem = pickle.load(handle)

        self.chat_agent.memory = mem

        # for could not parse LLM output
        try:
            response = self.chat_agent.run(input=prompt)
        except ValueError as e:
            response = str(e)
            if not response.startswith("Could not parse LLM output: `"):
                print(e)
                raise e
            response = response.removeprefix(
                "Could not parse LLM output: `").removesuffix("`")

        # save memory after response
        with open('conv_memory/' + id + '.pickle', 'wb') as handle:
            pickle.dump(mem, handle, protocol=pickle.HIGHEST_PROTOCOL)

        return {"answer": response}

[praneetreddy017]

I made some changes and removed whatever I wasn't too sure about (like the db, mem pickle and http stuff) Changed the db to chromadb too because that is what I was using on my end. Key thing seems to be the changes I made to the prompt So if something irrelevant is passed to the model as a question, the observation it made was "The context does not seem relevant to 'your random topic name here`"

So I changed the prompt to accommodate that and give us the result we desire

from langchain.memory import ConversationBufferMemory
from langchain import PromptTemplate
from langchain.agents import ConversationalChatAgent, Tool, AgentExecutor
import pickle
import os
import datetime
import logging
# from controllers.user_controller import UserController
from langchain.llms import OpenAI
from langchain.document_loaders import DirectoryLoader
from langchain.text_splitter import CharacterTextSplitter
# from langchain.vectorstores import FAISS
from langchain.embeddings.openai import OpenAIEmbeddings
from langchain.chains import RetrievalQA
from langchain.vectorstores import Chroma

# THE MODEL WHICH USES GENERAL GPT KNOWLEDGE, BUT DONT ALWAYS RESPONSE WITH THE RIGHT ANSWER

class ChatController(object):
    def __init__(self):
        self._create_chat_agent()

    def _create_chat_agent(self):

        self.llm = OpenAI(temperature=0, top_p=0.2, presence_penalty=0.4, frequency_penalty=0.2)
        embeddings = OpenAIEmbeddings()
        persist_directory = 'myvectordb'
        vectorstore = Chroma(persist_directory=persist_directory, embedding_function = embeddings)

        prompt_template = """If the context is not relevant, 
        please answer the question by using your own knowledge about the topic

        {context}

        Question: {question}
        """

        PROMPT = PromptTemplate(
            template=prompt_template, input_variables=["context", "question"]
        )

        chain_type_kwargs = {"prompt": PROMPT}
        # Initialise Langchain - QA chain
        qa = RetrievalQA.from_chain_type(llm=self.llm, 
                                         chain_type="stuff", 
                                         retriever=vectorstore.as_retriever(), 
                                         chain_type_kwargs=chain_type_kwargs)

        tools = [
            Tool(
                name="Document tool",
                func=qa.run,
                description="useful for when you need to answer questions."
            ),
        ]

        system_msg = "You are a helpful assistant."

        agent = ConversationalChatAgent.from_llm_and_tools(
            llm=self.llm,
            tools=tools,
            system_message=system_msg
        )

        self.chat_agent = AgentExecutor.from_agent_and_tools(
            agent=agent, tools=tools, verbose=True, memory=ConversationBufferMemory(memory_key="chat_history", 
                                                                                    return_messages=True)
        )

    def askAI(self, prompt: str):
#         try:
#             objId = ObjectId(id)
#         except:
#             raise HTTPException(status_code=400, detail="Not valid id.")

#         # create a conversation memory and save it if it not exists 
#         if not os.path.isfile('conv_memory/'+id+'.pickle'):
#             mem = ConversationBufferMemory(memory_key="chat_history", return_messages=True)
#             with open('conv_memory/' + id + '.pickle', 'wb') as handle:
#                 pickle.dump(mem, handle, protocol=pickle.HIGHEST_PROTOCOL)
#         else:
#             # load the memory according to the user id
#             with open('conv_memory/'+id+'.pickle', 'rb') as handle:
#                 mem = pickle.load(handle)

#         self.chat_agent.memory = mem

        # for could not parse LLM output
        try:
            response = self.chat_agent.run(input=prompt)
        except ValueError as e:
            response = str(e)
#             if not response.startswith("Could not parse LLM output: `"):
#                 print(e)
#                 raise e
#             response = response.removeprefix(
#                 "Could not parse LLM output: `").removesuffix("`")

#         # save memory after response
#         with open('conv_memory/' + id + '.pickle', 'wb') as handle:
#             pickle.dump(mem, handle, protocol=pickle.HIGHEST_PROTOCOL)

        return {"answer": response}

x = ChatController()
x.askAI("fifa")

[praneetreddy017]

Please read this and let me know if it works @pelyhe

[pelyhe]

Thats amazing, it works! I would never think that a simple rephrasing can solve this problem. Thank you so much for your time, I really appreciate it!

[praneetreddy017]

All good my man! This was a fun problem to work on

[ronasuti]

Hi, how did you two get around the issue of "cannot parse LLM output: "? I'm not sure what I'm doing wrong, the one big change in my setup compared to yours is a locally hosted model (gpt4all-j).

[praneetreddy017]

@ronasuti were you able to figure it out? Can you show me your code/data you are trying to answer over. Also, would be better if you could open a new issue post and tag me there tbh.

[ronasuti]

At this point, the latest updates to the library seem to let the chat agent at least get decent results when not using document information. However, the QA part is now broken: I only get empty responses back for things ChromaDB must know about. @praneetreddy017 I'll look into creating a new thread.

[akashlinux10may]

I have used following code and my problem is vice versa. Here if i am asking anything out of the scope or context it will still answer my question using its general knowledge. please help me in improving the same.

I only want it to stick to its vectorstore.

repo_id = "tiiuae/falcon-7b-instruct" llm = HuggingFaceHub(repo_id=repo_id, model_kwargs={"temperature":0.2, "max_new_tokens":500})

template = """You are an AI assistant for the USA Constitution. The documentation is located at SOURCE_DOCUMENTS.
You are given the following extracted parts of a long document and a question. 
You should only use source and data that are explicitly listed as a source in the context. 

Do NOT use any external resource, hyperlink or reference to answer that is not listed.

If you don't know the answer, just say "Hmm, I'm not sure." Don't try to make up an answer.
If the question is not about USA constitution, politely inform them that you are tuned to only answer questions about USA constitution.
If the context is not relevant, please dont answer the question by using your own knowledge about the topic

{context}

Question: {question}
"""

PROMPT = PromptTemplate(template=template, input_variables=["question", "context"])

chain_type_kwargs = {"prompt": PROMPT} qa = RetrievalQA.from_chain_type(llm, chain_type="stuff", retriever=retriever,chain_type_kwargs=chain_type_kwargs)

while True: query = input("\nEnter a query: ") if query == "exit": break if query.strip() == "": continue answer=qa.run(query)

end = time.time() print("\n\n> Question:") print(db.similarity_search_with_score(query)) print(query) print(f"\n> Answer (took {round(end - start, 2)} s.):") print(answer)

[praneetreddy017]

Hey, can you try this with some other model and let me know if the issue persists?

Please can you follow the code available on my git repo: https://github.com/praneetreddy017/Langchain-Cookbook/blob/main/Doc%20Q%26A.ipynb

I'm in the middle of the worst season of job hunting and I am not able to spend too much time on here helping people like I used to :(

[akashlinux10may]

see I have tried the above code but see nothing happened. Only OPENAI model is working fine. I have trained it on USA constitution but its also answering about famous personalities.

[praneetreddy017]

Ummm, can you try something like declare-lab/flan-gpt4all-xl

I wanna understand what's happening Because over I see that you're using an instruct model and I haven't experimented with falcon yet and how well it performs in these use cases

For example, models like FLAN T5 which work really well for NLP tasks don't output anything in these use cases

Can you get back to me after trying the model I mentioned above?

On Fri, Jun 16, 2023, 12:42 PM akashlinux10may @.***> wrote:

see I have tried the above code but see nothing happened. Only OPENAI model is working fine.

[image: image] https://user-images.githubusercontent.com/66991234/246477999-dd15ed0f-20ff-472d-b91e-4c4a2785ec26.png

— Reply to this email directly, view it on GitHub https://github.com/hwchase17/langchain/issues/4573#issuecomment-1594964866, or unsubscribe https://github.com/notifications/unsubscribe-auth/A32VR2CQQMIDVVL7ZSXF57LXLSEFTANCNFSM6AAAAAAX7N5OIM . You are receiving this because you were mentioned.Message ID: @.***>

[akashlinux10may]

I made some changes but still its working fine with openAI only. Now please help me how can I deploy my custom chatbot to production like AWS or GCP or any other reasonable platform. I have made front end in react for chat UI.

[gdagitrep]

@akashlinux10may Were you able to get what you want? What changes did you make?

[pradeepdas]

why not use SystemMessage for "context" -- if that's a general instruction to the model across all interactions

[hassanfar00q]

hi

[naveenfaclon]

Include In[7] and In[8] from my code after the line where you initialize the vector store So it should be something like this:

from langchain.prompts import PromptTemplate
.......
.......
vectorstore = FAISS.from_documents(documents, embeddings)

prompt_template = """Use the following pieces of context to answer the question at the end. 
If you don't know the answer, please think rationally and answer from your own knowledge base 

{context}

Question: {question}
"""
PROMPT = PromptTemplate(
    template=prompt_template, input_variables=["context", "question"]
)

chain_type_kwargs = {"prompt": PROMPT}
qa = RetrievalQA.from_chain_type(llm=OpenAI(), 
                                 chain_type="stuff", 
                                 retriever=vectordb.as_retriever(), 
                                 chain_type_kwargs=chain_type_kwargs)

i hve stored vectors locally using FAISS now how should i give that as context like you are giving in Prompt