search

Azure-Samples / azure-search-openai-demo

A sample app for the Retrieval-Augmented Generation pattern running in Azure, using Azure AI Search for retrieval and Azure OpenAI large language models to power ChatGPT-style and Q&A experiences.
https://azure.microsoft.com/products/search
MIT License
6.12k stars 4.16k forks source link

Answer only questions about the documents #1198

Open andreas-sommer-engenious opened 9 months ago

andreas-sommer-engenious commented 9 months ago

I have created the database with several pdf documents about our products. It works fine, when I ask a question about our products. The answer is correct. But when I ask a general question "Who is the first president of the USA" f.ex. then I also get an answer. I thought to change the retrivial mode to "Vector" should solve the issue, but it doesn't work. How can I prevent that?

Regards Andreas

advanced-flow commented 8 months ago

Hello Andreas, to prevent the LLMs from answering questions that are not justified by the data provided, you should specify this in the system query. This is actually already quite well described in the default of this repo, see the following snippet from chatreadretrieveread.py: "[...]Answer ONLY with the facts listed in the list of sources below. If there is not enough information, say you don't know. Do not create an answer that does not use the sources listed below. [...]" For me at least, it prevents questions that have nothing to do with the data in the index from being answered. The retrieval method (Vector, keyword, hybrid) usually has nothing to do with this.