search

Azure-Samples / azure-search-openai-demo

A sample app for the Retrieval-Augmented Generation pattern running in Azure, using Azure AI Search for retrieval and Azure OpenAI large language models to power ChatGPT-style and Q&A experiences.
https://azure.microsoft.com/products/search
MIT License
5.64k stars 3.78k forks source link

Summarizing pdf files but it is not using all of them #1497

Open vascofl opened 3 months ago

vascofl commented 3 months ago

Please provide us with the following information:

This issue is for a: (mark with an x)

- [x ] bug report -> please search issues before submitting
- [ ] feature request
- [ ] documentation issue or request
- [ ] regression (a behavior that used to work and stopped in a new release)

Minimal steps to reproduce

Loading and indexing multiple pdf files and asking to summarize them

Any log messages given by the failure

Expected/desired behavior

I'm building a medical records summarizer in which I load all the medical records from one person and I ask the chat to summarize them all in a single answer. The problem is that almost all the times it doesn't use all files provided. I already changed the top and temperature parameter but it didn't help. In the prompt I say multiple times to use all files provided but it doesn't do it. In this example I loaded and indexed 3 pdf files and it gives a summary using only 2 of them. image

OS and Version?

Windows 10

azd version?

azd version 1.7.0

Versions

Mention any other details that might be useful

Thanks! We'll be in touch soon.

pamelafox commented 3 months ago

Have you read through our guide on improving answer quality? https://github.com/Azure-Samples/azure-search-openai-demo/blob/main/docs/customization.md#improving-answer-quality

The first step is to look at the search results and determine if the problem is with it not finding the results. If it did find the results, then the problem is with the model ignoring a result. Can you clarify the situation here?

And are they very long documents? LLMs do have a tendency to get "lost in the middle" (https://arxiv.org/abs/2307.03172), so if it is receiving long documents, it might be ignoring some of them. In that case you may want to split your request into multiple calls.

vascofl commented 2 months ago

Hello Pamela, thanks for the answer!

Yes I read and I already tried changing those parameters but it didn't help.

I'm not sure if I understood the question but I know all files are indexed and if I ask multiple times to summarize using all the documents provided, the chatbot eventually does that, so I would say the models just chooses to ignore some results.

The documents are small, each one is around half a page.