search

Azure-Samples / azure-search-openai-demo

A sample app for the Retrieval-Augmented Generation pattern running in Azure, using Azure AI Search for retrieval and Azure OpenAI large language models to power ChatGPT-style and Q&A experiences.
https://azure.microsoft.com/products/search
MIT License
5.88k stars 4.02k forks source link

Broken citations #1912

Open bp3000bp opened 4 weeks ago

bp3000bp commented 4 weeks ago

Please provide us with the following information:

This issue is for a: (mark with an x)

- [x] bug report -> please search issues before submitting
- [ ] feature request
- [ ] documentation issue or request
- [ ] regression (a behavior that used to work and stopped in a new release)

Minimal steps to reproduce

usually on the first question, the chat will give the correct answer but the citation link is broken. it is referencing the correct document, and later in the chat it will give a correct citation. but usually on first question it rarely gets it right or gives a working citation. why.

Any log messages given by the failure

n/a

Expected/desired behavior

just work

OS and Version?

windows 11

azd version?

azd version 1.9.6 (commit 0b1a8ad350177078312141f4465ad72205d7c077)

Versions

n/a

Mention any other details that might be useful

why why why

Thanks! We'll be in touch soon.

pamelafox commented 3 weeks ago

Hm, is "source" at all similar to the document title? Which model are you using, gpt-35-turbo? I haven't seen this issue very frequently, but generally the approach to more reliable citations are: 1) Use a more powerful model like gpt-4, gpt-4o 2) Use more prompt engineering, like few-shot examples.

bp3000bp commented 3 weeks ago

I believe I am using 3.5. This is also for tabular data which you just mentioned is inconsistent on another issue. I only seem to have issues with this specific document. Is it because of that? I'd be okay with upgrading to 4o, but if the issue is the tabular data, then I don't want to upgrade for no reason.