Wrong Citation

[realdevmann]

Please provide us with the following information:

This issue is for a: (mark with an x)

- [ ] bug report -> please search issues before submitting
- [o ] feature request
- [ ] documentation issue or request
- [ ] regression (a behavior that used to work and stopped in a new release)

Minimal steps to reproduce

Any log messages given by the failure

Expected/desired behavior

Answer in according to uploaded files

OS and Version?

Windows 7, 8 or 10. Linux (which distribution). macOS (Yosemite? El Capitan? Sierra?) Window11

Versions

Mention any other details that might be useful

I upload my data to Azure Storage. I can run Azure Chat App correctly. But, I am disappointed with wrong answer of chatbot. My data file is written by Korean and English. My chatbot's answer is not consistent whenever I ask. Sometimes chatbot answer correctly, sometimes not.

[ Open Source Modification ]

1. create_search_index() in predocs.py
   • before : SearchableField(name="content", type="Edm.String", analyzer_name="en_microsoft"),

• after : SearchableField(name="content", type="Edm.String", analyzer_name="standard.lucene"),

2. query_prompt_template in chatretrieveread.py
   • before : If the question is not in English, translate the question to English before generating the search query.
   • After : Please search in the language of the original input of the question, never try to translate it into English.

[ Chatbot provided wrong citation ]

[ Azure Storage Explorer ] SACC is defined in different file(*Lecture-7.pdf)

How Can I improve Azure Search Service correctness?

---

Thanks! We'll be in touch soon.

[ericthomas1]

I'm seeing the same issue using the sample data.

• "What is my deductable?"
  • Answers correctly in the chat
  • But the citation is incorrect

[thiago-acn]

I also have the same issue. Have anyone try to fix the problem?

[ericthomas1]

I was unable to fix the issue. It used to work for me, in past deployments, but in the latest deployment, this issue popped up.

[dbae1145]

@ericthomas1 which of the previous deployments?

[ericthomas1]

@dbae1145 I've deployed the example multiple times in various Azure env's. In my previous, early deployments there were no issues. In my later deployments, this issue started popping up.

I'd say ~4-5 weeks ago.

[villepuntanen]

Similar issue reported in my case

[itmilos]

We should work on predocs functionally...

[kruselegal]

I think it's an off by one error in the pdf splitting. The pages index beginning at page 0, not page 1.

[ArunEPRO]

I am facing a similar issue. The previous file is always shown as citation.

[nickroseth]

Getting same thing. Any updates on this?

[jomieljaniuk]

Some cases can be caused because of the bug I described in #370 issue

[nickroseth]

Any update on this? is #370 the proper way to address? If so, is it because of how the chunking process works?

[WVSdigitaldojo]

Still a one off issue atm. Can we expect a fix?

[github-actions[bot]]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this issue will be closed.

[ericthomas1]

Still waiting for a fix! The issue of incorrect references still exists. Would be nice to know how to address, whether its with some sort of indexing or a Cognitive Search setting, etc.

[oizidbih]

Hey there, I'm facing the same issue, and it is causing a serious problem with our customers where the bot started showing wrong citations, are there any solutions for this?

[pamelafox]

@oizidbih When the wrong citations happen, have you dug into the thought process to see whether the search results have the correct filenames? Are the citations entirely made up, or syntactically incorrect? My general recommendation is to run an evaluation and change parameters to improve results. This repo is designed for evaluating this chat app: https://github.com/Azure-Samples/ai-rag-chat-evaluator

[oizidbih]

@pamelafox Thanks for your response. I have a website, a few adequately named documents, and an FAQ document as references. The citations are made up and happen more frequently now, and I'm still trying to figure out why. Interestingly, the answers are mostly correct, but why is the reference wrong? I also considered deactivating it for now, but I still need to figure out how. I'll try the evaluation and let you know