How to tell bot to reference specific document for certain user questions

[bp3000bp]

Please provide us with the following information:

This issue is for a: (mark with an x)

- [ ] bug report -> please search issues before submitting
- [x] feature request
- [ ] documentation issue or request
- [ ] regression (a behavior that used to work and stopped in a new release)

Minimal steps to reproduce

have knowledge base with overlapping keywords and content

Any log messages given by the failure

n/a

Expected/desired behavior

The bot keeps getting confused and returning incorrect answers because it's referencing the wrong document. I would like to find a way to manually override and tell the bot the correct document to reference for specific user questions that it frequently gets wrong. Is there a simple way that I can do this in the code with a dictionary or something? Or are there ways to improve the responses from the bot within its current form? I've already added metadata to all the PDF docs in the KB. Need this bot to be accurate. In the attached image is an example of my bot referring to windows shortcuts doc when it seems very clear it should refer to the doc dedicated to the question.

OS and Version?

Windows 11 Pro

azd version?

azd version 1.9.3 (commit e1624330dcc7dde440ecc1eda06aac40e68aa0a3)

Versions

n/a

Mention any other details that might be useful

---

Thanks! We'll be in touch soon.

[pamelafox]

Have you already read through our guide on improving answer quality? https://github.com/Azure-Samples/azure-search-openai-demo/blob/main/docs/customization.md#improving-answer-quality

You need to pinpoint if the issue is with the retrieval or the actual LLM stage. Usually the problem is at the retrieval stage. Please read through that guide for more specific ideas.