results with inScope: true (ie AZURE_SEARCH_ENABLE_IN_DOMAIN) cannot be trusted

[getpointful]

I seems to me that using inScope parameter of the AzureCognitiveSearch extension does not seem to work well enough to be useful.

Bad responses include:

• generated text (similar to gpt with no extensions) with no reference citations (i.e. not limited to my data)
• generated text with reference citations that demonstrably do not contain the stated "fact" (i.e. hallucination, not finding the correct part of my data)
• generated text with correct citations but the information presented is from unrelated parts of the reference (and hence incorrect) (i.e. not "understanding" my data)

I do understand that this is a hard problem, but presented as it is - a flag you can turn on with no further information about what it's doing or much ability to configure it, I think we can be forgiven for assuming that someone somewhere has signed off on it as "looks good to me" and that it should mostly work (this is preview stuff in the end...)

Where can we get more details on the implementation of the AzureCognitiveSearch extension, and how (if even possible) to tune it? There is some cool automagic going on there somewhere, but we need to be able to look behind the curtain a bit if we are going to be building on top of the work you have done on the platform. Particularly when some feature doesn't quite "just work"

Thanks.

[sarah-widder]

Hi @getpointful , we have documentation for using data with Azure OpenAI, including the inScope parameter, here: https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/use-your-data#limit-responses-to-your-data.

Limit responses to your data This option encourages the model to respond using your data only, and is selected by default. If you unselect this option, the model may more readily apply its internal knowledge to respond. Determine the correct selection based on your use case and scenario.

If you have specific quality concerns and would be comfortable sharing details about your data and scenario, please open a support ticket with Azure OpenAI service.