How can I tell if a search found or not something ?

[DrEight]

I would like to understand if a search found some data on the db vector. If the search didn't return anything, I would like to route the search on a different data source or maybe another index.

[dluc]

hi @DrEight if you are using the Search API, it will return an instance of SearchResult containing zero results (see Results property. Similarly, if you are looking at the web service response, the Results array will be empty.

[DrEight]

@dluc thank you for the answer. Let me explain better my use case, because I'm confused about the result I have and your answer. I'm using Azure Blob storage, Azure AI search and Azure OpenAI with chatgpt 3.5. I processed/imported 4 documents from the UPS website: FAQ, prepare the shipment and return services. With the call 'AskAsync': I'm asking a totally irrelevant question like: Can you tell me the best itinerary to visit Rome?

The answer is: Based on the information provided, I don't have enough information to provide the best itinerary to visit Rome. The facts provided are related to UPS shipping services and do not pertain to travel or tourism.

As relevant Sources I get 3 of the 4 documents I uploaded, that obviously, they don't make any sense in the context of the answer.

My expectation is: to have a way to tell that there are no information found, and if there are no information, the relevant source list is empty, not filled with random documents. With this info, then I can route the search in a different way.

[dluc]

I see, yes I've seen that behavior from GPT 3.5. Could you give it a try with GPT-4?

You are right, the service should reply with "INFO NOT FOUND" (an exact string, that you could check in your code). However, the service will still include the sources considered. In other words the service will say "I looked at these sources, that seemed relevant, but I couldn't find an answer".

In some instances GPT 3.5 doesn't follow the prompt instructions and generates the equivalent text. One quick solution is using GPT 4, another is tweaking the prompt. I'll see if I can find a more reliable prompt.

[DrEight]

I tried with chatGPT 4, but Instead of INFO NOT FOUND, it suggests to search for a travel agency... This is the response message, if you are curious:

_Answer: I'm sorry, but the information provided does not include any details about planning trips, tourist attractions, accommodations, or travel itineraries in Rome or any other location. The documents seem to be related to UPS shipping guidelines, packaging instructions, and shipping API details, which are not relevant to trip planning.

If you're looking to plan a trip to Rome, you might want to consider researching travel guides, contacting a travel agency, or using online travel planning tools that can help you with booking flights, hotels, and creating an itinerary that includes visiting popular tourist sites such as the Colosseum, Vatican City, and the Pantheon. Unfortunately, I cannot assist you with trip planning based on the UPS-related information provided._

Can you please tell me where is exactly the prompt to tweak ? I found the file 'answer-with-facts.txt'. Is it this the place ? I might try to help you to find a better prompt.

[dluc]

it suggests to search for a travel agency...

wow...

I found the file 'answer-with-facts.txt'. Is it this the place ?

yes that's the one. I think we could try using function calling for a more precise response, need to think about it. E.g. ask the model to call a specific function when no answer is available. The main challenge is how to fall back to a standard behavior with LLama and Gemini, ie when using other models...

[DrEight]

I'm very sorry. My fault. I was using a custom prompt provider than I forgot. I added the custom prompt provider, because the LLM was to much aggressive answering INFO NOT FOUND. So I added the the prompt to respond in a more polite way instead of INFO NOT FOUND. ( I modified with : If you don't have sufficient information, summarise the facts. However write to the user that you don't have enough information and you summarised the information you have. )

What misled me is that the array 'RelevantSources' is full of 3 reference even if the model return INFO NOT FOUND. ( I tried removing the custom prompt provider ) So the problem is not what the model answer, that was caused by me, but the RelevantSources that probably is filled by the DB.

[dluc]

I see, makes sense. Calling them "relevant" is misleading if they are not. The challenge is that the decision whether sources are relevant or not is made internally by the LLM when generating an answer. If you want the list to match exactly we will need a different strategy, e.g. asking the model to say which chunk/source was used.

[gopi-hl]

@dluc @DrEight : I am filtering on the score and only taking a reference if that scores above 0.75.