Open robinzhengwang opened 1 year ago
Similar issues here. Citations in a tool message but just the "not found" assistant message. Also: "not found" message with no citations, but plenty of highly scored hits in the data if you query the underlying cognitive search.
I noticed that some of the issues lie with the intent generated by the model. Keeping the search service default with keyword search I found less accurate results. However, testing with semantic search I noticed the intent generated was more accurate and so the model was capable of generating an answer.
@robinzhengwang @getpointful Feel free to try out semantic/vector search and let us know if it helps. If not, feel free to open a support ticket with Azure OpenAI. Thanks :)
@robinzhengwang @getpointful Feel free to try out semantic/vector search and let us know if it helps. If not, feel free to open a support ticket with Azure OpenAI. Thanks :)
Thanks for reply. In this case, I am using default semantic search, but I got the same result. I have also tried some approaches to raise the search score (in this case I am using synonym mapping), but the "assistant" still return "Not Found". I also watched the intent difference continuously, I guess OpenAI cannot work well directly with synonym settings from Cognitive Search, it might be completed by prompt engineering. I will try to test it in Vector Search.
After I use vector search, and vectorSemanticHybrid query mode, I still found the issue that citation has content, but "assistant" still says not found.
@lindazqli and @sarah-widder,
I am not able to get the data for the first query for the following configurations:
AZURE_SEARCH_USE_SEMANTIC_SEARCH=false
AZURE_SEARCH_QUERY_TYPE=vectorSemanticHybrid
AZURE_SEARCH_SEMANTIC_SEARCH_CONFIG=content-search
I ended up adding the following code to the initial chat request:
if len(request_messages) == 1 :
if not any("Tell me more about businesses." in message['content'] for message in request_messages) :
first_message = request_messages[0]
first_message = request_messages.pop(0)
request_messages.append({
"role": "tool",
"content": "{\"citations\": [], \"intent\": \"[]\"}",
"end_turn": "true"
})
request_messages.append({
"role": "assistant",
"content": "Please modify your question. I am looking for information about a specific product, company, or service.",
"end_turn": "true"
})
request_messages.append({
"role": "assistant",
"content": "return all possible products or services if an input only consist of alpha numeric value as a single word look for product number or company.",
"end_turn": "true"
})
request_messages.append(first_message)
It would be helpful if the integration between Azure OpenAI and Azure AI Search would be open sourced.
During the test, I found sometimes that the role "tool" has returned some citations within all related information. But the role "assistant" returned data not found. May I know how to deal with this issue? Is there something I can tune in OpenAI? Thank you.