API is not getting my datasource citations

[renatoromao]

Hi,

I already raised two bugs on Azure support but no solutions yet.

We have implemented the same code as you have on the app.py file, tested on Postman and then moved to a C# code.

Since this week, no citations have been displayed which is weird because the app worked early this month without changes.

I'm using this API below. https://[OPEN_AI].openai.azure.com/openai/deployments/gpt-4/extensions/chat/completions?api-version=2023-06-01-preview

The chat playground is working, but when I deploy it as an Azure Web App, there are no documents/answers (same question). And my API neither.

[raffaeler]

I can confirm it works as expected. The proof is the Azure dashboard that, at the end, is using the services.

Are you sending the correct headers? (just grab them from the Azure dashboard) They are fundamentals to make it work.

Also, please be specific on what you are seeing that is not correct or does not work.

[renatoromao]

hi @raffaeler ,

Thanks for your updates!

My main question is why is not working now? :D It was working last week, with the same API, no changes were made since last week. Including new apps deployed through Chat Playground are not working as well, I've tested them today.

As you can see below, I'm using 4 headers (the same headers suggested by Azure).

[raffaeler]

There have been major issue 2-3 weeks ago when they discontinued the old models. Did your issues start after that event?

What is the error returned by the service? Did you try in another region (I am working on East-US)? Did you check the index for your data is still good?

[HuapingLiu]

I got the similar issue this morning, but it is back to normal now.

[renatoromao]

@raffaeler ,

I didn't notice that.

Did your issues start after that event? I recreated new models last week, not old models (gpt-35-turbo-16k, gpt-35-turbo, gpt-4, gpt-4-32k).

What is the error returned by the service? It is returning no citations and a generic answer.

"messages": [
                {
                    "index": 0,
                    "role": "tool",
                    "content": "{\"citations\": [], \"intent\": \"[\\\"[MY QUESTION HERE]\\\"]\"}",
                    "end_turn": false
                },
                {
                    "index": 1,
                    "role": "assistant",
                    "content": "I don't find any information related to '[MY QUESTION]' from the fetched documents. As a result, I am unable to provide an answer.",
                    "end_turn": true
                }
            ]

Did you try in another region (I am working on East-US)? yes, tried now and no luck.

Did you check the index for your data is still good? yes, it is.

[renatoromao]

@HuapingLiu did you make any changes? Which region?

I'm using mainly North Europe.

[HuapingLiu]

East-US. I opened an issue this morning: https://github.com/microsoft/sample-app-aoai-chatGPT/issues/96. I suspected the root issue is the citations are empty, so they did not show up, instead to use some #doc1, #doc2.

I did not do any change, it is back to normal this afternoon.

I suspect the 2023-06-01-preview AZURE_OPENAI_PREVIEW_API_VERSION is not stable.

[raffaeler]

@renatoromao I just repeated the test on East-US. My code (modified Azure AI library and related app) worked even if with a long delay.

@HuapingLiu

I suspect the 2023-06-01-preview AZURE_OPENAI_PREVIEW_API_VERSION is not stable.

I totally agree. There have been several service disruptions in the last weeks.

[sarah-widder]

Hi @raffaeler can you confirm whether you see citations in the playground, but not the web app, for the same questions?

[raffaeler]

Hi @sarah-widder the issues I see are a quite different. It happened as well to me to see answers where the custom content was totally ignored but the answer stated that it could not find any information for my question.

I am sure this is a server-side issue because waiting for hours or a day and repeating the same question succeeded. Please consider that I am using my own code to make questions on the server side which normally works like a charm. All the issues began after you retiring some models. From that day on, the service began to hiccup..

HTH

[gingergenius]

We're having the same problem where there's definitely information related to the question in the retrieved documents, but the model says it could not find anything. We mostly experimented with the web app by running it locally.

[kjerusalmi]

Same problem, did you find a solution ?

[jonathanpv]

hi folks. I believe that having more clarity on what goes on under the hood will not only enhance our user experience but also optimize the time spent by developers in interacting with the API.

Could you please provide detailed information on the following aspects?

1. Could you share the entire workflow from prompt submission to final response?
2. Could you clarify the difference between how inScope: true vs inScope: false treats our prompts and subsequent data retrieval calls?
3. How is roleInformation handled vs a message with a system role. From my experimentation it appears a const pre-pended role takes precedence before our roleInformation and a {role: system, content: you are ...} system role is completely ignored.

I, along with many developers, find ourselves investing significant time adjusting prompts, possibly due to alterations in the data flow through your API. A clearer understanding of any changes and the overall workflow would be greatly beneficial.

Based on my current understanding, it seems like the system operates as follows:

1. A user submits a prompt.
2. The prompt populates the intent field for the citations tool.
3. There is a data retrieval step, the workings of which are not clear to us.
4. The final LLM prompt is constructed using this structure:

   <azure open ai talk to your data system role>
   <user's roleInformation value>
   <citation content>
   <user's prompt>

I must express that the current user experience and the quality of interactions are not meeting expectations. It would be highly appreciated if the system prompts could be made more user-friendly and less harsh of an experience. "There are no retrieved documents" compared to it functioning correctly before is a bit of a shock.

[raffaeler]

I don't get your points. If you use the latest preview, there is no need to make any special adjustment. Custom data can be queried very easily by adding the index name/secret in the extensions, and deserializing the answer to get the citations.

3. From my experimentation it appears a const pre-pended role takes precedence before our roleInformation and a {role: system, content: you are ...} system role is completely ignored.

What do you mean here? I always use the system prompt and it does absolutely work fine. I am probably not understanding your tests.

[jonathanpv]

I don't get your points. If you use the latest preview, there is no need to make any special adjustment. Custom data can be queried very easily by adding the index name/secret in the extensions, and deserializing the answer to get the citations.

3. From my experimentation it appears a const pre-pended role takes precedence before our roleInformation and a {role: system, content: you are ...} system role is completely ignored.

What do you mean here? I always use the system prompt and it does absolutely work fine. I am probably not understanding your tests.

It is obvious the system message is ignored by doing this:

1. enable inScope
2. embed system message to the array of messages, add a rule to your system prompt that states:

"if there is no retrieved information then say: I could not fine information related to your prompt, however based on my knowledge, then prompt the user nicely if there's anything else you can help them with".

3. send a message "hi"

In my own experimentation, doing roleInformation or both system + roleInformation customization leads to a response along the lines of:

There are no retrieved documents...

4. there is also a roleInformation field which is considered a system message, it is ambiguous what happens when you specify a system message in the messages field and a system message in the roleInformation field. Since these are LLMs, previous rulesets and instructions obstruct anything we customize and therefore lead us to undesired or unexpected behavior due to the changing nature of the API.

This was clearly an issue, since the UI for the azure chat test has been updated and two new fields have been added, notice "strictness" from 1-5, which is only modifiable with the inScope value.

[jonathanpv]

I don't get your points. If you use the latest preview, there is no need to make any special adjustment. Custom data can be queried very easily by adding the index name/secret in the extensions, and deserializing the answer to get the citations.

There are cases where the response says: "There are no retrieved documents" I inspect the context field and deserialize and find citation results from responses I knew should have been there. It seems like they added a classifier that is abstracted from us recently. And the suspicion I had was correct since the ui now has a strictness field.