Responses from LLM not being recorded in Chat History DB

[jack-vinitsky]

Describe the bug Currently the chat history is only recording the end user's prompts and not the responses from the LLM. Checking the Cosmos DB database only shows end user prompts.

To Reproduce Steps to reproduce the behavior:

1. Create a new chat and note the responses
2. Start a new chat and then go to chat history.
3. Click on a prior conversation.
4. Note that only the end user's prompts are displayed.

Expected behavior The history of the conversation includes responses from the LLM.

Screenshots Original Conversation:

Conversation Retrieved from Chat History:

Configuration: Please provide the following

• Azure OpenAI model name and version (e.g. 'gpt-35-turbo-16k, version 0613'): gpt-4o
• Is chat history enabled? - Yes
• Are you using data? If so, what data source? (e.g. Azure AI Search, Azure CosmosDB Mongo vCore, etc) - Azure AI Search

Additional context This is only happening on one of the 5 instances of this chatbot that we are running although this one is the most recent update that we have deployed (as of pull request #1073)

[dmytroHexagon]

Faced the same issue, also not present in the associated cosmos db

[Ashutosh2547]

As far as I have seen the backend, there isn't any logic to save the responses in the database. You can create your own endpoint to save the response in the conversations database and call it from the React code (to be precise Chat.tsx file).

[dmytroHexagon]

Yeap, I also investigated the code and haven't found saving of responses. They used to work. I didn't have time to find a commit where a problem was introduced. Probably easier just to fix by myself.

[Ashutosh2547]

Here is what I did:

• Created a new endpoint in app.py named it "conversation_response". This has a logic to set the "type" to "assistant".
• Called this endpoint from Chat.tsx (line 460). Here I was able to get the stringified content of stream response, otherwise the stream response was an array having text split into objects.

[ukiguy]

We're seeing same issues. Also deployed using 1073

[dmytroHexagon]

The issue is fixed for me, using latest available image

[jack-vinitsky]

I have confirmed as well. Latest update #1116 has resolved the issue.

[cdrguru]

Glad to see this issue resolved! We had the same problem with LLM responses not being captured in Cosmos DB, and this fix is crucial for teams relying on detailed logs for reporting.

For those exploring conversation analytics, capturing both user prompts and LLM responses helps track key metrics like response times or missed replies. This data is essential for optimizing AI interactions.

That’s why we built AInsights for Power BI, available on AppSource, to help teams analyze conversation data and improve system performance.

Thanks again for the quick fix—it’s a big help!