Stuck in "Generating response..." when using the History option and PromptFlow.

[OlegDStassenko]

Describe the bug

I encounter an issue with the message "Generating response..." after sending a chat.

My configuration:

UI_SHOW_CHAT_HISTORY_BUTTON = True AZURE_COSMOSDB_ACCOUNT = xxx AZURE_COSMOSDB_ACCOUNT_KEY = xxx AZURE_OPENAI_STREAM = False USE_PROMPTFLOW = True

To Reproduce If I leave these values empty it's work fine.

• AZURE_COSMOSDB_ACCOUNT = xxx
• AZURE_COSMOSDB_ACCOUNT_KEY = xxx

[abhahn]

Hi @OlegDStassenko , could you check the backend logs to see if there were any exceptions raised which may have caused this and share the result here?

[OlegDStassenko]

Hi @abhahn

Error in :

Frontend : An error occurred. Please try again. If the problem persists, please contact the site administrator.

Backend :

2024-07-14T22:25:38 Welcome, you are now connected to log-streaming service.Starting Log Tail -n 10 of existing logs ----/appsvctmp/volatile/logs/runtime/e9c64f35caa811a75e6fde308ac8a50a1c50f9db77491012d42df457fe7b5ec9.log 2024-07-14T22:10:22.545225038Z: [ERROR] WARNING:root:No datasource configuration found in the environment -- calls will be made to Azure OpenAI without grounding data. 2024-07-14T22:10:22.659260894Z: [ERROR] [2024-07-14 22:10:22 +0000] [73] [INFO] Started server process [73] 2024-07-14T22:10:22.660928673Z: [ERROR] [2024-07-14 22:10:22 +0000] [73] [INFO] Waiting for application startup. 2024-07-14T22:10:22.668054983Z: [ERROR] [2024-07-14 22:10:22 +0000] [73] [INFO] Application startup complete. 2024-07-14T22:10:22.688492824Z: [ERROR] [2024-07-14 22:10:22 +0000] [72] [INFO] Started server process [72] 2024-07-14T22:10:22.690371001Z: [ERROR] [2024-07-14 22:10:22 +0000] [72] [INFO] Waiting for application startup. 2024-07-14T22:10:22.697744807Z: [ERROR] [2024-07-14 22:10:22 +0000] [72] [INFO] Application startup complete. 2024-07-14T22:10:22.746670988Z: [ERROR] [2024-07-14 22:10:22 +0000] [74] [INFO] Started server process [74] 2024-07-14T22:10:22.746728687Z: [ERROR] [2024-07-14 22:10:22 +0000] [74] [INFO] Waiting for application startup. 2024-07-14T22:10:22.746734987Z: [ERROR] [2024-07-14 22:10:22 +0000] [74] [INFO] Application startup complete./appsvctmp/volatile/logs/runtime/03e14c32eb334eb631d93aeb7af40c40f850b0d64eb0f70acd3960f185c13fd8.log 2024-07-14T22:08:37.269760037Z: [INFO] Updated CA certificates 2024-07-14T22:08:45.646249819Z: [INFO] {"EventId":0,"LogLevel":"Information","Category":"Program","Message":"Entering app startup."} 2024-07-14T22:08:59.685470381Z: [INFO] Hosting environment: Production 2024-07-14T22:08:59.685998374Z: [INFO] Content root path: /app 2024-07-14T22:08:59.686009074Z: [INFO] Now listening on: http://[::]:8081/appsvctmp/volatile/logs/runtime/4c3aae71d0ee526a9ab05a3e699afa07d91b9732fc07ee1bee5cfc6824183326.log 2024-07-14T22:10:40.760417812Z: [INFO] Updated CA certificates 2024-07-14T22:10:44.945689012Z: [INFO] {"EventId":0,"LogLevel":"Information","Category":"Program","Message":"Entering app startup."} 2024-07-14T22:10:50.141111917Z: [INFO] Hosting environment: Production 2024-07-14T22:10:50.142548698Z: [INFO] Content root path: /app 2024-07-14T22:10:50.143455985Z: [INFO] Now listening on: http://[::]:8081/appsvctmp/volatile/logs/runtime/3edfcf06e9d6dd1af60626d1073cd8af5d43e2d011c73726a9f4f14295665da2.log 2024-07-14T22:11:25.192293758Z: [INFO] Updated CA certificates/appsvctmp/volatile/logs/runtime/4a12dbe184daae6d4cdfbb217dc83be635743959591858d3d99f0c0b1a24b3dd.log 2024-07-14T22:09:29.215578792Z: [INFO] Updated CA certificates/appsvctmp/volatile/logs/runtime/f9fb6054fc248762f95a750fab966e22a041d5519042f24b49b97c0b2de809ad.log 2024-07-14T22:12:28.287339606Z: [ERROR] await response.make_conditional(request, accept_ranges=True, complete_length=file_size) 2024-07-14T22:12:28.287343406Z: [ERROR] File "/tmp/8dca1da3bbdd5a0/antenv/lib/python3.11/site-packages/quart/wrappers/response.py", line 435, in make_conditional 2024-07-14T22:12:28.287347306Z: [ERROR] is206 = await self._process_range_request(request, complete_length, accept_ranges) 2024-07-14T22:12:28.287350905Z: [ERROR] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-07-14T22:12:28.287358405Z: [ERROR] File "/tmp/8dca1da3bbdd5a0/antenv/lib/python3.11/site-packages/quart/wrappers/response.py", line 417, in _process_range_request 2024-07-14T22:12:28.287362605Z: [ERROR] self.content_range = ContentRange( 2024-07-14T22:12:28.287366205Z: [ERROR] ^^^^^^^^^^^^^ 2024-07-14T22:12:28.287369705Z: [ERROR] File "/tmp/8dca1da3bbdd5a0/antenv/lib/python3.11/site-packages/werkzeug/datastructures/range.py", line 128, in init 2024-07-14T22:12:28.287373605Z: [ERROR] assert http.is_byte_range_valid(start, stop, length), "Bad range provided" 2024-07-14T22:12:28.287377405Z: [ERROR] AssertionError: Bad range providedEnding Log Tail of existing logs ---Starting Live Log Stream --- 2024-07-14T22:26:38 No new trace in the past 1 min(s). 2024-07-14T22:27:38 No new trace in the past 2 min(s). 2024-07-14T22:28:38 No new trace in the past 3 min(s).

---

I noticed that despite the error, my request is recorded in MongoDB, even if I don't get a response.

{ "id": "811bed96-11e9-4461-8eec-7bc70ab01a71", "type": "conversation", "createdAt": "2024-07-14T22:12:46.871155", "updatedAt": "2024-07-14T22:12:47.182418", "userId": "e723f0f3-b35e-4fe1-a93d-7f27f7f3eb7e", "title": "question, "_rid": "TWwEANqDHA4XAAAAAAAAAA==", "_self": "dbs/TWwEAA==/colls/TWwEANqDHA4=/docs/TWwEANqDHA4XAAAAAAAAAA==/", "_etag": "\"0200c048-0000-0e00-0000-66944d5f0000\"", "_attachments": "attachments/", "_ts": 1720995167 }

[karnArora]

I am facing the same issue when enabling chat history while creating the web app from ai studio) and then adding the promptflow variable values from the environment variables section in the azure web app services

[wheatsnackbread]

I am experiencing the exact same issue. When I enable both promptflow and chat history, creating new conversations results in an error. However, if I click into a previous existing chat history message and send a message again, it seems to work fine. Maybe it's an issue in chat.tsx? Examining rows 472 through 492 yields:

if (!result.history_metadata) {
            console.error('Error retrieving data.', result)
            let errorChatMsg: ChatMessage = {
              id: uuid(),
              role: ERROR,
              content: errorMessage,
              date: new Date().toISOString()
            }
            setMessages([...messages, userMessage, errorChatMsg])
            setIsLoading(false)
            setShowLoadingMessage(false)
            abortFuncs.current = abortFuncs.current.filter(a => a !== abortController)
            return
          }
          resultConversation = {
            id: result.history_metadata.conversation_id,
            title: result.history_metadata.title,
            messages: [userMessage],
            date: result.history_metadata.date
          }
          resultConversation.messages.push(errorChatMsg)

Which is interesting because

resultConversation = {
            id: result.history_metadata.conversation_id,
            title: result.history_metadata.title,
            messages: [userMessage],
            date: result.history_metadata.date
          }

tries to grab history_metadata directly from result, when it's previously referenced as result.choices[0] in lines 377 through 391. When I examine the result object being printed to console, I see that it's populated correctly as shown below.

Not sure if this is the issue or a red herring. Will try to deploy a version of the code with

resultConversation = {
            id: result.choices[0].history_metadata.conversation_id,
            title: result.choices[0].history_metadata.title,
            messages: [userMessage],
            date: result.choices[0].history_metadata.date
          }

to see what happens.

[karnArora]

@wheatsnackbread , you are correct , if you click on one of the previous chats and then continue the conversation , the app works fine , also the browser console clearly shows that the response is received by the web app , but thats when the issue comes up. did you have any luck with identifying where exactly the problem lies?

[wheatsnackbread]

@wheatsnackbread , you are correct , if you click on one of the previous chats and then continue the conversation , the app works fine , also the browser console clearly shows that the response is received by the web app , but thats when the issue comes up.

did you have any luck with identifying where exactly the problem lies?

@karnArora, I think it's definitely got to do with the way the returned "response" items are being parsed while creating a totally new conversation with prompt flow and conversation history enabled--either in terms of the logic or structure. After I get the error and refresh the page, I see that a new conversation has been created with everything correctly populated (title, my message, etc.) except there is no response from the bot. As you mentioned though, it begins working once I start responding to an existing conversation.

I have tried to redeploy the bot and modify the code without success. Can anyone on the product team help out here? 🙏 @abhahn

[mullergauthier]

Same issue and comments for me :)

[rossberry-foley]

I was able to resolve this issue by changing lines 173-184 of format_pf_non_streaming_response in backend/utils.py from this:

response_obj = {
            "id": chatCompletion["id"],
            "model": "",
            "created": "",
            "object": "",
            "choices": [
                {
                    "messages": messages,
                    "history_metadata": history_metadata,
                }
            ]
        }

To this:

response_obj = {
            "id": chatCompletion["id"],
            "model": "",
            "created": "",
            "object": "",
            "choices": [
                {
                    "messages": messages,
                }
            ],
            "history_metadata": history_metadata
        }