Error: Exception in /conversation

[nicolasworth]

Hi, I am using OpenAI Studio to test ChatGPT4 and when I deploy to App the application is being built fine and loading however. After the first chat and response the bot stops responding. When running the Log stream I can see the following error -

023-11-07T08:13:22.624770818Z: [ERROR] ERROR:root:Exception in /conversation 2023-11-07T08:13:22.624800518Z: [ERROR] Traceback (most recent call last): 2023-11-07T08:13:22.624805618Z: [ERROR] File "/tmp/8dbdeeb44e2fc04/app.py", line 451, in conversation_internal 2023-11-07T08:13:22.624809418Z: [ERROR] return conversation_without_data(request_body) 2023-11-07T08:13:22.624812819Z: [ERROR] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2023-11-07T08:13:22.624816019Z: [ERROR] File "/tmp/8dbdeeb44e2fc04/app.py", line 404, in conversation_without_data 2023-11-07T08:13:22.624819319Z: [ERROR] "role": message["role"] , 2023-11-07T08:13:22.624822719Z: [ERROR] ~~~^^^^^^^^ 2023-11-07T08:13:22.624825719Z: [ERROR] KeyError: 'role' 2023-11-07T08:13:22.636685890Z: [INFO] - - [07/Nov/2023:08:13:22 +0000] "POST /conversation HTTP/1.1" 500 19 "https://.azurewebsites.net/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36"

It was working previously when I built an app using the same method a few weeks ago.

Any ideas what could be causing this?

Thanks Nic

[AndreaCossio]

Which version of the model are you using? 2023-06-01-preview? The environment variable AZURE_OPENAI_STREAM is set to True or False?

[nicolasworth]

It was all done, via the OpenAI Studio interface, when viewing the code it is using model version '2023-07-01-preview'. Also looking at the WebApp configuration it was missing application setting 'AZURE_OPENAI_STREAM' so I have now added this and set this to 'True'. Then restarted the APP and the issue still remains. Thanks

[AndreaCossio]

Yes, I'm having the same problem. I think that it's related to the API <...>/extension/<...> endpoint. This new endpoint generates a different response from the standard one.

Swagger: https://github.com/Azure/azure-rest-api-specs/blob/main/specification/cognitiveservices/data-plane/AzureOpenAI/inference/preview/2023-07-01-preview/inference.json

[nicolasworth]

Ok glad its not just me, do you know of a way to get this working by using an older version of the model?

[wcwong]

The problem seems to be on line 197 of src/pages/chat/Chat.tsx: conversation.messages.push(toolMessage, assistantMessage)

The correct code probably should be

isEmpty(toolMessage) ?
    conversation.messages.push(assistantMessage) :
    conversation.messages.push(toolMessage, assistantMessage) 

where you don't push toolMessage to the message history if it is empty. There may also be other occurrences where this check is necessary.

Also, in app.py this could be made more generally robust if you made sure message wasn't empty in the for loop by adding the if clause at line 403

for message in request_messages:
  if message:
     message.append({

[sarah-widder]

Hi folks this was fixed in #372 - please give it a try.

[wcwong]

I took another look and verified that #372 isn't a complete fix. Empty elements are being inserted on the client side. So while the change keeps the server from crashing on invalid data, invalid data is still being inserted into the stream.

From my previous comment, lines 201 and 344 in frontend/src/pages/chat/Chat.tsx need to be wrapped so we don't insert an empty element into the array. specifically

isEmpty(toolMessage) ?
    conversation.messages.push(assistantMessage) :
    conversation.messages.push(toolMessage, assistantMessage) 

Note that the setMessages call later should also likely be wrapped:

isEmpty(toolMessage) ?
   setMessages([...messages, assistantMessage]) :
   setMessages([...messages, toolMessage, assistantMessage]); 

[sarah-widder]

Another fix was merged in #385. Closing this issue