WebSocket disconnects and does not return the response when authentication enabled

[mikitakandratsiuk]

Please provide us with the following information:

This issue is a: (mark with an x)

• [x] bug report -> please search issues before submitting
• [ ] documentation issue or request
• [ ] regression (a behavior that used to work and stopped in a new release)

Issue description

WebSocket disconnects and does not return the response when authentication enabled.

I have a sample Web Socket echo server mikitakandratsiuk/sample-websocket written in Python which appends all submitted messages on the page. It works fine in the ACA when authentication is disabled.

When the authentication is enabled (Azure AD), the WebSocket connection is being established, but when submitting a message the response is not returned (i.e. message is not appended on the page) and connection is closed with the exception on the server side (see below).

ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/uvicorn/protocols/websockets/websockets_impl.py", line 254, in run_asgi
    result = await self.app(self.scope, self.asgi_receive, self.asgi_send)
  File "/usr/local/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 78, in __call__
    return await self.app(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/fastapi/applications.py", line 284, in __call__
    await super().__call__(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/starlette/applications.py", line 122, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/starlette/middleware/errors.py", line 149, in __call__
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/starlette/middleware/httpsredirect.py", line 19, in __call__
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 79, in __call__
    raise exc
  File "/usr/local/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 68, in __call__
    await self.app(scope, receive, sender)
  File "/usr/local/lib/python3.10/site-packages/fastapi/middleware/asyncexitstack.py", line 20, in __call__
    raise e
  File "/usr/local/lib/python3.10/site-packages/fastapi/middleware/asyncexitstack.py", line 17, in __call__
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 718, in __call__
    await route.handle(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 341, in handle
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 82, in app
    await func(session)
  File "/usr/local/lib/python3.10/site-packages/fastapi/routing.py", line 292, in app
    await dependant.call(**values)
  File "/app/app.py", line 27, in websocket_endpoint
    data = await websocket.receive_text()
  File "/usr/local/lib/python3.10/site-packages/starlette/websockets.py", line 113, in receive_text
    self._raise_on_disconnect(message)
  File "/usr/local/lib/python3.10/site-packages/starlette/websockets.py", line 105, in _raise_on_disconnect
    raise WebSocketDisconnect(message["code"])
starlette.websockets.WebSocketDisconnect: 1002

Steps to reproduce

1. Deploy a websocket enabled Container App from this sample code (used Azure Portal) with Ingress enabled.
2. Test the WebSocket connection by pointing the browser to the container app: https://**********.germanywestcentral.azurecontainerapps.io/ws. It opens a WebSocket connection. When submit a text message into the form, it is sent to WebSocket and response is appended on the page.
3. Add an authentication provider to the Container App (used Azure AD), and enable it (used Azure Portal).
4. Go to the /ws page again. In the browser console log note that WebSocket connection is established. Submit a text message. Note that response is not returned from the WebSocket and not appended on the page. Additionally note the disconnect exception in the Container App logs.

Expected behavior [What you expected to happen.] Application with WebSocket and enabled authentication works fine (submitted messages are returned and appended on the page), same as it works when the authentication is disabled.

Actual behavior [What actually happened.] WebSocket disconnects when the message is submitted.

Screenshots

Additional context

All services are spined up using Azure Portal.

Additionally, I have followed the issue https://github.com/microsoft/azure-container-apps/issues/549. I tried to run jmalloc/echo-server, and it works fine with and without authentication. However, I'm not sure it is the same problem.

[ahmelsayed]

Thanks for the repro @mikitakandratsiuk.

I'm still looking at it, so I don't have much of an update. But I can repro it and setting uvicorn log_level to trace shows that the client -> server message is reaching it, but it gets 1002 ProtocolError when it's trying to write back

DEBUG:    < GET /ws HTTP/1.1
DEBUG:    < cache-control: no-cache
DEBUG:    < pragma: no-cache
DEBUG:    < connection: Upgrade
DEBUG:    < upgrade: websocket
DEBUG:    < sec-websocket-version: 13
[...]
DEBUG:    > HTTP/1.1 101 Switching Protocols
DEBUG:    > Upgrade: websocket
DEBUG:    > Connection: Upgrade
DEBUG:    > server: uvicorn
INFO:     connection open
DEBUG:    = connection is OPEN

DEBUG:    < TEXT 'Hello' [5 bytes]
TRACE:    ...:0 - ASGI [5] Receive {'type': 'websocket.receive', 'text': '<5 chars>'}
TRACE:    ...:0 - ASGI [5] Send {'type': 'websocket.send', 'bytes': '<26 bytes>'}
DEBUG:    > TEXT 'web socket response: Hello' [26 bytes]
DEBUG:    < CLOSE 1002 (protocol error) [2 bytes]
DEBUG:    = connection is CLOSING
DEBUG:    > CLOSE 1002 (protocol error) [2 bytes]
DEBUG:    x half-closing TCP connection

TRACE:    127.0.0.1:50996 - WebSocket connection lost
DEBUG:    = connection is CLOSED
TRACE:    ...:0 - ASGI [5] Receive {'type': 'websocket.disconnect', 'code': 1002}
TRACE:    ...:0 - ASGI [5] Raised exception
ERROR:    Exception in ASGI application 
[...]

I can also see from the edge loadbalancer that the upstream disconnected. so it's definitely something isolated inside the auth service that's rejecting the write from the python app. I've contacted the owners of the auth service as I don't see any helpful logs from there. Will update when I have an update

[mikitakandratsiuk]

Hi @ahmelsayed, do you have any news or updates about the issue? Thanks!

[fnavarrete83]

Hi @ahmelsayed, also interested in a solution to this issue with the websockets on authentication. Any news?

[ahmelsayed]

Sorry for the delay.

It seems that the problem is in the auth proxy not supporting compression; websocket's Sec-WebSocket-Extensions: permessage-deflate header which chrome/firefox set by default. I think the go server doesn't do compression by default either.

For a workaround you can disable compression in uvicorn by setting ws_per_message_deflate=False

e.g:

if __name__ == "__main__":
    uvicorn.run(app, host="0.0.0.0", port=8000, ws_per_message_deflate=False)

[mikitakandratsiuk]

Hi @ahmelsayed, this workaround worked and I can proceed with it. Thanks a lot!

Although, in my case it required some downstream workarounds, because I use third-party libraries and don't have direct access to uvicorn.Config object. So I'm wondering if there are plans to resolve the issue in a long-term?

Thanks again for the support!

[ahmelsayed]

Yes, I think it should ideally just work. So will mark it as bug for now