SocketIO behind two Nginx Proxies creates AssertionError: The application did not call start_response()

ghost commented 4 years ago

Describe the bug I am getting the following message when attempting to connect to a websocket. "AssertionError: The application did not call start_response()"

My application is hosted in Docker Enterprise on Kubernetes. I have 3 aliases set on the host ingress file.

Two of the host URLs are load balanced between data centers using Nginx and those produce the error below. So they use Nginx at a Global Traffic Manager Level and an Nginx ingress. THIS PRODUCES THE ERROR BELOW
One of the URLs hits the Kubernetes ingress directly without the GTM. THIS WORKS

So I know under normal circumstances my application works without the nginx GTM in front, but as soon as I try to access with the GTM URL on the same instance I get errors.

To Reproduce Running Docker-Enterprise Kubernetes, but not sure that matters. Using Kubernetes Nginx Ingress Using Nginx Proxy with websocket proxying and Upgrade cmds. Have two URLs controlled by DNS and GTM

nginx.ingress.kubernetes.io/server-alias: project.ekaas.bos.corp.com project.dapps.corp.com projectnongtm.ekaas.bos.corp.com

Set cors_allowed_origins to allow domains specified above.

Expected behavior A websocket is expected to upgrade.

After the message 'Sending packet MESSAGE data 0' shown below, we get an error. I expect the next messages to be similar to below.

lask-app-5749d6b67b-sw9sq flask-app f7a9cfd2d6944002bd285c9bd7e23c2e: Received request to upgrade to websocket
flask-app-5749d6b67b-sw9sq flask-app INFO:engineio.server:f7a9cfd2d6944002bd285c9bd7e23c2e: Received request to upgrade to websocket
flask-app-5749d6b67b-sw9sq flask-app f7a9cfd2d6944002bd285c9bd7e23c2e: Upgrade to websocket successful
flask-app-5749d6b67b-sw9sq flask-app INFO:engineio.server:f7a9cfd2d6944002bd285c9bd7e23c2e: Upgrade to websocket successful

Logs note: IP and HOST have been replaced

flask-app-c88d5ff74-6m48x flask-app 1b823f61364546798c5e0ddab5773a6e: Sending packet MESSAGE data 0
flask-app-c88d5ff74-6m48x flask-app INFO:engineio.server:1b823f61364546798c5e0ddab5773a6e: Sending packet MESSAGE data 0
flask-app-c88d5ff74-6m48x flask-app Traceback (most recent call last):
flask-app-c88d5ff74-6m48x flask-app   File "/venv/lib/python3.7/site-packages/gevent/pywsgi.py", line 968, in handle_one_response
flask-app-c88d5ff74-6m48x flask-app     self.run_application()
flask-app-c88d5ff74-6m48x flask-app   File "/venv/lib/python3.7/site-packages/geventwebsocket/handler.py", line 87, in run_application
flask-app-c88d5ff74-6m48x flask-app     return super(WebSocketHandler, self).run_application()
flask-app-c88d5ff74-6m48x flask-app   File "/venv/lib/python3.7/site-packages/gevent/pywsgi.py", line 916, in run_application
flask-app-c88d5ff74-6m48x flask-app     self.process_result()
flask-app-c88d5ff74-6m48x flask-app   File "/venv/lib/python3.7/site-packages/gevent/pywsgi.py", line 902, in process_result
flask-app-c88d5ff74-6m48x flask-app     self.write(data)
flask-app-c88d5ff74-6m48x flask-app   File "/venv/lib/python3.7/site-packages/gevent/pywsgi.py", line 748, in write
flask-app-c88d5ff74-6m48x flask-app     raise AssertionError("The application did not call start_response()")
flask-app-c88d5ff74-6m48x flask-app AssertionError: The application did not call start_response()
flask-app-c88d5ff74-6m48x flask-app 2020-06-02T18:19:16Z {'REMOTE_ADDR': '10.0.0.0', 'REMOTE_PORT': '7246', 'HTTP_HOST': 'project.dapps.corp.com', (hidden keys: 36)} failed with AssertionError

Additional context Add any other context about the problem here.

ghost commented 4 years ago

Ingress below.

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: project-ingress
  annotations:
    kubernetes.io/ingress.class: nginx
    # START ONLY FOR DEV
    # nginx.ingress.kubernetes.io/ssl-redirect: "true"
    # nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
    # END ONLY FOR DEV
    # START ONLY FOR PROD
    nginx.ingress.kubernetes.io/server-alias: project.ekaas.bos.corp.com project.dapps.corp.com projectnongtm.ekaas.bos.corp.com
    # END ONLY FOR PROD
spec:
  rules:
    - host: project.dapps.corp.com
      http:
        paths:
          - path: /
            backend:
              serviceName: flask-app
              servicePort: 8000
          - path: /api
            backend:
              serviceName: fastapi-app
              servicePort: 8001

miguelgrinberg commented 4 years ago

This error comes from the gevent web server. I can't really tell you what's wrong, but my guess is that something at the first nginx is corrupting the requests, in a way that causes Flask-SocketIO to mishandle it and gevent to flag it as an error.

It would be useful to get complete request dumps from the request that passed through the two nginx proxies versus one that passed only through one. Do you think you could modify the Flask-SocketIO code to log the environ dictionary in this middleware? That would probably help us determine what's wrong.

Also as a side note, is this top-level nginx set up to do session affinity? If not, that could be the problem, try setting ip_hash for your cluster.

miguelgrinberg / Flask-SocketIO

SocketIO behind two Nginx Proxies creates AssertionError: The application did not call start_response() #1293