gunicorn worker timeout when doing websocket reverse proxy

[haoming29]

Hi,

I'm using both flask-sock and websocket-client to set up a reverse proxy of WebSocket with Flask. It works fine on the development server. However, in the production server with Gunicorn, the WebSocket connection gets cut off after 30s, which is the timeout for Gunicorn worker. Any thoughts on why this happens? Is it becausethread.join() blocks the response of flask-sock? Thanks.

The error stack trace is as follows:

[2023-08-28 01:42:17,113] INFO in auth: Connected to backend server at ws://xxx.xxx.xxx.xxx:9090
Exception in thread Thread-5:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/usr/local/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/periscope-be/periscope/routes/auth.py", line 88, in recv_from_ros
    src_message = client_ws.recv()
  File "/opt/venv/lib/python3.8/site-packages/websocket/_core.py", line 362, in recv
    opcode, data = self.recv_data()
  File "/opt/venv/lib/python3.8/site-packages/websocket/_core.py", line 385, in recv_data
    opcode, frame = self.recv_data_frame(control_frame)
  File "/opt/venv/lib/python3.8/site-packages/websocket/_core.py", line 406, in recv_data_frame
    frame = self.recv_frame()
  File "/opt/venv/lib/python3.8/site-packages/websocket/_core.py", line 445, in recv_frame
    return self.frame_buffer.recv_frame()
  File "/opt/venv/lib/python3.8/site-packages/websocket/_abnf.py", line 338, in recv_frame
    self.recv_header()
  File "/opt/venv/lib/python3.8/site-packages/websocket/_abnf.py", line 294, in recv_header
    header = self.recv_strict(2)
  File "/opt/venv/lib/python3.8/site-packages/websocket/_abnf.py", line 373, in recv_strict
    bytes_ = self.recv(min(16384, shortage))
  File "/opt/venv/lib/python3.8/site-packages/websocket/_core.py", line 529, in _recv
    return recv(self.sock, bufsize)
  File "/opt/venv/lib/python3.8/site-packages/websocket/_socket.py", line 122, in recv
    raise WebSocketConnectionClosedException(
websocket._exceptions.WebSocketConnectionClosedException: Connection to remote host was lost.
[2023-08-28 01:42:47 +0000] [9] [CRITICAL] WORKER TIMEOUT (pid:25)
[2023-08-28 01:42:47,425] INFO in auth: Connection to backend server closed.
[2023-08-28 01:42:47 +0000] [25] [INFO] Worker exiting (pid: 25)
[2023-08-28 01:42:48 +0000] [9] [ERROR] Worker (pid:25) was sent SIGKILL! Perhaps out of memory?

This is the code for setting up the reverse proxy:

@sock.route('/backend')
def reverse_proxy_to_backend(ws):
    client_ws = websocket.WebSocket()

    def send_to_ros(ws, client_ws):
        while True:
            message = ws.receive()
            if message:
                client_ws.send(message)

    def recv_from_ros(ws, client_ws):
        while True:
            src_message = client_ws.recv()
            if src_message:
                ws.send(src_message)

    try:
        client_ws.connect(SERVER_URL)
        current_app.logger.info('Connected to backend server at ' + SERVER_URL)

        send_thread = threading.Thread(target=send_to_ros, args=(ws, client_ws))
        recv_thread = threading.Thread(target=recv_from_ros, args=(ws, client_ws))

        send_thread.start()
        recv_thread.start()

        send_thread.join()
        recv_thread.join()

    except ConnectionError:
        current_app.logger.error('Connection error to backend: ' + SERVER_URL)
    finally:
        client_ws.close()
        current_app.logger.info('Connection to backend server closed.')

[miguelgrinberg]

Did you enable pings on your flask-sock server? The pings can be sent every 25 seconds or so to prevent gunicorn from timing out.

[haoming29]

Hi, I didn't turn on pings but even if I did and kept the socket sending messages, gunicorn worker still exits after 30s.

[haoming29]

I figured out that I didn't set the number of threads in gunicorn configuration. Sorry, I should read the documentation more carefully.