Are Python WebSockets under SSL intended to work?

[acnebs]

Hi, I recently changed over our infrastructure to use Nginx Unit, and mostly it is going great. The only thing that is not going great is that this seems to have broken WebSockets for our app in production.

Our app is a Python ASGI app (FastAPI) so from what I've read it seems like WebSockets should be supported just fine, and while developing locally using Unit they do indeed work great. However in production they seem to be broken. As far as I can figure, the only difference between dev and prod is that prod is (of course) using an SSL cert, while dev on localhost is of course not. The stack before was Gunicorn w/ Uvicorn + Nginx + HAProxy, which had to have a number of tweaks and fiddles at the different layers to get WS working correctly, but working they were.

Is this a known issue? Are there any remediations I can try? A great thing about Unit is that it has massively reduced the config required to serve our application. The bad thing is that when something is broken it is hard to try to debug different layers of the stack because there is only the one 😅. I've tried debugging at the application level itself and that seems to be a deadend, so for now I've concluded the issue must be with how Unit is working alongside the application/WebSockets.

For reference, here are my Unit configs in dev and prod.

dev config

{
    "listeners": {
        "*:5000": {
            "pass": "applications/supernotes"
        }
    },
    "applications": {
        "supernotes": {
            "type": "python 3.11",
            "processes": 8,
            "path": "/Users/connor/supernotes/backend/",
            "working_directory": "/Users/connor/supernotes/backend/",
            "home": "/Users/connor/Library/Caches/pypoetry/virtualenvs/supernotes-0y77xQ4Y-py3.11",
            "module": "api.main",
            "callable": "app"
        }
    }
}

prod config

{
    "listeners": {
        "*:80": {
            "pass": "routes/acme"
        },
        "*:443": {
            "pass": "applications/supernotes",
            "tls": {
                "certificate": "certbot"
            }
        }
    },
    "routes": {
        "acme": [{
            "action": {
                "share": "/var/www/certs/$uri"
            }
        }]
    },
    "applications": {
        "supernotes": {
            "type": "python 3.11",
            "processes": 8,
            "path": "/var/www/sn-prod-deploy/backend",
            "working_directory": "/var/www/sn-prod-deploy/backend",
            "home": "/var/www/sn-prod-deploy/venv",
            "module": "api.main",
            "callable": "app"
        }
    }
}

Thanks for any help and thank you for making a great piece of software like Unit!

[tippexs]

Thanks for your honest feedback! We are ware of the current debugging capabilities and we are working on it making it better! Let's use this issue to identify challenges and what we can do to help you.

First of all, can you please share what OS and Unit Version you are using locally AND on your production server? (Dev looks like MacOS)

Did you installed the Python Module using our Packages? On Dev and Prod?

Have you tried using certs and websockets on your local dev machine? Is this working? Are there any errors in the unit log? Have you tried to run Unit in debug-mode on Prod to get a more detailed log output?

The Unit debug binaries comes with your package (based on the OS you are using on Prod).

[acnebs]

Thanks for the quick reply. Dev is macOS, prod is Ubuntu 23.04. Using the official packages for Python support. Haven't tried using certs locally, will try that now to see how that goes. Didn't see anything of note in the unit log, but also am not running in debug-mode in prod, so can try that if I don't find anything from running SSL certs locally.

[ac000]

Quick question. Are you using wss:// in production? does ws:// work there?

[acnebs]

We are indeed using wss:// in production, and ws:// locally. Have not tried to use ws:// in production, currently trying to get wss:// working locally first.

[acnebs]

EDIT: Ignore the below, misconfiguration on my end. I was able to get WebSockets working just fine locally with the wss:// scheme with a local SSL cert.

One thing I realized when doing this process, however, is that locally I am always specifying a port (5000), where as in production this value is implicit. Could it be that the port is somehow messing up here?

old comment

~I can confirm that when running locally using SSL and the wss:// scheme, WebSockets break. The error I'm receiving in unit.log is:~

~2023/12/07 23:15:21 [error] 40486#6372577 *217 SSL_do_handshake(9) failed (1046: sslv3 alert certificate unknown) (OpenSSL: error:0A000416:SSL routines::sslv3 alert certificate unknown:SSL alert number 46)~

~I'd assumed this would be fine since AFAIK wss:// should be happening on port :443 as well, so the cert should be used by the listener for that just fine. But I guess maybe not?~

[acnebs]

After messing around with the production environment and looking at the logs, I am actually seeing two errors repeatedly that definitely seem related:

2023/12/07 15:54:00 [info] 428141#428155 *735 websocket error 1001: Remote peer is going away

and

2023/12/07 15:54:08 [info] 428141#428155 *823 SSL_read(46, 7F2724245F48, 135) failed (104: Connection reset by peer)

[ac000]

If you hit the thing with curl -v ... does that provide any useful messages?

You could try explicitly specifying :443 in the url, but that would seem a little bonkers if that actually worked...

[acnebs]

I might be a bit dense, but what do you want me to hit with verbose curl?

Tried specifying 443 explicitly, indeed did not seem to help.

[ac000]

I might be a bit dense, but what do you want me to hit with verbose curl?

Yes, ignore me. Many years ago I wrote a websocket server and misremembered using curl to check the handshake stuff, it was most probably tcpdump/wireshark...

Tried specifying 443 explicitly, indeed did not seem to help.

This is likely going to need a reproducer app. Don't suppose you have a minimal example or a pointer to one?

[acnebs]

Yep, I will see if I can make a very simple repro in the next few days. Somewhat difficult because the websockets are working in dev, so it also might be an Ubuntu issue.