Jupyterlab websockets not being proxied correctly

s-andrews commented 1 month ago

The jupyter front end launches but we can't run any code. This seems to be because the web sockets aren't being proxied correctly.

I think this page has the information we'd need to help fix this.

From doing some testing I think the upshot is:

Apache can do the proxying of websockets
There is a difference betwen ws:// and wss:// which is the security layer
Proxying matches on the URL and the first match wins. If you use a simple proxypass then you need to specify the subset of URLs for websockets first and then everything else matches http
All of our wss requests were ending up with a 400 error
With the latest changes I can get a 500 error which has to be an improvement!

My latest version (which doesn't work still)

#andrewss compute-1-0 36116 89887 33bdfc4061dec2bf5d9322bb3fecb8349c329baaf0670fddb218f93ee18e3950
ProxyPass /jupyterlab/HKARTBUEJZTJBQGRRAPT/api/kernels/ wss://compute-1-0:36116/jupyterlab/HKARTBUEJZTJBQGRRAPT/api/kernels/
ProxyPassReverse /jupyterlab/HKARTBUEJZTJBQGRRAPT/api/kernels/ wss://compute-1-0:36116/jupyterlab/HKARTBUEJZTJBQGRRAPT/api/kernels/
ProxyPass /jupyterlab/HKARTBUEJZTJBQGRRAPT/api/events/ wss://compute-1-0:36116/jupyterlab/HKARTBUEJZTJBQGRRAPT/api/events/
ProxyPassReverse /jupyterlab/HKARTBUEJZTJBQGRRAPT/api/events/ wss://compute-1-0:36116/jupyterlab/HKARTBUEJZTJBQGRRAPT/api/events/
ProxyPass /jupyterlab/HKARTBUEJZTJBQGRRAPT/ http://compute-1-0:36116/jupyterlab/HKARTBUEJZTJBQGRRAPT/
ProxyPassReverse /jupyterlab/HKARTBUEJZTJBQGRRAPT/ http://compute-1-0:36116/jupyterlab/HKARTBUEJZTJBQGRRAPT/

s-andrews commented 1 month ago

If I change the proxy rules to use ws:// instead of wss:// I get a 403 forbidden error, which again seems like the traffic is getting to the right place at least.

s-andrews commented 1 month ago

Directly using websocat I can also see that a direct connection to ws fails, but a wss connection connects and then waits.

[andrewss@capstone ~]$ ./websocat ws://compute-1-0:36116/jupyterlab/HKARTBUEJZTJBQGRRAPT/api/events/subscribe
websocat: WebSocketError: WebSocketError: Received unexpected status code (403 Forbidden)
websocat: error running
[andrewss@capstone ~]$ ./websocat wss://compute-1-0:36116/jupyterlab/HKARTBUEJZTJBQGRRAPT/api/events/subscribe
[waits]

s-andrews commented 1 month ago

OK, some progress. I found this in the logs:

[Wed Sep 25 10:52:57.799309 2024] [core:error] [pid 2713139:tid 2713303] [remote 10.99.100.2:24063] AH01961:  failed to enable ssl support [Hint: if using mod_ssl, see SSLProxyEngine]

..which led me to this page which said I need to add:

SSLProxyEngine on

To the conf file. When I do this I get further, but the connections are now blocked, rather than giving a 500 error. It still fails, but in the logs I now see:

[Wed Sep 25 11:05:02.530978 2024] [proxy:error] [pid 2732607:tid 2732657] [client 149.155.134.75:58998] AH00898: Error during SSL Handshake with remote server returned by /jupyterlab/VYFRCHVLTUIKBDYPGKJL/api/events/subscribe
[Wed Sep 25 11:05:02.530987 2024] [proxy_http:error] [pid 2732607:tid 2732657] [client 149.155.134.75:58998] AH01097: pass request body failed to 10.99.100.2:35498 (compute-1-0) from 149.155.134.75 ()

Looking at this threadwe seem to see the problem which is that we're not using trusted certificates so this is going to fail. I therefore added:

SSLProxyEngine On
SSLProxyVerify none
SSLProxyCheckPeerCN off
SSLProxyCheckPeerName off
SSLProxyCheckPeerExpire off

..and tried again, but we get the same failure. I feel like I'm getting closer though.

s-andrews commented 1 month ago

Found it. After running a local instance here and seeing what it did I saw that it wasn't using wss at all, just plain ws. If you tried to connect to this directly then you got a 400 error, which was what I saw in the web app. The key was replicating the original headers sent to the head node and forwarding them on to the compute node. The correct working form of the config was therefore:

ProxyPreserveHost On
#andrewss compute-1-3 20387 89895 a1f75728bcf574a3eb955c63e12a172e7a04fa2f1fb9ab680e4e53b8e1b2c0f3
ProxyPass /jupyterlab/BRLWFIKHGKDYXQWLNLVP/api/kernels/ ws://compute-1-3:20387/jupyterlab/BRLWFIKHGKDYXQWLNLVP/api/kernels/
ProxyPassReverse /jupyterlab/BRLWFIKHGKDYXQWLNLVP/api/kernels/ ws://compute-1-3:20387/jupyterlab/BRLWFIKHGKDYXQWLNLVP/api/kernels/
ProxyPass /jupyterlab/BRLWFIKHGKDYXQWLNLVP/api/events/ ws://compute-1-3:20387/jupyterlab/BRLWFIKHGKDYXQWLNLVP/api/events/
ProxyPassReverse /jupyterlab/BRLWFIKHGKDYXQWLNLVP/api/events/ ws://compute-1-3:20387/jupyterlab/BRLWFIKHGKDYXQWLNLVP/api/events/
ProxyPass /jupyterlab/BRLWFIKHGKDYXQWLNLVP/ http://compute-1-3:20387/jupyterlab/BRLWFIKHGKDYXQWLNLVP/
ProxyPassReverse /jupyterlab/BRLWFIKHGKDYXQWLNLVP/ http://compute-1-3:20387/jupyterlab/BRLWFIKHGKDYXQWLNLVP/

s-andrews / capstone_monitor

Jupyterlab websockets not being proxied correctly #22