Proxying Xpra is not working

[ghost]

UDP: a way to reproduce: https://github.com/jupyterhub/nbserverproxy/issues/35#issuecomment-393025940 UPD: proposed fix: https://github.com/jupyterhub/nbserverproxy/issues/35#issuecomment-394719522

I've tried to proxy Xpra html5 client with nbserverproxy (version 0.8.3), but it doesn't seem to work. When I start Xpra on a local machine with the

xpra start --bind-tcp=0.0.0.0:14500 --html=on --start=xterm

command and open localhost:14500, I see xterm in my browser.

I've tried to run this command from the jupyter terminal and then access .../proxy/14500. It struggles to Update connection to WebSocket.

When I connect directly, the response to Upgrade request is

HTTP/1.1 101 Switching Protocols
Server: Xpra-WebSockify Python/2.7.5
Date: Wed, 30 May 2018 09:46:33 GMT
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Accept: ***
Sec-WebSocket-Protocol: binary
Expires: 0
Pragma: no-cache
Cache-Control: no-cache, no-store, must-revalidate

But behind nbserverproxy the response is

HTTP/1.1 302 Found
Server: TornadoServer/5.0.2
Content-Type: text/html; charset=UTF-8
Date: Wed, 30 May 2018 09:45:42 GMT
Location: //proxy/14500
Content-Length: 0

There is Wiki page about proxying Xpra with Nginx: https://www.xpra.org/trac/wiki/Nginx AFAIU it makes Nginx to do something to HTTP headers so that websockets work behind a proxy.

I know about https://github.com/ryanlovett/nbnovnc/, but I would like to see Xpra working.

[ghost]

Looks like it is Xpra issue, it doesn't handle websocket paths. Close for now.

[ghost]

I've updated Xpra and now it recognizes that it runs with some path in URL. Still it has problems with websockets. A way to reproduce: Dockerfile with Xpra and notebook:

FROM centos:7

RUN \
    yum -y install \
    epel-release \
    https://centos7.iuscommunity.org/ius-release.rpm \
 && rm -rf /var/cache/yum/

ADD https://xpra.org/repos/CentOS/xpra.repo /etc/yum.repos.d/

RUN \
    yum install -y \
    python36u python36u-libs python36u-devel python36u-pip \
    xpra python-websockify xterm \
 && rm -rf /var/cache/yum/

RUN pip3.6 --no-cache-dir install \
            'notebook==5.5.0' \
            'nbserverproxy'

RUN jupyter serverextension enable --py nbserverproxy

RUN dbus-uuidgen > /etc/machine-id

CMD ["jupyter", "notebook"]

docker build -t xpra-mwe:0.0.0 . to build.

Then run it with docker run --network=host --rm -ti xpra-mwe:0.0.0 jupyter notebook --allow-root. Connect to the notebook in a browser and open a terminal (in the notebook). In the terminal, run

xpra --daemon=no --dbus-launch=no --pulseaudio=no start --bind-tcp=0.0.0.0:14500 --html=on --start=xterm

and wait until it writes xpra is ready.

Go to http://localhost:14500/ and see xterm in your browser.

Go to http://localhost:8888/proxy/14500/ and see Xpra HTML5 client trying to connect to websocket with no luck

When I connect to http://localhost:14500/, the response to websocket Upgrade request is

HTTP/1.1 101 Switching Protocols
Server: Xpra-WebSockify Python/2.7.5
Date: Wed, 30 May 2018 09:46:33 GMT
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Accept: ***
Sec-WebSocket-Protocol: binary
Expires: 0
Pragma: no-cache
Cache-Control: no-cache, no-store, must-revalidate

But behind nbserverproxy the response is

HTTP/1.1 302 Found
Server: TornadoServer/5.0.2
Content-Type: text/html; charset=UTF-8
Date: Wed, 30 May 2018 09:45:42 GMT
Location: //proxy/14500
Content-Length: 0

[ryanlovett]

Hey @BerserkerTroll, thanks for reporting this issue. I think supporting Xpra would be cool, but I may not get to look at this for a few weeks. At first glance, /proxy/14500/ != //proxy/14500, so something may be amiss with slash parsing or requesting.

[ghost]

I think supporting Xpra would be cool

I think now it is not about supporting Xpra, it feels like a general WebSocket proxying issue.

At first glance, /proxy/14500/ != //proxy/14500, so something may be amiss with slash parsing or requesting.

Indeed, it helped. A bit. Now it is able to connect to WebSocket (The message "Opening WebSocket connection" changed to "WebSocket connection established"), but it never shows xterm and after 10-15 seconds throws me to /connect.html again. And I see a stack trace (UPD: probably, it is not the source of the problems, because it doesn't appear each time):

[I 07:38:08.021 NotebookApp] Client sent subprotocols: ['binary']
[I 07:38:08.023 NotebookApp] Trying to establish websocket connection to ws://127.0.0.1:14500/
[I 07:38:08.282 NotebookApp] Websocket connection established to ws://127.0.0.1:14500/
[E 07:38:13.566 NotebookApp] Uncaught exception in /proxy/14500/
    Traceback (most recent call last):
      File "/usr/lib64/python3.6/site-packages/tornado/websocket.py", line 801, in write_message
        fut = self._write_frame(True, opcode, message, flags=flags)
      File "/usr/lib64/python3.6/site-packages/tornado/websocket.py", line 780, in _write_frame
        return self.stream.write(frame)
      File "/usr/lib64/python3.6/site-packages/tornado/iostream.py", line 525, in write
        self._check_closed()
      File "/usr/lib64/python3.6/site-packages/tornado/iostream.py", line 1058, in _check_closed
        raise StreamClosedError(real_error=self.error)
    tornado.iostream.StreamClosedError: Stream is closed

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "/usr/lib64/python3.6/site-packages/tornado/websocket.py", line 498, in _run_callback
        result = callback(*args, **kwargs)
      File "/usr/lib/python3.6/site-packages/nbserverproxy/handlers.py", line 162, in on_message
        self.ws.write_message(message)
      File "/usr/lib64/python3.6/site-packages/tornado/websocket.py", line 1176, in write_message
        return self.protocol.write_message(message, binary=binary)
      File "/usr/lib64/python3.6/site-packages/tornado/websocket.py", line 803, in write_message
        raise WebSocketClosedError()
    tornado.websocket.WebSocketClosedError

Errors from Xpra (just in case):

2018-05-31 08:06:43,769 unknown or invalid packet type: suspend from Protocol(ws websocket: 127.0.0.1:14500 <- 127.0.0.1:42432)
2018-05-31 08:06:55,532 unknown or invalid packet type: resume from Protocol(ws websocket: 127.0.0.1:14500 <- 127.0.0.1:42448)
2018-05-31 08:06:59,484 unknown or invalid packet type: resume from Protocol(ws websocket: 127.0.0.1:14500 <- 127.0.0.1:42464)
2018-05-31 08:07:13,013 unknown or invalid packet type: sound-control from Protocol(ws websocket: 127.0.0.1:14500 <- 127.0.0.1:42480)

Ofc, I do not see these errors when I connect directly to localhost:14500.

Looks like the websocket data is corrupted somehow on the way from the Xpra client to the Xpra server.

[ghost]

I've debugged it in Wireshark a bit. I've spotted very strange things. The proxy answers to Upgrade request immediately with HTTP 101 "Switching protocol", before it even sends the request to the backend! Probably I'm bad at asynchronous programming, but this looks too asynchronous for me.

In details, when I connect directly, HTML5 client switches protocol to websocket and sends "hello" packet to the Xpra server:

1. Browser -> Server: Upgrade to WebSocket
2. Server -> Browser: HTTP 101 Switching protocol
3. Browser -> Server: WebSocket: Hello message with client details
4. Server -> Browser: answer to the hello, communication starts

Behind the proxy, the "hello" message never gets to the backend:

1. Browser -> Proxy: Upgrade to WebSocket
2. Proxy -> Browser: HTTP 101 Switching protocol — haven't even asked the backend!!!
3. Proxy -> Backend: Upgrade to WebSocket
4. Browser -> Proxy: WebSocket: Hello message with client details — Proxy sends it to /dev/null
5. Backend -> Proxy: HTTP 101 Switching protocol
6. (10 seconds later) Browser -> Proxy: WebSocket connection close — without getting the answer to the Hello request, the client closes the connection to a "dead" server
7. Proxy -> Backend: WebSocket connection close — at lest "connection close" was not redirected to /dev/null, thanks!

[ghost]

Well, I've added 1 second delay before "hello" message is sent and now the HTML5 client is able to connect and show me the xterm window. But not everything working as expected. Without a window manager running, the HTML5 client draws window frames itself and loads png-images of close and minimize buttons from the server. However, these images can't be loaded through the proxy.

It looks that the WebSocket proxying logics is flawed: the https://stackoverflow.com/questions/38663666/how-can-i-serve-a-http-page-and-a-websocket-on-the-same-url-in-tornado method is not intended for proxies. The proxy is answering immediately to the Upgrade request whilst it shall respond with the backend response.

[ghost]

I don't know how stupid is this asyncprogrammingwise, because I don't know much about Python and Tornado. It just works for me:

--- a/nbserverproxy/handlers.py
+++ b/nbserverproxy/handlers.py
@@ -93,12 +93,11 @@ class WebSocketHandlerMixin(websocket.WebSocketHandler):
     async def get(self, *args, **kwargs):
         if self.request.headers.get("Upgrade", "").lower() != 'websocket':
             return await self.http_get(*args, **kwargs)
-        # super get is not async
-        super().get(*args, **kwargs)
+        return await self.ws_get(*args, **kwargs)

 class LocalProxyHandler(WebSocketHandlerMixin, IPythonHandler):
-    async def open(self, port, proxied_path=''):
+    async def ws_get(self, port, proxied_path=''):
         """
         Called when a client opens a websocket connection.

@@ -144,12 +143,18 @@ class LocalProxyHandler(WebSocketHandlerMixin, IPythonHandler):
             self.log.info('Trying to establish websocket connection to {}'.format(client_uri))
             self._record_activity()
             request = httpclient.HTTPRequest(url=client_uri, headers=headers)
-            self.ws = await pingable_ws_connect(request=request,
-                on_message_callback=message_cb, on_ping_callback=ping_cb)
-            self._record_activity()
-            self.log.info('Websocket connection established to {}'.format(client_uri))
+            try:
+                self.ws = await pingable_ws_connect(request=request,
+                    on_message_callback=message_cb, on_ping_callback=ping_cb)
+            except Exception:
+                self.set_status(400)
+            else:
+                self._record_activity()
+                self.log.info('Websocket connection established to {}'.format(client_uri))
+                # super get is not async
+                super(WebSocketHandlerMixin, self).get(port, proxied_path)

-        ioloop.IOLoop.current().add_callback(start_websocket_connection)
+        return await start_websocket_connection()

     def on_message(self, message):
         """

[ryanlovett]

Just wanted to say thanks for your patience @BerserkerTroll! I'll be able to look at this and your PR soon.