Fix image tests: vncserver, websockify, jupyter-remote-desktop-proxy

[consideRatio]

This is work done in parallell to #93, initially thought to be a quick fix to ensure TurboVNC also works. Like @yuvipanda in #93 I ran into issues with different behavior locally and within the GitHub actions environment, but think the biggest difference got resolved by providing a TTY device.

@yuvipanda I propose #93 is completed in a dedicated job inside the test.yaml workflow, side by side to the image-test job finalized in this PR - and then finally in an optional dedicated PR we prune misc bash things I've done in favor of python things you've done.

I consider this to fix #98, which was bugfixed by #99 but not tested by automation to make it work as it is now to some extent at least.

Summary of misc changes

• added test for TurboVNC image in parallel to test for TigerVNC image
• added websockify --help and vncserver --help checks
• added a --fail flag to curl calls (detected by @yuvipanda in #93)
• added a test of vncserver startup
• added a test of websockify'ed vncserver startup
• added a test of jupyter-remote-desktop-proxy startup
  • typically fails once, then succeeds - tracked by #105
• provided a TTY when executing steps - a requirement for vncserver being started
• concluded that there is probably also ipv4 / ipv6 behaviors differing between my local computer and github actions environment, and it could influence testing of an isolated vncserver or websockify'ed vncserver, but not a jupyter server proxying via jupyter-server-proxy.

A blob of old notes, I've worked a lot of hours :scream:

Debugging the same issue Yuvi ran into I think: > This works perfectly fine locally, but unfortunately something is causing it to hang when the VNC client is trying to connect to the server on GitHub actions. will have to debug next time i get a chance. I've debugged this at length with minor progress. ## First attempt failure I've concluded that the initial websocket connection attempt often fails to get sensible responses etc, and I think this relates to jupyter-server-proxy finalizes a websocket handshake even if the handshake to the backend isn't finalized (https://github.com/jupyterhub/jupyter-server-proxy/issues/459). But this shouldn't cause issues for the second attempt that typically works locally. ## Future attempt failures The difference I observe locally and remote, comes down to this, where the green parts represent local and red parts represents failing remote test runs in github actions. (ignore timestamps, this is a composed set of log lines) ### websocat output ```diff [DEBUG websocat] Done third phase of interpreting options. [DEBUG websocat] Done fourth phase of interpreting options. [DEBUG websocat] Preparation done. Now actually starting. [DEBUG websocat::sessionserve] Serving ThreadedStdio to WsClient("ws://localhost:8888/desktop-websockify/?token=secret") with Options { websocket_text_mode: false, websocket_protocol: None, websocket_reply_protocol: None, udp_oneshot_mode: false, udp_broadcast: false, udp_multicast_loop: false, udp_ttl: None, udp_join_multicast_addr: [], udp_join_multicast_iface_v4: [], udp_join_multicast_iface_v6: [], udp_reuseaddr: false, unidirectional: false, unidirectional_reverse: false, max_messages: None, max_messages_rev: None, exit_on_eof: true, oneshot: false, unlink_unix_socket: false, unix_socket_accept_from_fd: false, exec_args: [], ws_c_uri: "ws://0.0.0.0/", linemode_strip_newlines: false, linemode_strict: false, origin: None, custom_headers: [], custom_reply_headers: [], websocket_version: None, websocket_dont_close: false, websocket_ignore_zeromsg: false, one_message: true, no_auto_linemode: false, buffer_size: 65536, broadcast_queue_len: 16, read_debt_handling: Silent, linemode_zero_terminated: false, restrict_uri: None, serve_static_files: [], exec_set_env: false, no_exit_on_zeromsg: false, reuser_send_zero_msg_on_disconnect: false, process_zero_sighup: false, process_exit_sighup: false, process_exit_on_disconnect: false, socks_destination: None, auto_socks5: None, socks5_bind_script: None, tls_domain: None, tls_insecure: false, headers_to_env: [], max_parallel_conns: None, ws_ping_interval: None, ws_ping_timeout: None, request_uri: None, request_method: None, request_headers: [], autoreconnect_delay_millis: 20, ws_text_prefix: None, ws_binary_prefix: None, ws_binary_base64: false, ws_text_base64: false, close_status_code: None, close_reason: None, asyncstdio: false, foreachmsg_wait_reads: false, announce_listens: false, timestamp_monotonic: false, print_ping_rtts: false, byte_to_exit_on: 28, max_ws_message_length: 209715200, max_ws_frame_length: 104857600, preamble: [], preamble_reverse: [], compress_deflate: false, compress_zlib: false, compress_gzip: false, uncompress_deflate: false, uncompress_zlib: false, uncompress_gzip: false, jsonrpc_omit_jsonrpc: false, inhibit_pongs: None, max_sent_pings: None } [INFO websocat::stdio_threaded_peer] get_stdio_peer (threaded) [INFO websocat::ws_client_peer] get_ws_client_peer [INFO websocat::ws_client_peer] Connected to ws +[DEBUG websocat::ws_peer] incoming binary +[DEBUG websocat::readdebt] Fulfilling the debt of 12 bytes +[DEBUG websocat::my_copy] Once mode requested, so aborting copy +[DEBUG websocat::my_copy] done +RFB 003.008 +[INFO websocat::sessionserve] Reverse finished +[DEBUG websocat::sessionserve] Reverse shutdown finished +[INFO websocat::sessionserve] One of directions finished +[DEBUG websocat::ws_peer] drop WsWriteWrapper -[DEBUG websocat::my_copy] zero len -[DEBUG websocat::my_copy] read_done -[DEBUG websocat::my_copy] done -[INFO websocat::sessionserve] Forward finished -[DEBUG websocat::sessionserve] Forward shutdown finished -[DEBUG websocat::ws_peer] drop WsWriteWrapper -[INFO websocat::sessionserve] One of directions finished ``` ### jupyter_server logs ```diff [I 2024-03-10 22:04:21.890 ServerApp] Trying to establish websocket connection to ws://localhost:56389/?token=secret 127.0.0.1: new handler Process 127.0.0.1 - - [10/Mar/2024 22:04:21] "GET /?token=secret HTTP/1.1" 101 - 127.0.0.1 - - [10/Mar/2024 22:04:21] 127.0.0.1: Plain non-SSL (ws://) WebSocket connection 127.0.0.1 - - [10/Mar/2024 22:04:21] 127.0.0.1: Path: '/?token=secret' 127.0.0.1 - - [10/Mar/2024 22:04:21] connecting to command: '/bin/sh -c cd /home/jovyan && /usr/bin/vncserver -rfbunixpath /tmp/tmpfm_suyv2/vnc-socket -xstartup /opt/install/jupyter_remote_desktop_proxy/share/xstartup -verbose -localhost -fg -geometry 1680x1050 -SecurityTypes None' (port 59467) [I 2024-03-10 22:04:21.894 ServerApp] Websocket connection established to ws://localhost:56389/?token=secret 127.0.0.1 - - [10/Mar/2024 22:04:21] 127.0.0.1:59467: Client closed connection 127.0.0.1 - - [10/Mar/2024 22:04:21] 127.0.0.1:59467: Closed target ``` ### websockify logs ```diff 127.0.0.1: new handler Process 127.0.0.1 - - [10/Mar/2024 21:15:57] "GET /?token=secret HTTP/1.1" 101 - 127.0.0.1 - - [10/Mar/2024 21:15:57] 127.0.0.1: Plain non-SSL (ws://) WebSocket connection 127.0.0.1 - - [10/Mar/2024 21:15:57] 127.0.0.1: Path: '/?token=secret' 127.0.0.1 - - [10/Mar/2024 21:15:57] connecting to command: '/bin/sh -c cd /home/jovyan && /usr/bin/vncserver -rfbunixpath /tmp/tmpp4fup6x4/vnc-socket -xstartup /opt/install/jupyter_remote_desktop_proxy/share/xstartup -verbose -localhost -fg -geometry 1680x1050 -SecurityTypes None' (port 35719) 127.0.0.1 - - [10/Mar/2024 21:15:57] 127.0.0.1:35719: Client closed connection 127.0.0.1 - - [10/Mar/2024 21:15:57] 127.0.0.1:35719: Closed target ``` ### vncserver logs ```diff Sun Mar 10 21:07:15 2024 Connections: accepted: /tmp/tmpn0g6yfd7/vnc-socket + VNCSConnST: closing /tmp/tmp6wsxj7wa/vnc-socket: Clean disconnection - VNCSConnST: closing /tmp/tmpn0g6yfd7/vnc-socket: read: Connection reset by - peer (104) EncodeManager: Framebuffer updates: 0 EncodeManager: Total: 0 rects, 0 pixels EncodeManager: 0 B (1:-nan ratio) Connections: closed: /tmp/tmpn0g6yfd7/vnc-socket ComparingUpdateTracker: 0 pixels in / 0 pixels out ComparingUpdateTracker: (1:-nan ratio) ```

[github-actions[bot]]

:point_left: Launch a binder notebook on this branch for commit 467d02459c1396c79a8b2630040629727851b0a1

I will automatically update this comment whenever this PR is modified

:point_left: Launch a binder notebook on this branch for commit a011cd11028279d51ef443fc1e595a034a526f8d

:point_left: Launch a binder notebook on this branch for commit c53623d52e3c0e375af8ce50349415796c228191

:point_left: Launch a binder notebook on this branch for commit b99190bc25e65cc8799e4249da0142d81a7f7554

:point_left: Launch a binder notebook on this branch for commit 0f78c1719536389e282d99fd5c99e1664e6b91f8

:point_left: Launch a binder notebook on this branch for commit 4ea09ec03af69e1ffdfa58aca6587496ba6f7e53

:point_left: Launch a binder notebook on this branch for commit 57d0147c02dc82196be08d367c072e84a958a3c2

:point_left: Launch a binder notebook on this branch for commit 709b31a0dc00f73aa3f10a3f7abe8756f54a2beb

:point_left: Launch a binder notebook on this branch for commit 05484588335887d911b8c62e158d7539ce979f5a

:point_left: Launch a binder notebook on this branch for commit 5a2d5a3c4b5276c7afc606affc5220b5053d3dec

:point_left: Launch a binder notebook on this branch for commit 41d2850fc739935d67cdabbfa061c49dc632fdb1

:point_left: Launch a binder notebook on this branch for commit beba2a4dcd5919932b95f945d1312f62aa1e0471

:point_left: Launch a binder notebook on this branch for commit b944f974f0bf7f57bae501bf421a5e0f17cc442c

:point_left: Launch a binder notebook on this branch for commit 8563a420de38e93ef6807483f22da7b06f6a1352

:point_left: Launch a binder notebook on this branch for commit abb7d477061ab82e56e9f60764840eb426cf8073

:point_left: Launch a binder notebook on this branch for commit 7dc6dc1beb2ac33a890dbd9cc682fc926c88d76d

:point_left: Launch a binder notebook on this branch for commit 21357822a900d4473ef31c426d363ef95986205b

:point_left: Launch a binder notebook on this branch for commit e11fdeeb81c7b8643526e077eeba6835a99bbeb9

:point_left: Launch a binder notebook on this branch for commit ac1d72aa8fc7cf809cee8f8f386c501bb700469d

:point_left: Launch a binder notebook on this branch for commit 7b35f1b37891e7504b6f34678dc44ffdf5dfa5c3

:point_left: Launch a binder notebook on this branch for commit 82da198b921159b7fd86c4584000d9b2aab01f88

:point_left: Launch a binder notebook on this branch for commit 6c00cf5989f75c2ffc3257cbfe4bfddf5497a256

:point_left: Launch a binder notebook on this branch for commit f071f707c0e3b964f1ca4a9c5dbe6e4d1fd51a06

:point_left: Launch a binder notebook on this branch for commit ff2a6a4d2a75304557d22564e1c57986d2e982fd

:point_left: Launch a binder notebook on this branch for commit 660185d45f27b29684e3cca42440b6b97f778efc

:point_left: Launch a binder notebook on this branch for commit 4ef52b972115f0e7d3264bdab60dddb9f4eea394

:point_left: Launch a binder notebook on this branch for commit 6c73a3d2b31af6b3f675720fc88235461ec2c114

:point_left: Launch a binder notebook on this branch for commit 21f0c903bd920e5714fb8953df4c5757ef763621

:point_left: Launch a binder notebook on this branch for commit d7d988f13402596942dcbacc346fb46ee3f8d75b

:point_left: Launch a binder notebook on this branch for commit 4ebde5682bf05bdc3d5c53c5824f9a56e2c66146

:point_left: Launch a binder notebook on this branch for commit fcd9a42cf8f7830a42a169765ea72f48c1dfb85e

:point_left: Launch a binder notebook on this branch for commit c5eef7806d6e9767fd51f17207c63ac98402f33f

:point_left: Launch a binder notebook on this branch for commit 711c568d0b31cac9aeffff424c2a07d5e542a9fa

:point_left: Launch a binder notebook on this branch for commit 3d4d983c825d8a2044064a968687135f1f6609c0

:point_left: Launch a binder notebook on this branch for commit c43d0250b9216dfa3364d8f9d8157163f50324b4

:point_left: Launch a binder notebook on this branch for commit 44e29a34a34888e92507cab34047cabd53292549

:point_left: Launch a binder notebook on this branch for commit bb5e397ecce5ee7cb62f480d868cc9fa478e7c0b

:point_left: Launch a binder notebook on this branch for commit d90be64dce8de98bec2c239ed0d58d9e3f127f03

:point_left: Launch a binder notebook on this branch for commit c4db618fa3ed917a1f7f75ea1f153e2014b42900

:point_left: Launch a binder notebook on this branch for commit 7eb2c508492b3b7a6e51fbf14146d4946175afeb

:point_left: Launch a binder notebook on this branch for commit fa6cd3a1b78eddbc0f2bd647d8ff856c31621908

:point_left: Launch a binder notebook on this branch for commit 4b20d1fa9699479dbb4022c80359ce3b912655c8

:point_left: Launch a binder notebook on this branch for commit bb2d1753636e0981a7438bcf241d31719f462161

:point_left: Launch a binder notebook on this branch for commit 3a51ed2fa6775a5ff2d16fff57c2ec9f49b86c71

:point_left: Launch a binder notebook on this branch for commit c93b9bb5234bf89719abad0ab62e56116a7ef42a

:point_left: Launch a binder notebook on this branch for commit 3a3ac30d04f3382d6493ddf95212bccfcdcf5a2a

:point_left: Launch a binder notebook on this branch for commit 1624b7ddc7bd09b8e8554fc5c9c1d54157a2aea5

:point_left: Launch a binder notebook on this branch for commit 0a7092c984c9f8497925f23c5569c1e060d536dd

:point_left: Launch a binder notebook on this branch for commit 13c7c40ae4dff8b31adec97e4960c5fa2dfafa02

:point_left: Launch a binder notebook on this branch for commit 01aa246fb714a6e22b251b9f73cd0a1b25a789d8

:point_left: Launch a binder notebook on this branch for commit b956e23c4112ef0a1e1de1c86f1e6f8882b9509a

:point_left: Launch a binder notebook on this branch for commit e2fb926bffa5ced7b5555830c42401691e0e2d8f

:point_left: Launch a binder notebook on this branch for commit 78e1f862cf64f7919cb3d7a8681a2ff5d82219d2

:point_left: Launch a binder notebook on this branch for commit b46676272b7b64ec06b71dd73012948087dcfcd3

:point_left: Launch a binder notebook on this branch for commit 6e89f4cf5974cc11e7f6ac8c96c64bd507b9eace

:point_left: Launch a binder notebook on this branch for commit 2c5c6dff25e95449f7e3b3bb8bb33995b60d5e63

:point_left: Launch a binder notebook on this branch for commit 42e8f28c5abf302a3f9362b68f9232970d22dc2c

:point_left: Launch a binder notebook on this branch for commit 9f0368b97e9facd437b7dbde66c1e03ae9a12533

:point_left: Launch a binder notebook on this branch for commit 036ff410536bf4ce1381d45418974b1e59918127

:point_left: Launch a binder notebook on this branch for commit da2459f40d8b04bd5b5347cd29fe0692da0ca6d2

:point_left: Launch a binder notebook on this branch for commit b0bb7937946777b640e4ea0d2bd0a558d72fc995

:point_left: Launch a binder notebook on this branch for commit 91ab7ae87e69efed2014fb895a11c489b45584c4

:point_left: Launch a binder notebook on this branch for commit bc144c9997f95884b11e9270295b0e470bd74ed2

:point_left: Launch a binder notebook on this branch for commit 4b9639a08e0aff2c2ab7466a971bea0e60708560

:point_left: Launch a binder notebook on this branch for commit f162d170c0ab35a4531a8973f681f399944b0d4c

:point_left: Launch a binder notebook on this branch for commit 45821a9f0f7f0f0002d7fce4cba0be4e458c434c

:point_left: Launch a binder notebook on this branch for commit 3edb33903d3bd41ab4e38de56b839b628d4be447

:point_left: Launch a binder notebook on this branch for commit 6e712d5804fc88fea1da7777ef0996e466903b93

:point_left: Launch a binder notebook on this branch for commit 722ae4d8af4895f14dec8d61c8c63d1db1198345

:point_left: Launch a binder notebook on this branch for commit b7259568fed66b7d9eb66f730db4b0cbb4c4298a

:point_left: Launch a binder notebook on this branch for commit d52e08ead6e17c181dda9f02f2f6cbca686cfa04

:point_left: Launch a binder notebook on this branch for commit 431691f30771931f248ffa7fcedebf3be1c67780

:point_left: Launch a binder notebook on this branch for commit 57fe7db50d9eb6a6c9e5d75751a61811b1ba835c

:point_left: Launch a binder notebook on this branch for commit 440c26ae9abd8333cf2d8ec4ffec08d40150eaf5

:point_left: Launch a binder notebook on this branch for commit 2dcb1ae95a7551e45ad8e2a452905e6d1170eb00

:point_left: Launch a binder notebook on this branch for commit 71de07dd227665f1c0d8ed49ce48360b23e1ac1b

:point_left: Launch a binder notebook on this branch for commit 3dd8f82790324ee3663992be6dd6559867473507

:point_left: Launch a binder notebook on this branch for commit e95a63eaa18cfeaf76d5a53239fa94b693e5dfd3

:point_left: Launch a binder notebook on this branch for commit 2d88476a97b7ed07571d8d809c86d96c426a1520

:point_left: Launch a binder notebook on this branch for commit 1858da67571d27d4783fa04c8165346d23949b4d

:point_left: Launch a binder notebook on this branch for commit 47ed4f7dfaa03dbff8ae32e41291bfbcf421984e

:point_left: Launch a binder notebook on this branch for commit e394d5dbdac278b9e8bf75141821b61a176dc92a

:point_left: Launch a binder notebook on this branch for commit 24b243b18a909ce91b44d554f0fc57f455b0ea90

:point_left: Launch a binder notebook on this branch for commit 91e24465755bead17a5c7755650413f60016c92b

:point_left: Launch a binder notebook on this branch for commit 780d98e3f6363073609cb104586528e19c73816f

:point_left: Launch a binder notebook on this branch for commit 0bd6429be7447c70dffd9dc96909aace88da8d1d

:point_left: Launch a binder notebook on this branch for commit e6f715ef2fce4d9c5dcba5f785eb81d083d330bf

:point_left: Launch a binder notebook on this branch for commit c6dd58308eaa3891119a4b962cfacb14a5d63e93

:point_left: Launch a binder notebook on this branch for commit e4e4d4f5b091f5eb6c0a7511785dd7ac939ca8f5

:point_left: Launch a binder notebook on this branch for commit 05c4f7dce6f14dc41afce696d6566a26a1bf9fc7

:point_left: Launch a binder notebook on this branch for commit 7865c22fb64c0263a5cccfc9025f0fbd08d3108e

:point_left: Launch a binder notebook on this branch for commit 7426daf3b8381873d89cacf9f51cb541d876bccc

:point_left: Launch a binder notebook on this branch for commit 3fdcb7c3a2e684dd8861ad92a573fe13c5cf277d

:point_left: Launch a binder notebook on this branch for commit 47a79ff9ab35592ca2c56e6f3ed7a36436bf6c74

[yuvipanda]

just casually passing by but i must say i absolutely love the commit messages

[consideRatio]

@yuvipanda I'll go for a merge here to trigger a rebuild with jupyterhub 4.1.4 instead of current 4.1.3 i the published images in the main branch, and then see if I can get it to function with the current xsrf complexities when booted with jupyterhub-singleuser - which it doesn't currently with jupyterhub 4.1.3 in the image at least.