serokell / deploy-rs

A simple multi-profile Nix-flake deploy tool.
Other
1.39k stars 101 forks source link

[Question] Is there a way to support `?remote-program=` in ssh url ? #298

Open 573 opened 2 days ago

573 commented 2 days ago

Related #290, would it be possible to support what https://github.com/NixOS/nix/issues/8292#issuecomment-1536230370 suggests, via ssh_opts probably ?

This way non-NixOS targets could more easily benefit of https://github.com/serokell/deploy-rs, i. e. :

$ nix copy --substitute-on-destination --no-check-sigs --to ssh://nix-on-droid@smartphone\?remote-program=/etc/profiles/per-user/nix-on-droid/bin/nix-store /nix/store/yq15m37688hzfpmqgmhppn0wp4ppvygr-activatable-neofetch-unstable-2021-12-10
debug1: auto-mux: Trying existing master at '/home/nixos/.ssh/socket-nix-on-droid@nn.nn.nn.nn-nn'
debug2: fd 3 setting O_NONBLOCK
debug2: mux_client_hello_exchange: master version 4
debug3: mux_client_forwards: request forwardings: 0 local, 0 remote
debug3: mux_client_request_session: entering
debug3: mux_client_request_alive: entering
debug3: mux_client_request_alive: done pid = 20718
debug3: mux_client_request_session: session request sent
debug1: mux_client_request_session: master session id: 3
copying path '/nix/store/r1ma6z6i44yj3lwv5kc29wsrgylml3j1-neofetch-unstable-2021-12-10-man' from 'https://cache.nixos.org'...
copying path '/nix/store/x66almqzcwwywmsyhvkx7vcwpgyz02lc-giflib-5.2.2' from 'https://cache.nixos.org'...
copying path '/nix/store/79grpcy67lpyz36a9q6m3dpa0waj7xpn-lerc-4.0.0' from 'https://cache.nixos.org'...
copying path '/nix/store/c27whjwb134h1ldmczxfjyig46y0dlg6-libimagequant-4.3.0' from 'https://cache.nixos.org'...
copying path '/nix/store/db07vavvjbc0n2c1qw22g7zy771idcdr-lcms2-2.16' from 'https://cache.nixos.org'...
copying path '/nix/store/zspkk5cva1sddii4k5a7mppa2lk8qg0n-python3.11-defusedxml-0.7.1' from 'https://cache.nixos.org'...
copying path '/nix/store/81rhymr34gwz87xbmagxn6988k8p2p3l-python3.11-olefile-0.47' from 'https://cache.nixos.org'...
copying path '/nix/store/x5m8anfh367mj19m2jj74cicmmjw6h1p-python3.11-psutil-5.9.8' from 'https://cache.nixos.org'...
copying path '/nix/store/5qw8s8sp8mv7qgz7q2pd7l4y0yrpfa50-libdeflate-1.20' from 'https://cache.nixos.org'...
copying path '/nix/store/li8dw789c5piy8xkm40735p475dbm21c-python3.11-six-1.16.0' from 'https://cache.nixos.org'...
copying path '/nix/store/55mn7bzrzzigjbffi6brm08k34z35ssk-python3.11-attrs-23.2.0' from 'https://cache.nixos.org'...
copying path '/nix/store/zzscwsgzryppyb2yybv8hnwx93jp406g-pciutils-3.12.0' from 'https://cache.nixos.org'...
copying path '/nix/store/bn06khc46v7bzw4czq1kd9djmy1cqk2r-python3.11-docopt-0.6.2' from 'https://cache.nixos.org'...
copying path '/nix/store/6ib3fxmjyh5jjv63aw694pj3j23s4n6j-python3.11-xlib-0.33' from 'https://cache.nixos.org'...
copying path '/nix/store/2hvgk61j3rrr968r3nkbdy1fhwjd3las-libtiff-4.6.0' from 'https://cache.nixos.org'...
copying path '/nix/store/j52s6fdiv6iqgd5zdwzyckz1vciqskjb-openjpeg-2.5.2' from 'https://cache.nixos.org'...
copying path '/nix/store/x6ghh5kl26hcdyiigjh06wbymsv7dxad-libwebp-1.4.0' from 'https://cache.nixos.org'...
copying path '/nix/store/ix26ydhww3gnnjxp7yxknqkhzzw80sl0-python3.11-pillow-10.3.0' from 'https://cache.nixos.org'...
copying path '/nix/store/7r5llhz5ri066b1ifp4a9s76jw7h6b8l-python3.11-ueberzug-18.1.9' from 'https://cache.nixos.org'...
copying path '/nix/store/hijxfyd0fbzf1h2q5vlcgs8wklkwfslv-neofetch-unstable-2021-12-10' from 'https://cache.nixos.org'...

vs. (without PATH set to /etc/profiles/per-user/nix-on-droid/bin/ for ssh, redacted log bits)

$ nix shell nixpkgs#deploy-rs --command deploy -s .#sams9__x86_64-linux -- --impure
🚀 ℹ️ [deploy] [INFO] The following profiles are going to be deployed:
[sams9__x86_64-linux.system]
user = "nix-on-droid"
ssh_user = "nix-on-droid"
path = "/nix/store/yq15m37688hzfpmqgmhppn0wp4ppvygr-activatable-neofetch-unstable-2021-12-10"
hostname = "smartphone"
ssh_opts = ["-p", "8022", "-vvv"]

🚀 ℹ️ [deploy] [INFO] Building profile `system` for node `sams9__x86_64-linux`
🚀 ℹ️ [deploy] [INFO] Copying profile `system` to node `sams9__x86_64-linux`
OpenSSH_9.7p1, OpenSSL 3.0.14 4 Jun 2024
debug1: Reading configuration data /home/nixos/.ssh/config
debug3: /home/nixos/.ssh/config line 1: Including file /home/nixos/.ssh/config.d/morehosts depth 0
debug1: Reading configuration data /home/nixos/.ssh/config.d/morehosts
debug1: /home/nixos/.ssh/config.d/morehosts line 15: Applying options for smartphone
debug1: /home/nixos/.ssh/config line 4: Applying options for *
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 5: Applying options for *
debug2: resolve_canonicalize: hostname nn.nn.nn.nn is address
debug1: multiplexing control connection
debug2: fd 5 setting O_NONBLOCK
debug3: fd 5 is O_NONBLOCK
debug1: channel 1: new mux-control [mux-control] (inactive timeout: 0)
debug3: channel_post_mux_listener: new mux channel 1 fd 5
debug3: mux_master_read_cb: channel 1: hello sent
debug2: set_control_persist_exit_time: cancel scheduled exit
debug3: mux_master_read_cb: channel 1 packet type 0x00000001 len 4
debug2: mux_master_process_hello: channel 1 client version 4
debug3: mux_master_read_cb: channel 1 packet type 0x10000004 len 4
debug2: mux_master_process_alive_check: channel 1: alive check
debug1: auto-mux: Trying existing master at '/home/nixos/.ssh/socket-nix-on-droid@nn.nn.nn.nn-8022'
debug2: fd 3 setting O_NONBLOCK
debug1: multiplexing control connection
debug2: fd 6 setting O_NONBLOCK
debug3: fd 6 is O_NONBLOCK
debug1: channel 2: new mux-control [mux-control] (inactive timeout: 0)
debug3: channel_post_mux_listener: new mux channel 2 fd 6
bash: line 1: nix-store: command not found
debug2: set_control_persist_exit_time: schedule exit in 600 seconds
debug3: mux_client_read_packet_timeout: read header failed: Broken pipe
debug2: Received exit status from master 127
error: cannot connect to 'nix-on-droid@smartphone'
🚀 ❌ [deploy] [ERROR] Failed to push profile to node sams9__x86_64-linux: sams9__x86_64-linux
573 commented 2 days ago

-

573 commented 23 hours ago

With ~/.ssh/environment

PATH=/data/data/com.termux.nix/files/usr/etc/profiles/per-user/nix-on-droid/bin:"$PATH"

and PermitUserEnvironment yes in sshd_config both on deploy target I get thus far that a nix-store --serve --write process is spawned there by nix shell nixpkgs#deploy-rs --command deploy -s .#sams9__x86_64-linux -- --impure. Other than that nothing more is happening:

$ nix shell nixpkgs#deploy-rs --command deploy -s .#sams9__x86_64-linux -- --impure -L
🚀 ℹ️ [deploy] [INFO] Evaluating flake in .
🚀 ℹ️ [deploy] [INFO] The following profiles are going to be deployed:
[sams9__x86_64-linux.system]
user = "nix-on-droid"
ssh_user = "nix-on-droid"
path = "/nix/store/fy1j25k13vckdk5qpr9xap8akawwdnna-activatable-tealdeer-1.6.1"
hostname = "smartphone"
ssh_opts = ["-v", "-oControlMaster=no"]

🚀 ℹ️ [deploy] [INFO] Building profile `system` for node `sams9__x86_64-linux`
activatable-tealdeer> created 4 symlinks in user environment
🚀 ℹ️ [deploy] [INFO] Copying profile `system` to node `sams9__x86_64-linux`
OpenSSH_9.7p1, OpenSSL 3.0.14 4 Jun 2024
debug1: Reading configuration data /home/nixos/.ssh/config
debug1: Reading configuration data /home/nixos/.ssh/config.d/morehosts
debug1: /home/nixos/.ssh/config.d/morehosts line 15: Applying options for smartphone
debug1: /home/nixos/.ssh/config line 4: Applying options for *
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 5: Applying options for *
debug1: Control socket "/home/nixos/.ssh/socket-nix-on-droid@nn.nn.nn.nn-8022" does not exist
debug1: Connecting to nn.nn.nn.nn [nn.nn.nn.nn] port 8022.
debug1: fd 3 clearing O_NONBLOCK
debug1: Connection established.
debug1: identity file /home/nixos/.ssh/id_ed25519 type 3
debug1: identity file /home/nixos/.ssh/id_ed25519-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_9.7
debug1: Remote protocol version 2.0, remote software version OpenSSH_9.7
debug1: compat_banner: match: OpenSSH_9.7 pat OpenSSH* compat 0x04000000
debug1: Authenticating to nn.nn.nn.nn:8022 as 'nix-on-droid'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: sntrup761x25519-sha512@openssh.com
debug1: kex: host key algorithm: ssh-ed25519
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: zlib@openssh.com
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: zlib@openssh.com
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: SSH2_MSG_KEX_ECDH_REPLY received
debug1: Server host key: ssh-ed25519 SHA256:XpLkjWVrFMqfJPKqLEgpwFBgUTVb/a3zyL9czQPMJS8
debug1: checking without port identifier
Warning: Permanently added '[nn.nn.nn.nn]:8022' (ED25519) to the list of known hosts.
debug1: ssh_packet_send2_wrapped: resetting send seqnr 3
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: Sending SSH2_MSG_EXT_INFO
debug1: expecting SSH2_MSG_NEWKEYS
debug1: ssh_packet_read_poll2: resetting read seqnr 3
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey in after 134217728 blocks
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_ext_info_client_parse: server-sig-algs=<ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ssh-ed25519@openssh.com,sk-ecdsa-sha2-nistp256@openssh.com,rsa-sha2-512,rsa-sha2-256>
debug1: kex_ext_info_check_ver: publickey-hostbound@openssh.com=<0>
debug1: kex_ext_info_check_ver: ping@openssh.com=<0>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_ext_info_client_parse: server-sig-algs=<ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ssh-ed25519@openssh.com,sk-ecdsa-sha2-nistp256@openssh.com,rsa-sha2-512,rsa-sha2-256>
debug1: Authentications that can continue: publickey,password,keyboard-interactive
debug1: Next authentication method: publickey
debug1: get_agent_identities: bound agent to hostkey
debug1: get_agent_identities: ssh_fetch_identitylist: agent contains no identities
debug1: Will attempt key: /home/nixos/.ssh/id_ed25519 ED25519 SHA256:UWqCm7Zsk2o+NJEpFp/0Z9mVzGUEXygGe79EGxa2kLg explicit
debug1: Offering public key: /home/nixos/.ssh/id_ed25519 ED25519 SHA256:UWqCm7Zsk2o+NJEpFp/0Z9mVzGUEXygGe79EGxa2kLg explicit
debug1: Server accepts key: /home/nixos/.ssh/id_ed25519 ED25519 SHA256:UWqCm7Zsk2o+NJEpFp/0Z9mVzGUEXygGe79EGxa2kLg explicit
debug1: Enabling compression at level 6.
Authenticated to nn.nn.nn.nn ([nn.nn.nn.nn]:8022) using "publickey".
debug1: channel 0: new session [client-session] (inactive timeout: 0)
debug1: Requesting no-more-sessions@openssh.com
Couldn't execute bash -c "echo started": No such file or directory
debug1: Entering interactive session.
debug1: pledge: exec
debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0
debug1: Remote: /data/data/com.termux.nix/files/home/.ssh/authorized_keys:2: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding
debug1: Remote: /data/data/com.termux.nix/files/home/.ssh/authorized_keys:2: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding
debug1: Sending environment.
debug1: channel 0: setting env LANG = "en_US.UTF-8"
debug1: channel 0: setting env LC_ADDRESS = "en_US.UTF-8"
debug1: channel 0: setting env LC_MEASUREMENT = "de_DE.UTF-8"
debug1: channel 0: setting env LC_MONETARY = "en_US.UTF-8"
debug1: channel 0: setting env LC_NAME = "en_US.UTF-8"
debug1: channel 0: setting env LC_PAPER = "de_DE.UTF-8"
debug1: channel 0: setting env LC_TELEPHONE = "en_US.UTF-8"
debug1: Sending command: nix-store --serve --write
573 commented 22 hours ago

Repeating deploy states:

$ nix shell nixpkgs#deploy-rs --command deploy -s .#sams9__x86_64-linux -- --impure
🚀 ℹ️ [deploy] [INFO] Evaluating flake in .
🚀 ℹ️ [deploy] [INFO] The following profiles are going to be deployed:
[sams9__x86_64-linux.system]
user = "nix-on-droid"
ssh_user = "nix-on-droid"
path = "/nix/store/0bzhbry0rjf10c0mw3pw532ncfrxp73z-activatable-ipfetch-unstable-2022-03-24"
hostname = "smartphone"
ssh_opts = ["-v"]
🚀 ℹ️ [deploy] [INFO] Building profile `system` for node `sams9__x86_64-linux`
🚀 ℹ️ [deploy] [INFO] Copying profile `system` to node `sams9__x86_64-linux`
OpenSSH_9.7p1, OpenSSL 3.0.14 4 Jun 2024
debug1: Reading configuration data /home/nixos/.ssh/config
debug1: Reading configuration data /home/nixos/.ssh/config.d/morehosts
debug1: /home/nixos/.ssh/config.d/morehosts line 15: Applying options for smartphone
debug1: /home/nixos/.ssh/config line 4: Applying options for *
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 5: Applying options for *
debug1: auto-mux: Trying existing master at '/home/nixos/.ssh/socket-nix-on-droid@nn.nn.nn.nn-8022'
debug1: multiplexing control connection
debug1: channel 1: new mux-control [mux-control] (inactive timeout: 0)
debug1: multiplexing control connection
debug1: channel 2: new mux-control [mux-control] (inactive timeout: 0)
debug1: channel 3: new session [client-session] (inactive timeout: 0)
debug1: channel 2: free: mux-control, nchannels 4
  #1 mux-control (t16 [mux-control] r3 i0/0 o0/0 e[closed]/0 fd 5/5/-1 sock 5 cc -1 io 0x00/0x00)
  #2 mux-control (t16 [mux-control] nr0 i3/0 o3/0 e[closed]/0 fd 6/6/-1 sock 6 cc -1 io 0x00/0x00)
  #3 client-session (t3 [session] nr0 i0/0 o0/0 e[write]/0 fd 7/8/9 sock -1 cc 1 io 0x00/0x00)
debug1: Sending environment.
debug1: channel 3: setting env LANG = "en_US.UTF-8"
debug1: channel 3: setting env LC_ADDRESS = "en_US.UTF-8"
debug1: channel 3: setting env LC_MEASUREMENT = "de_DE.UTF-8"
debug1: channel 3: setting env LC_MONETARY = "en_US.UTF-8"
debug1: channel 3: setting env LC_NAME = "en_US.UTF-8"
debug1: channel 3: setting env LC_PAPER = "de_DE.UTF-8"
debug1: channel 3: setting env LC_TELEPHONE = "en_US.UTF-8"
debug1: Sending command: nix-store --serve --write
debug1: mux_client_request_session: master session id: 3
debug1: channel 1: free: mux-control, nchannels 3
  #1 mux-control (t16 [mux-control] nr0 i3/0 o3/0 e[closed]/0 fd 5/5/-1 sock 5 cc -1 io 0x00/0x00)
  #3 client-session (t4 [session] r0 i3/0 o3/0 e[write]/0 fd -1/-1/9 sock -1 cc -1 io 0x00/0x00)
🚀 ℹ️ [deploy] [INFO] Activating profile `system` for node `sams9__x86_64-linux`
🚀 ℹ️ [deploy] [INFO] Creating activation waiter
OpenSSH_9.7p1, OpenSSL 3.0.14 4 Jun 2024
debug1: Reading configuration data /home/nixos/.ssh/config
debug1: Reading configuration data /home/nixos/.ssh/config.d/morehosts
debug1: /home/nixos/.ssh/config.d/morehosts line 15: Applying options for smartphone
debug1: /home/nixos/.ssh/config line 4: Applying options for *
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 5: Applying options for *
debug1: auto-mux: Trying existing master at '/home/nixos/.ssh/socket-nix-on-droid@nn.nn.nn.nn-8022'
debug1: multiplexing control connection
debug1: channel 1: new mux-control [mux-control] (inactive timeout: 0)
debug1: channel 2: new session [client-session] (inactive timeout: 0)
OpenSSH_9.7p1, OpenSSL 3.0.14 4 Jun 2024
debug1: Reading configuration data /home/nixos/.ssh/config
debug1: Reading configuration data /home/nixos/.ssh/config.d/morehosts
debug1: /home/nixos/.ssh/config.d/morehosts line 15: Applying options for smartphone
debug1: /home/nixos/.ssh/config line 4: Applying options for *
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 5: Applying options for *
debug1: auto-mux: Trying existing master at '/home/nixos/.ssh/socket-nix-on-droid@nn.nn.nn.nn-8022'
debug1: multiplexing control connection
debug1: channel 4: new mux-control [mux-control] (inactive timeout: 0)
debug1: channel 5: new session [client-session] (inactive timeout: 0)
debug1: Sending environment.
debug1: channel 2: setting env LC_ADDRESS = "en_US.UTF-8"
debug1: channel 2: setting env LC_NAME = "en_US.UTF-8"
debug1: channel 2: setting env LC_MONETARY = "en_US.UTF-8"
debug1: channel 2: setting env LC_PAPER = "de_DE.UTF-8"
debug1: channel 2: setting env LANG = "en_US.UTF-8"
debug1: channel 2: setting env LC_TELEPHONE = "en_US.UTF-8"
debug1: channel 2: setting env LC_MEASUREMENT = "de_DE.UTF-8"
debug1: Sending command: /nix/store/0bzhbry0rjf10c0mw3pw532ncfrxp73z-activatable-ipfetch-unstable-2022-03-24/activate-rs wait '/nix/store/0bzhbry0rjf10c0mw3pw532ncfrxp73z-activatable-ipfetch-unstable-2022-03-24' --temp-path '/tmp'
debug1: Sending environment.
debug1: channel 5: setting env LC_ADDRESS = "en_US.UTF-8"
debug1: channel 5: setting env LC_NAME = "en_US.UTF-8"
debug1: channel 5: setting env LC_MONETARY = "en_US.UTF-8"
debug1: channel 5: setting env LC_PAPER = "de_DE.UTF-8"
debug1: channel 5: setting env LANG = "en_US.UTF-8"
debug1: channel 5: setting env LC_TELEPHONE = "en_US.UTF-8"
debug1: channel 5: setting env LC_MEASUREMENT = "de_DE.UTF-8"
debug1: Sending command: /nix/store/0bzhbry0rjf10c0mw3pw532ncfrxp73z-activatable-ipfetch-unstable-2022-03-24/activate-rs activate '/nix/store/0bzhbry0rjf10c0mw3pw532ncfrxp73z-activatable-ipfetch-unstable-2022-03-24' --profile-user nix-on-droid --profile-name system --temp-path '/tmp' --confirm-timeout 30 --magic-rollback --auto-rollback
debug1: mux_client_request_session: master session id: 2
debug1: mux_client_request_session: master session id: 5
debug1: client_input_channel_req: channel 3 rtype exit-status reply 0
debug1: client_input_channel_req: no sink for exit-status on channel 3
debug1: channel 3: free: client-session, nchannels 6
  #1 mux-control (t16 [mux-control] r2 i0/0 o0/0 e[closed]/0 fd 5/5/-1 sock 5 cc -1 io 0x01/0x00)
  #2 client-session (t4 [session] r1 i0/0 o0/0 e[write]/0 fd 6/7/8 sock -1 cc 1 io 0x00/0x00)
  #3 client-session (t4 [session] r0 i3/0 o3/0 e[write]/0 fd -1/-1/9 sock -1 cc -1 io 0x00/0x00)
  #4 mux-control (t16 [mux-control] r5 i0/0 o0/0 e[closed]/0 fd 10/10/-1 sock 10 cc -1 io 0x00/0x00)
  #5 client-session (t4 [session] r2 i0/0 o0/0 e[write]/0 fd 11/12/13 sock -1 cc 4 io 0x00/0x00)
debug1: client_input_channel_req: channel 2 rtype exit-signal reply 0
debug1: channel 2: free: client-session, nchannels 5
  #1 mux-control (t16 [mux-control] nr0 i3/0 o1/0 e[closed]/0 fd 5/5/-1 sock 5 cc -1 io 0x01/0x00)
  #2 client-session (t4 [session] r1 i3/0 o3/0 e[write]/0 fd -1/-1/8 sock -1 cc -1 io 0x00/0x00)
  #4 mux-control (t16 [mux-control] r5 i0/0 o0/0 e[closed]/0 fd 10/10/-1 sock 10 cc -1 io 0x00/0x00)
  #5 client-session (t4 [session] r2 i3/0 o0/0 e[write]/0 fd -1/12/13 sock -1 cc 4 io 0x00/0x00)
debug1: channel 1: free: mux-control, nchannels 4
  #1 mux-control (t16 [mux-control] nr0 i3/0 o3/0 e[closed]/0 fd 5/5/-1 sock 5 cc -1 io 0x00/0x00)
  #4 mux-control (t16 [mux-control] r5 i0/0 o0/0 e[closed]/0 fd 10/10/-1 sock 10 cc -1 io 0x00/0x00)
  #5 client-session (t4 [session] r2 i3/0 o0/0 e[write]/0 fd -1/12/13 sock -1 cc 4 io 0x00/0x00)
🚀 ❌ [deploy] [ERROR] Waiting over SSH resulted in a bad exit code: Some(255)
🚀 ℹ️ [deploy] [INFO] Revoking previous deploys
🚀 ❌ [deploy] [ERROR] Deployment to node sams9__x86_64-linux failed, rolled back to previous generation
573 commented 5 hours ago

Related #226