Tasks No Longer Run

[Ashkaan]

I've had a great and working Semaphore docker for some time now. Suddenly, it hangs here when I run any task:

If I hit Stop, nothing happens. If I hit Force Stop, it sets it to a Stopped status. I don't have any tasks running. Is there a way clear the queue, or something?

I'm running this on docker using the default Bolt DB and I can access the CLI if needed.

[Ashkaan]

Good news: I tested a few versions, and the last version that was working is: semaphoreui/semaphore:v2.9.18

I tried v.2.9.21, v.2.9.23, whatever latest points to, and whatever develop points to, and none of those work.

[athomasse]

i'm experiencing the same issue. After a rollback to version v2.9.18 everything works again.

[Goldenflamer]

https://github.com/ansible-semaphore/semaphore/issues/1492

[Goldenflamer]

It seems that all versions above 2.9.4 are NOT considered as stable but as pre-releases. Not sure why there is no stable docker tag.

[Yuri-Lima]

Same issue: #1485

Consider to use v2.9.18

[fiftin]

Fix will be soon.

[fiftin]

Please check latest pre-release (latest docker image).

[Ashkaan]

Did not work for me. New issue: SSH key not working. I put it in and overwrote, now jobs are stuck on "Running".

[fiftin]

@Ashkaan need logs.

[fiftin]

@Ashkaan are you have custom config.json or passing settings via environment vars?

[Ashkaan]

I had to revert for production.

[fiftin]

Why I asked.

Container writes config.json only if it doesn't exist. So, if you stored config.json in volume and recreate container with different env vars, it will not affect to config.json stored in volume.

Currently this behaviour changed because we maps env vars to config fields. Env vars have priority over config.json.

[Ashkaan]

Oh interesting. Yes, that might be the case. The config.json is mapped externally.

[fiftin]

@Ashkaan thank you very much! I will think what to do... I didn't expect these to be breaking changes.

[fiftin]

@Ashkaan could you remove environment variables from docker image if you use mapped config.json?

[Ashkaan]

It doesn't load at all without env vars.

[fiftin]

@Ashkaan got it, thank you.

Please try latest pre-release: https://github.com/ansible-semaphore/semaphore/releases/tag/v2.9.36

[Ashkaan]

It just hangs. I tried to go back to 18, and it wouldn't run also, so I had to restore the database from backup.

[fiftin]

Hi @Ashkaan Cloud you run job in DEBUG mode?

[adaichendt-tv1]

I am facing the same problem, I think. It seems to be related to the ssh control path.

If I manually enter the command it is stuck on in the console, and delete the ControlPath, it works.

2.9.39-beta

``` TASK [Gathering Facts] ********************************************************* 5:07:44 PM task path: /var/lib/semaphore/tmp/repository_1_8/playbooks/netbox-agent/netbox-prod.yaml:2 5:07:44 PM <10.1.90.118> ESTABLISH SSH CONNECTION FOR USER: ansible 5:07:44 PM <10.1.90.118> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="ansible"' -o ConnectTimeout=10 -o ControlPath=/var/lib/semaphore/tmp/.ansible/cp/06174fc892 10.1.90.118 '/bin/sh -c '"'"'echo ~ansible && sleep 0'"'"'' 5:07:45 PM <10.1.90.118> (0, b'/home/ansible ', b'OpenSSH_8.4p1 Debian-5+deb11u1, OpenSSL 1.1.1n 15 Mar 2022\r debug1: Reading configuration data /etc/ssh/ssh_config\r debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files\r debug1: /etc/ssh/ssh_config line 21: Applying options for *\r debug2: resolve_canonicalize: hostname 10.1.90.118 is address\r debug3: expanded UserKnownHostsFile \'~/.ssh/known_hosts\' -> \'/var/lib/semaphore/.ssh/known_hosts\'\r debug3: expanded UserKnownHostsFile \'~/.ssh/known_hosts2\' -> \'/var/lib/semaphore/.ssh/known_hosts2\'\r debug1: auto-mux: Trying existing master\r debug1: Control socket "/var/lib/semaphore/tmp/.ansible/cp/06174fc892" does not exist\r debug2: ssh_connect_direct\r debug1: Connecting to 10.1.90.118 [10.1.90.118] port 22.\r debug2: fd 3 setting O_NONBLOCK\r debug1: fd 3 clearing O_NONBLOCK\r debug1: Connection established.\r debug3: timeout: 9976 ms remain after connect\r debug1: identity file /var/lib/semaphore/.ssh/id_rsa type -1\r debug1: identity file /var/lib/semaphore/.ssh/id_rsa-cert type -1\r debug1: identity file /var/lib/semaphore/.ssh/id_dsa type -1\r debug1: identity file /var/lib/semaphore/.ssh/id_dsa-cert type -1\r debug1: identity file /var/lib/semaphore/.ssh/id_ecdsa type -1\r debug1: identity file /var/lib/semaphore/.ssh/id_ecdsa-cert type -1\r debug1: identity file /var/lib/semaphore/.ssh/id_ecdsa_sk type -1\r debug1: identity file /var/lib/semaphore/.ssh/id_ecdsa_sk-cert type -1\r debug1: identity file /var/lib/semaphore/.ssh/id_ed25519 type -1\r debug1: identity file /var/lib/semaphore/.ssh/id_ed25519-cert type -1\r debug1: identity file /var/lib/semaphore/.ssh/id_ed25519_sk type -1\r debug1: identity file /var/lib/semaphore/.ssh/id_ed25519_sk-cert type -1\r debug1: identity file /var/lib/semaphore/.ssh/id_xmss type -1\r debug1: identity file /var/lib/semaphore/.ssh/id_xmss-cert type -1\r debug1: Local version string SSH-2.0-OpenSSH_8.4p1 Debian-5+deb11u1\r debug1: Remote protocol version 2.0, remote software version OpenSSH_8.9p1 Ubuntu-3ubuntu0.3\r debug1: match: OpenSSH_8.9p1 Ubuntu-3ubuntu0.3 pat OpenSSH* compat 0x04000000\r debug2: fd 3 setting O_NONBLOCK\r debug1: Authenticating to 10.1.90.118:22 as \'ansible\'\r debug3: hostkeys_foreach: reading file "/var/lib/semaphore/.ssh/known_hosts"\r debug3: record_hostkey: found key type ECDSA in file /var/lib/semaphore/.ssh/known_hosts:21\r debug3: load_hostkeys: loaded 1 keys from 10.1.90.118\r debug3: order_hostkeyalgs: have matching best-preference key type ecdsa-sha2-nistp256-cert-v01@openssh.com, using HostkeyAlgorithms verbatim\r debug3: send packet: type 20\r debug1: SSH2_MSG_KEXINIT sent\r debug3: receive packet: type 20\r debug1: SSH2_MSG_KEXINIT received\r debug2: local client KEXINIT proposal\r debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,ext-info-c\r debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,sk-ecdsa-sha2-nistp256-cert-v01@openssh.com,ssh-ed25519-cert-v01@openssh.com,sk-ssh-ed25519-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ecdsa-sha2-nistp256@openssh.com,ssh-ed25519,sk-ssh-ed25519@openssh.com,rsa-sha2-512,rsa-sha2-256,ssh-rsa\r debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com\r debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com\r debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1\r debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1\r debug2: compression ctos: zlib@openssh.com,zlib,none\r debug2: compression stoc: zlib@openssh.com,zlib,none\r debug2: languages ctos: \r debug2: languages stoc: \r debug2: first_kex_follows 0 \r debug2: reserved 0 \r debug2: peer server KEXINIT proposal\r debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,sntrup761x25519-sha512@openssh.com,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256\r debug2: host key algorithms: rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519\r debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com\r debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com\r debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1\r debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1\r debug2: compression ctos: none,zlib@openssh.com\r debug2: compression stoc: none,zlib@openssh.com\r debug2: languages ctos: \r debug2: languages stoc: \r debug2: first_kex_follows 0 \r debug2: reserved 0 \r debug1: kex: algorithm: curve25519-sha256\r debug1: kex: host key algorithm: ecdsa-sha2-nistp256\r debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: compression: zlib@openssh.com\r debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: compression: zlib@openssh.com\r debug3: send packet: type 30\r debug1: expecting SSH2_MSG_KEX_ECDH_REPLY\r debug3: receive packet: type 31\r debug1: Server host key: ecdsa-sha2-nistp256 SHA256:fQ+zqbLm2yy3oXbA3nb6K3Fr/CEoWEaYWasKrnvd5MI\r debug3: hostkeys_foreach: reading file "/var/lib/semaphore/.ssh/known_hosts"\r debug3: record_hostkey: found key type ECDSA in file /var/lib/semaphore/.ssh/known_hosts:21\r debug3: load_hostkeys: loaded 1 keys from 10.1.90.118\r debug1: Host \'10.1.90.118\' is known and matches the ECDSA host key.\r debug1: Found key in /var/lib/semaphore/.ssh/known_hosts:21\r debug3: send packet: type 21\r debug2: set_newkeys: mode 1\r debug1: rekey out after 134217728 blocks\r debug1: SSH2_MSG_NEWKEYS sent\r debug1: expecting SSH2_MSG_NEWKEYS\r debug3: receive packet: type 21\r debug1: SSH2_MSG_NEWKEYS received\r debug2: set_newkeys: mode 0\r debug1: rekey in after 134217728 blocks\r debug1: Will attempt key: RSA SHA256:TVCa0RIxoPXjPcPMISoxT/7CFuGILYKEdArGUOu/Avs agent\r debug1: Will attempt key: /var/lib/semaphore/.ssh/id_rsa \r debug1: Will attempt key: /var/lib/semaphore/.ssh/id_dsa \r debug1: Will attempt key: /var/lib/semaphore/.ssh/id_ecdsa \r debug1: Will attempt key: /var/lib/semaphore/.ssh/id_ecdsa_sk \r debug1: Will attempt key: /var/lib/semaphore/.ssh/id_ed25519 \r debug1: Will attempt key: /var/lib/semaphore/.ssh/id_ed25519_sk \r debug1: Will attempt key: /var/lib/semaphore/.ssh/id_xmss \r debug2: pubkey_prepare: done\r debug3: send packet: type 5\r debug3: receive packet: type 7\r debug1: SSH2_MSG_EXT_INFO received\r debug1: kex_input_ext_info: server-sig-algs=\r debug1: kex_input_ext_info: publickey-hostbound@openssh.com (unrecognised)\r debug3: receive packet: type 6\r debug2: service_accept: ssh-userauth\r debug1: SSH2_MSG_SERVICE_ACCEPT received\r debug3: send packet: type 50\r debug3: receive packet: type 51\r debug1: Authentications that can continue: publickey,password\r debug3: start over, passed a different list publickey,password\r debug3: preferred gssapi-with-mic,gssapi-keyex,hostbased,publickey\r debug3: authmethod_lookup publickey\r debug3: remaining preferred: ,gssapi-keyex,hostbased,publickey\r debug3: authmethod_is_enabled publickey\r debug1: Next authentication method: publickey\r debug1: Offering public key: RSA SHA256:TVCa0RIxoPXjPcPMISoxT/7CFuGILYKEdArGUOu/Avs agent\r debug3: send packet: type 50\r debug2: we sent a publickey packet, wait for reply\r debug3: receive packet: type 60\r debug1: Server accepts key: RSA SHA256:TVCa0RIxoPXjPcPMISoxT/7CFuGILYKEdArGUOu/Avs agent\r debug3: sign_and_send_pubkey: RSA SHA256:TVCa0RIxoPXjPcPMISoxT/7CFuGILYKEdArGUOu/Avs\r debug3: sign_and_send_pubkey: signing using rsa-sha2-512 SHA256:TVCa0RIxoPXjPcPMISoxT/7CFuGILYKEdArGUOu/Avs\r debug3: send packet: type 50\r debug3: receive packet: type 52\r debug1: Enabling compression at level 6.\r debug1: Authentication succeeded (publickey).\r Authenticated to 10.1.90.118 ([10.1.90.118]:22).\r debug1: setting up multiplex master socket\r debug3: muxserver_listen: temporary control path /var/lib/semaphore/tmp/.ansible/cp/06174fc892.6J0AgcQ00icYVocV\r debug2: fd 4 setting O_NONBLOCK\r debug3: fd 4 is O_NONBLOCK\r debug3: fd 4 is O_NONBLOCK\r debug1: channel 0: new [/var/lib/semaphore/tmp/.ansible/cp/06174fc892]\r debug3: muxserver_listen: mux listener channel 0 fd 4\r debug2: fd 3 setting TCP_NODELAY\r debug3: ssh_packet_set_tos: set IP_TOS 0x08\r debug1: control_persist_detach: backgrounding master process\r debug2: control_persist_detach: background process is 628532\r debug2: fd 4 setting O_NONBLOCK\r debug1: forking to background\r debug1: Entering interactive session.\r debug1: pledge: id\r debug2: set_control_persist_exit_time: schedule exit in 60 seconds\r debug1: multiplexing control connection\r debug2: fd 5 setting O_NONBLOCK\r debug3: fd 5 is O_NONBLOCK\r debug1: channel 1: new [mux-control]\r debug3: channel_post_mux_listener: new mux channel 1 fd 5\r debug3: mux_master_read_cb: channel 1: hello sent\r debug2: set_control_persist_exit_time: cancel scheduled exit\r debug3: mux_master_read_cb: channel 1 packet type 0x00000001 len 4\r debug2: mux_master_process_hello: channel 1 client version 4\r debug2: mux_client_hello_exchange: master version 4\r debug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r debug3: mux_client_request_session: entering\r debug3: mux_client_request_alive: entering\r debug3: mux_master_read_cb: channel 1 packet type 0x10000004 len 4\r debug2: mux_master_process_alive_check: channel 1: alive check\r debug3: mux_client_request_alive: done pid = 628534\r debug3: mux_client_request_session: session request sent\r debug3: mux_master_read_cb: channel 1 packet type 0x10000002 len 89\r debug2: mux_master_process_new_session: channel 1: request tty 0, X 0, agent 0, subsys 0, term "", cmd "/bin/sh -c \'echo ~ansible && sleep 0\'", env 1\r debug3: mux_master_process_new_session: got fds stdin 6, stdout 7, stderr 8\r debug2: fd 7 setting O_NONBLOCK\r debug2: fd 8 setting O_NONBLOCK\r debug1: channel 2: new [client-session]\r debug2: mux_master_process_new_session: channel_new: 2 linked to control channel 1\r debug2: channel 2: send open\r debug3: send packet: type 90\r debug3: receive packet: type 80\r debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0\r debug3: receive packet: type 4\r debug1: Remote: /home/ansible/.ssh/authorized_keys:1: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding\r debug3: receive packet: type 4\r debug1: Remote: /home/ansible/.ssh/authorized_keys:1: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding\r debug3: receive packet: type 91\r debug2: channel_input_open_confirmation: channel 2: callback start\r debug2: client_session2_setup: id 2\r debug1: Sending environment.\r debug1: Sending env LANG = C.UTF-8\r debug2: channel 2: request env confirm 0\r debug3: send packet: type 98\r debug1: Sending command: /bin/sh -c \'echo ~ansible && sleep 0\'\r debug2: channel 2: request exec confirm 1\r debug3: send packet: type 98\r debug3: mux_session_confirm: sending success reply\r debug2: channel_input_open_confirmation: channel 2: callback done\r debug2: channel 2: open confirm rwindow 0 rmax 32768\r debug1: mux_client_request_session: master session id: 2\r debug2: channel 2: rcvd adjust 2097152\r debug3: receive packet: type 99\r debug2: channel_input_status_confirm: type 99 id 2\r debug2: exec request accepted on channel 2\r debug3: receive packet: type 96\r debug2: channel 2: rcvd eof\r debug2: channel 2: output open -> drain\r debug2: channel 2: obuf empty\r debug2: channel 2: chan_shutdown_write (i0 o1 sock -1 wfd 7 efd 8 [write])\r debug2: channel 2: output drain -> closed\r debug3: receive packet: type 98\r debug1: client_input_channel_req: channel 2 rtype exit-status reply 0\r debug3: mux_exit_message: channel 2: exit message, exitval 0\r debug3: receive packet: type 98\r debug1: client_input_channel_req: channel 2 rtype eow@openssh.com reply 0\r debug2: channel 2: rcvd eow\r debug2: channel 2: chan_shutdown_read (i0 o3 sock -1 wfd 6 efd 8 [write])\r debug2: channel 2: input open -> closed\r debug3: receive packet: type 97\r debug2: channel 2: rcvd close\r debug3: channel 2: will not send data after close\r debug2: channel 2: send close\r debug3: send packet: type 97\r debug2: channel 2: is dead\r debug2: channel 2: gc: notify user\r debug3: mux_master_session_cleanup_cb: entering for channel 2\r debug2: channel 1: rcvd close\r debug2: channel 1: output open -> drain\r debug2: channel 1: chan_shutdown_read (i0 o1 sock 5 wfd 5 efd -1 [closed])\r debug2: channel 1: input open -> closed\r debug2: channel 2: gc: user detached\r debug2: channel 2: is dead\r debug2: channel 2: garbage collecting\r debug1: channel 2: free: client-session, nchannels 3\r debug3: channel 2: status: The following connections are open:\r #1 mux-control (t16 nr0 i3/0 o1/16 e[closed]/0 fd 5/5/-1 sock 5 cc -1)\r #2 client-session (t4 r0 i3/0 o3/0 e[write]/0 fd -1/-1/8 sock -1 cc -1)\r \r debug2: channel 1: obuf empty\r debug2: channel 1: chan_shutdown_write (i3 o1 sock 5 wfd 5 efd -1 [closed])\r debug3: mux_client_read_packet: read header failed: Broken pipe\r debug2: Received exit status from master 0\r debug2: channel 1: output drain -> closed\r debug2: channel 1: is dead (local)\r debug2: channel 1: gc: notify user\r debug3: mux_master_control_cleanup_cb: entering for channel 1\r debug2: channel 1: gc: user detached\r debug2: channel 1: is dead (local)\r debug2: channel 1: garbage collecting\r debug1: channel 1: free: mux-control, nchannels 2\r debug3: channel 1: status: The following connections are open:\r #1 mux-control (t16 nr0 i3/0 o3/0 e[closed]/0 fd 5/5/-1 sock 5 cc -1)\r \r debug2: set_control_persist_exit_time: schedule exit in 60 seconds\r ') 5:07:45 PM <10.1.90.118> ESTABLISH SSH CONNECTION FOR USER: ansible 5:07:45 PM <10.1.90.118> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="ansible"' -o ConnectTimeout=10 -o ControlPath=/var/lib/semaphore/tmp/.ansible/cp/06174fc892 10.1.90.118 '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo /home/ansible/.ansible/tmp `"&& mkdir "` echo /home/ansible/.ansible/tmp/ansible-tmp-1696518465.4432235-628530-55829167661651 `" && echo ansible-tmp-1696518465.4432235-628530-55829167661651="` echo /home/ansible/.ansible/tmp/ansible-tmp-1696518465.4432235-628530-55829167661651 `" ) && sleep 0'"'"'' 5:07:45 PM <10.1.90.118> (0, b'ansible-tmp-1696518465.4432235-628530-55829167661651=/home/ansible/.ansible/tmp/ansible-tmp-1696518465.4432235-628530-55829167661651 ', b"OpenSSH_8.4p1 Debian-5+deb11u1, OpenSSL 1.1.1n 15 Mar 2022\r debug1: Reading configuration data /etc/ssh/ssh_config\r debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files\r debug1: /etc/ssh/ssh_config line 21: Applying options for *\r debug2: resolve_canonicalize: hostname 10.1.90.118 is address\r debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/var/lib/semaphore/.ssh/known_hosts'\r debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/var/lib/semaphore/.ssh/known_hosts2'\r debug1: auto-mux: Trying existing master\r debug2: fd 3 setting O_NONBLOCK\r debug2: mux_client_hello_exchange: master version 4\r debug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r debug3: mux_client_request_session: entering\r debug3: mux_client_request_alive: entering\r debug3: mux_client_request_alive: done pid = 628534\r debug3: mux_client_request_session: session request sent\r debug1: mux_client_request_session: master session id: 2\r debug3: mux_client_read_packet: read header failed: Broken pipe\r debug2: Received exit status from master 0\r ") 5:07:45 PM <10.1.90.118> Attempting python interpreter discovery 5:07:45 PM <10.1.90.118> ESTABLISH SSH CONNECTION FOR USER: ansible 5:07:45 PM <10.1.90.118> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="ansible"' -o ConnectTimeout=10 -o ControlPath=/var/lib/semaphore/tmp/.ansible/cp/06174fc892 10.1.90.118 '/bin/sh -c '"'"'echo PLATFORM; uname; echo FOUND; command -v '"'"'"'"'"'"'"'"'/usr/bin/python'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python3.9'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python3.8'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python3.7'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python3.6'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python3.5'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python2.7'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python2.6'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'/usr/libexec/platform-python'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'/usr/bin/python3'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python'"'"'"'"'"'"'"'"'; echo ENDFOUND && sleep 0'"'"'' 5:07:45 PM <10.1.90.118> (0, b'PLATFORM Linux FOUND /usr/bin/python3 ENDFOUND ', b"OpenSSH_8.4p1 Debian-5+deb11u1, OpenSSL 1.1.1n 15 Mar 2022\r debug1: Reading configuration data /etc/ssh/ssh_config\r debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files\r debug1: /etc/ssh/ssh_config line 21: Applying options for *\r debug2: resolve_canonicalize: hostname 10.1.90.118 is address\r debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/var/lib/semaphore/.ssh/known_hosts'\r debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/var/lib/semaphore/.ssh/known_hosts2'\r debug1: auto-mux: Trying existing master\r debug2: fd 3 setting O_NONBLOCK\r debug2: mux_client_hello_exchange: master version 4\r debug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r debug3: mux_client_request_session: entering\r debug3: mux_client_request_alive: entering\r debug3: mux_client_request_alive: done pid = 628534\r debug3: mux_client_request_session: session request sent\r debug1: mux_client_request_session: master session id: 2\r debug3: mux_client_read_packet: read header failed: Broken pipe\r debug2: Received exit status from master 0\r ") 5:07:45 PM <10.1.90.118> ESTABLISH SSH CONNECTION FOR USER: ansible 5:07:45 PM <10.1.90.118> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="ansible"' -o ConnectTimeout=10 -o ControlPath=/var/lib/semaphore/tmp/.ansible/cp/06174fc892 10.1.90.118 '/bin/sh -c '"'"'/usr/bin/python3 && sleep 0'"'"'' 5:07:45 PM <10.1.90.118> (0, b'{"platform_dist_result": [], "osrelease_content": "PRETTY_NAME=\\"Ubuntu 22.04.2 LTS\\"\ NAME=\\"Ubuntu\\"\ VERSION_ID=\\"22.04\\"\ VERSION=\\"22.04.2 LTS (Jammy Jellyfish)\\"\ VERSION_CODENAME=jammy\ ID=ubuntu\ ID_LIKE=debian\ HOME_URL=\\"https://www.ubuntu.com/\\"\ SUPPORT_URL=\\"https://help.ubuntu.com/\\"\ BUG_REPORT_URL=\\"https://bugs.launchpad.net/ubuntu/\\"\ PRIVACY_POLICY_URL=\\"https://www.ubuntu.com/legal/terms-and-policies/privacy-policy\\"\ UBUNTU_CODENAME=jammy\ "} ', b"OpenSSH_8.4p1 Debian-5+deb11u1, OpenSSL 1.1.1n 15 Mar 2022\r debug1: Reading configuration data /etc/ssh/ssh_config\r debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files\r debug1: /etc/ssh/ssh_config line 21: Applying options for *\r debug2: resolve_canonicalize: hostname 10.1.90.118 is address\r debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/var/lib/semaphore/.ssh/known_hosts'\r debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/var/lib/semaphore/.ssh/known_hosts2'\r debug1: auto-mux: Trying existing master\r debug2: fd 3 setting O_NONBLOCK\r debug2: mux_client_hello_exchange: master version 4\r debug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r debug3: mux_client_request_session: entering\r debug3: mux_client_request_alive: entering\r debug3: mux_client_request_alive: done pid = 628534\r debug3: mux_client_request_session: session request sent\r debug1: mux_client_request_session: master session id: 2\r debug3: mux_client_read_packet: read header failed: Broken pipe\r debug2: Received exit status from master 0\r ") 5:07:45 PM Using module file /usr/lib/python3/dist-packages/ansible/modules/setup.py 5:07:45 PM <10.1.90.118> PUT /var/lib/semaphore/tmp/.ansible/tmp/ansible-local-628527pdlgrgxf/tmp1in2w444 TO /home/ansible/.ansible/tmp/ansible-tmp-1696518465.4432235-628530-55829167661651/AnsiballZ_setup.py 5:07:45 PM <10.1.90.118> SSH: EXEC sftp -b - -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="ansible"' -o ConnectTimeout=10 -o ControlPath=/var/lib/semaphore/tmp/.ansible/cp/06174fc892 '[10.1.90.118]' ```

[Goldenflamer]

I am facing the same problem, I think. It seems to be related to the ssh control path.

If I manually enter the command it is stuck on in the console, and delete the ControlPath, it works.

2.9.39-beta

Use version 2.9.37

[adaichendt-tv1]

I am facing the same problem, I think. It seems to be related to the ssh control path. If I manually enter the command it is stuck on in the console, and delete the ControlPath, it works. 2.9.39-beta

Use version 2.9.37

Downgrading to 2.9.37 did not resolve the issue.

 11:17:30 AM
PLAY [all] *********************************************************************
11:17:30 AM
11:17:30 AM
TASK [Gathering Facts] *********************************************************
11:17:30 AM
task path: /var/lib/semaphore/tmp/repository_1_8/playbooks/netbox-agent/netbox-prod.yaml:2
11:17:30 AM
<10.1.90.118> ESTABLISH SSH CONNECTION FOR USER: lxadmin
11:17:30 AM
<10.1.90.118> SSH: EXEC sshpass -d9 ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'User="lxadmin"' -o ConnectTimeout=10 -o ControlPath=/var/lib/semaphore/tmp/.ansible/cp/c3de483d93 10.1.90.118 '/bin/sh -c '"'"'echo ~lxadmin && sleep 0'"'"''

stuck here.

[Ashkaan]

2.9.18 is the latest version that works great for me.

[Ashkaan]

I just tested several of the lastest version including betas and they're still not working. Any word on when I can go back to "latest"?

Thanks so much

[dennisaucoin]

I can agree. I just installed the newest version of the release version and the screen is not refreshing. v2.9.37 is not working. If you refresh the screen it show the job progress but it is not auto refreshing. I'd go the beta, but I tried install and it and my keys stop working.

[fiftin]

@dennisaucoin is this issue actual? What screen? Need more details. Thank you

[Ashkaan]

I'd go the beta, but I tried install and it and my keys stop working.

Your keys always stop working with any version change. Just input them again.

[fiftin]

@Ashkaan Keys encrypted in database.

It was a difficult choice between not encrypting data by default, or generating a new key when deploying the container without SEMAPHORE_ACCESS_KEY_ENCRYPTION. We chose the second option.

So, you have 2 options: 1) Backup /etc/semaphore/config.json from container to host and mount it to container. 2) Pass environment variable SEMAPHORE_ACCESS_KEY_ENCRYPTION to container: https://docs.semui.co/administration-guide/security.