search

techno-tim / k3s-ansible

The easiest way to bootstrap a self-hosted High Availability Kubernetes cluster. A fully automated HA k3s etcd install with kube-vip, MetalLB, and more. Build. Destroy. Repeat.
https://technotim.live/posts/k3s-etcd-ansible/
Apache License 2.0
2.41k stars 1.05k forks source link

Automation hangs on step Enable and check K3s service #314

Closed mraczka closed 1 year ago

mraczka commented 1 year ago

After creating six clean Ubuntu VMs on proxmox I am trying to setup k3s cluster with automation. Basic configuration, local IPs adjusted and nothing else. Everything works for master nodes but for workers automation hangs indefinitely on TASK [k3s/node : Enable and check K3s service].

Expected Behavior

Expected behavior is that everything works as advertised ;)

Current Behavior

Everything works to the point of TASK [k3s/node : Enable and check K3s service]. Services are not created on worker nodes. I have checked previous issues with this step but none was the case.

TASK [k3s/node : Enable and check K3s service] **************************************************************************************************************************************************************************************************************************************
task path: /home/mrd/git/k3s-ansible/roles/k3s/node/tasks/main.yml:11
<192.168.1.112> ESTABLISH SSH CONNECTION FOR USER: mrd
<192.168.1.112> SSH: ansible.cfg set ssh_args: (-C)(-o)(ControlMaster=auto)(-o)(ControlPersist=60s)
<192.168.1.112> SSH: ansible_password/ansible_ssh_password not set: (-o)(KbdInteractiveAuthentication=no)(-o)(PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey)(-o)(PasswordAuthentication=no)
<192.168.1.112> SSH: ANSIBLE_REMOTE_USER/remote_user/ansible_user/user/-u set: (-o)(User="mrd")
<192.168.1.112> SSH: ANSIBLE_TIMEOUT/timeout set: (-o)(ConnectTimeout=10)
<192.168.1.112> SSH: Set ssh_common_args: ()
<192.168.1.112> SSH: Set ssh_extra_args: ()
<192.168.1.112> SSH: found only ControlPersist; added ControlPath: (-o)(ControlPath="/home/mrd/.ansible/cp/1e5ab0657b")
<192.168.1.112> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="mrd"' -o ConnectTimeout=10 -o 'ControlPath="/home/mrd/.ansible/cp/1e5ab0657b"' 192.168.1.112 '/bin/sh -c '"'"'echo ~mrd && sleep 0'"'"''
<192.168.1.114> ESTABLISH SSH CONNECTION FOR USER: mrd
<192.168.1.114> SSH: ansible.cfg set ssh_args: (-C)(-o)(ControlMaster=auto)(-o)(ControlPersist=60s)
<192.168.1.114> SSH: ansible_password/ansible_ssh_password not set: (-o)(KbdInteractiveAuthentication=no)(-o)(PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey)(-o)(PasswordAuthentication=no)
<192.168.1.114> SSH: ANSIBLE_REMOTE_USER/remote_user/ansible_user/user/-u set: (-o)(User="mrd")
<192.168.1.114> SSH: ANSIBLE_TIMEOUT/timeout set: (-o)(ConnectTimeout=10)
<192.168.1.114> SSH: Set ssh_common_args: ()
<192.168.1.114> SSH: Set ssh_extra_args: ()
<192.168.1.114> SSH: found only ControlPersist; added ControlPath: (-o)(ControlPath="/home/mrd/.ansible/cp/70c7f1c985")
<192.168.1.114> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="mrd"' -o ConnectTimeout=10 -o 'ControlPath="/home/mrd/.ansible/cp/70c7f1c985"' 192.168.1.114 '/bin/sh -c '"'"'echo ~mrd && sleep 0'"'"''
<192.168.1.112> (0, b'/home/mrd\n', b"OpenSSH_8.9p1 Ubuntu-3ubuntu0.1, OpenSSL 3.0.2 15 Mar 2022\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files\r\ndebug1: /etc/ssh/ssh_config line 21: Applying options for *\r\ndebug2: resolve_canonicalize: hostname 192.168.1.112 is address\r\ndebug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/home/mrd/.ssh/known_hosts'\r\ndebug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/home/mrd/.ssh/known_hosts2'\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 38192\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n")
<192.168.1.112> ESTABLISH SSH CONNECTION FOR USER: mrd
<192.168.1.112> SSH: ansible.cfg set ssh_args: (-C)(-o)(ControlMaster=auto)(-o)(ControlPersist=60s)
<192.168.1.112> SSH: ansible_password/ansible_ssh_password not set: (-o)(KbdInteractiveAuthentication=no)(-o)(PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey)(-o)(PasswordAuthentication=no)
<192.168.1.112> SSH: ANSIBLE_REMOTE_USER/remote_user/ansible_user/user/-u set: (-o)(User="mrd")
<192.168.1.112> SSH: ANSIBLE_TIMEOUT/timeout set: (-o)(ConnectTimeout=10)
<192.168.1.112> SSH: Set ssh_common_args: ()
<192.168.1.112> SSH: Set ssh_extra_args: ()
<192.168.1.112> SSH: found only ControlPersist; added ControlPath: (-o)(ControlPath="/home/mrd/.ansible/cp/1e5ab0657b")
<192.168.1.112> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="mrd"' -o ConnectTimeout=10 -o 'ControlPath="/home/mrd/.ansible/cp/1e5ab0657b"' 192.168.1.112 '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo /home/mrd/.ansible/tmp `"&& mkdir "` echo /home/mrd/.ansible/tmp/ansible-tmp-1686658183.5355954-38293-118510171344726 `" && echo ansible-tmp-1686658183.5355954-38293-118510171344726="` echo /home/mrd/.ansible/tmp/ansible-tmp-1686658183.5355954-38293-118510171344726 `" ) && sleep 0'"'"''
<192.168.1.114> (0, b'/home/mrd\n', b"OpenSSH_8.9p1 Ubuntu-3ubuntu0.1, OpenSSL 3.0.2 15 Mar 2022\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files\r\ndebug1: /etc/ssh/ssh_config line 21: Applying options for *\r\ndebug2: resolve_canonicalize: hostname 192.168.1.114 is address\r\ndebug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/home/mrd/.ssh/known_hosts'\r\ndebug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/home/mrd/.ssh/known_hosts2'\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 38195\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n")
<192.168.1.114> ESTABLISH SSH CONNECTION FOR USER: mrd
<192.168.1.114> SSH: ansible.cfg set ssh_args: (-C)(-o)(ControlMaster=auto)(-o)(ControlPersist=60s)
<192.168.1.114> SSH: ansible_password/ansible_ssh_password not set: (-o)(KbdInteractiveAuthentication=no)(-o)(PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey)(-o)(PasswordAuthentication=no)
<192.168.1.114> SSH: ANSIBLE_REMOTE_USER/remote_user/ansible_user/user/-u set: (-o)(User="mrd")
<192.168.1.114> SSH: ANSIBLE_TIMEOUT/timeout set: (-o)(ConnectTimeout=10)
<192.168.1.114> SSH: Set ssh_common_args: ()
<192.168.1.114> SSH: Set ssh_extra_args: ()
<192.168.1.114> SSH: found only ControlPersist; added ControlPath: (-o)(ControlPath="/home/mrd/.ansible/cp/70c7f1c985")
<192.168.1.114> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="mrd"' -o ConnectTimeout=10 -o 'ControlPath="/home/mrd/.ansible/cp/70c7f1c985"' 192.168.1.114 '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo /home/mrd/.ansible/tmp `"&& mkdir "` echo /home/mrd/.ansible/tmp/ansible-tmp-1686658183.561984-38295-97990290856469 `" && echo ansible-tmp-1686658183.561984-38295-97990290856469="` echo /home/mrd/.ansible/tmp/ansible-tmp-1686658183.561984-38295-97990290856469 `" ) && sleep 0'"'"''
<192.168.1.112> (0, b'ansible-tmp-1686658183.5355954-38293-118510171344726=/home/mrd/.ansible/tmp/ansible-tmp-1686658183.5355954-38293-118510171344726\n', b"OpenSSH_8.9p1 Ubuntu-3ubuntu0.1, OpenSSL 3.0.2 15 Mar 2022\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files\r\ndebug1: /etc/ssh/ssh_config line 21: Applying options for *\r\ndebug2: resolve_canonicalize: hostname 192.168.1.112 is address\r\ndebug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/home/mrd/.ssh/known_hosts'\r\ndebug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/home/mrd/.ssh/known_hosts2'\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 38192\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n")
Using module file /home/mrd/.local/lib/python3.10/site-packages/ansible/modules/systemd.py
<192.168.1.112> PUT /home/mrd/.ansible/tmp/ansible-local-358592ujn1_o_/tmpfp5hak96 TO /home/mrd/.ansible/tmp/ansible-tmp-1686658183.5355954-38293-118510171344726/AnsiballZ_systemd.py
<192.168.1.112> SSH: ansible.cfg set ssh_args: (-C)(-o)(ControlMaster=auto)(-o)(ControlPersist=60s)
<192.168.1.112> SSH: ansible_password/ansible_ssh_password not set: (-o)(KbdInteractiveAuthentication=no)(-o)(PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey)(-o)(PasswordAuthentication=no)
<192.168.1.112> SSH: ANSIBLE_REMOTE_USER/remote_user/ansible_user/user/-u set: (-o)(User="mrd")
<192.168.1.112> SSH: ANSIBLE_TIMEOUT/timeout set: (-o)(ConnectTimeout=10)
<192.168.1.112> SSH: Set ssh_common_args: ()
<192.168.1.112> SSH: Set sftp_extra_args: ()
<192.168.1.112> SSH: found only ControlPersist; added ControlPath: (-o)(ControlPath="/home/mrd/.ansible/cp/1e5ab0657b")
<192.168.1.112> SSH: EXEC sftp -b - -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="mrd"' -o ConnectTimeout=10 -o 'ControlPath="/home/mrd/.ansible/cp/1e5ab0657b"' '[192.168.1.112]'
<192.168.1.114> (0, b'ansible-tmp-1686658183.561984-38295-97990290856469=/home/mrd/.ansible/tmp/ansible-tmp-1686658183.561984-38295-97990290856469\n', b"OpenSSH_8.9p1 Ubuntu-3ubuntu0.1, OpenSSL 3.0.2 15 Mar 2022\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files\r\ndebug1: /etc/ssh/ssh_config line 21: Applying options for *\r\ndebug2: resolve_canonicalize: hostname 192.168.1.114 is address\r\ndebug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/home/mrd/.ssh/known_hosts'\r\ndebug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/home/mrd/.ssh/known_hosts2'\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 38195\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n")
Using module file /home/mrd/.local/lib/python3.10/site-packages/ansible/modules/systemd.py
<192.168.1.114> PUT /home/mrd/.ansible/tmp/ansible-local-358592ujn1_o_/tmpzpwsphgi TO /home/mrd/.ansible/tmp/ansible-tmp-1686658183.561984-38295-97990290856469/AnsiballZ_systemd.py
<192.168.1.114> SSH: ansible.cfg set ssh_args: (-C)(-o)(ControlMaster=auto)(-o)(ControlPersist=60s)
<192.168.1.114> SSH: ansible_password/ansible_ssh_password not set: (-o)(KbdInteractiveAuthentication=no)(-o)(PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey)(-o)(PasswordAuthentication=no)
<192.168.1.114> SSH: ANSIBLE_REMOTE_USER/remote_user/ansible_user/user/-u set: (-o)(User="mrd")
<192.168.1.114> SSH: ANSIBLE_TIMEOUT/timeout set: (-o)(ConnectTimeout=10)
<192.168.1.114> SSH: Set ssh_common_args: ()
<192.168.1.114> SSH: Set sftp_extra_args: ()
<192.168.1.114> SSH: found only ControlPersist; added ControlPath: (-o)(ControlPath="/home/mrd/.ansible/cp/70c7f1c985")
<192.168.1.114> SSH: EXEC sftp -b - -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="mrd"' -o ConnectTimeout=10 -o 'ControlPath="/home/mrd/.ansible/cp/70c7f1c985"' '[192.168.1.114]'
<192.168.1.116> ESTABLISH SSH CONNECTION FOR USER: mrd
<192.168.1.116> SSH: ansible.cfg set ssh_args: (-C)(-o)(ControlMaster=auto)(-o)(ControlPersist=60s)
<192.168.1.116> SSH: ansible_password/ansible_ssh_password not set: (-o)(KbdInteractiveAuthentication=no)(-o)(PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey)(-o)(PasswordAuthentication=no)
<192.168.1.116> SSH: ANSIBLE_REMOTE_USER/remote_user/ansible_user/user/-u set: (-o)(User="mrd")
<192.168.1.116> SSH: ANSIBLE_TIMEOUT/timeout set: (-o)(ConnectTimeout=10)
<192.168.1.116> SSH: Set ssh_common_args: ()
<192.168.1.116> SSH: Set ssh_extra_args: ()
<192.168.1.116> SSH: found only ControlPersist; added ControlPath: (-o)(ControlPath="/home/mrd/.ansible/cp/82653e3056")
<192.168.1.116> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="mrd"' -o ConnectTimeout=10 -o 'ControlPath="/home/mrd/.ansible/cp/82653e3056"' 192.168.1.116 '/bin/sh -c '"'"'echo ~mrd && sleep 0'"'"''
<192.168.1.116> (0, b'/home/mrd\n', b"OpenSSH_8.9p1 Ubuntu-3ubuntu0.1, OpenSSL 3.0.2 15 Mar 2022\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files\r\ndebug1: /etc/ssh/ssh_config line 21: Applying options for *\r\ndebug2: resolve_canonicalize: hostname 192.168.1.116 is address\r\ndebug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/home/mrd/.ssh/known_hosts'\r\ndebug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/home/mrd/.ssh/known_hosts2'\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 38198\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n")
<192.168.1.116> ESTABLISH SSH CONNECTION FOR USER: mrd
<192.168.1.116> SSH: ansible.cfg set ssh_args: (-C)(-o)(ControlMaster=auto)(-o)(ControlPersist=60s)
<192.168.1.116> SSH: ansible_password/ansible_ssh_password not set: (-o)(KbdInteractiveAuthentication=no)(-o)(PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey)(-o)(PasswordAuthentication=no)
<192.168.1.116> SSH: ANSIBLE_REMOTE_USER/remote_user/ansible_user/user/-u set: (-o)(User="mrd")
<192.168.1.116> SSH: ANSIBLE_TIMEOUT/timeout set: (-o)(ConnectTimeout=10)
<192.168.1.116> SSH: Set ssh_common_args: ()
<192.168.1.116> SSH: Set ssh_extra_args: ()
<192.168.1.116> SSH: found only ControlPersist; added ControlPath: (-o)(ControlPath="/home/mrd/.ansible/cp/82653e3056")
<192.168.1.116> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="mrd"' -o ConnectTimeout=10 -o 'ControlPath="/home/mrd/.ansible/cp/82653e3056"' 192.168.1.116 '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo /home/mrd/.ansible/tmp `"&& mkdir "` echo /home/mrd/.ansible/tmp/ansible-tmp-1686658183.6560056-38300-69096513580153 `" && echo ansible-tmp-1686658183.6560056-38300-69096513580153="` echo /home/mrd/.ansible/tmp/ansible-tmp-1686658183.6560056-38300-69096513580153 `" ) && sleep 0'"'"''
<192.168.1.112> (0, b'sftp> put /home/mrd/.ansible/tmp/ansible-local-358592ujn1_o_/tmpfp5hak96 /home/mrd/.ansible/tmp/ansible-tmp-1686658183.5355954-38293-118510171344726/AnsiballZ_systemd.py\n', b'OpenSSH_8.9p1 Ubuntu-3ubuntu0.1, OpenSSL 3.0.2 15 Mar 2022\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files\r\ndebug1: /etc/ssh/ssh_config line 21: Applying options for *\r\ndebug2: resolve_canonicalize: hostname 192.168.1.112 is address\r\ndebug3: expanded UserKnownHostsFile \'~/.ssh/known_hosts\' -> \'/home/mrd/.ssh/known_hosts\'\r\ndebug3: expanded UserKnownHostsFile \'~/.ssh/known_hosts2\' -> \'/home/mrd/.ssh/known_hosts2\'\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 38192\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug2: Remote version: 3\r\ndebug2: Server supports extension "posix-rename@openssh.com" revision 1\r\ndebug2: Server supports extension "statvfs@openssh.com" revision 2\r\ndebug2: Server supports extension "fstatvfs@openssh.com" revision 2\r\ndebug2: Server supports extension "hardlink@openssh.com" revision 1\r\ndebug2: Server supports extension "fsync@openssh.com" revision 1\r\ndebug2: Server supports extension "lsetstat@openssh.com" revision 1\r\ndebug2: Server supports extension "limits@openssh.com" revision 1\r\ndebug2: Server supports extension "expand-path@openssh.com" revision 1\r\ndebug3: Sent message limits@openssh.com I:1\r\ndebug3: Received limits reply T:201 I:1\r\ndebug1: Using server download size 261120\r\ndebug1: Using server upload size 261120\r\ndebug1: Server handle limit 1019; using 64\r\ndebug2: Sending SSH2_FXP_REALPATH "."\r\ndebug3: Sent message fd 3 T:16 I:2\r\ndebug3: SSH2_FXP_REALPATH . -> /home/mrd\r\ndebug3: Looking up /home/mrd/.ansible/tmp/ansible-local-358592ujn1_o_/tmpfp5hak96\r\ndebug2: Sending SSH2_FXP_STAT "/home/mrd/.ansible/tmp/ansible-tmp-1686658183.5355954-38293-118510171344726/AnsiballZ_systemd.py"\r\ndebug3: Sent message fd 3 T:17 I:3\r\ndebug1: stat remote: No such file or directory\r\ndebug2: do_upload: upload local "/home/mrd/.ansible/tmp/ansible-local-358592ujn1_o_/tmpfp5hak96" to remote "/home/mrd/.ansible/tmp/ansible-tmp-1686658183.5355954-38293-118510171344726/AnsiballZ_systemd.py"\r\ndebug2: Sending SSH2_FXP_OPEN "/home/mrd/.ansible/tmp/ansible-tmp-1686658183.5355954-38293-118510171344726/AnsiballZ_systemd.py"\r\ndebug3: Sent dest message SSH2_FXP_OPEN I:4 P:/home/mrd/.ansible/tmp/ansible-tmp-1686658183.5355954-38293-118510171344726/AnsiballZ_systemd.py M:0x001a\r\ndebug3: Sent message SSH2_FXP_WRITE I:6 O:0 S:261120\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 6 261120 bytes at 0\r\ndebug3: Sent message SSH2_FXP_WRITE I:7 O:261120 S:45579\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 7 45579 bytes at 261120\r\ndebug3: Sent message SSH2_FXP_CLOSE I:5\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
<192.168.1.112> ESTABLISH SSH CONNECTION FOR USER: mrd
<192.168.1.112> SSH: ansible.cfg set ssh_args: (-C)(-o)(ControlMaster=auto)(-o)(ControlPersist=60s)
<192.168.1.112> SSH: ansible_password/ansible_ssh_password not set: (-o)(KbdInteractiveAuthentication=no)(-o)(PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey)(-o)(PasswordAuthentication=no)
<192.168.1.112> SSH: ANSIBLE_REMOTE_USER/remote_user/ansible_user/user/-u set: (-o)(User="mrd")
<192.168.1.112> SSH: ANSIBLE_TIMEOUT/timeout set: (-o)(ConnectTimeout=10)
<192.168.1.112> SSH: Set ssh_common_args: ()
<192.168.1.112> SSH: Set ssh_extra_args: ()
<192.168.1.112> SSH: found only ControlPersist; added ControlPath: (-o)(ControlPath="/home/mrd/.ansible/cp/1e5ab0657b")
<192.168.1.112> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="mrd"' -o ConnectTimeout=10 -o 'ControlPath="/home/mrd/.ansible/cp/1e5ab0657b"' 192.168.1.112 '/bin/sh -c '"'"'chmod u+x /home/mrd/.ansible/tmp/ansible-tmp-1686658183.5355954-38293-118510171344726/ /home/mrd/.ansible/tmp/ansible-tmp-1686658183.5355954-38293-118510171344726/AnsiballZ_systemd.py && sleep 0'"'"''
<192.168.1.116> (0, b'ansible-tmp-1686658183.6560056-38300-69096513580153=/home/mrd/.ansible/tmp/ansible-tmp-1686658183.6560056-38300-69096513580153\n', b"OpenSSH_8.9p1 Ubuntu-3ubuntu0.1, OpenSSL 3.0.2 15 Mar 2022\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files\r\ndebug1: /etc/ssh/ssh_config line 21: Applying options for *\r\ndebug2: resolve_canonicalize: hostname 192.168.1.116 is address\r\ndebug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/home/mrd/.ssh/known_hosts'\r\ndebug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/home/mrd/.ssh/known_hosts2'\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 38198\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n")
Using module file /home/mrd/.local/lib/python3.10/site-packages/ansible/modules/systemd.py
<192.168.1.116> PUT /home/mrd/.ansible/tmp/ansible-local-358592ujn1_o_/tmpchb4cz30 TO /home/mrd/.ansible/tmp/ansible-tmp-1686658183.6560056-38300-69096513580153/AnsiballZ_systemd.py
<192.168.1.116> SSH: ansible.cfg set ssh_args: (-C)(-o)(ControlMaster=auto)(-o)(ControlPersist=60s)
<192.168.1.116> SSH: ansible_password/ansible_ssh_password not set: (-o)(KbdInteractiveAuthentication=no)(-o)(PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey)(-o)(PasswordAuthentication=no)
<192.168.1.116> SSH: ANSIBLE_REMOTE_USER/remote_user/ansible_user/user/-u set: (-o)(User="mrd")
<192.168.1.116> SSH: ANSIBLE_TIMEOUT/timeout set: (-o)(ConnectTimeout=10)
<192.168.1.116> SSH: Set ssh_common_args: ()
<192.168.1.116> SSH: Set sftp_extra_args: ()
<192.168.1.116> SSH: found only ControlPersist; added ControlPath: (-o)(ControlPath="/home/mrd/.ansible/cp/82653e3056")
<192.168.1.116> SSH: EXEC sftp -b - -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="mrd"' -o ConnectTimeout=10 -o 'ControlPath="/home/mrd/.ansible/cp/82653e3056"' '[192.168.1.116]'
<192.168.1.112> (0, b'', b"OpenSSH_8.9p1 Ubuntu-3ubuntu0.1, OpenSSL 3.0.2 15 Mar 2022\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files\r\ndebug1: /etc/ssh/ssh_config line 21: Applying options for *\r\ndebug2: resolve_canonicalize: hostname 192.168.1.112 is address\r\ndebug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/home/mrd/.ssh/known_hosts'\r\ndebug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/home/mrd/.ssh/known_hosts2'\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 38192\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n")
<192.168.1.112> ESTABLISH SSH CONNECTION FOR USER: mrd
<192.168.1.112> SSH: ansible.cfg set ssh_args: (-C)(-o)(ControlMaster=auto)(-o)(ControlPersist=60s)
<192.168.1.112> SSH: ansible_password/ansible_ssh_password not set: (-o)(KbdInteractiveAuthentication=no)(-o)(PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey)(-o)(PasswordAuthentication=no)
<192.168.1.112> SSH: ANSIBLE_REMOTE_USER/remote_user/ansible_user/user/-u set: (-o)(User="mrd")
<192.168.1.112> SSH: ANSIBLE_TIMEOUT/timeout set: (-o)(ConnectTimeout=10)
<192.168.1.112> SSH: Set ssh_common_args: ()
<192.168.1.112> SSH: Set ssh_extra_args: ()
<192.168.1.112> SSH: found only ControlPersist; added ControlPath: (-o)(ControlPath="/home/mrd/.ansible/cp/1e5ab0657b")
<192.168.1.112> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="mrd"' -o ConnectTimeout=10 -o 'ControlPath="/home/mrd/.ansible/cp/1e5ab0657b"' -tt 192.168.1.112 '/bin/sh -c '"'"'sudo -H -S -n  -u root /bin/sh -c '"'"'"'"'"'"'"'"'echo BECOME-SUCCESS-ehcwqqxrbkjrrciowdfebwalquplvzhk ; /usr/bin/python3 /home/mrd/.ansible/tmp/ansible-tmp-1686658183.5355954-38293-118510171344726/AnsiballZ_systemd.py'"'"'"'"'"'"'"'"' && sleep 0'"'"''
<192.168.1.114> (0, b'sftp> put /home/mrd/.ansible/tmp/ansible-local-358592ujn1_o_/tmpzpwsphgi /home/mrd/.ansible/tmp/ansible-tmp-1686658183.561984-38295-97990290856469/AnsiballZ_systemd.py\n', b'OpenSSH_8.9p1 Ubuntu-3ubuntu0.1, OpenSSL 3.0.2 15 Mar 2022\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files\r\ndebug1: /etc/ssh/ssh_config line 21: Applying options for *\r\ndebug2: resolve_canonicalize: hostname 192.168.1.114 is address\r\ndebug3: expanded UserKnownHostsFile \'~/.ssh/known_hosts\' -> \'/home/mrd/.ssh/known_hosts\'\r\ndebug3: expanded UserKnownHostsFile \'~/.ssh/known_hosts2\' -> \'/home/mrd/.ssh/known_hosts2\'\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 38195\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug2: Remote version: 3\r\ndebug2: Server supports extension "posix-rename@openssh.com" revision 1\r\ndebug2: Server supports extension "statvfs@openssh.com" revision 2\r\ndebug2: Server supports extension "fstatvfs@openssh.com" revision 2\r\ndebug2: Server supports extension "hardlink@openssh.com" revision 1\r\ndebug2: Server supports extension "fsync@openssh.com" revision 1\r\ndebug2: Server supports extension "lsetstat@openssh.com" revision 1\r\ndebug2: Server supports extension "limits@openssh.com" revision 1\r\ndebug2: Server supports extension "expand-path@openssh.com" revision 1\r\ndebug3: Sent message limits@openssh.com I:1\r\ndebug3: Received limits reply T:201 I:1\r\ndebug1: Using server download size 261120\r\ndebug1: Using server upload size 261120\r\ndebug1: Server handle limit 1019; using 64\r\ndebug2: Sending SSH2_FXP_REALPATH "."\r\ndebug3: Sent message fd 3 T:16 I:2\r\ndebug3: SSH2_FXP_REALPATH . -> /home/mrd\r\ndebug3: Looking up /home/mrd/.ansible/tmp/ansible-local-358592ujn1_o_/tmpzpwsphgi\r\ndebug2: Sending SSH2_FXP_STAT "/home/mrd/.ansible/tmp/ansible-tmp-1686658183.561984-38295-97990290856469/AnsiballZ_systemd.py"\r\ndebug3: Sent message fd 3 T:17 I:3\r\ndebug1: stat remote: No such file or directory\r\ndebug2: do_upload: upload local "/home/mrd/.ansible/tmp/ansible-local-358592ujn1_o_/tmpzpwsphgi" to remote "/home/mrd/.ansible/tmp/ansible-tmp-1686658183.561984-38295-97990290856469/AnsiballZ_systemd.py"\r\ndebug2: Sending SSH2_FXP_OPEN "/home/mrd/.ansible/tmp/ansible-tmp-1686658183.561984-38295-97990290856469/AnsiballZ_systemd.py"\r\ndebug3: Sent dest message SSH2_FXP_OPEN I:4 P:/home/mrd/.ansible/tmp/ansible-tmp-1686658183.561984-38295-97990290856469/AnsiballZ_systemd.py M:0x001a\r\ndebug3: Sent message SSH2_FXP_WRITE I:6 O:0 S:261120\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 6 261120 bytes at 0\r\ndebug3: Sent message SSH2_FXP_WRITE I:7 O:261120 S:45577\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 7 45577 bytes at 261120\r\ndebug3: Sent message SSH2_FXP_CLOSE I:5\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
<192.168.1.114> ESTABLISH SSH CONNECTION FOR USER: mrd
<192.168.1.114> SSH: ansible.cfg set ssh_args: (-C)(-o)(ControlMaster=auto)(-o)(ControlPersist=60s)
<192.168.1.114> SSH: ansible_password/ansible_ssh_password not set: (-o)(KbdInteractiveAuthentication=no)(-o)(PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey)(-o)(PasswordAuthentication=no)
<192.168.1.114> SSH: ANSIBLE_REMOTE_USER/remote_user/ansible_user/user/-u set: (-o)(User="mrd")
<192.168.1.114> SSH: ANSIBLE_TIMEOUT/timeout set: (-o)(ConnectTimeout=10)
<192.168.1.114> SSH: Set ssh_common_args: ()
<192.168.1.114> SSH: Set ssh_extra_args: ()
<192.168.1.114> SSH: found only ControlPersist; added ControlPath: (-o)(ControlPath="/home/mrd/.ansible/cp/70c7f1c985")
<192.168.1.114> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="mrd"' -o ConnectTimeout=10 -o 'ControlPath="/home/mrd/.ansible/cp/70c7f1c985"' 192.168.1.114 '/bin/sh -c '"'"'chmod u+x /home/mrd/.ansible/tmp/ansible-tmp-1686658183.561984-38295-97990290856469/ /home/mrd/.ansible/tmp/ansible-tmp-1686658183.561984-38295-97990290856469/AnsiballZ_systemd.py && sleep 0'"'"''
<192.168.1.114> (0, b'', b"OpenSSH_8.9p1 Ubuntu-3ubuntu0.1, OpenSSL 3.0.2 15 Mar 2022\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files\r\ndebug1: /etc/ssh/ssh_config line 21: Applying options for *\r\ndebug2: resolve_canonicalize: hostname 192.168.1.114 is address\r\ndebug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/home/mrd/.ssh/known_hosts'\r\ndebug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/home/mrd/.ssh/known_hosts2'\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 38195\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n")
<192.168.1.114> ESTABLISH SSH CONNECTION FOR USER: mrd
<192.168.1.114> SSH: ansible.cfg set ssh_args: (-C)(-o)(ControlMaster=auto)(-o)(ControlPersist=60s)
<192.168.1.114> SSH: ansible_password/ansible_ssh_password not set: (-o)(KbdInteractiveAuthentication=no)(-o)(PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey)(-o)(PasswordAuthentication=no)
<192.168.1.114> SSH: ANSIBLE_REMOTE_USER/remote_user/ansible_user/user/-u set: (-o)(User="mrd")
<192.168.1.114> SSH: ANSIBLE_TIMEOUT/timeout set: (-o)(ConnectTimeout=10)
<192.168.1.114> SSH: Set ssh_common_args: ()
<192.168.1.114> SSH: Set ssh_extra_args: ()
<192.168.1.114> SSH: found only ControlPersist; added ControlPath: (-o)(ControlPath="/home/mrd/.ansible/cp/70c7f1c985")
<192.168.1.114> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="mrd"' -o ConnectTimeout=10 -o 'ControlPath="/home/mrd/.ansible/cp/70c7f1c985"' -tt 192.168.1.114 '/bin/sh -c '"'"'sudo -H -S -n  -u root /bin/sh -c '"'"'"'"'"'"'"'"'echo BECOME-SUCCESS-ytfnadgrhzhxkgjnevbklodpcrwdmrsz ; /usr/bin/python3 /home/mrd/.ansible/tmp/ansible-tmp-1686658183.561984-38295-97990290856469/AnsiballZ_systemd.py'"'"'"'"'"'"'"'"' && sleep 0'"'"''
Escalation succeeded
<192.168.1.116> (0, b'sftp> put /home/mrd/.ansible/tmp/ansible-local-358592ujn1_o_/tmpchb4cz30 /home/mrd/.ansible/tmp/ansible-tmp-1686658183.6560056-38300-69096513580153/AnsiballZ_systemd.py\n', b'OpenSSH_8.9p1 Ubuntu-3ubuntu0.1, OpenSSL 3.0.2 15 Mar 2022\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files\r\ndebug1: /etc/ssh/ssh_config line 21: Applying options for *\r\ndebug2: resolve_canonicalize: hostname 192.168.1.116 is address\r\ndebug3: expanded UserKnownHostsFile \'~/.ssh/known_hosts\' -> \'/home/mrd/.ssh/known_hosts\'\r\ndebug3: expanded UserKnownHostsFile \'~/.ssh/known_hosts2\' -> \'/home/mrd/.ssh/known_hosts2\'\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 38198\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug2: Remote version: 3\r\ndebug2: Server supports extension "posix-rename@openssh.com" revision 1\r\ndebug2: Server supports extension "statvfs@openssh.com" revision 2\r\ndebug2: Server supports extension "fstatvfs@openssh.com" revision 2\r\ndebug2: Server supports extension "hardlink@openssh.com" revision 1\r\ndebug2: Server supports extension "fsync@openssh.com" revision 1\r\ndebug2: Server supports extension "lsetstat@openssh.com" revision 1\r\ndebug2: Server supports extension "limits@openssh.com" revision 1\r\ndebug2: Server supports extension "expand-path@openssh.com" revision 1\r\ndebug3: Sent message limits@openssh.com I:1\r\ndebug3: Received limits reply T:201 I:1\r\ndebug1: Using server download size 261120\r\ndebug1: Using server upload size 261120\r\ndebug1: Server handle limit 1019; using 64\r\ndebug2: Sending SSH2_FXP_REALPATH "."\r\ndebug3: Sent message fd 3 T:16 I:2\r\ndebug3: SSH2_FXP_REALPATH . -> /home/mrd\r\ndebug3: Looking up /home/mrd/.ansible/tmp/ansible-local-358592ujn1_o_/tmpchb4cz30\r\ndebug2: Sending SSH2_FXP_STAT "/home/mrd/.ansible/tmp/ansible-tmp-1686658183.6560056-38300-69096513580153/AnsiballZ_systemd.py"\r\ndebug3: Sent message fd 3 T:17 I:3\r\ndebug1: stat remote: No such file or directory\r\ndebug2: do_upload: upload local "/home/mrd/.ansible/tmp/ansible-local-358592ujn1_o_/tmpchb4cz30" to remote "/home/mrd/.ansible/tmp/ansible-tmp-1686658183.6560056-38300-69096513580153/AnsiballZ_systemd.py"\r\ndebug2: Sending SSH2_FXP_OPEN "/home/mrd/.ansible/tmp/ansible-tmp-1686658183.6560056-38300-69096513580153/AnsiballZ_systemd.py"\r\ndebug3: Sent dest message SSH2_FXP_OPEN I:4 P:/home/mrd/.ansible/tmp/ansible-tmp-1686658183.6560056-38300-69096513580153/AnsiballZ_systemd.py M:0x001a\r\ndebug3: Sent message SSH2_FXP_WRITE I:6 O:0 S:261120\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 6 261120 bytes at 0\r\ndebug3: Sent message SSH2_FXP_WRITE I:7 O:261120 S:45578\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 7 45578 bytes at 261120\r\ndebug3: Sent message SSH2_FXP_CLOSE I:5\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
<192.168.1.116> ESTABLISH SSH CONNECTION FOR USER: mrd
<192.168.1.116> SSH: ansible.cfg set ssh_args: (-C)(-o)(ControlMaster=auto)(-o)(ControlPersist=60s)
<192.168.1.116> SSH: ansible_password/ansible_ssh_password not set: (-o)(KbdInteractiveAuthentication=no)(-o)(PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey)(-o)(PasswordAuthentication=no)
<192.168.1.116> SSH: ANSIBLE_REMOTE_USER/remote_user/ansible_user/user/-u set: (-o)(User="mrd")
<192.168.1.116> SSH: ANSIBLE_TIMEOUT/timeout set: (-o)(ConnectTimeout=10)
<192.168.1.116> SSH: Set ssh_common_args: ()
<192.168.1.116> SSH: Set ssh_extra_args: ()
<192.168.1.116> SSH: found only ControlPersist; added ControlPath: (-o)(ControlPath="/home/mrd/.ansible/cp/82653e3056")
<192.168.1.116> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="mrd"' -o ConnectTimeout=10 -o 'ControlPath="/home/mrd/.ansible/cp/82653e3056"' 192.168.1.116 '/bin/sh -c '"'"'chmod u+x /home/mrd/.ansible/tmp/ansible-tmp-1686658183.6560056-38300-69096513580153/ /home/mrd/.ansible/tmp/ansible-tmp-1686658183.6560056-38300-69096513580153/AnsiballZ_systemd.py && sleep 0'"'"''
Escalation succeeded
<192.168.1.116> (0, b'', b"OpenSSH_8.9p1 Ubuntu-3ubuntu0.1, OpenSSL 3.0.2 15 Mar 2022\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files\r\ndebug1: /etc/ssh/ssh_config line 21: Applying options for *\r\ndebug2: resolve_canonicalize: hostname 192.168.1.116 is address\r\ndebug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/home/mrd/.ssh/known_hosts'\r\ndebug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/home/mrd/.ssh/known_hosts2'\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 38198\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n")
<192.168.1.116> ESTABLISH SSH CONNECTION FOR USER: mrd
<192.168.1.116> SSH: ansible.cfg set ssh_args: (-C)(-o)(ControlMaster=auto)(-o)(ControlPersist=60s)
<192.168.1.116> SSH: ansible_password/ansible_ssh_password not set: (-o)(KbdInteractiveAuthentication=no)(-o)(PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey)(-o)(PasswordAuthentication=no)
<192.168.1.116> SSH: ANSIBLE_REMOTE_USER/remote_user/ansible_user/user/-u set: (-o)(User="mrd")
<192.168.1.116> SSH: ANSIBLE_TIMEOUT/timeout set: (-o)(ConnectTimeout=10)
<192.168.1.116> SSH: Set ssh_common_args: ()
<192.168.1.116> SSH: Set ssh_extra_args: ()
<192.168.1.116> SSH: found only ControlPersist; added ControlPath: (-o)(ControlPath="/home/mrd/.ansible/cp/82653e3056")
<192.168.1.116> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="mrd"' -o ConnectTimeout=10 -o 'ControlPath="/home/mrd/.ansible/cp/82653e3056"' -tt 192.168.1.116 '/bin/sh -c '"'"'sudo -H -S -n  -u root /bin/sh -c '"'"'"'"'"'"'"'"'echo BECOME-SUCCESS-svurncrgpqsfdcppyshptjrcnlsnpnos ; /usr/bin/python3 /home/mrd/.ansible/tmp/ansible-tmp-1686658183.6560056-38300-69096513580153/AnsiballZ_systemd.py'"'"'"'"'"'"'"'"' && sleep 0'"'"''
Escalation succeeded
<192.168.1.116> (1, b'\r\n{"failed": true, "msg": "Unable to start service k3s-node: Job for k3s-node.service failed because the control process exited with error code.\\nSee \\"systemctl status k3s-node.service\\" and \\"journalctl -xeu k3s-node.service\\" for details.\\n", "invocation": {"module_args": {"name": "k3s-node", "daemon_reload": true, "state": "restarted", "enabled": true, "daemon_reexec": false, "scope": "system", "no_block": false, "force": null, "masked": null}}}\r\n', b"OpenSSH_8.9p1 Ubuntu-3ubuntu0.1, OpenSSL 3.0.2 15 Mar 2022\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files\r\ndebug1: /etc/ssh/ssh_config line 21: Applying options for *\r\ndebug2: resolve_canonicalize: hostname 192.168.1.116 is address\r\ndebug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/home/mrd/.ssh/known_hosts'\r\ndebug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/home/mrd/.ssh/known_hosts2'\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 38198\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 1\r\nShared connection to 192.168.1.116 closed.\r\n")
<192.168.1.116> Failed to connect to the host via ssh: OpenSSH_8.9p1 Ubuntu-3ubuntu0.1, OpenSSL 3.0.2 15 Mar 2022
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug2: resolve_canonicalize: hostname 192.168.1.116 is address
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/home/mrd/.ssh/known_hosts'
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/home/mrd/.ssh/known_hosts2'
debug1: auto-mux: Trying existing master
debug2: fd 3 setting O_NONBLOCK
debug2: mux_client_hello_exchange: master version 4
debug3: mux_client_forwards: request forwardings: 0 local, 0 remote
debug3: mux_client_request_session: entering
debug3: mux_client_request_alive: entering
debug3: mux_client_request_alive: done pid = 38198
debug3: mux_client_request_session: session request sent
debug1: mux_client_request_session: master session id: 2
debug3: mux_client_read_packet: read header failed: Broken pipe
debug2: Received exit status from master 1
Shared connection to 192.168.1.116 closed.
<192.168.1.116> ESTABLISH SSH CONNECTION FOR USER: mrd
<192.168.1.116> SSH: ansible.cfg set ssh_args: (-C)(-o)(ControlMaster=auto)(-o)(ControlPersist=60s)
<192.168.1.116> SSH: ansible_password/ansible_ssh_password not set: (-o)(KbdInteractiveAuthentication=no)(-o)(PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey)(-o)(PasswordAuthentication=no)
<192.168.1.116> SSH: ANSIBLE_REMOTE_USER/remote_user/ansible_user/user/-u set: (-o)(User="mrd")
<192.168.1.116> SSH: ANSIBLE_TIMEOUT/timeout set: (-o)(ConnectTimeout=10)
<192.168.1.116> SSH: Set ssh_common_args: ()
<192.168.1.116> SSH: Set ssh_extra_args: ()
<192.168.1.116> SSH: found only ControlPersist; added ControlPath: (-o)(ControlPath="/home/mrd/.ansible/cp/82653e3056")
<192.168.1.116> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="mrd"' -o ConnectTimeout=10 -o 'ControlPath="/home/mrd/.ansible/cp/82653e3056"' 192.168.1.116 '/bin/sh -c '"'"'rm -f -r /home/mrd/.ansible/tmp/ansible-tmp-1686658183.6560056-38300-69096513580153/ > /dev/null 2>&1 && sleep 0'"'"''
<192.168.1.116> (0, b'', b"OpenSSH_8.9p1 Ubuntu-3ubuntu0.1, OpenSSL 3.0.2 15 Mar 2022\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files\r\ndebug1: /etc/ssh/ssh_config line 21: Applying options for *\r\ndebug2: resolve_canonicalize: hostname 192.168.1.116 is address\r\ndebug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/home/mrd/.ssh/known_hosts'\r\ndebug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/home/mrd/.ssh/known_hosts2'\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 38198\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n")
fatal: [kubernetes-worker6]: FAILED! => {
    "changed": false,
    "invocation": {
        "module_args": {
            "daemon_reexec": false,
            "daemon_reload": true,
            "enabled": true,
            "force": null,
            "masked": null,
            "name": "k3s-node",
            "no_block": false,
            "scope": "system",
            "state": "restarted"
        }
    },
    "msg": "Unable to start service k3s-node: Job for k3s-node.service failed because the control process exited with error code.\nSee \"systemctl status k3s-node.service\" and \"journalctl -xeu k3s-node.service\" for details.\n"
}
<192.168.1.112> (1, b'\r\n{"failed": true, "msg": "Unable to start service k3s-node: Job for k3s-node.service failed because the control process exited with error code.\\nSee \\"systemctl status k3s-node.service\\" and \\"journalctl -xeu k3s-node.service\\" for details.\\n", "invocation": {"module_args": {"name": "k3s-node", "daemon_reload": true, "state": "restarted", "enabled": true, "daemon_reexec": false, "scope": "system", "no_block": false, "force": null, "masked": null}}}\r\n', b"OpenSSH_8.9p1 Ubuntu-3ubuntu0.1, OpenSSL 3.0.2 15 Mar 2022\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files\r\ndebug1: /etc/ssh/ssh_config line 21: Applying options for *\r\ndebug2: resolve_canonicalize: hostname 192.168.1.112 is address\r\ndebug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/home/mrd/.ssh/known_hosts'\r\ndebug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/home/mrd/.ssh/known_hosts2'\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 38192\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 1\r\nShared connection to 192.168.1.112 closed.\r\n")
<192.168.1.112> Failed to connect to the host via ssh: OpenSSH_8.9p1 Ubuntu-3ubuntu0.1, OpenSSL 3.0.2 15 Mar 2022
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug2: resolve_canonicalize: hostname 192.168.1.112 is address
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/home/mrd/.ssh/known_hosts'
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/home/mrd/.ssh/known_hosts2'
debug1: auto-mux: Trying existing master
debug2: fd 3 setting O_NONBLOCK
debug2: mux_client_hello_exchange: master version 4
debug3: mux_client_forwards: request forwardings: 0 local, 0 remote
debug3: mux_client_request_session: entering
debug3: mux_client_request_alive: entering
debug3: mux_client_request_alive: done pid = 38192
debug3: mux_client_request_session: session request sent
debug1: mux_client_request_session: master session id: 2
debug3: mux_client_read_packet: read header failed: Broken pipe
debug2: Received exit status from master 1
Shared connection to 192.168.1.112 closed.
<192.168.1.112> ESTABLISH SSH CONNECTION FOR USER: mrd
<192.168.1.112> SSH: ansible.cfg set ssh_args: (-C)(-o)(ControlMaster=auto)(-o)(ControlPersist=60s)
<192.168.1.112> SSH: ansible_password/ansible_ssh_password not set: (-o)(KbdInteractiveAuthentication=no)(-o)(PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey)(-o)(PasswordAuthentication=no)
<192.168.1.112> SSH: ANSIBLE_REMOTE_USER/remote_user/ansible_user/user/-u set: (-o)(User="mrd")
<192.168.1.112> SSH: ANSIBLE_TIMEOUT/timeout set: (-o)(ConnectTimeout=10)
<192.168.1.112> SSH: Set ssh_common_args: ()
<192.168.1.112> SSH: Set ssh_extra_args: ()
<192.168.1.112> SSH: found only ControlPersist; added ControlPath: (-o)(ControlPath="/home/mrd/.ansible/cp/1e5ab0657b")
<192.168.1.112> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="mrd"' -o ConnectTimeout=10 -o 'ControlPath="/home/mrd/.ansible/cp/1e5ab0657b"' 192.168.1.112 '/bin/sh -c '"'"'rm -f -r /home/mrd/.ansible/tmp/ansible-tmp-1686658183.5355954-38293-118510171344726/ > /dev/null 2>&1 && sleep 0'"'"''
<192.168.1.112> (0, b'', b"OpenSSH_8.9p1 Ubuntu-3ubuntu0.1, OpenSSL 3.0.2 15 Mar 2022\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files\r\ndebug1: /etc/ssh/ssh_config line 21: Applying options for *\r\ndebug2: resolve_canonicalize: hostname 192.168.1.112 is address\r\ndebug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/home/mrd/.ssh/known_hosts'\r\ndebug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/home/mrd/.ssh/known_hosts2'\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 38192\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n")
fatal: [kubernetes-worker2]: FAILED! => {
    "changed": false,
    "invocation": {
        "module_args": {
            "daemon_reexec": false,
            "daemon_reload": true,
            "enabled": true,
            "force": null,
            "masked": null,
            "name": "k3s-node",
            "no_block": false,
            "scope": "system",
            "state": "restarted"
        }
    },
    "msg": "Unable to start service k3s-node: Job for k3s-node.service failed because the control process exited with error code.\nSee \"systemctl status k3s-node.service\" and \"journalctl -xeu k3s-node.service\" for details.\n"
}
<192.168.1.114> (1, b'\r\n{"failed": true, "msg": "Unable to start service k3s-node: Job for k3s-node.service failed because the control process exited with error code.\\nSee \\"systemctl status k3s-node.service\\" and \\"journalctl -xeu k3s-node.service\\" for details.\\n", "invocation": {"module_args": {"name": "k3s-node", "daemon_reload": true, "state": "restarted", "enabled": true, "daemon_reexec": false, "scope": "system", "no_block": false, "force": null, "masked": null}}}\r\n', b"OpenSSH_8.9p1 Ubuntu-3ubuntu0.1, OpenSSL 3.0.2 15 Mar 2022\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files\r\ndebug1: /etc/ssh/ssh_config line 21: Applying options for *\r\ndebug2: resolve_canonicalize: hostname 192.168.1.114 is address\r\ndebug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/home/mrd/.ssh/known_hosts'\r\ndebug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/home/mrd/.ssh/known_hosts2'\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 38195\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 1\r\nShared connection to 192.168.1.114 closed.\r\n")
<192.168.1.114> Failed to connect to the host via ssh: OpenSSH_8.9p1 Ubuntu-3ubuntu0.1, OpenSSL 3.0.2 15 Mar 2022
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug2: resolve_canonicalize: hostname 192.168.1.114 is address
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/home/mrd/.ssh/known_hosts'
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/home/mrd/.ssh/known_hosts2'
debug1: auto-mux: Trying existing master
debug2: fd 3 setting O_NONBLOCK
debug2: mux_client_hello_exchange: master version 4
debug3: mux_client_forwards: request forwardings: 0 local, 0 remote
debug3: mux_client_request_session: entering
debug3: mux_client_request_alive: entering
debug3: mux_client_request_alive: done pid = 38195
debug3: mux_client_request_session: session request sent
debug1: mux_client_request_session: master session id: 2
debug3: mux_client_read_packet: read header failed: Broken pipe
debug2: Received exit status from master 1
Shared connection to 192.168.1.114 closed.
<192.168.1.114> ESTABLISH SSH CONNECTION FOR USER: mrd
<192.168.1.114> SSH: ansible.cfg set ssh_args: (-C)(-o)(ControlMaster=auto)(-o)(ControlPersist=60s)
<192.168.1.114> SSH: ansible_password/ansible_ssh_password not set: (-o)(KbdInteractiveAuthentication=no)(-o)(PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey)(-o)(PasswordAuthentication=no)
<192.168.1.114> SSH: ANSIBLE_REMOTE_USER/remote_user/ansible_user/user/-u set: (-o)(User="mrd")
<192.168.1.114> SSH: ANSIBLE_TIMEOUT/timeout set: (-o)(ConnectTimeout=10)
<192.168.1.114> SSH: Set ssh_common_args: ()
<192.168.1.114> SSH: Set ssh_extra_args: ()
<192.168.1.114> SSH: found only ControlPersist; added ControlPath: (-o)(ControlPath="/home/mrd/.ansible/cp/70c7f1c985")
<192.168.1.114> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="mrd"' -o ConnectTimeout=10 -o 'ControlPath="/home/mrd/.ansible/cp/70c7f1c985"' 192.168.1.114 '/bin/sh -c '"'"'rm -f -r /home/mrd/.ansible/tmp/ansible-tmp-1686658183.561984-38295-97990290856469/ > /dev/null 2>&1 && sleep 0'"'"''
<192.168.1.114> (0, b'', b"OpenSSH_8.9p1 Ubuntu-3ubuntu0.1, OpenSSL 3.0.2 15 Mar 2022\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files\r\ndebug1: /etc/ssh/ssh_config line 21: Applying options for *\r\ndebug2: resolve_canonicalize: hostname 192.168.1.114 is address\r\ndebug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/home/mrd/.ssh/known_hosts'\r\ndebug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/home/mrd/.ssh/known_hosts2'\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 38195\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n")
fatal: [kubernetes-worker4]: FAILED! => {
    "changed": false,
    "invocation": {
        "module_args": {
            "daemon_reexec": false,
            "daemon_reload": true,
            "enabled": true,
            "force": null,
            "masked": null,
            "name": "k3s-node",
            "no_block": false,
            "scope": "system",
            "state": "restarted"
        }
    },
    "msg": "Unable to start service k3s-node: Job for k3s-node.service failed because the control process exited with error code.\nSee \"systemctl status k3s-node.service\" and \"journalctl -xeu k3s-node.service\" for details.\n"
}

journalctl -xeu k3s-node.service output

░░ Subject: A start job for unit k3s-node.service has failed
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ A start job for unit k3s-node.service has finished with a failure.
░░
░░ The job identifier is 16322 and the job result is failed.
Jun 13 14:20:47 ubuntu6 systemd[1]: k3s-node.service: Scheduled restart job, restart counter is at 65.
░░ Subject: Automatic restarting of a unit has been scheduled
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ Automatic restarting of the unit k3s-node.service has been scheduled, as the result for
░░ the configured Restart= setting for the unit.
Jun 13 14:20:47 ubuntu6 systemd[1]: Stopped Lightweight Kubernetes.
░░ Subject: A stop job for unit k3s-node.service has finished
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ A stop job for unit k3s-node.service has finished.
░░
░░ The job identifier is 16409 and the job result is done.
Jun 13 14:20:47 ubuntu6 systemd[1]: Starting Lightweight Kubernetes...
░░ Subject: A start job for unit k3s-node.service has begun execution
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ A start job for unit k3s-node.service has begun execution.
░░
░░ The job identifier is 16409.
Jun 13 14:20:47 ubuntu6 k3s[8186]: time="2023-06-13T14:20:47Z" level=info msg="Starting k3s agent v1.26.5+k3s1 (7cefebea)"
Jun 13 14:20:47 ubuntu6 k3s[8186]: time="2023-06-13T14:20:47Z" level=info msg="Adding server to load balancer k3s-agent-load-balancer: 192.168.1.110:6443"
Jun 13 14:20:47 ubuntu6 k3s[8186]: time="2023-06-13T14:20:47Z" level=info msg="Adding server to load balancer k3s-agent-load-balancer: 192.168.1.111:6443"
Jun 13 14:20:47 ubuntu6 k3s[8186]: time="2023-06-13T14:20:47Z" level=info msg="Adding server to load balancer k3s-agent-load-balancer: 192.168.1.113:6443"
Jun 13 14:20:47 ubuntu6 k3s[8186]: time="2023-06-13T14:20:47Z" level=info msg="Adding server to load balancer k3s-agent-load-balancer: 192.168.1.115:6443"
Jun 13 14:20:47 ubuntu6 k3s[8186]: time="2023-06-13T14:20:47Z" level=info msg="Removing server from load balancer k3s-agent-load-balancer: 192.168.1.110:6443"
Jun 13 14:20:47 ubuntu6 k3s[8186]: time="2023-06-13T14:20:47Z" level=info msg="Running load balancer k3s-agent-load-balancer 127.0.0.1:6444 -> [192.168.1.111:6443 192.168.1.113:6443 192.168.1.115:6443] [default: 192.168.1.>
lines 1666-1702/1702 (END)

Steps to Reproduce

  1. Create six vms based on ubuntu server 2204
  2. configure inventory
  3. configure all.yaml variables - api server IP, IP range for metallb, ansible user, cluster token, kubernetes version
  4. trigger automation

Context (variables)

Operating system: Ubuntu 22 04

Hardware: Proxmox on Dell Wyse 5070

Variables Used

all.yml

k3s_version: v1.27.2+k3s1
# this is the user that has ssh access to these machines
ansible_user: mrd
systemd_dir: /etc/systemd/system

# Set your timezone
system_timezone: "UTC"

# interface which will be used for flannel
flannel_iface: "eth0"

# apiserver_endpoint is virtual ip-address which will be configured on each master
apiserver_endpoint: "192.168.1.110"

# k3s_token is required  masters can talk together securely
# this token should be alpha numeric only
k3s_token: "rKm!&Ubgc554gZ*r*VPujmcf5PN@pcawYcD%vexJK4uj8R@N7vCzxWGK2W3C4u9&ABs"

# The IP on which the node is reachable in the cluster.
# Here, a sensible default is provided, you can still override
# it for each of your hosts, though.
k3s_node_ip: '{{ ansible_facts[flannel_iface]["ipv4"]["address"] }}'

# Disable the taint manually by setting: k3s_master_taint = false
k3s_master_taint: "{{ true if groups['node'] | default([]) | length >= 1 else false }}"

# these arguments are recommended for servers as well as agents:
extra_args: >-
  --flannel-iface={{ flannel_iface }}
  --node-ip={{ k3s_node_ip }}

# change these to your liking, the only required are: --disable servicelb, --tls-san {{ apiserver_endpoint }}
extra_server_args: >-
  {{ extra_args }}
  {{ '--node-taint node-role.kubernetes.io/master=true:NoSchedule' if k3s_master_taint else '' }}
  --tls-san {{ apiserver_endpoint }}
  --disable servicelb
  --disable traefik
extra_agent_args: >-
  {{ extra_args }}

# image tag for kube-vip
kube_vip_tag_version: "v0.5.12"

# metallb type frr or native
metal_lb_type: "native"

# metallb mode layer2 or bgp
metal_lb_mode: "layer2"

# bgp options
# metal_lb_bgp_my_asn: "64513"
# metal_lb_bgp_peer_asn: "64512"
# metal_lb_bgp_peer_address: "192.168.30.1"

# image tag for metal lb
metal_lb_frr_tag_version: "v7.5.1"
metal_lb_speaker_tag_version: "v0.13.9"
metal_lb_controller_tag_version: "v0.13.9"

# metallb ip range for load balancer
metal_lb_ip_range: "192.168.1.80-192.168.1.90"

# Only enable if your nodes are proxmox LXC nodes, make sure to configure your proxmox nodes
# in your hosts.ini file.
# Please read https://gist.github.com/triangletodd/02f595cd4c0dc9aac5f7763ca2264185 before using this.
# Most notably, your containers must be privileged, and must not have nesting set to true.
# Please note this script disables most of the security of lxc containers, with the trade off being that lxc
# containers are significantly more resource efficent compared to full VMs.
# Mixing and matching VMs and lxc containers is not supported, ymmv if you want to do this.
# I would only really recommend using this if you have partiularly low powered proxmox nodes where the overhead of
# VMs would use a significant portion of your available resources.
proxmox_lxc_configure: false
# the user that you would use to ssh into the host, for example if you run ssh some-user@my-proxmox-host,
# set this value to some-user
proxmox_lxc_ssh_user: root
# the unique proxmox ids for all of the containers in the cluster, both worker and master nodes
proxmox_lxc_ct_ids:
  - 200
  - 201
  - 202
  - 203
  - 204

Hosts

host.ini

kubernetes-master1 ansible_host=192.168.1.111   ansible_user=mrd
kubernetes-master3 ansible_host=192.168.1.113   ansible_user=mrd
kubernetes-master5 ansible_host=192.168.1.115   ansible_user=mrd

[node]
kubernetes-worker2 ansible_host=192.168.1.112   ansible_user=mrd
kubernetes-worker4 ansible_host=192.168.1.114   ansible_user=mrd
kubernetes-worker6 ansible_host=192.168.1.116   ansible_user=mrd

# only required if proxmox_lxc_configure: true
# must contain all proxmox instances that have a master or worker node
# [proxmox]
# 192.168.30.43

[k3s_cluster:children]
master
node

Possible Solution

FrostyFitz commented 1 year ago

I am currently having the same issue. 6 Ubuntu Jammy VMs on proxmox. Stuck in the same spot. My config looks very similar to OP's and I have not changed much from the example. Would love to know what is going wrong here.