Failed to set execute bit on remote files

[FatmanUK]

TASK [k3s_reboot_slow : Wait for all nodes to exist]
 ***************************
fatal: [k3s-mother-01]: FAILED! => {"msg": "Failed to
 set execute bit on remote files (rc: 1, err:
 mux_client_request_session: session request failed:
 Session open refused by peer\r\nWarning:
 Permanently added '10.11.0.1' (ECDSA) to the list
 of known hosts.\r\nControlSocket
 /home/fatman/.ansible/cp/9b11b6834e already
 exists, disabling multiplexing\r\nX11 forwarding
 request failed on channel 0\r\nserver exited
 unexpectedly\n)"}

[FatmanUK]

[WARNING]: Error deleting remote temporary files
 (rc: 1, stderr: X11 forwarding

Related?

[FatmanUK]

fatal: [k3s-child-06 -> k3s-mother-01]:
 UNREACHABLE! => {"changed": false, "msg": "Failed
 to create temporary directory. In some cases, you
 may have been able to authenticate and did not
 have permissions on the target directory. Consider
 changing the remote tmp path in ansible.cfg to a
 path rooted in \"/tmp\", for more error information
 use -vvv. Failed command was: ( umask 77 &&
 mkdir -p \"` echo /tmp `\"&& mkdir \"` echo
 /tmp/ansible-tmp-1709590387.7094378-21535-2
5183337048722 `\" && echo ansible-
tmp-1709590387.7094378-21535-25183337048722=
\"` echo /tmp/ansible-tmp-1709590387.7094378-2153
5-25183337048722 `\" ), exited with result 1",
 "unreachable": true}

Related?

[FatmanUK]

fatal: [k3s-child-01 -> k3s-mother-01({{ network
 | ipv4(ip_fourth_byte) | ipv4("address") }})]:
 FAILED! => {"attempts": 60, "changed": true, "cmd":
 ["/usr/local/bin/kubectl", "get", "node",
 "k3s-child-01.dreamtrack.test"], "delta":
 "0:00:00.123127", "end": "2024-03-04 22:20:27.
834238", "msg": "non-zero return code", "rc": 1,
 "start": "2024-03-04 22:20:27.711111", "stderr":
 "Error from server (NotFound): nodes
 \"k3s-child-01.dreamtrack.test\" not found",
 "stderr_lines": ["Error from server (NotFound):
 nodes \"k3s-child-01.dreamtrack.test\" not
 found"], "stdout": "", "stdout_lines": []}

Related?

[athajaerel]

TASK [k3s_reboot_slow : Wait for all nodes to exist]
 ******************************
fatal: [k3s-mother-02 -> k3s-mother-01]: UNREACHABLE!
 => {"changed": false, "msg": "Failed to create temporary
 directory. In some cases, you may have been able to
 authenticate and did not have permissions on the target directory.
 Consider changing the remote tmp path in ansible.cfg to a path
 rooted in \"/tmp\", for more error information use -vvv. Failed
 command was: ( umask 77 && mkdir -p \"` echo /tmp `\"&&
 mkdir \"` echo /tmp/ansible-tmp-1709631714.6867297-1921279
-155302194164978 `\" && echo ansible-tmp-1709631714.6867297
-1921279-155302194164978=\"` echo /tmp/ansible-tmp
-1709631714.6867297-1921279-155302194164978 `\" ), exited
 with result 1", "unreachable": true}

At work site. Interesting.

[FatmanUK]

TASK [k3s_reboot_slow : Wait for all nodes to exist]
 ***************************
fatal: [k3s-child-03 -> k3s-mother-01]: FAILED! =>
 {"msg": "Failed to set execute bit on remote files
 (rc: 1, err: X11 forwarding request failed\r\nserver
 exited unexpectedly\n)"}
...
fatal: [k3s-child-04 -> k3s-mother-01]: FAILED! =>
 {"msg": "Failed to set execute bit on remote files
 (rc: 1, err: X11 forwarding request failed\r\nserver
 exited unexpectedly\n)"}

[FatmanUK]

For tasks that wait or run for a long time without producing output, I found it necessary to add -o ServerAliveInterval=50 to ssh parameters to enable keepalives.

[FatmanUK]

pipelining

[FatmanUK]

Reboot task might be "racy": https://github.com/ostreedev/ostree/pull/1548

[athajaerel]

Can't be the reboot module --- it's the "wait for nodes" task which is a command.

- name: 'Wait for all nodes to exist'
  register: 'reg_nodes'
  retries: 120
  delay: 20
  until: 'reg_nodes is success'
  delegate_to: '{{ groups.k3s_primary_masters | first }}'
  command:
    argv:
      - '{{ k3s_binary_dir }}/kubectl'
      - 'get'
      - 'node'
      - '{{ fqdn }}'

Would be fixed by #175 , most likely.

[athajaerel]

Not forgetting to fork off a ticket for the outputs which don't fit...

[FatmanUK]

https://github.com/containers/ansible-podman-collections/issues/389 https://github.com/ansible/ansible/issues/40434 https://stackoverflow.com/questions/52496020/mux-client-request-session-session-request-failed-session-open-refused-by-peer