Cannot connect to a server via ssh after creating them

[torsina]

Summary

I have a first play that creates a set of node to create a k8 cluster, the first play takes care of creting the servers (hetzner), as well as using add_host for the next play to use said servers

but said servers refuses to connect to said servers via ssh while I can ssh into those servers at the same time if I type the command ssh manually the first action that Ansible wants to do with said server leads to a crash, the server be it in rescue mode or not. I tried running the playbook in sudo, but that didn't change anything

I an running on a bootstrap server from hetzner as to have a reproducible environement, I copied my ssh key on that server

Issue Type

Bug Report

Component Name

hcloud

Ansible Version

$ ansible [core 2.16.3]
  config file = /root/self-host/ansible-playbook/ansible.cfg
  configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3/dist-packages/ansible
  ansible collection location = /root/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/bin/ansible
  python version = 3.12.3 (main, Apr 10 2024, 05:33:47) [GCC 13.2.0] (/usr/bin/python3)
  jinja version = 3.1.2
  libyaml = True

Configuration

CONFIG_FILE() = /root/self-host/ansible-playbook/ansible.cfg
DEFAULT_HOST_LIST(/root/self-host/ansible-playbook/ansible.cfg) = ['/root/self-host/ansible-playbook/inventory.ini']
DEFAULT_PRIVATE_KEY_FILE(/root/self-host/ansible-playbook/ansible.cfg) = /home/aelis/self-host/ansible-playbook/id_ed25519.pem
DEPRECATION_WARNINGS(/root/self-host/ansible-playbook/ansible.cfg) = True
ENABLE_TASK_DEBUGGER(/root/self-host/ansible-playbook/ansible.cfg) = True
HOST_KEY_CHECKING(/root/self-host/ansible-playbook/ansible.cfg) = False
INTERPRETER_PYTHON(/root/self-host/ansible-playbook/ansible.cfg) = auto_silent

CONNECTION:
==========

paramiko_ssh:
____________
host_key_checking(/root/self-host/ansible-playbook/ansible.cfg) = False
private_key_file(/root/self-host/ansible-playbook/ansible.cfg) = /home/aelis/self-host/ansible-playbook/id_ed25519.pem

ssh:
___
host_key_checking(/root/self-host/ansible-playbook/ansible.cfg) = False
private_key_file(/root/self-host/ansible-playbook/ansible.cfg) = /home/aelis/self-host/ansible-playbook/id_ed25519.pem

OS / Environment

Ubuntu 22

Steps to Reproduce

very long gist, removed secrets like private keys or vault secrets https://gist.github.com/torsina/0b9ed4c1784c81b55de4d7e387de9cdf

Expected Results

That the second play can connect via ssh to those newly created servers and install arch linux accordingly for now and then continue to develop roles to install k8

Actual Results

Error with -vvvv

TASK [install_arch : Downlad ISO] *************************************************************************************************************************************
task path: /root/self-host/ansible-playbook/roles/install_arch/tasks/main.yaml:1
<49.12.4.129> ESTABLISH SSH CONNECTION FOR USER: root
<49.12.4.129> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'IdentityFile="./group_vars/all/id_ed25519.pem"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="root"' -o ConnectTimeout=10 -o 'ControlPath="/root/.ansible/cp/aa19e5cbe1"' 49.12.4.129 '/bin/sh -c '"'"'echo ~root && sleep 0'"'"''
<49.12.4.129> (255, b'', b'OpenSSH_9.6p1 Ubuntu-3ubuntu13, OpenSSL 3.0.13 30 Jan 2024\r\ndebug1: Reading configuration data /root/.ssh/config\r\ndebug1: /root/.ssh/config line 1: Applying options for *\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files\r\ndebug1: /etc/ssh/ssh_config line 21: Applying options for *\r\ndebug2: resolve_canonicalize: hostname 49.12.4.129 is address\r\ndebug3: expanded UserKnownHostsFile \'~/.ssh/known_hosts\' -> \'/root/.ssh/known_hosts\'\r\ndebug3: expanded UserKnownHostsFile \'~/.ssh/known_hosts2\' -> \'/root/.ssh/known_hosts2\'\r\ndebug1: auto-mux: Trying existing master at \'/root/.ansible/cp/aa19e5cbe1\'\r\ndebug1: Control socket "/root/.ansible/cp/aa19e5cbe1" does not exist\r\ndebug3: channel_clear_timeouts: clearing\r\ndebug3: ssh_connect_direct: entering\r\ndebug1: Connecting to 49.12.4.129 [49.12.4.129] port 22.\r\ndebug3: set_sock_tos: set socket 3 IP_TOS 0x10\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug1: connect to address 49.12.4.129 port 22: No route to host\r\nssh: connect to host 49.12.4.129 port 22: No route to host\r\n')
fatal: [49.12.4.129]: UNREACHABLE! => {
    "changed": false,
    "msg": "Failed to connect to the host via ssh: OpenSSH_9.6p1 Ubuntu-3ubuntu13, OpenSSL 3.0.13 30 Jan 2024\r\ndebug1: Reading configuration data /root/.ssh/config\r\ndebug1: /root/.ssh/config line 1: Applying options for *\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files\r\ndebug1: /etc/ssh/ssh_config line 21: Applying options for *\r\ndebug2: resolve_canonicalize: hostname 49.12.4.129 is address\r\ndebug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/root/.ssh/known_hosts'\r\ndebug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/root/.ssh/known_hosts2'\r\ndebug1: auto-mux: Trying existing master at '/root/.ansible/cp/aa19e5cbe1'\r\ndebug1: Control socket \"/root/.ansible/cp/aa19e5cbe1\" does not exist\r\ndebug3: channel_clear_timeouts: clearing\r\ndebug3: ssh_connect_direct: entering\r\ndebug1: Connecting to 49.12.4.129 [49.12.4.129] port 22.\r\ndebug3: set_sock_tos: set socket 3 IP_TOS 0x10\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug1: connect to address 49.12.4.129 port 22: No route to host\r\nssh: connect to host 49.12.4.129 port 22: No route to host",
    "unreachable": true
}```

Code of Conduct

• [X] I agree to follow the Ansible Code of Conduct

[torsina]

If it helps I am on a bootstrap node also on hetzner on ubuntu that I haven't touched apart from installing ansible, git and setting up my ssh key

[jooola]

Are you waiting for the server to be booted and listening for connections before tying to connect to it?

I recommend adding the following task at the end of a playbook that creates servers:

    - name: Wait to become reachable
      ansible.builtin.wait_for_connection:

You could replace the following task with what I suggested above:

- name: Ensure the server is started
  async: 45
  poll: 0
  hetzner.hcloud.server:
    name: "{{item.server_name}}"
    state: started
    api_token: "{{item.api_token}}"