search

andreygubarev / molecule-qemu

Molecule QEMU driver for testing Ansible roles
https://pypi.org/project/molecule-qemu/
MIT License
11 stars 8 forks source link

SSH shows as disabled on Ubuntu 24.04 hosts. cloud-init sudo issue #43

Open aitorpazos opened 1 month ago

aitorpazos commented 1 month ago

Trying to use molecule-qemu to test on Ubuntu 24.04 it reports UNREACHABLE on converge start:

...
TASK [Prepare VMs config dict] *************************************************
ok: [localhost] => (item=service-exporter-default-ubuntu2404-amd64)
ok: [localhost] => (item=service-exporter-default-ubuntu2204-amd64)
ok: [localhost] => (item=service-exporter-default-ubuntu2004-amd64)

TASK [Prepare VMs config list] *************************************************
ok: [localhost]

TASK [Dump VMs config] *********************************************************
changed: [localhost]

PLAY RECAP *********************************************************************
localhost                  : ok=34   changed=11   unreachable=0    failed=0    skipped=5    rescued=0    ignored=0

INFO     Running default > prepare
WARNING  Skipping, prepare playbook not configured.
INFO     Running default > converge

PLAY [MOLECULE CONVERGE] *******************************************************

TASK [Gathering Facts] *********************************************************
fatal: [service-exporter-default-ubuntu2404-amd64]: UNREACHABLE! => {
    "changed": false,
    "unreachable": true
}

MSG:

Failed to connect to the host via ssh: 
ok: [service-exporter-default-ubuntu2004-amd64]
ok: [service-exporter-default-ubuntu2204-amd64]

TASK [set_fact] ****************************************************************
ok: [service-exporter-default-ubuntu2004-amd64]
ok: [service-exporter-default-ubuntu2204-amd64]
...

If I run converge after some time, it becomes reachable, so there seems to be some sort of race condition on 24.04.

Checking ssh service on 22.04 vs 24.04, I noticed ssh service shows as disabled (but running):

# 24.04
root@service-exporter-default-ubuntu2404-amd64:~# systemctl status ssh
● ssh.service - OpenBSD Secure Shell server
     Loaded: loaded (/usr/lib/systemd/system/ssh.service; disabled; preset: enabled)
     Active: active (running) since Thu 2024-08-01 09:38:11 UTC; 18min ago
TriggeredBy: ● ssh.socket
       Docs: man:sshd(8)
             man:sshd_config(5)
    Process: 966 ExecStartPre=/usr/sbin/sshd -t (code=exited, status=0/SUCCESS)
   Main PID: 967 (sshd)
      Tasks: 1 (limit: 1106)
     Memory: 2.2M (peak: 2.8M)
        CPU: 305ms
     CGroup: /system.slice/ssh.service
             └─967 "sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups"

Aug 01 09:38:11 service-exporter-default-ubuntu2404-amd64 systemd[1]: Starting ssh.service - OpenBSD Secure Shell server...
Aug 01 09:38:11 service-exporter-default-ubuntu2404-amd64 sshd[967]: Server listening on :: port 22.
Aug 01 09:38:11 service-exporter-default-ubuntu2404-amd64 systemd[1]: Started ssh.service - OpenBSD Secure Shell server.
Aug 01 09:38:54 service-exporter-default-ubuntu2404-amd64 sshd[1029]: Accepted publickey for ansible from 10.0.2.2 port 53113 ssh2: ED25519 SHA256:tgi>
Aug 01 09:38:54 service-exporter-default-ubuntu2404-amd64 sshd[1029]: pam_unix(sshd:session): session opened for user ansible(uid=1000) by ansible(uid>

# 22.04
root@service-exporter-default-ubuntu2204-amd64:~# systemctl status ssh
● ssh.service - OpenBSD Secure Shell server
     Loaded: loaded (/lib/systemd/system/ssh.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2024-08-01 09:38:27 UTC; 18min ago
       Docs: man:sshd(8)
             man:sshd_config(5)
   Main PID: 1416 (sshd)
      Tasks: 1 (limit: 1095)
     Memory: 4.3M
        CPU: 294ms
     CGroup: /system.slice/ssh.service
             └─1416 "sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups"

Aug 01 09:38:26 service-exporter-default-ubuntu2204-amd64 systemd[1]: Starting OpenBSD Secure Shell server...
Aug 01 09:38:26 service-exporter-default-ubuntu2204-amd64 sshd[1416]: Server listening on 0.0.0.0 port 22.
Aug 01 09:38:26 service-exporter-default-ubuntu2204-amd64 sshd[1416]: Server listening on :: port 22.
Aug 01 09:38:27 service-exporter-default-ubuntu2204-amd64 systemd[1]: Started OpenBSD Secure Shell server.
Aug 01 09:44:33 service-exporter-default-ubuntu2204-amd64 sshd[4803]: Accepted publickey for ansible from 10.0.2.2 port 53914 ssh2: ED25519 SHA256:tgiv+1a>
Aug 01 09:44:33 service-exporter-default-ubuntu2204-amd64 sshd[4803]: pam_unix(sshd:session): session opened for user ansible(uid=1000) by (uid=0)

24.04 ssh related logs

root@service-exporter-default-ubuntu2404-amd64:~# journalctl | grep ssh
Aug 01 09:38:03 service-exporter-default-ubuntu2404-amd64 cloud-init[664]: Your identification has been saved in /etc/ssh/ssh_host_rsa_key
Aug 01 09:38:03 service-exporter-default-ubuntu2404-amd64 cloud-init[664]: Your public key has been saved in /etc/ssh/ssh_host_rsa_key.pub
Aug 01 09:38:03 service-exporter-default-ubuntu2404-amd64 cloud-init[664]: Your identification has been saved in /etc/ssh/ssh_host_ecdsa_key
Aug 01 09:38:03 service-exporter-default-ubuntu2404-amd64 cloud-init[664]: Your public key has been saved in /etc/ssh/ssh_host_ecdsa_key.pub
Aug 01 09:38:03 service-exporter-default-ubuntu2404-amd64 cloud-init[664]: Your identification has been saved in /etc/ssh/ssh_host_ed25519_key
Aug 01 09:38:03 service-exporter-default-ubuntu2404-amd64 cloud-init[664]: Your public key has been saved in /etc/ssh/ssh_host_ed25519_key.pub
Aug 01 09:38:03 service-exporter-default-ubuntu2404-amd64 systemd[1]: Listening on ssh.socket - OpenBSD Secure Shell server socket.
Aug 01 09:38:08 service-exporter-default-ubuntu2404-amd64 systemd[1]: Starting ssh.service - OpenBSD Secure Shell server...
Aug 01 09:38:09 service-exporter-default-ubuntu2404-amd64 sshd[937]: Server listening on :: port 22.
Aug 01 09:38:09 service-exporter-default-ubuntu2404-amd64 systemd[1]: Started ssh.service - OpenBSD Secure Shell server.
Aug 01 09:38:09 service-exporter-default-ubuntu2404-amd64 sshd[941]: Connection closed by 10.0.2.2 port 52969
Aug 01 09:38:11 service-exporter-default-ubuntu2404-amd64 sshd[961]: fatal: Access denied for user ansible by PAM account configuration [preauth]
Aug 01 09:38:11 service-exporter-default-ubuntu2404-amd64 sshd[937]: Received signal 15; terminating.
Aug 01 09:38:11 service-exporter-default-ubuntu2404-amd64 systemd[1]: Stopping ssh.service - OpenBSD Secure Shell server...
Aug 01 09:38:11 service-exporter-default-ubuntu2404-amd64 systemd[1]: ssh.service: Deactivated successfully.
Aug 01 09:38:11 service-exporter-default-ubuntu2404-amd64 systemd[1]: Stopped ssh.service - OpenBSD Secure Shell server.
Aug 01 09:38:11 service-exporter-default-ubuntu2404-amd64 systemd[1]: Starting ssh.service - OpenBSD Secure Shell server...
Aug 01 09:38:11 service-exporter-default-ubuntu2404-amd64 sshd[967]: Server listening on :: port 22.
Aug 01 09:38:11 service-exporter-default-ubuntu2404-amd64 systemd[1]: Started ssh.service - OpenBSD Secure Shell server.
Aug 01 09:38:54 service-exporter-default-ubuntu2404-amd64 sshd[1029]: Accepted publickey for ansible from 10.0.2.2 port 53113 ssh2: ED25519 SHA256:tgiv+1a4CmVgYkrfzvaiR8N87t1xTbE2GNuyY7kv4k0
Aug 01 09:38:54 service-exporter-default-ubuntu2404-amd64 sshd[1029]: pam_unix(sshd:session): session opened for user ansible(uid=1000) by ansible(uid=0)
Aug 01 09:38:56 service-exporter-default-ubuntu2404-amd64 systemd[1034]: Starting gpg-agent-ssh.socket - GnuPG cryptographic agent (ssh-agent emulation)...
Aug 01 09:38:56 service-exporter-default-ubuntu2404-amd64 systemd[1034]: Listening on gpg-agent-ssh.socket - GnuPG cryptographic agent (ssh-agent emulation).

22.04 ssh related logs

root@service-exporter-default-ubuntu2204-amd64:~# journalctl | grep ssh
Aug 01 09:38:01 service-exporter-default-ubuntu2204-amd64 cloud-init[525]: Your identification has been saved in /etc/ssh/ssh_host_rsa_key
Aug 01 09:38:01 service-exporter-default-ubuntu2204-amd64 cloud-init[525]: Your public key has been saved in /etc/ssh/ssh_host_rsa_key.pub
Aug 01 09:38:01 service-exporter-default-ubuntu2204-amd64 cloud-init[525]: Your identification has been saved in /etc/ssh/ssh_host_ecdsa_key
Aug 01 09:38:01 service-exporter-default-ubuntu2204-amd64 cloud-init[525]: Your public key has been saved in /etc/ssh/ssh_host_ecdsa_key.pub
Aug 01 09:38:01 service-exporter-default-ubuntu2204-amd64 cloud-init[525]: Your identification has been saved in /etc/ssh/ssh_host_ed25519_key
Aug 01 09:38:01 service-exporter-default-ubuntu2204-amd64 cloud-init[525]: Your public key has been saved in /etc/ssh/ssh_host_ed25519_key.pub
Aug 01 09:38:08 service-exporter-default-ubuntu2204-amd64 sshd[758]: Server listening on 0.0.0.0 port 22.
Aug 01 09:38:08 service-exporter-default-ubuntu2204-amd64 sshd[758]: Server listening on :: port 22.
Aug 01 09:38:09 service-exporter-default-ubuntu2204-amd64 sshd[776]: error: kex_exchange_identification: Connection closed by remote host
Aug 01 09:38:09 service-exporter-default-ubuntu2204-amd64 sshd[776]: Connection closed by 10.0.2.2 port 52985
Aug 01 09:38:10 service-exporter-default-ubuntu2204-amd64 sshd[783]: Accepted publickey for ansible from 10.0.2.2 port 52993 ssh2: ED25519 SHA256:tgiv+1a4CmVgYkrfzvaiR8N87t1xTbE2GNuyY7kv4k0
Aug 01 09:38:10 service-exporter-default-ubuntu2204-amd64 sshd[783]: pam_unix(sshd:session): session opened for user ansible(uid=1000) by (uid=0)
Aug 01 09:38:11 service-exporter-default-ubuntu2204-amd64 systemd[789]: Listening on GnuPG cryptographic agent (ssh-agent emulation).
Aug 01 09:38:26 service-exporter-default-ubuntu2204-amd64 sshd[758]: Received signal 15; terminating.
Aug 01 09:38:26 service-exporter-default-ubuntu2204-amd64 systemd[1]: ssh.service: Deactivated successfully.
Aug 01 09:38:26 service-exporter-default-ubuntu2204-amd64 sshd[1416]: Server listening on 0.0.0.0 port 22.
Aug 01 09:38:26 service-exporter-default-ubuntu2204-amd64 sshd[1416]: Server listening on :: port 22.
Aug 01 09:42:19 service-exporter-default-ubuntu2204-amd64 python3.10[4247]: ansible-docker_login ...
aitorpazos commented 1 month ago

Looking at the timings, it seems that on 24.04 cloud-init is not yet done when converge starts:

root@service-exporter-default-ubuntu2404-amd64:~# cloud-init status --long
status: done
extended_status: degraded done
boot_status_code: enabled-by-generator
last_update: Thu, 01 Aug 2024 11:45:58 +0000
detail: DataSourceNoCloud [seed=/dev/sr0][dsmode=net]
errors: []
recoverable_errors:
WARNING:
        - Invalid cloud-config provided: Please run 'sudo cloud-init schema --system' to see the schema errors.
root@service-exporter-default-ubuntu2404-amd64:~# cloud-init schema --system
Found cloud-config data types: user-data, network-config

1. user-data at /var/lib/cloud/instances/service-exporter-default-ubuntu2404-amd64/cloud-config.txt:
  Invalid user-data /var/lib/cloud/instances/service-exporter-default-ubuntu2404-amd64/cloud-config.txt
  Error: Cloud config schema errors: users.1.sudo: ['ALL=(ALL) NOPASSWD:ALL'] is not of type 'boolean', users.1.sudo: ['ALL=(ALL) NOPASSWD:ALL'] is not of type 'string', 'null'

2. network-config at /var/lib/cloud/instances/service-exporter-default-ubuntu2404-amd64/network-config.json:
  Valid schema network-config
Error: Invalid schema: user-data

Ubuntu 24.04 is shipping with a bug fixed in https://github.com/canonical/cloud-init/pull/5418