Unable to connect after run & Amazon heath checks fail

[four43]

Describe the Issue

After running the playbook I restart the instance and access it. If I take an AMI of the instance and try and run it again however, it won't start properly.

After running:

sudo ANSIBLE_HOST_KEY_CHECKING=False ansible-playbook \
    -i localhost, --connection=local \
    site.yml \
    -e os_gpg_key_pubkey_name=gpg-pubkey-d832c631-6515c85e \
    -e amzn2023cis_syslog_service=journald \
    --tags level1-server \
    --skip-tags rule_1.2.4,rule_4.6.6 | tee cis-ansible-harden.log

I can pull logs from the instance that is failing:

Boot Log

``` [=3h[=3h[=3h[=3h Booting `Amazon Linux (6.1.92-99.174.amzn2023.x86_64) 2023' [ 0.071111] RETBleed: WARNING: Spectre v2 mitigation leaves CPU vulnerable to RETBleed attacks, data leaks possible! [ 11.677731] kauditd_printk_skb: 37 callbacks suppressed [ 11.677733] audit: type=1305 audit(1718920166.950:71): op=set audit_enabled=1 old=1 auid=4294967295 ses=4294967295 subj=system_u:system_r:syslogd_t:s0 res=1 [ 11.679489] audit: type=1300 audit(1718920166.950:71): arch=c000003e syscall=46 success=yes exit=60 a0=3 a1=7ffdb7634340 a2=4000 a3=7ffdb76343cc items=0 ppid=1 pid=833 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="systemd-journal" exe="/usr/lib/systemd/systemd-journald" subj=system_u:system_r:syslogd_t:s0 key=(null) [ 11.683252] audit: type=1327 audit(1718920166.950:71): proctitle="/usr/lib/systemd/systemd-journald" [ 11.688457] systemd[1]: Started systemd-journald.service - Journal Service. [ 11.691369] audit: type=1130 audit(1718920166.960:72): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-journald comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 11.746154] systemd-journald[833]: Received client request to flush runtime journal. [ 11.796820] audit: type=1130 audit(1718920167.070:73): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-sysctl comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 12.015400] audit: type=1130 audit(1718920167.290:74): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-sysusers comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 12.025959] systemd-journald[833]: Data hash table of /var/log/journal/ec25f52d066115e854db78d38b68bbcc/system.journal has a fill level at 78.5 (1785 of 2275 items, 1310720 file size, 734 bytes per hash table item), suggesting rotation. [ 12.028004] systemd-journald[833]: /var/log/journal/ec25f52d066115e854db78d38b68bbcc/system.journal: Journal header limits reached or header out-of-date, rotating. [ 12.105780] audit: type=1130 audit(1718920167.380:75): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-journal-flush comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 12.147423] audit: type=1130 audit(1718920167.420:76): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-tmpfiles-setup-dev comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 12.176941] audit: type=1130 audit(1718920167.450:77): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=dracut-shutdown comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 12.423940] audit: type=1130 audit(1718920167.700:78): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-machine-id-commit comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 13.071902] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 13.112759] ACPI: button: Power Button [PWRF] [ 13.113313] input: Sleep Button as /devices/LNXSYSTM:00/LNXSLPBN:00/input/input1 [ 13.114258] ACPI: button: Sleep Button [SLPF] [ 13.140928] cryptd: max_cpu_qlen set to 1000 [ 13.142801] pps_core: LinuxPPS API ver. 1 registered [ 13.143429] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 13.146413] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 13.147349] i8042: Warning: Keylock active [ 13.148826] PTP clock support registered [ 13.159199] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 13.159739] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 13.230539] AVX2 version of gcm_enc/dec engaged. [ 13.231253] AES CTR mode by8 optimization enabled [ 13.231933] ena 0000:00:05.0: Elastic Network Adapter (ENA) v2.12.0g [ 13.242607] ena 0000:00:05.0: ENA device version: 0.10 [ 13.243190] ena 0000:00:05.0: ENA controller version: 0.0.1 implementation version 1 [ 13.322668] ena 0000:00:05.0: LLQ is not supported Fallback to host mode policy. [ 13.334765] ena 0000:00:05.0: Elastic Network Adapter (ENA) found at mem c0400000, mac addr 16:ff:ef:c7:8f:25 [ 13.432768] ena 0000:00:05.0 ens5: renamed from eth0 [ 13.685470] zram_generator::config[1496]: zram0: system has too much memory (7811MB), limit is 800MB, ignoring. [ 15.322988] ena 0000:00:05.0 ens5: Local page cache is disabled for less than 16 channels [FAILED] Failed to start dbus-broke…ce - D-Bus System Message Bus. [FAILED] Failed to start systemd-ho…d.service - Home Area Manager. [FAILED] Failed to start systemd-lo…rvice - User Login Management. [FAILED] Failed to start dbus-broke…ce - D-Bus System Message Bus. [FAILED] Failed to start policy-rou…m - Set up policy routes for ens5. [FAILED] Failed to start systemd-ne…Wait for Network to be Configured. [ 135.957908] cloud-init[3385]: Cloud-init v. 22.2.2 running 'init' at Thu, 20 Jun 2024 21:51:31 +0000. Up 135.92 seconds. [ 136.047154] cloud-init[3385]: ci-info: ++++++++++++++++++++++++++++++++++++Net device info+++++++++++++++++++++++++++++++++++++ [ 136.048922] cloud-init[3385]: ci-info: +--------+------+------------------------------+-----------+-------+-------------------+ [ 136.050499] cloud-init[3385]: ci-info: | Device | Up | Address | Mask | Scope | Hw-Address | [ 136.052055] cloud-init[3385]: ci-info: +--------+------+------------------------------+-----------+-------+-------------------+ [ 136.053562] cloud-init[3385]: ci-info: | ens5 | True | fe80::14ff:efff:fec7:8f25/64 | . | link | 16:ff:ef:c7:8f:25 | [ 136.055157] cloud-init[3385]: ci-info: | lo | True | 127.0.0.1 | 255.0.0.0 | host | . | [ 136.056710] cloud-init[3385]: ci-info: +--------+------+------------------------------+-----------+-------+-------------------+ [ 136.058206] cloud-init[3385]: ci-info: +++++++++++++++++++Route IPv6 info+++++++++++++++++++ [ 136.059320] cloud-init[3385]: ci-info: +-------+-------------+---------+-----------+-------+ [ 136.060500] cloud-init[3385]: ci-info: | Route | Destination | Gateway | Interface | Flags | [ 136.061734] cloud-init[3385]: ci-info: +-------+-------------+---------+-----------+-------+ [ 136.062895] cloud-init[3385]: ci-info: | 0 | fe80::/64 | :: | ens5 | U | [ 136.064000] cloud-init[3385]: ci-info: | 1 | local | :: | ens5 | U | [ 136.065106] cloud-init[3385]: ci-info: | 2 | multicast | :: | ens5 | U | [ 136.066211] cloud-init[3385]: ci-info: +-------+-------------+---------+-----------+-------+ [ 136.221822] cloud-init[3385]: 2024-06-20 21:51:31,506 - url_helper.py[WARNING]: Exception(s) [UrlError("HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 101] Network is unreachable'))"), UrlError("HTTPConnectionPool(host='fd00:ec2::254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: [Errno -9] Address family for hostname not supported'))")] during request to http://[fd00:ec2::254]:80/latest/api/token, raising last exception [ 136.232889] cloud-init[3385]: 2024-06-20 21:51:31,506 - url_helper.py[WARNING]: Calling 'None' failed [0/120s]: request error [HTTPConnectionPool(host='fd00:ec2::254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: [Errno -9] Address family for hostname not supported'))] [ 136.633570] cloud-init[3385]: 2024-06-20 21:51:31,917 - url_helper.py[WARNING]: Exception(s) [UrlError("HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 101] Network is unreachable'))"), UrlError("HTTPConnectionPool(host='fd00:ec2::254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: [Errno -9] Address family for hostname not supported'))")] during request to http://[fd00:ec2::254]:80/latest/api/token, raising last exception [ 136.642945] cloud-init[3385]: 2024-06-20 21:51:31,918 - url_helper.py[WARNING]: Calling 'None' failed [0/120s]: request error [HTTPConnectionPool(host='fd00:ec2::254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: [Errno -9] Address family for hostname not supported'))] [ 137.037221] cloud-init[3385]: 2024-06-20 21:51:32,321 - url_helper.py[WARNING]: Exception(s) [UrlError("HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 101] Network is unreachable'))"), UrlError("HTTPConnectionPool(host='fd00:ec2::254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: [Errno -9] Address family for hostname not supported'))")] during request to http://[fd00:ec2::254]:80/latest/api/token, raising last exception [ 137.046367] cloud-init[3385]: 2024-06-20 21:51:32,321 - url_helper.py[WARNING]: Calling 'None' failed [0/120s]: request error [HTTPConnectionPool(host='fd00:ec2::254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: [Errno -9] Address family for hostname not supported'))] [ 137.439805] cloud-init[3385]: 2024-06-20 21:51:32,724 - url_helper.py[WARNING]: Exception(s) [UrlError("HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 101] Network is unreachable'))"), UrlError("HTTPConnectionPool(host='fd00:ec2::254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: [Errno -9] Address family for hostname not supported'))")] during request to http://[fd00:ec2::254]:80/latest/api/token, raising last exception [ 137.448930] cloud-init[3385]: 2024-06-20 21:51:32,724 - url_helper.py[WARNING]: Calling 'None' failed [1/120s]: request error [HTTPConnectionPool(host='fd00:ec2::254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: [Errno -9] Address family for hostname not supported'))] [ 137.851360] cloud-init[3385]: 2024-06-20 21:51:33,135 - url_helper.py[WARNING]: Exception(s) [UrlError("HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 101] Network is unreachable'))"), UrlError("HTTPConnectionPool(host='fd00:ec2::254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: [Errno -9] Address family for hostname not supported'))")] during request to http://[fd00:ec2::254]:80/latest/api/token, raising last exception [ 137.860560] cloud-init[3385]: 2024-06-20 21:51:33,135 - url_helper.py[WARNING]: Calling 'None' failed [1/120s]: request error [HTTPConnectionPool(host='fd00:ec2::254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: [Errno -9] Address family for hostname not supported'))] [ 138.254861] cloud-init[3385]: 2024-06-20 21:51:33,538 - url_helper.py[WARNING]: Exception(s) [UrlError("HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 101] Network is unreachable'))"), UrlError("HTTPConnectionPool(host='fd00:ec2::254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: [Errno -9] Address family for hostname not supported'))")] during request to http://[fd00:ec2::254]:80/latest/api/token, raising last exception [ 138.263932] cloud-init[3385]: 2024-06-20 21:51:33,539 - url_helper.py[WARNING]: Calling 'None' failed [2/120s]: request error [HTTPConnectionPool(host='fd00:ec2::254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: [Errno -9] Address family for hostname not supported'))] [ 139.659081] cloud-init[3385]: 2024-06-20 21:51:34,942 - url_helper.py[WARNING]: Exception(s) [UrlError("HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 101] Network is unreachable'))"), UrlError("HTTPConnectionPool(host='fd00:ec2::254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: [Errno -9] Address family for hostname not supported'))")] during request to http://[fd00:ec2::254]:80/latest/api/token, raising last exception [ 139.668717] cloud-init[3385]: 2024-06-20 21:51:34,943 - url_helper.py[WARNING]: Calling 'None' failed [3/120s]: request error [HTTPConnectionPool(host='fd00:ec2::254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: [Errno -9] Address family for hostname not supported'))] [ 142.062814] cloud-init[3385]: 2024-06-20 21:51:37,346 - url_helper.py[WARNING]: Exception(s) [UrlError("HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 101] Network is unreachable'))"), UrlError("HTTPConnectionPool(host='fd00:ec2::254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: [Errno -9] Address family for hostname not supported'))")] during request to http://[fd00:ec2::254]:80/latest/api/token, raising last exception [ 142.072069] cloud-init[3385]: 2024-06-20 21:51:37,346 - url_helper.py[WARNING]: Calling 'None' failed [5/120s]: request error [HTTPConnectionPool(host='fd00:ec2::254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: [Errno -9] Address family for hostname not supported'))] [ 145.469382] cloud-init[3385]: 2024-06-20 21:51:40,753 - url_helper.py[WARNING]: Exception(s) [UrlError("HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 101] Network is unreachable'))"), UrlError("HTTPConnectionPool(host='fd00:ec2::254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: [Errno -9] Address family for hostname not supported'))")] during request to http://[fd00:ec2::254]:80/latest/api/token, raising last exception [ 145.478307] cloud-init[3385]: 2024-06-20 21:51:40,753 - url_helper.py[WARNING]: Calling 'None' failed [9/120s]: request error [HTTPConnectionPool(host='fd00:ec2::254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: [Errno -9] Address family for hostname not supported'))] [ 149.874896] cloud-init[3385]: 2024-06-20 21:51:45,158 - url_helper.py[WARNING]: Exception(s) [UrlError("HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 101] Network is unreachable'))"), UrlError("HTTPConnectionPool(host='fd00:ec2::254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: [Errno -9] Address family for hostname not supported'))")] during request to http://[fd00:ec2::254]:80/latest/api/token, raising last exception [ 149.886574] cloud-init[3385]: 2024-06-20 21:51:45,159 - url_helper.py[WARNING]: Calling 'None' failed [13/120s]: request error [HTTPConnectionPool(host='fd00:ec2::254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: [Errno -9] Address family for hostname not supported'))] [ 155.282642] cloud-init[3385]: 2024-06-20 21:51:50,566 - url_helper.py[WARNING]: Exception(s) [UrlError("HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 101] Network is unreachable'))"), UrlError("HTTPConnectionPool(host='fd00:ec2::254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: [Errno -9] Address family for hostname not supported'))")] during request to http://[fd00:ec2::254]:80/latest/api/token, raising last exception [ 155.293011] cloud-init[3385]: 2024-06-20 21:51:50,566 - url_helper.py[WARNING]: Calling 'None' failed [19/120s]: request error [HTTPConnectionPool(host='fd00:ec2::254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: [Errno -9] Address family for hostname not supported'))] [ 162.692613] cloud-init[3385]: 2024-06-20 21:51:57,976 - url_helper.py[WARNING]: Exception(s) [UrlError("HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 101] Network is unreachable'))"), UrlError("HTTPConnectionPool(host='fd00:ec2::254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: [Errno -9] Address family for hostname not supported'))")] during request to http://[fd00:ec2::254]:80/latest/api/token, raising last exception [ 162.702151] cloud-init[3385]: 2024-06-20 21:51:57,976 - url_helper.py[WARNING]: Calling 'None' failed [26/120s]: request error [HTTPConnectionPool(host='fd00:ec2::254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: [Errno -9] Address family for hostname not supported'))] [ 172.103163] cloud-init[3385]: 2024-06-20 21:52:07,387 - url_helper.py[WARNING]: Exception(s) [UrlError("HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 101] Network is unreachable'))"), UrlError("HTTPConnectionPool(host='fd00:ec2::254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: [Errno -9] Address family for hostname not supported'))")] during request to http://[fd00:ec2::254]:80/latest/api/token, raising last exception [ 172.112474] cloud-init[3385]: 2024-06-20 21:52:07,387 - url_helper.py[WARNING]: Calling 'None' failed [36/120s]: request error [HTTPConnectionPool(host='fd00:ec2::254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: [Errno -9] Address family for hostname not supported'))] [ 183.512976] cloud-init[3385]: 2024-06-20 21:52:18,796 - url_helper.py[WARNING]: Exception(s) [UrlError("HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 101] Network is unreachable'))"), UrlError("HTTPConnectionPool(host='fd00:ec2::254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: [Errno -9] Address family for hostname not supported'))")] during request to http://[fd00:ec2::254]:80/latest/api/token, raising last exception [ 183.522415] cloud-init[3385]: 2024-06-20 21:52:18,797 - url_helper.py[WARNING]: Calling 'None' failed [47/120s]: request error [HTTPConnectionPool(host='fd00:ec2::254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: [Errno -9] Address family for hostname not supported'))] [FAILED] Failed to start systemd-lo…rvice - User Login Management. [FAILED] Failed to start dbus-broke…ce - D-Bus System Message Bus. [ 195.745473] systemd-journald[833]: Data hash table of /var/log/journal/ec25f52d066115e854db78d38b68bbcc/system.journal has a fill level at 75.0 (1707 of 2275 items, 1310720 file size, 767 bytes per hash table item), suggesting rotation. [ 195.748522] systemd-journald[833]: /var/log/journal/ec25f52d066115e854db78d38b68bbcc/system.journal: Journal header limits reached or header out-of-date, rotating. [ 196.929195] cloud-init[3385]: 2024-06-20 21:52:32,213 - url_helper.py[WARNING]: Exception(s) [UrlError("HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 101] Network is unreachable'))"), UrlError("HTTPConnectionPool(host='fd00:ec2::254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: [Errno -9] Address family for hostname not supported'))")] during request to http://[fd00:ec2::254]:80/latest/api/token, raising last exception [ 196.938671] cloud-init[3385]: 2024-06-20 21:52:32,213 - url_helper.py[WARNING]: Calling 'None' failed [60/120s]: request error [HTTPConnectionPool(host='fd00:ec2::254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: [Errno -9] Address family for hostname not supported'))] [ 212.342925] cloud-init[3385]: 2024-06-20 21:52:47,626 - url_helper.py[WARNING]: Exception(s) [UrlError("HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 101] Network is unreachable'))"), UrlError("HTTPConnectionPool(host='fd00:ec2::254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: [Errno -9] Address family for hostname not supported'))")] during request to http://[fd00:ec2::254]:80/latest/api/token, raising last exception [ 212.352846] cloud-init[3385]: 2024-06-20 21:52:47,627 - url_helper.py[WARNING]: Calling 'None' failed [76/120s]: request error [HTTPConnectionPool(host='fd00:ec2::254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: [Errno -9] Address family for hostname not supported'))] [ 230.763157] cloud-init[3385]: 2024-06-20 21:53:06,047 - url_helper.py[WARNING]: Exception(s) [UrlError("HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 101] Network is unreachable'))"), UrlError("HTTPConnectionPool(host='fd00:ec2::254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: [Errno -9] Address family for hostname not supported'))")] during request to http://[fd00:ec2::254]:80/latest/api/token, raising last exception [ 230.772492] cloud-init[3385]: 2024-06-20 21:53:06,047 - url_helper.py[WARNING]: Calling 'None' failed [94/120s]: request error [HTTPConnectionPool(host='fd00:ec2::254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: [Errno -9] Address family for hostname not supported'))] [ 252.181711] cloud-init[3385]: 2024-06-20 21:53:27,465 - url_helper.py[WARNING]: Exception(s) [UrlError("HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 101] Network is unreachable'))"), UrlError("HTTPConnectionPool(host='fd00:ec2::254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: [Errno -9] Address family for hostname not supported'))")] during request to http://[fd00:ec2::254]:80/latest/api/token, raising last exception [ 252.190589] cloud-init[3385]: 2024-06-20 21:53:27,465 - url_helper.py[WARNING]: Calling 'None' failed [116/120s]: request error [HTTPConnectionPool(host='fd00:ec2::254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: [Errno -9] Address family for hostname not supported'))] [FAILED] Failed to start refresh-po… - Refresh policy routes for ens5. [ 276.457125] cloud-init[3385]: 2024-06-20 21:53:51,741 - url_helper.py[WARNING]: Timed out waiting for addresses: http://169.254.169.254:80/latest/api/token http://[fd00:ec2::254]:80/latest/api/token, exception(s) raised while waiting: [ 276.460149] cloud-init[3385]: 2024-06-20 21:53:51,741 - url_helper.py[WARNING]: Calling 'None' failed [140/120s]: unexpected error ['NoneType' object has no attribute 'contents'] [ 276.462266] cloud-init[3385]: 2024-06-20 21:53:51,741 - url_helper.py[ERROR]: Timed out, no response from urls: ['http://169.254.169.254:80/latest/api/token', 'http://[fd00:ec2::254]:80/latest/api/token'] [ 276.464809] cloud-init[3385]: 2024-06-20 21:53:51,746 - DataSourceEc2.py[WARNING]: IMDS's HTTP endpoint is probably disabled [ 276.672411] cloud-init[3385]: 2024-06-20 21:53:51,956 - cc_write_metadata.py[WARNING]: there is no identity dataset [ 276.674120] cloud-init[3385]: 2024-06-20 21:53:51,956 - cc_write_metadata.py[WARNING]: using path services/domain against metadata failed: KeyError: 'services' [ 276.754607] cloud-init[3385]: 2024-06-20 21:53:52,038 - util.py[WARNING]: Failed to set the hostname to localhost (localhost) [ 276.761129] cloud-init[3385]: 2024-06-20 21:53:52,045 - util.py[WARNING]: Running module set_hostname () failed [ 276.860316] cloud-init[3385]: Generating public/private ed25519 key pair. [ 276.861437] cloud-init[3385]: Your identification has been saved in /etc/ssh/ssh_host_ed25519_key [ 276.862742] cloud-init[3385]: Your public key has been saved in /etc/ssh/ssh_host_ed25519_key.pub [ 276.863892] cloud-init[3385]: The key fingerprint is: [ 276.864601] cloud-init[3385]: SHA256:KSUtM749HGhNcaxaihffWqpT0SjNo3qHAe9XIQ0CrD8 root@localhost [ 276.865747] cloud-init[3385]: The key's randomart image is: [ 276.866492] cloud-init[3385]: +--[ED25519 256]--+ [ 276.867225] cloud-init[3385]: | ... ... | [ 276.868073] cloud-init[3385]: | . ...o. | [ 276.869823] cloud-init[3385]: | . =++* | [ 276.870506] cloud-init[3385]: | . ..oXX.+ | [ 276.871162] cloud-init[3385]: | . +=OS= . | [ 276.871871] cloud-init[3385]: | E.B=o.+ | [ 276.872700] cloud-init[3385]: | =.++= | [ 276.873385] cloud-init[3385]: | . = =. | [ 276.874034] cloud-init[3385]: | ..= | [ 276.874685] cloud-init[3385]: +----[SHA256]-----+ [ 276.875330] cloud-init[3385]: Generating public/private ecdsa key pair. [ 276.876282] cloud-init[3385]: Your identification has been saved in /etc/ssh/ssh_host_ecdsa_key [ 276.877605] cloud-init[3385]: Your public key has been saved in /etc/ssh/ssh_host_ecdsa_key.pub [ 276.879196] cloud-init[3385]: The key fingerprint is: [ 276.881608] cloud-init[3385]: SHA256:Hu0fsLaeoN8TcPhmxvJWUsFSXnOkE/gFdetjijLVcIA root@localhost [ 276.882799] cloud-init[3385]: The key's randomart image is: [ 276.883589] cloud-init[3385]: +---[ECDSA 256]---+ [ 276.884430] cloud-init[3385]: | .+..=+=| [ 276.885387] cloud-init[3385]: | E..=. =+| [ 276.886109] cloud-init[3385]: | . o.+oo | [ 276.886793] cloud-init[3385]: | o.. = o. | [ 276.887546] cloud-init[3385]: | S=oo . + | [ 276.888336] cloud-init[3385]: | ..oXoo o .| [ 276.889038] cloud-init[3385]: | oO+=.. | [ 276.889733] cloud-init[3385]: | . +*+ . | [ 276.890904] cloud-init[3385]: | ...o=.. | [ 276.891653] cloud-init[3385]: +----[SHA256]-----+ [FAILED] Failed to start cloud-init…it job (metadata service crawler). [ 277.397792] cloud-init[5851]: Cloud-init v. 22.2.2 running 'modules:config' at Thu, 20 Jun 2024 21:53:52 +0000. Up 277.34 seconds. [ 277.839259] cloud-init[5856]: Cloud-init v. 22.2.2 running 'modules:final' at Thu, 20 Jun 2024 21:53:53 +0000. Up 277.78 seconds. ci-info: no authorized SSH keys fingerprints found for user ec2-user. <14>Jun 20 21:53:53 cloud-init: ############################################################# <14>Jun 20 21:53:53 cloud-init: -----BEGIN SSH HOST KEY FINGERPRINTS----- <14>Jun 20 21:53:53 cloud-init: 256 SHA256:Hu0fsLaeoN8TcPhmxvJWUsFSXnOkE/gFdetjijLVcIA root@localhost (ECDSA) <14>Jun 20 21:53:53 cloud-init: 256 SHA256:KSUtM749HGhNcaxaihffWqpT0SjNo3qHAe9XIQ0CrD8 root@localhost (ED25519) <14>Jun 20 21:53:53 cloud-init: -----END SSH HOST KEY FINGERPRINTS----- <14>Jun 20 21:53:53 cloud-init: ############################################################# -----BEGIN SSH HOST KEY KEYS----- ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNAX7PY1EB+vmSy3xCeOMZtkbRmy4xr+9kfgQ4q7YDQlkhcP6WAcViRKa1hohdk9y6SziYifO3owtpX74pOoJso= root@localhost ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDqtskN6MD+/+WAicLEraMPfSkqqHQXYSIFs6rUKIWkm root@localhost -----END SSH HOST KEY KEYS----- [ 277.965751] cloud-init[5856]: Cloud-init v. 22.2.2 finished at Thu, 20 Jun 2024 21:53:53 +0000. Datasource DataSourceNone. Up 277.96 seconds [ 277.968961] cloud-init[5856]: 2024-06-20 21:53:53,253 - cc_final_message.py[WARNING]: Used fallback datasource Authorized uses only. All activity may be monitored and reported. ```

Expected Behavior

Instance fully boots without failures

Actual Behavior

See log above in repro steps

Control(s) Affected What controls are being affected by the issue

I have no idea! I was hoping someone here might have an idea of what it nuking those systemd units.

Environment (please complete the following information):

• branch being used: release 1.1.0 ansible 2.10.17 python version = 3.9.17 (main, Jun 13 2023, 16:05:09) [GCC 8.3.0]

Additional Notes Thanks for any insight or ideas!

Possible Solution Unknown