search

ansible-lockdown / AMAZON2023-CIS

Ansible role for Amazon2023 CIS Baseline
https://ansible-lockdown.readthedocs.io/en/latest/
MIT License
27 stars 20 forks source link

Unable to connect after run & Amazon heath checks fail #80

Open four43 opened 5 months ago

four43 commented 5 months ago

Describe the Issue

After running the playbook I restart the instance and access it. If I take an AMI of the instance and try and run it again however, it won't start properly.

After running:

sudo ANSIBLE_HOST_KEY_CHECKING=False ansible-playbook \
    -i localhost, --connection=local \
    site.yml \
    -e os_gpg_key_pubkey_name=gpg-pubkey-d832c631-6515c85e \
    -e amzn2023cis_syslog_service=journald \
    --tags level1-server \
    --skip-tags rule_1.2.4,rule_4.6.6 | tee cis-ansible-harden.log

I can pull logs from the instance that is failing:

Boot Log ``` [=3h[=3h[=3h[=3h Booting `Amazon Linux (6.1.92-99.174.amzn2023.x86_64) 2023' [ 0.071111] RETBleed: WARNING: Spectre v2 mitigation leaves CPU vulnerable to RETBleed attacks, data leaks possible! [ 11.677731] kauditd_printk_skb: 37 callbacks suppressed [ 11.677733] audit: type=1305 audit(1718920166.950:71): op=set audit_enabled=1 old=1 auid=4294967295 ses=4294967295 subj=system_u:system_r:syslogd_t:s0 res=1 [ 11.679489] audit: type=1300 audit(1718920166.950:71): arch=c000003e syscall=46 success=yes exit=60 a0=3 a1=7ffdb7634340 a2=4000 a3=7ffdb76343cc items=0 ppid=1 pid=833 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="systemd-journal" exe="/usr/lib/systemd/systemd-journald" subj=system_u:system_r:syslogd_t:s0 key=(null) [ 11.683252] audit: type=1327 audit(1718920166.950:71): proctitle="/usr/lib/systemd/systemd-journald" [ 11.688457] systemd[1]: Started systemd-journald.service - Journal Service. [ 11.691369] audit: type=1130 audit(1718920166.960:72): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-journald comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 11.746154] systemd-journald[833]: Received client request to flush runtime journal. [ 11.796820] audit: type=1130 audit(1718920167.070:73): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-sysctl comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 12.015400] audit: type=1130 audit(1718920167.290:74): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-sysusers comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 12.025959] systemd-journald[833]: Data hash table of /var/log/journal/ec25f52d066115e854db78d38b68bbcc/system.journal has a fill level at 78.5 (1785 of 2275 items, 1310720 file size, 734 bytes per hash table item), suggesting rotation. [ 12.028004] systemd-journald[833]: /var/log/journal/ec25f52d066115e854db78d38b68bbcc/system.journal: Journal header limits reached or header out-of-date, rotating. [ 12.105780] audit: type=1130 audit(1718920167.380:75): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-journal-flush comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 12.147423] audit: type=1130 audit(1718920167.420:76): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-tmpfiles-setup-dev comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 12.176941] audit: type=1130 audit(1718920167.450:77): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=dracut-shutdown comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 12.423940] audit: type=1130 audit(1718920167.700:78): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-machine-id-commit comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 13.071902] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 13.112759] ACPI: button: Power Button [PWRF] [ 13.113313] input: Sleep Button as /devices/LNXSYSTM:00/LNXSLPBN:00/input/input1 [ 13.114258] ACPI: button: Sleep Button [SLPF] [ 13.140928] cryptd: max_cpu_qlen set to 1000 [ 13.142801] pps_core: LinuxPPS API ver. 1 registered [ 13.143429] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 13.146413] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 13.147349] i8042: Warning: Keylock active [ 13.148826] PTP clock support registered [ 13.159199] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 13.159739] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 13.230539] AVX2 version of gcm_enc/dec engaged. [ 13.231253] AES CTR mode by8 optimization enabled [ 13.231933] ena 0000:00:05.0: Elastic Network Adapter (ENA) v2.12.0g [ 13.242607] ena 0000:00:05.0: ENA device version: 0.10 [ 13.243190] ena 0000:00:05.0: ENA controller version: 0.0.1 implementation version 1 [ 13.322668] ena 0000:00:05.0: LLQ is not supported Fallback to host mode policy. [ 13.334765] ena 0000:00:05.0: Elastic Network Adapter (ENA) found at mem c0400000, mac addr 16:ff:ef:c7:8f:25 [ 13.432768] ena 0000:00:05.0 ens5: renamed from eth0 [ 13.685470] zram_generator::config[1496]: zram0: system has too much memory (7811MB), limit is 800MB, ignoring. [ 15.322988] ena 0000:00:05.0 ens5: Local page cache is disabled for less than 16 channels [FAILED] Failed to start dbus-broke…ce - D-Bus System Message Bus. [FAILED] Failed to start systemd-ho…d.service - Home Area Manager. [FAILED] Failed to start systemd-lo…rvice - User Login Management. [FAILED] Failed to start dbus-broke…ce - D-Bus System Message Bus. [FAILED] Failed to start policy-rou…m - Set up policy routes for ens5. [FAILED] Failed to start systemd-ne…Wait for Network to be Configured. [ 135.957908] cloud-init[3385]: Cloud-init v. 22.2.2 running 'init' at Thu, 20 Jun 2024 21:51:31 +0000. Up 135.92 seconds. [ 136.047154] cloud-init[3385]: ci-info: ++++++++++++++++++++++++++++++++++++Net device info+++++++++++++++++++++++++++++++++++++ [ 136.048922] cloud-init[3385]: ci-info: +--------+------+------------------------------+-----------+-------+-------------------+ [ 136.050499] cloud-init[3385]: ci-info: | Device | Up | Address | Mask | Scope | Hw-Address | [ 136.052055] cloud-init[3385]: ci-info: +--------+------+------------------------------+-----------+-------+-------------------+ [ 136.053562] cloud-init[3385]: ci-info: | ens5 | True | fe80::14ff:efff:fec7:8f25/64 | . | link | 16:ff:ef:c7:8f:25 | [ 136.055157] cloud-init[3385]: ci-info: | lo | True | 127.0.0.1 | 255.0.0.0 | host | . | [ 136.056710] cloud-init[3385]: ci-info: +--------+------+------------------------------+-----------+-------+-------------------+ [ 136.058206] cloud-init[3385]: ci-info: +++++++++++++++++++Route IPv6 info+++++++++++++++++++ [ 136.059320] cloud-init[3385]: ci-info: +-------+-------------+---------+-----------+-------+ [ 136.060500] cloud-init[3385]: ci-info: | Route | Destination | Gateway | Interface | Flags | [ 136.061734] cloud-init[3385]: ci-info: +-------+-------------+---------+-----------+-------+ [ 136.062895] cloud-init[3385]: ci-info: | 0 | fe80::/64 | :: | ens5 | U | [ 136.064000] cloud-init[3385]: ci-info: | 1 | local | :: | ens5 | U | [ 136.065106] cloud-init[3385]: ci-info: | 2 | multicast | :: | ens5 | U | [ 136.066211] cloud-init[3385]: ci-info: +-------+-------------+---------+-----------+-------+ [ 136.221822] cloud-init[3385]: 2024-06-20 21:51:31,506 - url_helper.py[WARNING]: Exception(s) [UrlError("HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 101] Network is unreachable'))"), UrlError("HTTPConnectionPool(host='fd00:ec2::254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: [Errno -9] Address family for hostname not supported'))")] during request to http://[fd00:ec2::254]:80/latest/api/token, raising last exception [ 136.232889] cloud-init[3385]: 2024-06-20 21:51:31,506 - url_helper.py[WARNING]: Calling 'None' failed [0/120s]: request error [HTTPConnectionPool(host='fd00:ec2::254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: [Errno -9] Address family for hostname not supported'))] [ 136.633570] cloud-init[3385]: 2024-06-20 21:51:31,917 - url_helper.py[WARNING]: Exception(s) [UrlError("HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 101] Network is unreachable'))"), UrlError("HTTPConnectionPool(host='fd00:ec2::254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: [Errno -9] Address family for hostname not supported'))")] during request to http://[fd00:ec2::254]:80/latest/api/token, raising last exception [ 136.642945] cloud-init[3385]: 2024-06-20 21:51:31,918 - url_helper.py[WARNING]: Calling 'None' failed [0/120s]: request error [HTTPConnectionPool(host='fd00:ec2::254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: [Errno -9] Address family for hostname not supported'))] [ 137.037221] cloud-init[3385]: 2024-06-20 21:51:32,321 - url_helper.py[WARNING]: Exception(s) [UrlError("HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 101] Network is unreachable'))"), UrlError("HTTPConnectionPool(host='fd00:ec2::254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: [Errno -9] Address family for hostname not supported'))")] during request to http://[fd00:ec2::254]:80/latest/api/token, raising last exception [ 137.046367] cloud-init[3385]: 2024-06-20 21:51:32,321 - url_helper.py[WARNING]: Calling 'None' failed [0/120s]: request error [HTTPConnectionPool(host='fd00:ec2::254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: [Errno -9] Address family for hostname not supported'))] [ 137.439805] cloud-init[3385]: 2024-06-20 21:51:32,724 - url_helper.py[WARNING]: Exception(s) [UrlError("HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 101] Network is unreachable'))"), UrlError("HTTPConnectionPool(host='fd00:ec2::254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: [Errno -9] Address family for hostname not supported'))")] during request to http://[fd00:ec2::254]:80/latest/api/token, raising last exception [ 137.448930] cloud-init[3385]: 2024-06-20 21:51:32,724 - url_helper.py[WARNING]: Calling 'None' failed [1/120s]: request error [HTTPConnectionPool(host='fd00:ec2::254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: [Errno -9] Address family for hostname not supported'))] [ 137.851360] cloud-init[3385]: 2024-06-20 21:51:33,135 - url_helper.py[WARNING]: Exception(s) [UrlError("HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 101] Network is unreachable'))"), UrlError("HTTPConnectionPool(host='fd00:ec2::254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: [Errno -9] Address family for hostname not supported'))")] during request to http://[fd00:ec2::254]:80/latest/api/token, raising last exception [ 137.860560] cloud-init[3385]: 2024-06-20 21:51:33,135 - url_helper.py[WARNING]: Calling 'None' failed [1/120s]: request error [HTTPConnectionPool(host='fd00:ec2::254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: [Errno -9] Address family for hostname not supported'))] [ 138.254861] cloud-init[3385]: 2024-06-20 21:51:33,538 - url_helper.py[WARNING]: Exception(s) [UrlError("HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 101] Network is unreachable'))"), UrlError("HTTPConnectionPool(host='fd00:ec2::254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: [Errno -9] Address family for hostname not supported'))")] during request to http://[fd00:ec2::254]:80/latest/api/token, raising last exception [ 138.263932] cloud-init[3385]: 2024-06-20 21:51:33,539 - url_helper.py[WARNING]: Calling 'None' failed [2/120s]: request error [HTTPConnectionPool(host='fd00:ec2::254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: [Errno -9] Address family for hostname not supported'))] [ 139.659081] cloud-init[3385]: 2024-06-20 21:51:34,942 - url_helper.py[WARNING]: Exception(s) [UrlError("HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 101] Network is unreachable'))"), UrlError("HTTPConnectionPool(host='fd00:ec2::254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: [Errno -9] Address family for hostname not supported'))")] during request to http://[fd00:ec2::254]:80/latest/api/token, raising last exception [ 139.668717] cloud-init[3385]: 2024-06-20 21:51:34,943 - url_helper.py[WARNING]: Calling 'None' failed [3/120s]: request error [HTTPConnectionPool(host='fd00:ec2::254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: [Errno -9] Address family for hostname not supported'))] [ 142.062814] cloud-init[3385]: 2024-06-20 21:51:37,346 - url_helper.py[WARNING]: Exception(s) [UrlError("HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 101] Network is unreachable'))"), UrlError("HTTPConnectionPool(host='fd00:ec2::254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: [Errno -9] Address family for hostname not supported'))")] during request to http://[fd00:ec2::254]:80/latest/api/token, raising last exception [ 142.072069] cloud-init[3385]: 2024-06-20 21:51:37,346 - url_helper.py[WARNING]: Calling 'None' failed [5/120s]: request error [HTTPConnectionPool(host='fd00:ec2::254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: [Errno -9] Address family for hostname not supported'))] [ 145.469382] cloud-init[3385]: 2024-06-20 21:51:40,753 - url_helper.py[WARNING]: Exception(s) [UrlError("HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 101] Network is unreachable'))"), UrlError("HTTPConnectionPool(host='fd00:ec2::254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: [Errno -9] Address family for hostname not supported'))")] during request to http://[fd00:ec2::254]:80/latest/api/token, raising last exception [ 145.478307] cloud-init[3385]: 2024-06-20 21:51:40,753 - url_helper.py[WARNING]: Calling 'None' failed [9/120s]: request error [HTTPConnectionPool(host='fd00:ec2::254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: [Errno -9] Address family for hostname not supported'))] [ 149.874896] cloud-init[3385]: 2024-06-20 21:51:45,158 - url_helper.py[WARNING]: Exception(s) [UrlError("HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 101] Network is unreachable'))"), UrlError("HTTPConnectionPool(host='fd00:ec2::254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: [Errno -9] Address family for hostname not supported'))")] during request to http://[fd00:ec2::254]:80/latest/api/token, raising last exception [ 149.886574] cloud-init[3385]: 2024-06-20 21:51:45,159 - url_helper.py[WARNING]: Calling 'None' failed [13/120s]: request error [HTTPConnectionPool(host='fd00:ec2::254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: [Errno -9] Address family for hostname not supported'))] [ 155.282642] cloud-init[3385]: 2024-06-20 21:51:50,566 - url_helper.py[WARNING]: Exception(s) [UrlError("HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 101] Network is unreachable'))"), UrlError("HTTPConnectionPool(host='fd00:ec2::254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: [Errno -9] Address family for hostname not supported'))")] during request to http://[fd00:ec2::254]:80/latest/api/token, raising last exception [ 155.293011] cloud-init[3385]: 2024-06-20 21:51:50,566 - url_helper.py[WARNING]: Calling 'None' failed [19/120s]: request error [HTTPConnectionPool(host='fd00:ec2::254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: [Errno -9] Address family for hostname not supported'))] [ 162.692613] cloud-init[3385]: 2024-06-20 21:51:57,976 - url_helper.py[WARNING]: Exception(s) [UrlError("HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 101] Network is unreachable'))"), UrlError("HTTPConnectionPool(host='fd00:ec2::254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: [Errno -9] Address family for hostname not supported'))")] during request to http://[fd00:ec2::254]:80/latest/api/token, raising last exception [ 162.702151] cloud-init[3385]: 2024-06-20 21:51:57,976 - url_helper.py[WARNING]: Calling 'None' failed [26/120s]: request error [HTTPConnectionPool(host='fd00:ec2::254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: [Errno -9] Address family for hostname not supported'))] [ 172.103163] cloud-init[3385]: 2024-06-20 21:52:07,387 - url_helper.py[WARNING]: Exception(s) [UrlError("HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 101] Network is unreachable'))"), UrlError("HTTPConnectionPool(host='fd00:ec2::254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: [Errno -9] Address family for hostname not supported'))")] during request to http://[fd00:ec2::254]:80/latest/api/token, raising last exception [ 172.112474] cloud-init[3385]: 2024-06-20 21:52:07,387 - url_helper.py[WARNING]: Calling 'None' failed [36/120s]: request error [HTTPConnectionPool(host='fd00:ec2::254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: [Errno -9] Address family for hostname not supported'))] [ 183.512976] cloud-init[3385]: 2024-06-20 21:52:18,796 - url_helper.py[WARNING]: Exception(s) [UrlError("HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 101] Network is unreachable'))"), UrlError("HTTPConnectionPool(host='fd00:ec2::254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: [Errno -9] Address family for hostname not supported'))")] during request to http://[fd00:ec2::254]:80/latest/api/token, raising last exception [ 183.522415] cloud-init[3385]: 2024-06-20 21:52:18,797 - url_helper.py[WARNING]: Calling 'None' failed [47/120s]: request error [HTTPConnectionPool(host='fd00:ec2::254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: [Errno -9] Address family for hostname not supported'))] [FAILED] Failed to start systemd-lo…rvice - User Login Management. [FAILED] Failed to start dbus-broke…ce - D-Bus System Message Bus. [ 195.745473] systemd-journald[833]: Data hash table of /var/log/journal/ec25f52d066115e854db78d38b68bbcc/system.journal has a fill level at 75.0 (1707 of 2275 items, 1310720 file size, 767 bytes per hash table item), suggesting rotation. [ 195.748522] systemd-journald[833]: /var/log/journal/ec25f52d066115e854db78d38b68bbcc/system.journal: Journal header limits reached or header out-of-date, rotating. [ 196.929195] cloud-init[3385]: 2024-06-20 21:52:32,213 - url_helper.py[WARNING]: Exception(s) [UrlError("HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 101] Network is unreachable'))"), UrlError("HTTPConnectionPool(host='fd00:ec2::254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: [Errno -9] Address family for hostname not supported'))")] during request to http://[fd00:ec2::254]:80/latest/api/token, raising last exception [ 196.938671] cloud-init[3385]: 2024-06-20 21:52:32,213 - url_helper.py[WARNING]: Calling 'None' failed [60/120s]: request error [HTTPConnectionPool(host='fd00:ec2::254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: [Errno -9] Address family for hostname not supported'))] [ 212.342925] cloud-init[3385]: 2024-06-20 21:52:47,626 - url_helper.py[WARNING]: Exception(s) [UrlError("HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 101] Network is unreachable'))"), UrlError("HTTPConnectionPool(host='fd00:ec2::254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: [Errno -9] Address family for hostname not supported'))")] during request to http://[fd00:ec2::254]:80/latest/api/token, raising last exception [ 212.352846] cloud-init[3385]: 2024-06-20 21:52:47,627 - url_helper.py[WARNING]: Calling 'None' failed [76/120s]: request error [HTTPConnectionPool(host='fd00:ec2::254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: [Errno -9] Address family for hostname not supported'))] [ 230.763157] cloud-init[3385]: 2024-06-20 21:53:06,047 - url_helper.py[WARNING]: Exception(s) [UrlError("HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 101] Network is unreachable'))"), UrlError("HTTPConnectionPool(host='fd00:ec2::254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: [Errno -9] Address family for hostname not supported'))")] during request to http://[fd00:ec2::254]:80/latest/api/token, raising last exception [ 230.772492] cloud-init[3385]: 2024-06-20 21:53:06,047 - url_helper.py[WARNING]: Calling 'None' failed [94/120s]: request error [HTTPConnectionPool(host='fd00:ec2::254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: [Errno -9] Address family for hostname not supported'))] [ 252.181711] cloud-init[3385]: 2024-06-20 21:53:27,465 - url_helper.py[WARNING]: Exception(s) [UrlError("HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 101] Network is unreachable'))"), UrlError("HTTPConnectionPool(host='fd00:ec2::254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: [Errno -9] Address family for hostname not supported'))")] during request to http://[fd00:ec2::254]:80/latest/api/token, raising last exception [ 252.190589] cloud-init[3385]: 2024-06-20 21:53:27,465 - url_helper.py[WARNING]: Calling 'None' failed [116/120s]: request error [HTTPConnectionPool(host='fd00:ec2::254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: [Errno -9] Address family for hostname not supported'))] [FAILED] Failed to start refresh-po… - Refresh policy routes for ens5. [ 276.457125] cloud-init[3385]: 2024-06-20 21:53:51,741 - url_helper.py[WARNING]: Timed out waiting for addresses: http://169.254.169.254:80/latest/api/token http://[fd00:ec2::254]:80/latest/api/token, exception(s) raised while waiting: [ 276.460149] cloud-init[3385]: 2024-06-20 21:53:51,741 - url_helper.py[WARNING]: Calling 'None' failed [140/120s]: unexpected error ['NoneType' object has no attribute 'contents'] [ 276.462266] cloud-init[3385]: 2024-06-20 21:53:51,741 - url_helper.py[ERROR]: Timed out, no response from urls: ['http://169.254.169.254:80/latest/api/token', 'http://[fd00:ec2::254]:80/latest/api/token'] [ 276.464809] cloud-init[3385]: 2024-06-20 21:53:51,746 - DataSourceEc2.py[WARNING]: IMDS's HTTP endpoint is probably disabled [ 276.672411] cloud-init[3385]: 2024-06-20 21:53:51,956 - cc_write_metadata.py[WARNING]: there is no identity dataset [ 276.674120] cloud-init[3385]: 2024-06-20 21:53:51,956 - cc_write_metadata.py[WARNING]: using path services/domain against metadata failed: KeyError: 'services' [ 276.754607] cloud-init[3385]: 2024-06-20 21:53:52,038 - util.py[WARNING]: Failed to set the hostname to localhost (localhost) [ 276.761129] cloud-init[3385]: 2024-06-20 21:53:52,045 - util.py[WARNING]: Running module set_hostname () failed [ 276.860316] cloud-init[3385]: Generating public/private ed25519 key pair. [ 276.861437] cloud-init[3385]: Your identification has been saved in /etc/ssh/ssh_host_ed25519_key [ 276.862742] cloud-init[3385]: Your public key has been saved in /etc/ssh/ssh_host_ed25519_key.pub [ 276.863892] cloud-init[3385]: The key fingerprint is: [ 276.864601] cloud-init[3385]: SHA256:KSUtM749HGhNcaxaihffWqpT0SjNo3qHAe9XIQ0CrD8 root@localhost [ 276.865747] cloud-init[3385]: The key's randomart image is: [ 276.866492] cloud-init[3385]: +--[ED25519 256]--+ [ 276.867225] cloud-init[3385]: | ... ... | [ 276.868073] cloud-init[3385]: | . ...o. | [ 276.869823] cloud-init[3385]: | . =++* | [ 276.870506] cloud-init[3385]: | . ..oXX.+ | [ 276.871162] cloud-init[3385]: | . +=OS= . | [ 276.871871] cloud-init[3385]: | E.B=o.+ | [ 276.872700] cloud-init[3385]: | =.++= | [ 276.873385] cloud-init[3385]: | . = =. | [ 276.874034] cloud-init[3385]: | ..= | [ 276.874685] cloud-init[3385]: +----[SHA256]-----+ [ 276.875330] cloud-init[3385]: Generating public/private ecdsa key pair. [ 276.876282] cloud-init[3385]: Your identification has been saved in /etc/ssh/ssh_host_ecdsa_key [ 276.877605] cloud-init[3385]: Your public key has been saved in /etc/ssh/ssh_host_ecdsa_key.pub [ 276.879196] cloud-init[3385]: The key fingerprint is: [ 276.881608] cloud-init[3385]: SHA256:Hu0fsLaeoN8TcPhmxvJWUsFSXnOkE/gFdetjijLVcIA root@localhost [ 276.882799] cloud-init[3385]: The key's randomart image is: [ 276.883589] cloud-init[3385]: +---[ECDSA 256]---+ [ 276.884430] cloud-init[3385]: | .+..=+=| [ 276.885387] cloud-init[3385]: | E..=. =+| [ 276.886109] cloud-init[3385]: | . o.+oo | [ 276.886793] cloud-init[3385]: | o.. = o. | [ 276.887546] cloud-init[3385]: | S=oo . + | [ 276.888336] cloud-init[3385]: | ..oXoo o .| [ 276.889038] cloud-init[3385]: | oO+=.. | [ 276.889733] cloud-init[3385]: | . +*+ . | [ 276.890904] cloud-init[3385]: | ...o=.. | [ 276.891653] cloud-init[3385]: +----[SHA256]-----+ [FAILED] Failed to start cloud-init…it job (metadata service crawler). [ 277.397792] cloud-init[5851]: Cloud-init v. 22.2.2 running 'modules:config' at Thu, 20 Jun 2024 21:53:52 +0000. Up 277.34 seconds. [ 277.839259] cloud-init[5856]: Cloud-init v. 22.2.2 running 'modules:final' at Thu, 20 Jun 2024 21:53:53 +0000. Up 277.78 seconds. ci-info: no authorized SSH keys fingerprints found for user ec2-user. <14>Jun 20 21:53:53 cloud-init: ############################################################# <14>Jun 20 21:53:53 cloud-init: -----BEGIN SSH HOST KEY FINGERPRINTS----- <14>Jun 20 21:53:53 cloud-init: 256 SHA256:Hu0fsLaeoN8TcPhmxvJWUsFSXnOkE/gFdetjijLVcIA root@localhost (ECDSA) <14>Jun 20 21:53:53 cloud-init: 256 SHA256:KSUtM749HGhNcaxaihffWqpT0SjNo3qHAe9XIQ0CrD8 root@localhost (ED25519) <14>Jun 20 21:53:53 cloud-init: -----END SSH HOST KEY FINGERPRINTS----- <14>Jun 20 21:53:53 cloud-init: ############################################################# -----BEGIN SSH HOST KEY KEYS----- ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNAX7PY1EB+vmSy3xCeOMZtkbRmy4xr+9kfgQ4q7YDQlkhcP6WAcViRKa1hohdk9y6SziYifO3owtpX74pOoJso= root@localhost ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDqtskN6MD+/+WAicLEraMPfSkqqHQXYSIFs6rUKIWkm root@localhost -----END SSH HOST KEY KEYS----- [ 277.965751] cloud-init[5856]: Cloud-init v. 22.2.2 finished at Thu, 20 Jun 2024 21:53:53 +0000. Datasource DataSourceNone. Up 277.96 seconds [ 277.968961] cloud-init[5856]: 2024-06-20 21:53:53,253 - cc_final_message.py[WARNING]: Used fallback datasource Authorized uses only. All activity may be monitored and reported. ```

Expected Behavior

Instance fully boots without failures

Actual Behavior

See log above in repro steps

Control(s) Affected What controls are being affected by the issue

I have no idea! I was hoping someone here might have an idea of what it nuking those systemd units.

Environment (please complete the following information):

Additional Notes Thanks for any insight or ideas!

Possible Solution Unknown

uk-bolly commented 2 months ago

hi @four43

The original AMI works as expected i am assuming. Just when you take a copy to a new AMI? I believe this could be something to do with cloud-init, it doesnt appear to be anything to do with this role specifically. Hopefully someone may have seen similar. If you do find the root cause that comes from the role please let us know and we can see if we can add something for you.

Many thanks

uk-bolly

vireshsolanki commented 2 months ago

Yes, it is issue with AMI's

herman-wong-cf commented 2 months ago

I've spotted this issue in my own build.

@four43, in the role => tasks => section_4 => cis_4.6.x.yml => "4.6.5 | PATCH | Ensure default user umask is 027 or more restrictive | Set umask for /etc/login.defs pam_umask settings", try comment out "/etc/bashrc" from the "loop".

@uk-bolly I don't know what the ultimate root cause is, but excluding that file from the loop allowed me to launch the instance normally. I'm guessing something runs in cloud-init that depends on a loose umask in /etc/bashrc.

This is specifically an issue where there are no problems/errors with the Packer AMI build, but launching an instance from the AMI leads to basic systemd service failures.

uk-bolly commented 2 months ago

hi @herman-wong-cf @four43

Thank you both for the feedback, Having a quick read up it is indeed cloud-init. In order to be compliant that will need to be adjusted, to either skip as you have mentioned or set the permissions back in cloud init and once completed fix it to be compliant.

Its not something that we would change as part of the role. It would be a great article once resolved on how to fix it.

Kindest

uk-bolly

vireshsolanki commented 2 weeks ago

hi @uk-bolly , your script changes the /etc/fstab entry so when creating the AMI it got crashes because of that and @herman-wong-cf the role => tasks => section_4 => cis_4.6.x.yml => "4.6.5 | PATCH | Ensure default user umask is 027 or more restrictive | Set umask for /etc/login.defs pam_umask settings", try comment out "/etc/bashrc" from the "loop". this one is issue as well, after implementing it cloud-init runs successfully.