`/hardening/host-os/ansible` doesn't seem to catch playbook errors

[comps]

05:59:58                 out: 2024-07-17 05:59:58 test.py:29: running: ansible-playbook -v -c local -i localhost, --skip-tags accounts_password_set_max_life_existing,accounts_password_set_max_life_root,package_dnf-plugin-subscription-manager_installed,package_screen_installed,sysctl_kernel_unprivileged_bpf_disabled,coredump_disable_backtraces,coredump_disable_storage,package_tmux_installed,package_rsyslog-gnutls_installed,package_audispd-plugins_installed /root/content/build/ansible/rhel10-playbook-cis_workstation_l2.yml
05:59:59                 out: Using /etc/ansible/ansible.cfg as config file
05:59:59                 out:
05:59:59                 out: PLAY [Ansible Playbook for xccdf_org.ssgproject.content_profile_cis_workstation_l2] ***
05:59:59                 out:
05:59:59                 out: TASK [Gathering Facts] *********************************************************
06:00:00                 out: ok: [localhost]

...

06:00:12                 out: TASK [Verify that Crypto Policy is Set (runtime)] ******************************
06:00:12                 out: fatal: [localhost]: FAILED! => {"changed": true, "cmd": ["/usr/bin/update-crypto-policies", "--set", "DEFAULT:NO-SHA1"], "delta": "0:00:00.036241", "end": "2024-07-17 06:00:12.798464", "msg": "non-zero return code", "rc": 1, "start": "2024-07-17 06:00:12.762223", "stderr": "Unknown policy `NO-SHA1`: file `NO-SHA1.pmod` not found in (., policies/modules, /etc/crypto-policies/policies/modules, /usr/share/crypto-policies/policies/modules)", "stderr_lines": ["Unknown policy `NO-SHA1`: file `NO-SHA1.pmod` not found in (., policies/modules, /etc/crypto-policies/policies/modules, /usr/share/crypto-policies/policies/modules)"], "stdout": "", "stdout_lines": []}
06:00:12                 out: 2024-07-17 06:00:12 test.py:30: lib.waive.collect_waivers:141: using /tmp/runcontest-results/task20/plans/default/discover/default-0/tests/conf/waivers for waiving
06:00:12                 out: 2024-07-17 06:00:12 test.py:30: lib.results.report_plain:182: ERROR playbook: Verify that Crypto Policy is Set (runtime) ({"changed": true, "cmd": ["/usr/bin/update-crypto-policies", "--set", "DEFAULT:NO-SHA1"], "delta": "0:00:00.036241", "end": "2024-07-17 06:00:12.798464", "msg": "non-zero return code", "rc": 1, "start": "2024-07-17 06:00:12.762223", "stderr": "Unknown policy `NO-SHA1`: file `NO-SHA1.pmod` not found in (., policies/modules, /etc/crypto-policies/policies/modules, /usr/share/crypto-policies/policies/modules)", "stderr_lines": ["Unknown policy `NO-SHA1`: file `NO-SHA1.pmod` not found in (., policies/modules, /etc/crypto-policies/policies/modules, /usr/share/crypto-policies/policies/modules)"], "stdout": "", "stdout_lines": []})
06:00:12                 out:
06:00:12                 out: PLAY RECAP *********************************************************************
06:00:12                 out: localhost                  : ok=16   changed=6    unreachable=0    failed=1    skipped=1    rescued=0    ignored=0
06:00:12                 out:
06:00:12                 out: 2024-07-17 06:00:12 test.py:38: lib.util.environment.reboot:19: running: tmt-reboot

the test continued on with the reboot despite an obvious error

[comps]

related code:

    cmd = [
        'ansible-playbook', '-v', '-c', 'local', '-i', 'localhost,',
        *skip_tags_arg,
        playbook,
    ]
    proc, lines = util.subprocess_stream(cmd)
    failed = ansible.report_from_output(lines)
    if proc.returncode not in [0,2] or proc.returncode == 2 and not failed:
        raise RuntimeError(f"ansible-playbook failed with {proc.returncode}")

maybe we should util.log() from inside report_from_output() how many failures were found?

[comps]

Actually, the error is reported:

16:34:00                 out: TASK [Configure Firewalld to Restrict Loopback Traffic - Informative Message Based on Service State] ***
16:34:00                 out: fatal: [localhost]: FAILED! => {
16:34:00                 out: 2024-07-18 16:34:00 test.py:30: lib.waive.collect_waivers:141: using /var/tmp/runcontest-results/task28/plans/default/discover/default-0/tests/conf/waivers for waiving
16:34:00                 out: 2024-07-18 16:34:00 test.py:30: lib.results.report_plain:182: ERROR playbook: Configure Firewalld to Restrict Loopback Traffic - Informative Message Based on Service State ({)
16:34:00                 out:     "assertion": "ansible_facts.services['firewalld.service'].state == 'running'",

see the

lib.results.report_plain:182: ERROR playbook: Configure Firewalld to Restrict Loopback Traffic - Informative Message Based on Service State

it's just that we can't automagically figure out which fatal errors render the system unbootable and which ones don't..