search

ComplianceAsCode / content

Security automation content in SCAP, Bash, Ansible, and other formats
https://complianceascode.readthedocs.io/en/latest/
Other
2.22k stars 698 forks source link

Rule configure_crypto_policy is failing tests #10895

Open jan-cerny opened 1 year ago

jan-cerny commented 1 year ago

Description of problem:

During the review of the productization test run test /CoreOS/scap-security-guide/Sanity/test-rules-scenarios-per-profile OSPP 4/5 we discovered that the rule configure_crypto_policy fails the cis_l2.pass.sh test scenario for both Ansible and Bash remediations when Automatus is executed in a combined mode.

SCAP Security Guide Version:

current upstream master branch as of 2023-07-22 as of HEAD a96ccb9

Operating System Version:

RHEL 9

Steps to Reproduce:

  1. python3 /tmp/tmp.GwTs8FwB4g/rpmbuild/BUILD/scap-security-guide-0.1.69/tests/test_suite.py combined --slice 5 5 --libvirt qemu:///system test_suite_vm --datastream /usr/share/xml/scap/ssg/content/ssg-rhel9-ds.xml --mode online --remediate-using bash --duplicate-templates --no-reports xccdf_org.ssgproject.content_profile_ospp
  2. python3 /tmp/tmp.GwTs8FwB4g/rpmbuild/BUILD/scap-security-guide-0.1.69/tests/test_suite.py combined --slice 5 5 --libvirt qemu:///system test_suite_vm --datastream /usr/share/xml/scap/ssg/content/ssg-rhel9-ds.xml --mode online --remediate-using ansible --duplicate-templates --no-reports xccdf_org.ssgproject.content_profile_ospp

Actual Results:

INFO - xccdf_org.ssgproject.content_rule_configure_crypto_policy
INFO - Script cis_l2.pass.sh using profile xccdf_org.ssgproject.content_profile_cis OK
ERROR - Script cis_l2.pass.sh using profile xccdf_org.ssgproject.content_profile_cis_workstation_l2 found issue:
ERROR - Rule xccdf_org.ssgproject.content_rule_configure_crypto_policy has not been evaluated! Wrong profile selected in test scenario?
ERROR - The initial scan failed for rule 'xccdf_org.ssgproject.content_rule_configure_crypto_policy'.
INFO - Script config_and_current_same_time.pass.sh using profile (all) OK
INFO - Script config_newer_than_current.fail.sh using profile (all) OK
INFO - Script missing_nss_config.fail.sh using profile xccdf_org.ssgproject.content_profile_ospp OK
INFO - Script missing_policy.fail.sh using profile xccdf_org.ssgproject.content_profile_ospp OK
INFO - Script missing_policy_file.fail.sh using profile xccdf_org.ssgproject.content_profile_ospp OK
INFO - Script nss_config_as_file.pass.sh using profile xccdf_org.ssgproject.content_profile_ospp OK
INFO - Script nss_config_as_symlink.pass.sh using profile xccdf_org.ssgproject.content_profile_ospp OK
INFO - Script policy_default_cis_l1.pass.sh using profile xccdf_org.ssgproject.content_profile_cis_server_l1 OK
INFO - Script policy_default_cis_l1.pass.sh using profile xccdf_org.ssgproject.content_profile_cis_workstation_l1 OK
INFO - Script policy_default_nosha1_set.pass.sh using profile xccdf_org.ssgproject.content_profile_e8 OK
WARNING - Script policy_default_set.pass.sh - profile xccdf_org.ssgproject.content_profile_standard not found in datastream
INFO - Script policy_fips_ospp_set.pass.sh using profile xccdf_org.ssgproject.content_profile_ospp OK
INFO - Script wrong_policy.fail.sh using profile xccdf_org.ssgproject.content_profile_ospp OK

Expected Results:

no errors are reported by Automatus

Additional Information/Debugging Steps:

I was also able to reproduce the problem in rule mode locally with a RHEL 9 VM back end.

[jcerny@fedora scap-security-guide{master}]$ tests/automatus.py rule --libvirt qemu:///system ssgts_rhel9 --scenario cis_l2.pass.sh configure_crypto_policy
Setting console output to log level INFO
INFO - The base image option has not been specified, choosing libvirt-based test environment.
INFO - Logging into /home/jcerny/work/git/scap-security-guide/logs/rule-custom-2023-07-24-1537/test_suite.log
INFO - xccdf_org.ssgproject.content_rule_configure_crypto_policy
INFO - Script cis_l2.pass.sh using profile xccdf_org.ssgproject.content_profile_cis OK
ERROR - Script cis_l2.pass.sh using profile xccdf_org.ssgproject.content_profile_cis_workstation_l2 found issue:
ERROR - Rule xccdf_org.ssgproject.content_rule_configure_crypto_policy has not been evaluated! Wrong profile selected in test scenario?
ERROR - The initial scan failed for rule 'xccdf_org.ssgproject.content_rule_configure_crypto_policy'.

Also, please check if the Automatus's output isn't misleading in this case. We had a similar case where the Automatus's output was misleading: https://github.com/ComplianceAsCode/content/issues/10823

jan-cerny commented 1 year ago

Today we saw a similar situation also on RHEL 8 when reviewing the /CoreOS/scap-security-guide/Sanity/test-rules-scenarios-per-profile PCI-DSS 5/5 productization test run. We used current upstream master branch as of 2023-07-22 as of HEAD https://github.com/ComplianceAsCode/content/commit/a96ccb9e2eb05e6706b4fb0144dad15e3ef6b60a.

However, in the RHEL 8 run, the issue is more rich - there are 2 scenarios error out and these are different test scenarios.

INFO - xccdf_org.ssgproject.content_rule_configure_crypto_policy
INFO - Script cis_l2.pass.sh using profile xccdf_org.ssgproject.content_profile_cis OK
INFO - Script cis_l2.pass.sh using profile xccdf_org.ssgproject.content_profile_cis_workstation_l2 OK
INFO - Script config_and_current_same_time.pass.sh using profile (all) OK
INFO - Script config_newer_than_current.fail.sh using profile (all) OK
INFO - Script missing_nss_config.fail.sh using profile xccdf_org.ssgproject.content_profile_ospp OK
INFO - Script missing_policy.fail.sh using profile xccdf_org.ssgproject.content_profile_ospp OK
INFO - Script missing_policy.fail.sh using profile xccdf_org.ssgproject.content_profile_standard OK
INFO - Script missing_policy_file.fail.sh using profile xccdf_org.ssgproject.content_profile_ospp OK
ERROR - Script missing_policy_file.fail.sh using profile xccdf_org.ssgproject.content_profile_standard found issue:
ERROR - Rule xccdf_org.ssgproject.content_rule_configure_crypto_policy has not been evaluated! Wrong profile selected in test scenario?
ERROR - The initial scan failed for rule 'xccdf_org.ssgproject.content_rule_configure_crypto_policy'.
INFO - Script nss_config_as_file.pass.sh using profile xccdf_org.ssgproject.content_profile_ospp OK
INFO - Script nss_config_as_symlink.pass.sh using profile xccdf_org.ssgproject.content_profile_ospp OK
INFO - Script policy_default_cis_l1.pass.sh using profile xccdf_org.ssgproject.content_profile_cis_server_l1 OK
ERROR - Script policy_default_cis_l1.pass.sh using profile xccdf_org.ssgproject.content_profile_cis_workstation_l1 found issue:
ERROR - Rule xccdf_org.ssgproject.content_rule_configure_crypto_policy has not been evaluated! Wrong profile selected in test scenario?
ERROR - The initial scan failed for rule 'xccdf_org.ssgproject.content_rule_configure_crypto_policy'.
INFO - Script policy_default_nosha1_set.pass.sh using profile xccdf_org.ssgproject.content_profile_e8 OK
INFO - Script policy_default_set.pass.sh using profile xccdf_org.ssgproject.content_profile_standard OK
INFO - Script policy_fips_ospp_set.pass.sh using profile xccdf_org.ssgproject.content_profile_ospp OK
INFO - Script wrong_policy.fail.sh using profile xccdf_org.ssgproject.content_profile_ospp OK
INFO - Script wrong_policy.fail.sh using profile xccdf_org.ssgproject.content_profile_standard OK
INFO - xccdf_org.ssgproject.content_rule_configure_kerberos_crypto_policy
INFO - Script kerberos_correct_policy.pass.sh using profile (all) OK
INFO - Script kerberos_missing_policy.fail.sh using profile (all) OK
INFO - Script kerberos_wrong_policy.fail.sh using profile (all) OK
INFO - xccdf_org.ssgproject.content_rule_configure_libreswan_crypto_policy
INFO - Script libreswan_not_installed.pass.sh using profile (all) OK
INFO - Script line_commented.fail.sh using profile (all) OK
INFO - Script line_is_there.pass.sh using profile (all) OK
INFO - Script line_not_there.fail.sh using profile (all) OK
INFO - Script wrong_value.fail.sh using profile (all) OK
vojtapolasek commented 1 year ago

I can confirm it happened also in stabilization branch as of 2023-07-20 while scanning for /CoreOS/scap-security-guide/Sanity/test-rules-scenarios-per-profile OSPP 5/5 on RHEL8: INFO - xccdf_org.ssgproject.content_rule_configure_crypto_policy ERROR - Script cis_l2.pass.sh using profile xccdf_org.ssgproject.content_profile_cis_workstation_l2 found issue: ERROR - Rule xccdf_org.ssgproject.content_rule_configure_crypto_policy has not been evaluated! Wrong profile selected in test scenario? ERROR - The initial scan failed for rule 'xccdf_org.ssgproject.content_rule_configure_crypto_policy'. ERROR - Script missing_policy.fail.sh using profile xccdf_org.ssgproject.content_profile_standard found issue: ERROR - Rule xccdf_org.ssgproject.content_rule_configure_crypto_policy has not been evaluated! Wrong profile selected in test scenario? ERROR - The initial scan failed for rule 'xccdf_org.ssgproject.content_rule_configure_crypto_policy'. ERROR - Script missing_policy_file.fail.sh using profile xccdf_org.ssgproject.content_profile_standard found issue: ERROR - Rule xccdf_org.ssgproject.content_rule_configure_crypto_policy has not been evaluated! Wrong profile selected in test scenario? ERROR - The initial scan failed for rule 'xccdf_org.ssgproject.content_rule_configure_crypto_policy'. ERROR - Script policy_default_cis_l1.pass.sh using profile xccdf_org.ssgproject.content_profile_cis_workstation_l1 found issue: ERROR - Rule xccdf_org.ssgproject.content_rule_configure_crypto_policy has not been evaluated! Wrong profile selected in test scenario? ERROR - The initial scan failed for rule 'xccdf_org.ssgproject.content_rule_configure_crypto_policy'. ERROR - Script wrong_policy.fail.sh using profile xccdf_org.ssgproject.content_profile_standard found issue: ERROR - Rule xccdf_org.ssgproject.content_rule_configure_crypto_policy has not been evaluated! Wrong profile selected in test scenario? ERROR - The initial scan failed for rule 'xccdf_org.ssgproject.content_rule_configure_crypto_policy'.

While scanning /CoreOS/scap-security-guide/Sanity/test-rules-scenarios-per-profile PCI-DSS 5/5 on RHEL9, I see: INFO - xccdf_org.ssgproject.content_rule_configure_crypto_policy ERROR - Script policy_default_cis_l1.pass.sh using profile xccdf_org.ssgproject.content_profile_cis_workstation_l1 found issue: ERROR - Rule xccdf_org.ssgproject.content_rule_configure_crypto_policy has not been evaluated! Wrong profile selected in test scenario? ERROR - The initial scan failed for rule 'xccdf_org.ssgproject.content_rule_configure_crypto_policy'.

jan-cerny commented 1 year ago

Hi, this looks very similar or like a duplicate of https://github.com/ComplianceAsCode/content/issues/9058

vojtapolasek commented 1 year ago

I was not able to reliably reproduce this problem.

marcusburghardt commented 1 year ago

Still present in last productization review.

vojtapolasek commented 1 year ago

Still ppresent in the latest productization review.

jan-cerny commented 1 year ago

This issue is now present in the latest productization run in the test results of the test /CoreOS/scap-security-guide/Sanity/test-rules-scenarios-per-profile OSPP 5/5 on RHEL-9.3.0-20230909.0 with the latest upstream version as of 2023-09-09 as of HEAD 7c741f2.

marcusburghardt commented 1 year ago

Still present in the latest productization review.

Mab879 commented 1 year ago

This issue is still present and I can reproduce locally with ./automatus.py rule --datastream ../build/ssg-rhel9-ds.xml --libvirt qemu:///system automatus_rhel9_4 --scenario cis_l2.pass.sh configure_crypto_policy but trying to run in combined mode locally I can't reproduce it.

jan-cerny commented 1 year ago

Thanks for noticing this!

Moreover, In the latest productization as of HEAD 8c9ed4f as of 2023-10-14 I can see also a lot of problems in the combined mode as well. Specifically, I observe this in test /CoreOS/scap-security-guide/Sanity/test-rules-scenarios-per-profile OSPP 5/5 on RHEL 9.

:: [ 15:54:15 ] :: [  BEGIN   ] :: Test suite combined mode for ospp profile - bash remediations :: actually running 'python3 /tmp/tmp.3fdSZr7RxU/rpmbuild/BUILD/scap-security-guide-0.1.71/tests/test_suite.py combined                 --slice 5 5                 --libvirt qemu:///system test_suite_vm                 --datastream /usr/share/xml/scap/ssg/content/ssg-rhel9-ds.xml                 --mode online                 --remediate-using bash                 --duplicate-templates                 --no-reports                 xccdf_org.ssgproject.content_profile_ospp'

[ ... snip ... ]

INFO - xccdf_org.ssgproject.content_rule_configure_crypto_policy
WARNING - The script cis_l2.pass.sh is not applicable for the xccdf_org.ssgproject.content_profile_ospp profile.
INFO - Script config_and_current_same_time.pass.sh using profile (all) OK
INFO - Script config_newer_than_current.fail.sh using profile (all) OK
ERROR - Script missing_nss_config.fail.sh using profile xccdf_org.ssgproject.content_profile_ospp found issue:
ERROR - Rule xccdf_org.ssgproject.content_rule_configure_crypto_policy has not been evaluated! Wrong profile selected in test scenario?
ERROR - The initial scan failed for rule 'xccdf_org.ssgproject.content_rule_configure_crypto_policy'.
INFO - Script missing_policy.fail.sh using profile xccdf_org.ssgproject.content_profile_ospp OK
ERROR - Rule xccdf_org.ssgproject.content_rule_configure_crypto_policy has not been evaluated! Wrong profile selected in test scenario?
ERROR - The check after remediation failed for rule 'xccdf_org.ssgproject.content_rule_configure_crypto_policy'.
INFO - Script missing_policy_file.fail.sh using profile xccdf_org.ssgproject.content_profile_ospp OK
ERROR - Rule xccdf_org.ssgproject.content_rule_configure_crypto_policy has not been evaluated! Wrong profile selected in test scenario?
ERROR - The check after remediation failed for rule 'xccdf_org.ssgproject.content_rule_configure_crypto_policy'.
ERROR - Script nss_config_as_file.pass.sh using profile xccdf_org.ssgproject.content_profile_ospp found issue:
ERROR - Rule xccdf_org.ssgproject.content_rule_configure_crypto_policy has not been evaluated! Wrong profile selected in test scenario?
ERROR - The initial scan failed for rule 'xccdf_org.ssgproject.content_rule_configure_crypto_policy'.
ERROR - Script nss_config_as_symlink.pass.sh using profile xccdf_org.ssgproject.content_profile_ospp found issue:
ERROR - Rule xccdf_org.ssgproject.content_rule_configure_crypto_policy has not been evaluated! Wrong profile selected in test scenario?
ERROR - The initial scan failed for rule 'xccdf_org.ssgproject.content_rule_configure_crypto_policy'.
WARNING - The script policy_default_cis_l1.pass.sh is not applicable for the xccdf_org.ssgproject.content_profile_ospp profile.
WARNING - The script policy_default_nosha1_set.pass.sh is not applicable for the xccdf_org.ssgproject.content_profile_ospp profile.
WARNING - The script policy_default_set.pass.sh is not applicable for the xccdf_org.ssgproject.content_profile_ospp profile.
ERROR - Script policy_fips_ospp_set.pass.sh using profile xccdf_org.ssgproject.content_profile_ospp found issue:
ERROR - Rule xccdf_org.ssgproject.content_rule_configure_crypto_policy has not been evaluated! Wrong profile selected in test scenario?
ERROR - The initial scan failed for rule 'xccdf_org.ssgproject.content_rule_configure_crypto_policy'.
INFO - Script wrong_policy.fail.sh using profile xccdf_org.ssgproject.content_profile_ospp OK
ERROR - Rule xccdf_org.ssgproject.content_rule_configure_crypto_policy has not been evaluated! Wrong profile selected in test scenario?
ERROR - The check after remediation failed for rule 'xccdf_org.ssgproject.content_rule_configure_crypto_policy'.
jan-cerny commented 1 year ago

This seems to be a case of a misleading output of automatus, because a detailed look into the logs shows that permission has been denied. For example, configure_crypto_policy-nss_config_as_symlink.pass.sh-initial.verbose.log:

Warning: Permanently added '192.168.122.126' (RSA) to the list of known hosts.
Permission denied, please try again.
Permission denied, please try again.
root@192.168.122.126: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).
Failed to connect!

Other .verbose.log files have a similar content.

Additional information: this is RHEL-9.3.0-updates-20231014.4

jan-cerny commented 1 year ago

I tried to reproduce the problem from the previous comment (https://github.com/ComplianceAsCode/content/issues/10895#issuecomment-1766389306) locally with the same compose but I haven't reproduced it.

@Mab879 I can't reproduce the issue that you mentioned in https://github.com/ComplianceAsCode/content/issues/10895#issuecomment-1755502479 . Could you share some more details?

Mab879 commented 1 year ago

I tried to reproduce the problem from the previous comment (#10895 (comment)) locally with the same compose but I haven't reproduced it.

@Mab879 I can't reproduce the issue that you mentioned in #10895 (comment) . Could you share some more details?

I can't reproduce it anymore as well.

mildas commented 1 year ago

Setting blocked label. We will plan the investigation for some of upcoming quarters.