search

ComplianceAsCode / content

Security automation content in SCAP, Bash, Ansible, and other formats
https://complianceascode.readthedocs.io/en/latest/
Other
2.14k stars 677 forks source link

Automatúš in combined mode executes irrelevant scenarios #9058

Open jan-cerny opened 2 years ago

jan-cerny commented 2 years ago

Description of problem:

When Automatus is executed in combined mode, it executes test scenarios that are marked as scenarios for a different profile than the profile being tested. Then it produces an error message that the rule hasn't been evaluated and asks whether a wrong profile is used in the test scenario.

SCAP Security Guide Version:

scap-security-guide-0.1.63-1.git46faa00.el9.noarch

Operating System Version:

RHEL 9.1.0

Steps to Reproduce:

This issue has been discovered during "productization" task /CoreOS/scap-security-guide/Sanity/test-rules-scenarios-per-profile.

  1. python3 /tmp/tmp.Kz4GTNtOGv/rpmbuild/BUILD/scap-security-guide-0.1.63/tests/test_suite.py combined --slice 3 3 --libvirt qemu:///system test_suite_vm --datastream /usr/share/xml/scap/ssg/content/ssg-rhel9-ds.xml --mode online --remediate-using bash --duplicate-templates --no-reports xccdf_org.ssgproject.content_profile_ospp

Actual Results:

......

INFO - xccdf_org.ssgproject.content_rule_configure_crypto_policy
INFO - Script config_and_current_same_time.pass.sh using profile (all) OK
INFO - Script config_newer_than_current.fail.sh using profile (all) OK
INFO - Script missing_nss_config.fail.sh using profile xccdf_org.ssgproject.content_profile_ospp OK
INFO - Script missing_policy.fail.sh using profile xccdf_org.ssgproject.content_profile_ospp OK
INFO - Script missing_policy_file.fail.sh using profile xccdf_org.ssgproject.content_profile_ospp OK
INFO - Script nss_config_as_file.pass.sh using profile xccdf_org.ssgproject.content_profile_ospp OK
INFO - Script nss_config_as_symlink.pass.sh using profile xccdf_org.ssgproject.content_profile_ospp OK
INFO - Script policy_default_cis_l1.pass.sh using profile xccdf_org.ssgproject.content_profile_cis_server_l1 OK
ERROR - Script policy_default_cis_l1.pass.sh using profile xccdf_org.ssgproject.content_profile_cis_workstation_l1 found issue:
ERROR - Rule xccdf_org.ssgproject.content_rule_configure_crypto_policy has not been evaluated! Wrong profile selected in test scenario?
ERROR - The initial scan failed for rule 'xccdf_org.ssgproject.content_rule_configure_crypto_policy'.
INFO - Script policy_default_nosha1_set.pass.sh using profile xccdf_org.ssgproject.content_profile_e8 OK
WARNING - Script policy_default_set.pass.sh - profile xccdf_org.ssgproject.content_profile_standard not found in datastream
INFO - Script policy_fips_ospp_set.pass.sh using profile xccdf_org.ssgproject.content_profile_ospp OK
INFO - Script policy_future_cis_l2.pass.sh using profile xccdf_org.ssgproject.content_profile_cis OK
ERROR - Script policy_future_cis_l2.pass.sh using profile xccdf_org.ssgproject.content_profile_cis_workstation_l2 found issue:
ERROR - Rule xccdf_org.ssgproject.content_rule_configure_crypto_policy has not been evaluated! Wrong profile selected in test scenario?
ERROR - The initial scan failed for rule 'xccdf_org.ssgproject.content_rule_configure_crypto_policy'.
INFO - Script wrong_policy.fail.sh using profile xccdf_org.ssgproject.content_profile_ospp OK

.........

Expected Results:

No errors.

The expected results needs to be clarified. I expect that the scenarios policy_default_cis_l1.pass.sh and policy_future_cis_l2.pass.sh will not be executed at all in this situation because we are testing OSPP profile and these scenarios contain # profiles header that limits them to CIS L1 or CIS L2 profile.

Additional Information/Debugging Steps:

no

jan-cerny commented 2 years ago

I don't have this problem when I run it on Fedora 35.

jan-cerny commented 1 year ago

A very similar problem was found today (2022-10-24) during productization test run of the test case "/CoreOS/scap-security-guide/Sanity/test-rules-scenarios-per-profile PCI-DSS 3/3" on RHEL 9.1 with the current upstream as of 2022-10-22 as of HEAD 3748e7b.

:: [ 06:53:49 ] :: [  BEGIN   ] :: Test suite combined mode for pci-dss profile - bash remediations :: actually running 'python3 /tmp/tmp.EbwRuXtpdO/rpmbuild/BUILD/scap-security-guide-0.1.65/tests/test_suite.py combined                 --slice 3 3                 --libvirt qemu:///system test_suite_vm                 --datastream /usr/share/xml/scap/ssg/content/ssg-rhel9-ds.xml                 --mode online                 --remediate-using bash                 --duplicate-templates                 --no-reports                 xccdf_org.ssgproject.content_profile_pci-dss'
WARNING - You call Automatus using the legacy 'test_suite.py' script, use the 'automatus.py' instead

INFO - The base image option has not been specified, choosing libvirt-based test environment.
INFO - Logging into /tmp/tmp.EbwRuXtpdO/logs/combined-custom-2022-10-22-0653/test_suite.log
INFO - Performing combined test using profile: xccdf_org.ssgproject.content_profile_pci-dss
WARNING - Script sssd_parameter_false.fail.sh is not applicable on given platform
WARNING - Script sssd_parameter_missing.fail.sh is not applicable on given platform
WARNING - Script sssd_parameter_missing_file.fail.sh is not applicable on given platform
WARNING - Script sssd_parameter_true.pass.sh is not applicable on given platform
WARNING - Script correct_value.pass.sh is not applicable on given platform
WARNING - Script default_config.fail.sh is not applicable on given platform
WARNING - Script silent_present.fail.sh is not applicable on given platform
WARNING - Script wrong_control.fail.sh is not applicable on given platform
WARNING - Script wrong_value.fail.sh is not applicable on given platform
WARNING - Script argument_missing.fail.sh is not applicable on given platform
WARNING - Script correct_value.pass.sh is not applicable on given platform
WARNING - Script wrong_value.fail.sh is not applicable on given platform
WARNING - Script pam_faillock_expected_pam_files.pass.sh is not applicable on given platform
WARNING - Script pam_faillock_lenient_pam_files.fail.sh is not applicable on given platform
WARNING - Script pam_faillock_multiple_pam_unix_pam_files.fail.sh is not applicable on given platform
WARNING - Script pam_faillock_stricter_pam_files.pass.sh is not applicable on given platform
WARNING - Script pam_faillock_expected_pam_files.pass.sh is not applicable on given platform
WARNING - Script pam_faillock_lenient_pam_files.fail.sh is not applicable on given platform
WARNING - Script pam_faillock_multiple_pam_unix_pam_files.fail.sh is not applicable on given platform
WARNING - Script pam_faillock_stricter_pam_files.pass.sh is not applicable on given platform
WARNING - Script correct.pass.sh is not applicable on given platform
WARNING - Script missing.fail.sh is not applicable on given platform
WARNING - Script wrong_control.fail.sh is not applicable on given platform
WARNING - Script correct.pass.sh is not applicable on given platform
WARNING - Script missing.fail.sh is not applicable on given platform
WARNING - Script wrong_control.fail.sh is not applicable on given platform
WARNING - Script commented_standard_fedora.fail.sh is not applicable on given platform
WARNING - Script correct_standard_fedora.pass.sh is not applicable on given platform
WARNING - Script incorrect_standard_fedora.fail.sh is not applicable on given platform
WARNING - Script no_nullok.pass.sh is not applicable on given platform
WARNING - Script nullok_commented.pass.sh is not applicable on given platform
WARNING - Script nullok_present.fail.sh is not applicable on given platform
WARNING - Script nullok_present_password_auth.fail.sh is not applicable on given platform
WARNING - Script missing_auid_filter.fail.sh is not applicable on given platform
WARNING - Script missing_auid_filter.fail.sh is not applicable on given platform
WARNING - Script missing_auid_filter.fail.sh is not applicable on given platform
WARNING - Script auditctl_default.fail.sh is not applicable on given platform
WARNING - Script auditctl_missing_rule.fail.sh is not applicable on given platform
WARNING - Script auditctl_one_rule.fail.sh is not applicable on given platform
WARNING - Script auditctl_rules_configured.pass.sh is not applicable on given platform
WARNING - Script auditctl_rules_with_perm_x.fail.sh is not applicable on given platform
WARNING - Script augenrules_default.fail.sh is not applicable on given platform
WARNING - Script augenrules_duplicated.fail.sh is not applicable on given platform
WARNING - Script augenrules_missing_rule.fail.sh is not applicable on given platform
WARNING - Script augenrules_one_rule.fail.sh is not applicable on given platform
WARNING - Script augenrules_rules_configured.pass.sh is not applicable on given platform
WARNING - Script augenrules_rules_configured_mixed_keys.pass.sh is not applicable on given platform
WARNING - Script augenrules_rules_with_perm_x.fail.sh is not applicable on given platform
WARNING - Script augenrules_two_rules_mixed_keys.fail.sh is not applicable on given platform
WARNING - Script augenrules_two_rules_sep_files.fail.sh is not applicable on given platform
WARNING - Script rules_with_own_key.pass.sh is not applicable on given platform
WARNING - Script correct_permissions.pass.sh is not applicable on given platform
WARNING - Script incorrect_permissions.fail.sh is not applicable on given platform
WARNING - Script audisp_syslog_plugin_activated.pass.sh is not applicable on given platform
WARNING - Script audisp_syslog_plugin_activated_not_there.fail.sh is not applicable on given platform
WARNING - Script audisp_syslog_plugin_not_activated.fail.sh is not applicable on given platform
WARNING - Script arg_not_there_grubenv.fail.sh is not applicable on given platform
WARNING - Script wrong_value.fail.sh is not applicable on given platform
WARNING - Script arg_not_there_etcdefaultgrub_recovery_disabled.fail.sh is not applicable on given platform
WARNING - Script arg_not_there_rhel7.fail.sh is not applicable on given platform
WARNING - Script blank_grubenv_rhel8.fail.sh is not applicable on given platform
WARNING - Script correct_recovery_disabled.pass.sh is not applicable on given platform
WARNING - Script double_value_rhel7.fail.sh is not applicable on given platform
WARNING - Script double_value_rhel8.fail.sh is not applicable on given platform
WARNING - Script wrong_value_etcdefaultgrub.fail.sh is not applicable on given platform
WARNING - Script wrong_value_etcdefaultgrub_recovery_disabled.fail.sh is not applicable on given platform
WARNING - Script wrong_value_rhel7.fail.sh is not applicable on given platform
WARNING - Script wrong_value_rhel8.fail.sh is not applicable on given platform
WARNING - Script include_is_other.fail.sh is not applicable on given platform
WARNING - Script include_is_root.pass.sh is not applicable on given platform
WARNING - Script include_is_root_IncludeConfig_is_other.fail.sh is not applicable on given platform
WARNING - Script include_is_root_IncludeConfig_is_root.pass.sh is not applicable on given platform
WARNING - Script include_multiline_is_root.pass.sh is not applicable on given platform
WARNING - Script include_is_other.fail.sh is not applicable on given platform
WARNING - Script include_is_root.pass.sh is not applicable on given platform
WARNING - Script include_is_root_IncludeConfig_is_other.fail.sh is not applicable on given platform
WARNING - Script include_is_root_IncludeConfig_is_root.pass.sh is not applicable on given platform
WARNING - Script include_multiline_is_root.pass.sh is not applicable on given platform
WARNING - Script include_multiline_perms_0600.pass.sh is not applicable on given platform
WARNING - Script include_perms_0600.pass.sh is not applicable on given platform
WARNING - Script include_perms_0600_IncludeConfig_perms_0600.pass.sh is not applicable on given platform
WARNING - Script include_perms_0600_IncludeConfig_perms_0601.fail.sh is not applicable on given platform
WARNING - Script include_perms_0600_IncludeConfig_perms_0601_hidden.pass.sh is not applicable on given platform
WARNING - Script include_perms_0600_IncludeConfig_perms_0601_missing.pass.sh is not applicable on given platform
WARNING - Script include_perms_0601.fail.sh is not applicable on given platform
WARNING - Script logrotate_configured.pass.sh is not applicable on given platform
WARNING - Script fedora_key.fail.sh is not applicable on given platform
INFO - xccdf_org.ssgproject.content_rule_file_owner_grub2_cfg
INFO - Script incorrect_owner.fail.sh using profile (all) OK
INFO - Script correct_owner.pass.sh using profile (all) OK
INFO - Script missing_file_test.pass.sh using profile (all) OK
INFO - xccdf_org.ssgproject.content_rule_rsyslog_files_groupownership
INFO - Script IncludeConfig_is_other.fail.sh using profile (all) OK
WARNING - No remediation is available for rule 'xccdf_org.ssgproject.content_rule_rsyslog_files_groupownership'.
INFO - Script IncludeConfig_is_root.pass.sh using profile (all) OK
INFO - Script is_other.fail.sh using profile (all) OK
WARNING - No remediation is available for rule 'xccdf_org.ssgproject.content_rule_rsyslog_files_groupownership'.
INFO - Script is_root.pass.sh using profile (all) OK
INFO - xccdf_org.ssgproject.content_rule_rsyslog_files_ownership
INFO - Script IncludeConfig_is_other.fail.sh using profile (all) OK
WARNING - No remediation is available for rule 'xccdf_org.ssgproject.content_rule_rsyslog_files_ownership'.
INFO - Script IncludeConfig_is_root.pass.sh using profile (all) OK
INFO - Script is_other.fail.sh using profile (all) OK
WARNING - No remediation is available for rule 'xccdf_org.ssgproject.content_rule_rsyslog_files_ownership'.
INFO - Script is_root.pass.sh using profile (all) OK
INFO - xccdf_org.ssgproject.content_rule_rsyslog_files_permissions
INFO - Script IncludeConfig_glob_perms_0600.pass.sh using profile (all) OK
INFO - Script IncludeConfig_glob_perms_0601.fail.sh using profile (all) OK
INFO - Script IncludeConfig_perms_0600.pass.sh using profile (all) OK
INFO - Script IncludeConfig_perms_0601.fail.sh using profile (all) OK
INFO - Script include_config_syntax_perms_0600.pass.sh using profile (all) OK
INFO - Script include_config_syntax_perms_0601.fail.sh using profile (all) OK
INFO - Script perms_0600.pass.sh using profile (all) OK
INFO - Script perms_0601.fail.sh using profile (all) OK
INFO - xccdf_org.ssgproject.content_rule_ensure_logrotate_activated
INFO - Script logrotate_conf_extra_monthly.fail.sh using profile (all) OK
INFO - Script logrotate_conf_weekly.fail.sh using profile (all) OK
INFO - Script logrotate_no_config.fail.sh using profile (all) OK
INFO - Script logrotate_no_cron_daily.fail.sh using profile (all) OK
INFO - xccdf_org.ssgproject.content_rule_file_permissions_etc_group
INFO - Script correct_permissions.pass.sh using profile (all) OK
INFO - Script stricter_permisions.pass.sh using profile (all) OK
INFO - Script lenient_permissions.fail.sh using profile (all) OK
INFO - xccdf_org.ssgproject.content_rule_file_permissions_etc_shadow
INFO - Script correct_permissions.pass.sh using profile (all) OK
INFO - xccdf_org.ssgproject.content_rule_dconf_db_up_to_date
INFO - Script db_not_up_to_date.fail.sh using profile (all) OK
INFO - Script db_up_to_date.pass.sh using profile (all) OK
INFO - Script no_db_files.fail.sh using profile (all) OK
INFO - Script no_keyfiles.pass.sh using profile (all) OK
INFO - xccdf_org.ssgproject.content_rule_dconf_gnome_screensaver_idle_activation_enabled
INFO - Script comment.fail.sh using profile (all) OK
INFO - Script correct_value.pass.sh using profile (all) OK
INFO - Script missing_lock.fail.sh using profile (all) OK
INFO - Script setting_not_there.fail.sh using profile (all) OK
INFO - Script wrong_value.fail.sh using profile (all) OK
INFO - xccdf_org.ssgproject.content_rule_dconf_gnome_screensaver_idle_delay
INFO - Script comment.fail.sh using profile (all) OK
INFO - Script correct_value.pass.sh using profile (all) OK
INFO - Script setting_not_there.fail.sh using profile (all) OK
INFO - Script wrong_value.fail.sh using profile (all) OK
INFO - xccdf_org.ssgproject.content_rule_dconf_gnome_screensaver_lock_enabled
INFO - Script comment.fail.sh using profile (all) OK
INFO - Script correct_value.pass.sh using profile (all) OK
INFO - Script correct_value_unlocked.fail.sh using profile (all) OK
INFO - Script setting_not_there.fail.sh using profile (all) OK
INFO - Script wrong_value.fail.sh using profile (all) OK
INFO - xccdf_org.ssgproject.content_rule_dconf_gnome_screensaver_mode_blank
INFO - Script comment.fail.sh using profile (all) OK
INFO - Script correct_value.pass.sh using profile (all) OK
INFO - Script correct_value_not_locked.fail.sh using profile (all) OK
INFO - Script setting_not_there.fail.sh using profile (all) OK
INFO - Script wrong_value.fail.sh using profile (all) OK
INFO - xccdf_org.ssgproject.content_rule_dconf_gnome_session_idle_user_locks
INFO - Script comented_value.fail.sh using profile (all) OK
INFO - Script correct_value.pass.sh using profile (all) OK
INFO - Script missing_value.fail.sh using profile (all) OK
INFO - xccdf_org.ssgproject.content_rule_configure_bind_crypto_policy
INFO - Script absent.fail.sh using profile (all) OK
INFO - Script bind_not_installed.pass.sh using profile (all) OK
INFO - Script no_config_file.fail.sh using profile (all) OK
INFO - Script ok.pass.sh using profile (all) OK
INFO - Script overrides.fail.sh using profile (all) OK
INFO - xccdf_org.ssgproject.content_rule_configure_crypto_policy
INFO - Script cis_l2.pass.sh using profile xccdf_org.ssgproject.content_profile_cis OK
ERROR - Script cis_l2.pass.sh using profile xccdf_org.ssgproject.content_profile_cis_workstation_l2 found issue:
ERROR - Rule xccdf_org.ssgproject.content_rule_configure_crypto_policy has not been evaluated! Wrong profile selected in test scenario?
ERROR - The initial scan failed for rule 'xccdf_org.ssgproject.content_rule_configure_crypto_policy'.
INFO - Script config_and_current_same_time.pass.sh using profile (all) OK
INFO - Script config_newer_than_current.fail.sh using profile (all) OK
INFO - Script missing_nss_config.fail.sh using profile xccdf_org.ssgproject.content_profile_ospp OK
INFO - Script missing_policy.fail.sh using profile xccdf_org.ssgproject.content_profile_ospp OK
INFO - Script missing_policy_file.fail.sh using profile xccdf_org.ssgproject.content_profile_ospp OK
INFO - Script nss_config_as_file.pass.sh using profile xccdf_org.ssgproject.content_profile_ospp OK
INFO - Script nss_config_as_symlink.pass.sh using profile xccdf_org.ssgproject.content_profile_ospp OK
INFO - Script policy_default_cis_l1.pass.sh using profile xccdf_org.ssgproject.content_profile_cis_server_l1 OK
ERROR - Script policy_default_cis_l1.pass.sh using profile xccdf_org.ssgproject.content_profile_cis_workstation_l1 found issue:
ERROR - Rule xccdf_org.ssgproject.content_rule_configure_crypto_policy has not been evaluated! Wrong profile selected in test scenario?
ERROR - The initial scan failed for rule 'xccdf_org.ssgproject.content_rule_configure_crypto_policy'.
INFO - Script policy_default_nosha1_set.pass.sh using profile xccdf_org.ssgproject.content_profile_e8 OK
WARNING - Script policy_default_set.pass.sh - profile xccdf_org.ssgproject.content_profile_standard not found in datastream
INFO - Script policy_fips_ospp_set.pass.sh using profile xccdf_org.ssgproject.content_profile_ospp OK
INFO - Script wrong_policy.fail.sh using profile xccdf_org.ssgproject.content_profile_ospp OK
INFO - xccdf_org.ssgproject.content_rule_configure_kerberos_crypto_policy
INFO - Script kerberos_correct_policy.pass.sh using profile (all) OK
INFO - Script kerberos_missing_policy.fail.sh using profile (all) OK
INFO - Script kerberos_wrong_policy.fail.sh using profile (all) OK
INFO - xccdf_org.ssgproject.content_rule_configure_libreswan_crypto_policy
INFO - Script libreswan_not_installed.pass.sh using profile (all) OK
INFO - Script line_commented.fail.sh using profile (all) OK
INFO - Script line_is_there.pass.sh using profile (all) OK
INFO - Script line_not_there.fail.sh using profile (all) OK
INFO - Script wrong_value.fail.sh using profile (all) OK
INFO - xccdf_org.ssgproject.content_rule_configure_openssl_crypto_policy
INFO - Script include_with_equal_sign.pass.sh using profile (all) OK
INFO - Script nothing.fail.sh using profile (all) OK
INFO - Script ok.pass.sh using profile (all) OK
INFO - Script section_not_include.fail.sh using profile (all) OK
INFO - Script wrong.fail.sh using profile (all) OK
INFO - xccdf_org.ssgproject.content_rule_configure_ssh_crypto_policy
INFO - Script absent.pass.sh using profile (all) OK
INFO - Script case_insensitive_present.fail.sh using profile (all) OK
INFO - Script comment.pass.sh using profile (all) OK
INFO - Script no_config_file.pass.sh using profile (all) OK
INFO - Script overrides.fail.sh using profile (all) OK
INFO - xccdf_org.ssgproject.content_rule_aide_build_database
INFO - Script db_malformed.fail.sh using profile (all) OK
INFO - Script db_not_present.fail.sh using profile (all) OK
INFO - xccdf_org.ssgproject.content_rule_aide_periodic_cron_checking
INFO - Script aide_not_installed.fail.sh using profile (all) OK
INFO - Script crontab_daily.pass.sh using profile (all) OK
INFO - Script crontab_daily_shortcut.pass.sh using profile (all) OK
INFO - Script crontab_monthly.fail.sh using profile (all) OK
INFO - Script crontab_two_days_week.pass.sh using profile (all) OK
INFO - Script crontab_weekly_on_exact_day.pass.sh using profile (all) OK
INFO - Script crontab_weekly_shortcut.pass.sh using profile (all) OK
INFO - Script crontab_weekly_word.pass.sh using profile (all) OK
INFO - Script crontab_yearly.fail.sh using profile (all) OK
INFO - Script not_in_cron.fail.sh using profile (all) OK
INFO - xccdf_org.ssgproject.content_rule_rpm_verify_hashes
INFO - Script bad_document.fail.sh using profile (all) OK
INFO - Script fresh_system.pass.sh using profile (all) OK
INFO - xccdf_org.ssgproject.content_rule_rpm_verify_permissions
INFO - Script all_permissions_ok.pass.sh using profile (all) OK
INFO - Script bad_permissions.fail.sh using profile (all) OK
INFO - xccdf_org.ssgproject.content_rule_ensure_gpgcheck_globally_activated
INFO - Script comment.fail.sh using profile (all) OK
INFO - Script correct_value.pass.sh using profile (all) OK
INFO - Script line_not_there.fail.sh using profile (all) OK
INFO - Script wrong_value.fail.sh using profile (all) OK
INFO - xccdf_org.ssgproject.content_rule_ensure_gpgcheck_never_disabled
INFO - Script gpgcheck_disabled.fail.sh using profile (all) OK
INFO - Script gpgcheck_enabled.pass.sh using profile (all) OK
INFO - xccdf_org.ssgproject.content_rule_ensure_redhat_gpgkey_installed
INFO - Script key_installed.pass.sh using profile (all) OK
INFO - Script missing_key.fail.sh using profile (all) OK
INFO - The following rule(s) were not tested:
INFO - account_disable_post_pw_expiration
INFO - account_unique_name
INFO - accounts_maximum_age_login_defs
INFO - accounts_password_all_shadowed
INFO - accounts_password_pam_dcredit
INFO - accounts_password_pam_lcredit
INFO - accounts_password_pam_minlen
INFO - accounts_password_pam_ucredit
INFO - accounts_password_pam_unix_remember
INFO - accounts_passwords_pam_faillock_deny
INFO - accounts_passwords_pam_faillock_unlock_time
INFO - audit_rules_dac_modification_chmod
INFO - audit_rules_dac_modification_chown
INFO - audit_rules_dac_modification_fchmod
INFO - audit_rules_dac_modification_fchmodat
INFO - audit_rules_dac_modification_fchown
INFO - audit_rules_dac_modification_fchownat
INFO - audit_rules_dac_modification_fremovexattr
INFO - audit_rules_dac_modification_fsetxattr
INFO - audit_rules_dac_modification_lchown
INFO - audit_rules_dac_modification_lremovexattr
INFO - audit_rules_dac_modification_lsetxattr
INFO - audit_rules_dac_modification_removexattr
INFO - audit_rules_dac_modification_setxattr
INFO - audit_rules_file_deletion_events_rename
INFO - audit_rules_file_deletion_events_renameat
INFO - audit_rules_file_deletion_events_rmdir
INFO - audit_rules_file_deletion_events_unlink
INFO - audit_rules_file_deletion_events_unlinkat
INFO - audit_rules_immutable
INFO - audit_rules_kernel_module_loading_delete
INFO - audit_rules_kernel_module_loading_finit
INFO - audit_rules_kernel_module_loading_init
INFO - audit_rules_login_events
INFO - audit_rules_mac_modification
INFO - audit_rules_media_export
INFO - audit_rules_networkconfig_modification
INFO - audit_rules_privileged_commands
INFO - audit_rules_session_events
INFO - audit_rules_sysadmin_actions
INFO - audit_rules_time_adjtimex
INFO - audit_rules_time_clock_settime
INFO - audit_rules_time_settimeofday
INFO - audit_rules_time_stime
INFO - audit_rules_time_watch_localtime
INFO - audit_rules_unsuccessful_file_modification_creat
INFO - audit_rules_unsuccessful_file_modification_ftruncate
INFO - audit_rules_unsuccessful_file_modification_open
INFO - audit_rules_unsuccessful_file_modification_open_by_handle_at
INFO - audit_rules_unsuccessful_file_modification_openat
INFO - audit_rules_unsuccessful_file_modification_truncate
INFO - audit_rules_usergroup_modification_group
INFO - audit_rules_usergroup_modification_gshadow
INFO - audit_rules_usergroup_modification_opasswd
INFO - audit_rules_usergroup_modification_passwd
INFO - audit_rules_usergroup_modification_shadow
INFO - auditd_audispd_syslog_plugin_activated
INFO - auditd_data_retention_action_mail_acct
INFO - auditd_data_retention_admin_space_left_action
INFO - auditd_data_retention_max_log_file
INFO - auditd_data_retention_max_log_file_action
INFO - auditd_data_retention_num_logs
INFO - auditd_data_retention_space_left_action
INFO - chronyd_specify_remote_server
INFO - configure_opensc_card_drivers
INFO - display_login_attempts
INFO - enable_authselect
INFO - file_groupowner_etc_group
INFO - file_groupowner_etc_passwd
INFO - file_groupowner_etc_shadow
INFO - file_groupowner_grub2_cfg
INFO - file_owner_etc_group
INFO - file_owner_etc_passwd
INFO - file_owner_etc_shadow
INFO - file_ownership_var_log_audit
INFO - file_permissions_etc_passwd
INFO - file_permissions_var_log_audit
INFO - force_opensc_card_drivers
INFO - gid_passwd_group_same
INFO - grub2_audit_argument
INFO - install_hids
INFO - no_empty_passwords
INFO - package_aide_installed
INFO - package_audispd-plugins_installed
INFO - package_libreswan_installed
INFO - package_opensc_installed
INFO - package_pcsc-lite_installed
INFO - security_patches_up_to_date
INFO - service_auditd_enabled
INFO - service_chronyd_enabled
INFO - service_pcscd_enabled
INFO - set_password_hashing_algorithm_libuserconf
INFO - set_password_hashing_algorithm_logindefs
INFO - set_password_hashing_algorithm_passwordauth
INFO - set_password_hashing_algorithm_systemauth
INFO - sshd_use_directory_configuration
INFO - sssd_enable_smartcards

WARNING - You call Automatus using the legacy 'test_suite.py' script, use the 'automatus.py' instead
Setting console output to log level INFO
jan-cerny commented 1 year ago

Today (2022-11-01), we found a very similar problem, but maybe not the exact one.

This time, we used RPM built from current upstream as of 2022-10-29 as of HEAD 4b5551f.

We executed the combined mode for OSPP profile on a RHEL 8.7 system.

python3 /tmp/tmp.og8FEgo9zl/rpmbuild/BUILD/scap-security-guide-0.1.65/tests/test_suite.py combined --slice 2 3 --libvirt qemu:///system test_suite_vm --datastream /usr/share/xml/scap/ssg/content/ssg-rhel8-ds.xml --mode online --remediate-using ansible --duplicate-templates --no-reports xccdf_org.ssgproject.content_profile_ospp

In the output, we can see an awkward output for the rule accounts_umask_etc_bashrc:

...
INFO - xccdf_org.ssgproject.content_rule_accounts_umask_etc_bashrc
INFO - Script missing.fail.sh using profile (all) OK
INFO - Script ospp_cis_correct.pass.sh using profile xccdf_org.ssgproject.content_profile_cis OK
ERROR - Script ospp_cis_correct.pass.sh using profile xccdf_org.ssgproject.content_profile_ospp found issue:
ERROR - Rule xccdf_org.ssgproject.content_rule_accounts_umask_etc_bashrc has not been evaluated! Wrong profile selected in test scenario?
ERROR - The initial scan failed for rule 'xccdf_org.ssgproject.content_rule_accounts_umask_etc_bashrc'.
INFO - Script stig_correct.pass.sh using profile xccdf_org.ssgproject.content_profile_stig OK
INFO - Script super_compliant.pass.sh using profile (all) OK
INFO - Script wrong.fail.sh using profile (all) OK
INFO - Script wrong_multiple.fail.sh using profile (all) OK
...

The offended test scenario ospp_cis_correct.pass.sh has the following profiles keyword in its header:

# profiles = xccdf_org.ssgproject.content_profile_cis, xccdf_org.ssgproject.content_profile_ospp

I have found that the rule accounts_umask_etc_bashrc is part of both profiles. However, we test the OSPP profile by the test_suite.py command so I would expect that only the OSPP would be evaluated. Instead, it first run the scenario with the CIS profile with result OK and then the same test scenario with OSPP and returns that the rule has not been evaulated and again it asks if a wrong profile has been selected.