search

ComplianceAsCode / content

Security automation content in SCAP, Bash, Ansible, and other formats
https://complianceascode.readthedocs.io/en/latest/
Other
2.22k stars 698 forks source link

SSG test suite fails when rules marked with 'platform: machine' are run in container environment #12511

Open mpurg opened 1 month ago

mpurg commented 1 month ago

Description of problem:

The ssg test suite does not take into account the platform: machine definitions in the rule, when deciding which tests to run. The evaluation of applicability (machine vs container) is performed only after the test environment is already prepared, which can result in the test failing if the environment is not setup correctly in a non-supported environment.

An example rule which tests fine in a VM but fails to test in a container is all_apparmor_profiles_in_enforce_complain_mode, when the machine profile is added to its rule defintion.

$ tests/test_rule_in_container.sh --no-make-applicable-in-containers --dontclean --logdir logs_bash --remediate-using bash --name ssg_test_suite --datastream build/ssg-ubuntu2204-ds.xml all_apparmor_profiles_in_enforce_complain_mode

ERROR - Rule 'all_apparmor_profiles_in_enforce_complain_mode' test setup script 'correct_all_apparmor_profiles_in_enforce.pass.sh' failed with exit code 1
ERROR - Environment failed to prepare, skipping test

By adding the following to the test scenario, the issue is resolved:

if [ -f /.dockerenv ] || [ -f /run/.containerenv ]; then
    exit 0
fi
$ tests/test_rule_in_container.sh --no-make-applicable-in-containers --dontclean --logdir logs_bash --remediate-using bash --name ssg_test_suite --datastream build/ssg-ubuntu2204-ds.xml all_apparmor_profiles_in_enforce_complain_mode

WARNING - Script correct_all_apparmor_profiles_in_enforce.pass.sh using profile (all) notapplicable
WARNING - Rule xccdf_org.ssgproject.content_rule_all_apparmor_profiles_in_enforce_complain_mode evaluation resulted in notapplicable

SCAP Security Guide Version:

master branch (https://github.com/ComplianceAsCode/content/commit/a11711d14d3252adc05916d5a9419eabd9cce10c)

Operating System Version:

Steps to Reproduce:

  1. Add platform: machine to rule all_apparmor_profiles_in_enforce_complain_mode
  2. ./build_product ubuntu2204
  3. podman build --build-arg "CLIENT_PUBLIC_KEY=$(cat ~/.ssh/id_rsa.pub)" -t ssg_test_suite -f Dockefiles/test_suite-ubuntu2204
  4. tests/test_rule_in_container.sh --no-make-applicable-in-containers --dontclean --logdir logs_bash --remediate-using bash --name ssg_test_suite --datastream build/ssg-ubuntu2204-ds.xml all_apparmor_profiles_in_enforce_complain_mode

Actual Results:

See error above

Expected Results:

The expected result is that the tests pass when inside a container because the rule should not be evaluated.

Additional Information/Debugging Steps:

Possible solutions that come to mind: