tests/test_suite.py rule fails with Rule xccdf_org.ssgproject.content_rule_coredump_disable_backtraces has not been evaluated! Wrong profile selected in test scenario?

adelton commented 5 years ago

Description of problem:

Running tests/test_suite.py rule now fails with

Rule xccdf_org.ssgproject.content_rule_coredump_disable_backtraces has not been evaluated! Wrong profile selected in test scenario?

SCAP Security Guide Version:

The master: v0.1.45-195-g68aacd11a

Operating System Version:

On the host, Fedora 30 with docker-1.13.1-67.git1185cfd.fc30.x86_64. In the container, RHEL 8.

Steps to Reproduce:

1.

docker build --build-arg=CLIENT_PUBLIC_KEY="$( cat /dir/.ssh/id_rsa.pub )" -t test_suite-rhel8 -f Dockerfiles/test_suite-rhel8 Dockerfiles

2.

./build_product rhel8

3.

tests/test_suite.py rule --docker test_suite-rhel8 --datastream build/ssg-rhel8-ds.xml coredump_disable_backtraces

Actual Results:

Setting console output to log level INFO
INFO - The base image option has been specified, choosing Docker-based test environment.
INFO - Logging into /dir/scap-content/logs/rule-custom-2019-08-09-1655/test_suite.log
INFO - xccdf_org.ssgproject.content_rule_coredump_disable_backtraces
ERROR - Failed to cleanup file '/dir/scap-content/logs/rule-custom-2019-08-09-1655/xccdf_org.ssgproject.content_rule_coredump_disable_backtraces-coredumps_processsizemax_nonzero.fail.sh-initial.html'
ERROR - Script coredumps_processsizemax_nonzero.fail.sh using profile xccdf_org.ssgproject.content_profile_ospp found issue:
ERROR - Rule xccdf_org.ssgproject.content_rule_coredump_disable_backtraces has not been evaluated! Wrong profile selected in test scenario?
ERROR - The initial scan failed for rule 'xccdf_org.ssgproject.content_rule_coredump_disable_backtraces'.
ERROR - Failed to cleanup file '/dir/scap-content/logs/rule-custom-2019-08-09-1655/xccdf_org.ssgproject.content_rule_coredump_disable_backtraces-coredumps_processsizemax_0.pass.sh-initial.html'
ERROR - Script coredumps_processsizemax_0.pass.sh using profile xccdf_org.ssgproject.content_profile_ospp found issue:
ERROR - Rule xccdf_org.ssgproject.content_rule_coredump_disable_backtraces has not been evaluated! Wrong profile selected in test scenario?
ERROR - The initial scan failed for rule 'xccdf_org.ssgproject.content_rule_coredump_disable_backtraces'.
ERROR - Failed to cleanup file '/dir/scap-content/logs/rule-custom-2019-08-09-1655/xccdf_org.ssgproject.content_rule_coredump_disable_backtraces-coredumps_processsizemax_default.fail.sh-initial.html'
ERROR - Script coredumps_processsizemax_default.fail.sh using profile xccdf_org.ssgproject.content_profile_ospp found issue:
ERROR - Rule xccdf_org.ssgproject.content_rule_coredump_disable_backtraces has not been evaluated! Wrong profile selected in test scenario?
ERROR - The initial scan failed for rule 'xccdf_org.ssgproject.content_rule_coredump_disable_backtraces'.

Expected Results:

Pass or fail but not error.

Addition Information/Debugging Steps:

adelton commented 5 years ago

There is openscap-1.3.1-2.fc30.x86_64 on the Fedora 30 and openscap-scanner-1.3.0-7.el8.x86_64 in that RHEL 8 container.

matejak commented 5 years ago

The ERROR comes from the Python logging module string. What happens is that the rule is not selected for some reason. This points out to an inconsistency of profile composition vs test scenarios, which is an error, but not a scanner error. ~Could it be that the rule is not part of any profile due to profile modification?~

ggbecker commented 5 years ago

I had similar issue, it turned out to be that the openscap version which was used to scan didn't contain the fix for the --fetch-remote-resources (versions prior to 1.3.1) and then the scan failed but the test suite doesn't know how to detect the openscap error and it throws this error about rule no selected in profile. This only happens on datastreams with SCAP 1.3. But currently there is no support in build_product script to build SCAP1.2 content only, one needs to parametrize cmake command manually.

adelton commented 5 years ago

Is there a plan for adding some option / environment variable to make build_product build 1.2 content? Patching build_product and keeping that local change around will mess up subsequent git operations.

jan-cerny commented 5 years ago

After #4708 has been merged the issue should be fixed by updating tests/README.md.

ggbecker commented 5 years ago

Proposed documentation by #4730 should address the issue.

ComplianceAsCode / content