search

ComplianceAsCode / content

Security automation content in SCAP, Bash, Ansible, and other formats
https://complianceascode.readthedocs.io/en/latest/
Other
2.23k stars 698 forks source link

Profile selection overloading in SSGTS combined mode causes issues #4952

Closed matusmarhefka closed 2 years ago

matusmarhefka commented 5 years ago

Description of problem:

Profile selections in test case metadata are ignored when running SSGTS in combined mode for certain profile. This causes that the test scenarios which are not targeted for the tested profile are also run and will fail.

Example: The test policy_default_set.pass.sh for the configure_crypto_policy rule fails when running SSGTS for the ospp profile because it is implemented only for the standard profile and it is N/A for the ospp. This is caused by the SSGTS (combined mode) overloading profile metadata in test scenarios.

jan-cerny commented 5 years ago

I think that this behavior has been created because the test scenarios aren't labeled properly - eg. some rule is part of multiple profiles on multiple products but the test scenario is labeled only that it belongs to 1 profile. Labeling them properly is very costly, though. I think you would have to add a lot of profiles to the headers and then test it. Also maintaining it is be difficult - each time anybody changes a profile he should change all the test scenarios that test rules added/removed from the profile. I think we can't ask the our contributors to do this. Instead, we should rethink the profile metadata.

matusmarhefka commented 2 years ago

Closing, this has been addressed by https://github.com/ComplianceAsCode/content/pull/6123 and https://github.com/ComplianceAsCode/content/pull/6146