OpenSCAP generates incomplete Ansible Playbooks

jan-cerny commented 1 year ago

Description of Problem:

OpenSCAP oscap xccdf generate fix submodule can't generate Ansible Playbooks in the results oriented mode if the given ARF has been created by oscap running in --rule mode and the provided rule isn't a part of the profile provided in the --profile argument.

This situation can happen for example in Automatus when executing TSs that contain profile in their headers. This has been discovered during review of https://github.com/ComplianceAsCode/content/pull/10348.

OpenSCAP Version:

openscap-1.3.7-1.fc37.x86_64

Operating System & Version:

F 37

Steps to Reproduce:

Download reproducer_content.tar.gz. This archive contains ARF file arf.xml and also the original ssg-rhel8-ds.xml.
tar xzf reproducer_content.tar.gz
oscap xccdf generate fix --fix-type ansible --result-id xccdf_org.open-scap_testresult_xccdf_org.ssgproject.content_profile_stig arf.xml

Actual Results:

Dummy playbook with no contents

---
###############################################################################
#
# Ansible Playbook generated from evaluation of DISA STIG for Red Hat Enterprise Linux 8
#
# Profile ID: xccdf_org.ssgproject.content_profile_stig
# XCCDF Version:  1.2
#
# Evaluation Start Time:  2023-03-27T07:11:08-05:00
# Evaluation End Time:  2023-03-27T07:11:08-05:00
#
# This file was generated by OpenSCAP 1.3.7 using:
# $ oscap xccdf generate fix --result-id xccdf_org.open-scap_testresult_xccdf_org.ssgproject.content_profile_stig --fix-type ansible xccdf-results.xml
#
# This Ansible Playbook is generated from the results of a profile evaluation.
# It attempts to remediate all issues from the selected rules that failed the test.
#
# How to apply this Ansible Playbook:
# $ ansible-playbook -i "localhost," -c local playbook.yml
# $ ansible-playbook -i "192.168.1.155," playbook.yml
# $ ansible-playbook -i inventory.ini playbook.yml
#
###############################################################################

- hosts: all
  vars:
  tasks:

Expected Results:

The generated Playbook contains tasks for remediation of rule xccdf_org.ssgproject.content_rule_sssd_ldap_configure_tls_ca_dir. The reason for this expectation is that the ARF file contains a "fail" result for this rule. The rule has been also reported as a "fail" during the scan that created this ARF. The behavior of the results-oriented Playbook generator should be that it the generated Playbook contains remediations of failed rules.

Additional Information / Debugging Steps:

evgenyz commented 1 year ago

So, in this scenario, when --rule is provided the scanner should just ignore --profile completely? Is it actually desirable? What is the behaviour where the is no --profile argument at all?

jan-cerny commented 1 year ago

What is the behaviour where the is no --profile argument at all?

If there is no --profile provided, oscap evaluates so-called virtual (default) profile. This virtual (default) profile honors the explicit selections of rules and other items - ie. values of select attributes in rule elements and other elements. For example, in SCAP source data streams provided by scap-security-guide, all rules have selected attribute set to "false", so the virtual (default) profile is empty, so when evaluating these data streams and no --profile argument is provided by user it doesn't evaluate anything. However, this was just an example, and other SCAP data streams might set all or some rules selected attribute to true, so so when evaluating these data streams and no --profile argument is provided by user these rules will be evaluated.

So, in this scenario, when --rule is provided the scanner should just ignore --profile completely? Is it actually desirable?

I believe that the --rule option hard overrides the rule selection in the profile IOW deselects all rules and selects this specific rule.

When --rule is provided the scanner shouldn't simply ignore --profile, because of the values of XCCDF values. But, on the other hand, we could change the behavior so that the --rule won't add new selections to the profile provided by --profile.

evgenyz commented 1 year ago

jan-cerny commented 1 year ago

Thank you for opening this issue! We have discussed this internally. We realize that there are multiple concerns that we need to take into account.

First problem is that we aren't consistent across different oscap submodules. We need to make sure that the options --rule and --profile and their combination behave consistently give the same results in xccdf eval, xccdf generate fix and all other submodules where --rule is available.

Also, we should keep in mind that the profile choice doesn't influence only selections of rules to be evaluated but it also has an influence over values of XCCDF Values. We can imagine multiple use-cases when a content author or user would like to run a specific rule with a specific value of an XCCDF value determined by the profile. Therefore, we should allow user to use --profile and --rule on the command line at the same time.

We need to realize that the combination of --rule and --profile has historically been used frequently in Automatus in ComplianceAsContent repository where a profiles header in test scenario can be used to set specific values of XCCDF Values. However, this approached had serious drawbacks that it depended on changes in profiles and control files that aren't under influence of the test scenario authors and are hard to track and keep synchronize. Therefore, for setting specific values of XCCDF Values a different header variables has been introduced. This variables header allows to set specific values directly in the test scenarios, and therefore create test scenarios with no dependency on profiles. That means that the profiles header doesn't have the use case that is used to have and therefore the combination of --rule and --profile is used less often in Automatus.

With this in mind, we have reached a conclusion which is to keep the ability of the users to specify --rule and --profile at the same time. However, if the given rule isn't part of the given profile, we will throw an error message in oscap xccdf eval. Then, we will copy this behavior to all other affected submodules.

jan-cerny commented 1 year ago

@matejak please submit your opinion

evgenyz commented 9 months ago

@mildas @jan-cerny How actual this problem is nowadays?

OpenSCAP / openscap