Update XCCDF rule-result to clearly define where detailed results from checking system should be returned to

[vanderpol]

In the NIWC developed SCAP Compliance Checker (SCC) application, we are using the rule-result 'message' with level of 'info' to convey the failure details from the check system. We have convinced result consumers such as DISA's STIG viewer, to process this data, but we feel that either a new 'result-details' element should be added, or the existing documentation should be updated to make it clear to all XCCDF result creators and consumers, that result details should be put in as 'info' level 'messages'.

Our current method, using 'message' field

<cdf:rule-result idref="xccdf_mil.disa.stig_rule_SV-253261r958504_rule" role="full" severity="medium" time="2024-10-28T13:39:18" version="WN11-00-000032" weight="10.0">
      <cdf:result>fail</cdf:result>
      <cdf:ident system="http://cyber.mil/cci">CCI-000804</cdf:ident>
      <cdf:message severity="info">Criteria        : false (All child tests must be true.)
Criteria        :   false (All child tests must be true.)
Test            :     false (BitLocker must be 6 digits or more.)
Result Analysis :          Registry Name Not found: 'HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\FVE\MinimumPIN'.  Existence requirement is 'at_least_one_exists' and no items were collected, so result is false.
</cdf:message>
      <cdf:fix id="F-56664r828866_fix"></cdf:fix>
      <cdf:check system="http://oval.mitre.org/XMLSchema/oval-definitions-5">
            <cdf:check-content-ref href="#scap_mil.disa.stig.fso_comp_U_MS_Windows_11_V2R1_STIG_SCAP_1-3_Benchmark-enhancedV7-oval.xml" name="oval:mil.disa.stig.windows11:def:253261"></cdf:check-content-ref>
      </cdf:check>
</cdf:rule-result>

Suggested update:

<cdf:rule-result idref="xccdf_mil.disa.stig_rule_SV-253261r958504_rule" role="full" severity="medium" time="2024-10-28T13:39:18" version="WN11-00-000032" weight="10.0">
      <cdf:result>fail</cdf:result>
      <cdf:result-details">Criteria        : false (All child tests must be true.)
Criteria        :   false (All child tests must be true.)
Test            :     false (BitLocker must be 6 digits or more.)
Result Analysis :          Registry Name Not found: 'HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\FVE\MinimumPIN'.  Existence requirement is 'at_least_one_exists' and no items were collected, so result is false.
</cdf:result-details>
      <cdf:ident system="http://cyber.mil/cci">CCI-000804</cdf:ident>
      <cdf:fix id="F-56664r828866_fix"></cdf:fix>
      <cdf:check system="http://oval.mitre.org/XMLSchema/oval-definitions-5">
            <cdf:check-content-ref href="#scap_mil.disa.stig.fso_comp_U_MS_Windows_11_V2R1_STIG_SCAP_1-3_Benchmark-enhancedV7-oval.xml" name="oval:mil.disa.stig.windows11:def:253261"></cdf:check-content-ref>
      </cdf:check>
</cdf:rule-result>

[vanderpol]

@solind do you have any opinions on this? We are just wanting to make sure that SCAP 3.0 is clear on where detailed results from the checking language should be stored in XCCDF results.

[solind]

In SCAP 1.3, all the results are actually available in the ARF. So do we really need another place to store them?

Having said that, before we figured out ARF, we did create an XML structure to embed in XCCDF TestResults that pulled in a lot of details from the check systems, that we could transform into a pretty report using XSLT. But, I no longer have access to the details. @maxullman or @a-biggs might have some thoughts.

[vanderpol]

@solind somehow I missed your response from a few weeks ago. In discussions with DISA and NIST, ARF has been universally ignored, only used in order to pass SCAP 1.3 validation, and we have recommended either redesigning it based on what people need, or dropping it entirely. Maybe you are aware of an ARF consumer? I am not. The DOD exclusively uses XCCDF or their own proprietary CKL format (which is like a subset of XCCDF), for consuming results. ARF from what we have heard was too large and bloated, and there are no OVAL results consumers either... In order to send some kind of useful data into XCCDF results, we found the "info" messages as part of the the rule result and used that. It works, and XCCDF result consumers are now starting to process that field, but we felt that SCAP needs to clearly define either a different field, or to use that obscure one.

[solind]

@vanderpol that's hilarious. At Joval I built all sorts of tools for working with ARF files. You could generate scan reports from them, extract content from them, even merge them together and split them apart.

So, Arctic Wolf, their managed risk customers, and all of their Joval OEM customers are making use of ARF reports in some way.

[vanderpol]

Well at least we know somebody is using ARF, that helps clarify if it should or should not be removed from the spec. We disabled ARF from SCC by default years ago and have never once had anyone ask why.

[vanderpol]

@solind and if you want more of a laugh, from what I heard via the grapevine was that DISA/NSA created ARF, and then turned it over to NIST, who then changed it completely and published it in SCAP 1.3. DISA ignored NIST ARF as it wasn't what they needed and continued on with their own DOD ARF, and still maintain it to this day. SCC used to support DOD ARF for a while.

[solind]

@vanderpol That's what "DoD ARF" was about?! I finally know!

Like most of these NIST specifications, I hated it before I learned to love it. As a standardized result format that contains everything you could need to debug a configuration failure, it's actually pretty good.

[vanderpol]

@solind I agree that NIST ARF has everything you might need, I think the DOD just didn't want such much data coming back from millions of endpoints.