Remote resources mapping by catalog element in datastreams doesn't work

[jan-cerny]

The remote resources support in datastreams is incomplete in OpenSCAP in my opinion.

I have run into this issue when examining test content from the SCAP validation test suite released by NIST. The following snippet is copy-pasted for NIST validation test suite, requirement R2910. Explanation of the problem will follow below.

  <data-stream id="scap_gov.nist_datastream_r2910-datastream" scap-version="1.2" timestamp="2016-05-25T14:00:00" use-case="CONFIGURATION">
    <dictionaries>
      <component-ref id="scap_gov.nist_cref_r2910-cpe-dictionary.xml" xlink:href="#scap_gov.nist_comp_r2910-cpe-dictionary.xml">
        <cat:catalog>
          <cat:uri name="r2910-cpe-oval.xml" uri="#scap_gov.nist_cref_r2910-cpe-oval.xml"/>
        </cat:catalog>
      </component-ref>
    </dictionaries>
    <checklists>
      <component-ref id="scap_gov.nist_cref_r2910-xccdf.xml" xlink:href="#scap_gov.nist_comp_r2910-xccdf.xml">
        <cat:catalog>
          <cat:uri name="r2910-oval.xml" uri="#scap_gov.nist_cref_r2910-oval.xml"/>
        </cat:catalog>
      </component-ref>
    </checklists>
    <checks>
      <component-ref id="scap_gov.nist_cref_r2910-oval.xml" xlink:href="http://scap.nist.gov/validation/downloads/r2910-external-oval-component.xml"/>
      <component-ref id="scap_gov.nist_cref_r2910-cpe-oval.xml" xlink:href="#scap_gov.nist_comp_r2910-cpe-oval.xml"/>
    </checks>
  </data-stream>

XCCDF Rules in this datastream reference the OVAL check as eg. <xccdf:check-content-ref href="r2910-oval.xml" name="oval:nist.validation.variable:def:11"/> .

However the "r2910-oval.xml" is not a local file, but it is a remote resource and it needs to be downloaded from the internet.

I will now explain how OpenSCAP should figure out that "r2910-oval.xml" is a remote resource.

1. It should process the <checklist> element to find an XCCDF checklist.
2. It should find an XCCDF component there.
3. That component contains an catalog. The purpose of a catalog is to map URIs to datastream IDs. The @name attribute is the source of the mapping, and the @uri attribute is the destination of the mapping.
4. It should realise that name="r2910-oval.xml" maps to uri="#scap_gov.nist_cref_r2910-oval.xml", because <cat:uri> element says that.
5. To resolve the uri="#scap_gov.nist_cref_r2910-oval.xml", it should find a ds:component-ref element with id="scap_gov.nist_cref_r2910-oval.xml"
6. This component-ref element exists, however it is in another section, in this case it is a child of element.
7. From the <component-ref id="scap_gov.nist_cref_r2910-oval.xml" xlink:href="http://scap.nist.gov/validation/downloads/r2910-external-oval-component.xml"/> it should see that it needs to download some file.
8. Finally it should realise that every reference to "r2910-oval.xml" is that downloaded file from the internet.

Please have a look to SCAP 1.2 scarification, page 16, table 9, definition of the "catalog" element. http://dx.doi.org/10.6028/NIST.SP.800-126r2

[jan-cerny]

The issue appears on OpenSCAP 1.2.8, but from git log I don't see if there was something done about that since 1.2.8 has been released.

[ybznek]

It will probably cause changes in same lines as fixing r3005.

[ybznek]

I will assign it to myself, because it seems to be related to problem I am dealing with.

[ybznek]

Remote(http) components - (not only OVAL)

<component-ref id="scap_gov.nist_cref_r2910-oval.xml" xlink:href="http://scap.nist.gov/validation/downloads/r2910-external-oval-component.xml"/>  

are required during oscap info. We can add --fetch-remote-resources to be in datastream session too. But what if remote component is referenced from datastream and --fetch-remote-resources is not enabled?

• oscap can try to skip the component (but I don't know if it will not get broken in some next code paths)
• or oscap can exit.

[mpreisler]

Warn and skip IMO.

[ybznek]

To be transparent:

• https://github.com/OpenSCAP/openscap/compare/maint-1.2...ybznek:maint-1.2-tailor2?expand=1
• loading of external file seems to be working
• loading of external component (part of external datastream) seems to be working too
• downloading of content from http seems to be working, but
  • files are not cached
  • during oscap info downloading progress do mess in output
  • without --fetch-remote-resources it fails

Fail after skip:

WARNING: Skipping 'http://localhost:8000/validation/downloads/r2910-external-oval-component.xml' file which is referenced from datastream
OpenSCAP Error: Internal error: Could not acquire handle to xccdf.xml source. [ds_sds_session.c:216]

Mess in output: Downloading: http://localhost:8000/validation/downloads/r2910-external-oval-component.xml ... ok

$ myoscap info --fetch-remote-resources ./r2910-datastream.xml
Document type: Source Data Stream
Imported: 2016-09-19T15:27:23

Stream: scap_gov.nist_datastream_r2910-datastream
Generated: 2016-05-25T14:00:00
Version: 1.2
Checklists:
    Ref-Id: scap_gov.nist_cref_r2910-oval.xml
Downloading: http://localhost:8000/validation/downloads/r2910-external-oval-component.xml ... ok        Status: draft
        Generated: 2016-06-27
        Resolved: true
        Profiles:
            standard
            pci-dss
            C2S
            rht-ccp
            common
            stig-rhel7-workstation-upstream
            stig-rhel7-server-gui-upstream
            stig-rhel7-server-upstream
            ospp-rhel7-server
            nist-cl-il-al
            cjis-rhel7-server

I suggest to postpone fixing of the "mess bug", because we don't have many external components.

[ybznek]

Part of changes here https://github.com/OpenSCAP/openscap/pull/531

[ybznek]

It should be already done. Before merging I want to wait for https://github.com/OpenSCAP/openscap/pull/547 which contains related changes.

[jan-cerny]

547 has been already merged, so I'm closing this as done.