[SCAP.R.400] OpenSCAP doesn't reject an invalid DS

jan-cerny commented 3 years ago

Description of Problem:

The file r400.2.3-datastream.xml from SCAP 1.3 validation test suite is invalid according to the Schematron schema representation for the source data stream. OpenSCAP should reject the file.

The r400.2.3-datastream.xml is invalid, SCAPVal says that it violates rule: The @use-case attribute in the element SHALL be set to "CONFIGURATION", "VULNERABILITY", "INVENTORY" or "OTHER".

Indeed, on line 39, there is use-case="CONFIGURATIOM" typo

This file is a part of the test procedure:

SCAP.T.400.2: The tester SHALL import an invalid SCAP source data stream into the vendor product and ensure that the imported content is not available for execution.

OpenSCAP Version:

1.3.4

Operating System & Version:

Steps to Reproduce:

oscap xccdf eval r400.2.3-datastream.xml

Actual Results:

The content is evaluated, no error is printed

Expected Results:

The content is rejected, not available for execution, error that the file is invalid is printed

Additional Information / Debugging Steps:

Also, I can't perform schematron validation:

oscap xccdf validate --schematron r400.2.3-datastream.xml
OpenSCAP Error: Schematron rules not found when trying to validate 'r400.2.3-datastream.xml' [/builddir/build/BUILD/openscap-1.3.4/src/source/schematron.c:124

For more details about schemematron validation, please read ticket https://github.com/OpenSCAP/openscap/issues/1637

jan-cerny commented 3 years ago

The 'r400.2.3-datastream.xml' is also used in tests for SCAP R.1300 which is about validation and I believe it also fails the requirement R1300 or the test procedure SCAP.T.1300.1.

SCAP.R.1300: The product SHALL report and MAY reject SCAP source data stream collection content that is invalid according to the SCAP source data stream and\or its component XML schemas and Schematron schemas.

SCAP.T.1300.1: The tester SHALL attempt to import known-invalid SCAP source data stream collection content into the vendor product and examine the product output to validate that the product reports the invalid SCAP source data stream collection content. The product MAY reject the content as invalid according to the SCAP source data stream collection schema and Schematron schemas.

jan-cerny commented 3 years ago

@evgenyz The current upstream oscap (as of HEAD 2e804671d765475138c41393f552c193882fbe01) is now able to perform a schematron validation and it correctly recognizes the error SRC-324-1 which is the SCAPVAL error mentioned in this ticket. However, I have to run the oscap xccdf validate --schematron command to do that. That is different than the reproducer in this issue. This issue says that reproducer is the oscap xccdf eval command. On a different place we have decided that the schematron validation is optional. However, that might be in conflict with the SCAP.R.400:

SCAP.R.400: The product SHALL be able to import SCAP source data streams for the target platform and correctly load the included Rules and their associated Check System Definitions, rejecting any invalid content.

What do you think?

evgenyz commented 3 years ago

The SCAP.T.400.2 is actually way more explicit:

The tester SHALL import an invalid SCAP source
data stream into the vendor product and
ensure that the imported content is not
available for execution.

This requirement requires an unsigned data stream
applied to the product under test. The r400.2.1-
datastream.xml data stream is invalid according to the
SCAP source data stream collection schema; the r400.2.2-
datastream.xml is not well formed XML; and r400.2.3-
datastream.xml is invalid according to the SCAP source
data stream collection schematron schema.
This requirement is intended to identify, communicate,
and prevent software failure due to errors that MAY exist
in content. The tester SHALL attempt to import the data
stream and the test report SHALL indicate the cause
(corruption) of the error and provide evidence of the error.
The test report SHALL provide evidence indicating the
error. The invalid content SHOULD NOT be available for
execution.

jan-cerny commented 3 years ago

This is fixed in current upstream as of head 596933b478a3800d51689688a0ee0c7b12c4d3d8 as a consequence of #1730 and #1732.

OpenSCAP / openscap