usnistgov / oscal-content

NIST SP 800-53 content and other OSCAL content examples
Other
295 stars 123 forks source link

Integrate profile checker Schematron into CI/CD #128

Open wendellpiez opened 4 years ago

wendellpiez commented 4 years ago

User Story:

Responding to usnistgov/OSCAL#534 we implemented a simple profile link checker that reports when profiles call controls that can't be found in an imported catalog. It doesn't work on profiles importing profiles but it is better than nothing for detecting simple pointer errors.

The Schematron is in this branch: https://github.com/wendellpiez/OSCAL/tree/Issue534-profile-validation. The PR is usnistgov/OSCAL#539.

(Link to commit: https://github.com/wendellpiez/OSCAL/commit/aa223f340dd5887a5c8f4abd05262f0d429fc253#diff-ae6cae8616ab5f465e28e103376e036e )

Goals:

Help ensure profiles committed to the repo are correct.

This Schematron can also be the basis for other checks to be run on profiles.

Dependencies:

Acceptance Criteria

{The items above are general acceptance criteria for all User Stories. Please describe anything else that must be completed for this issue to be considered resolved.}

wendellpiez commented 3 years ago

Revamping this Schematron in working branch behind usnistgov/OSCAL#970.

wendellpiez commented 3 years ago

See WIP cited above.

Also compare to this and possibly consolidate: https://github.com/wendellpiez/oscal-content/blob/master/src/nist.gov/SP800-53/rev5/xml/validate-profiles_SP800-53B-baselines.sch

wendellpiez commented 3 years ago

See PR usnistgov/OSCAL#1013 for updated Schematron.

It is located in the OSCAL repo here: src/utils/schematron/oscal-profile.sch

It does basic testing of profiles importing catalogs, confirming that controls and parameters that are addressed by the profile are actually present in the imported catalog.

aj-stein-nist commented 1 year ago

Heads up to @david-waltermire-nist and @wendellpiez. I thought I would do a quick one to clear my head. Hit some minor speedbumps.

  1. Not urgent, will not block this work: we want to figure out the path forward for tech debt items on supporting XSLT 3 for any Schematron or Schematron that imports util XSLT functions from a stylesheet using XSLT3 features.
  2. More urgent, and I will file a bug: it seems the Saxon-executed use of a command-line invocated XPath essentially looking for any sad path Schematron result in the SVRL, even with intentionally crafted bad profiles do not bubble up an error. This code seems to evaluate a 0 even in the existence of a bad profile with a non-existent control-id inserted (one of the clear-cut assertions in the profile checker): https://github.com/usnistgov/metaschema/blob/675d385d65c85639d0e2d8e395cb80e4d66d63d3/scripts/include/init-schematron.sh#L60-L81 Will file a bug around this later in the evening.
aj-stein-nist commented 1 year ago

Heads up to @david-waltermire-nist and @wendellpiez. I thought I would do a quick one to clear my head. Hit some minor speedbumps.

1. Not urgent, will not block this work: we want to figure out the path forward for tech debt items on supporting XSLT 3 for any Schematron _or_ Schematron that imports util XSLT functions from a stylesheet using XSLT3 features.

2. More urgent, and I will file a bug: it seems the Saxon-executed use of a command-line invocated XPath essentially looking for any sad path Schematron result in the SVRL, even with intentionally crafted bad profiles do not bubble up an error. This code seems to evaluate a 0 even in the existence of a bad profile with a non-existent control-id inserted (one of the clear-cut assertions in the profile checker): https://github.com/usnistgov/metaschema/blob/675d385d65c85639d0e2d8e395cb80e4d66d63d3/scripts/include/init-schematron.sh#L60-L81 Will file a bug around this later in the evening.

Most of this content has been handled in the interim, but I updated to properly identify the blocker around CI/CD failures rooted in usnistgov/metaschema#240. There may be other issues, but I will need to update accordingly by end of week.

aj-stein-nist commented 1 year ago

Moving to Sprint 61, and leaving open until the updated CI/CD code is merged into the oscal-content repo where it will actually validate profiles.

aj-stein-nist commented 1 year ago

This can be merged with very limited effort once CI/CD configuration is resolved. Moving to Sprint 63 as it very close to complete.