An OSCAL developer might wish to test OSCAL schemas (of whatever sort) against sets of documents known to be valid and invalid, for purposes of testing conformance to expectations of respective metaschema designers.
I.e. for OSCAL catalog we want a library of catalogs and near-catalogs, along with a way to tell if a given schema validator (tool or schema) can tell the difference.
A pipeline could then be used to test an arbitrary XSD or JSON Schema against a test sample set, appropriately designed and linked with specifications.
Ultimately (not immediately) such a deployment should support:
replication in other contexts (for schema field testing) especially OSCAL
refitting by other parties testing their own or one another's schemas
applicability to other validation technologies such as Schematron or Inspector XSLT
extensibility and ongoing enhancement of the test sets and functionality
Goals
[ ] A JSON Schema test harness with test samples (OSCAL JSON catalogs) demonstrating the reported bug
Ideally, we should be able to try any/all available JSON Schemas (v07 or any supported) to assess their conformance to the group construct semantics
[ ] A commit to this repository showing a duplicable runtime (under bash and/or a Windows command shell) for demonstrating this test, live (and see Considerations below)
[ ] Any XSpec or other tests for testing the testing
[ ] The architecture should be portable into OSCAL repo (or any other with schemas or a way to get schemas)
Nb: This process must not fail under CI/CD, while any tests for its functionality or integrity can do so. Being able to set up and run this testing pipeline is a goal, and this includes being able to set it up under CI/CD. Doing so is not a goal, except for in service of OSCAL 1989 (and for demonstration).
Secondary goals
[ ] A similar test for an XSD
[ ] Address #14, a blocker for dynamic refresh of OSCAL schemas from their metaschema sources (the next step after static schemas)
To do this in the XML stack we need XProc 3.0 for its JSON Schema support. This is a minus as well as a plus, as this will require a new processor (MorganaXProcIII). Documenting and scripting this is in scope for this Issue.
An OSCAL developer might wish to test OSCAL schemas (of whatever sort) against sets of documents known to be valid and invalid, for purposes of testing conformance to expectations of respective metaschema designers.
This will give us a basis for detecting and preventing problems such as reported in https://github.com/usnistgov/OSCAL/issues/1989.
I.e. for OSCAL catalog we want a library of catalogs and near-catalogs, along with a way to tell if a given schema validator (tool or schema) can tell the difference.
A pipeline could then be used to test an arbitrary XSD or JSON Schema against a test sample set, appropriately designed and linked with specifications.
Ultimately (not immediately) such a deployment should support:
Goals
groupconstruct semanticsbashand/or a Windows command shell) for demonstrating this test, live (and see Considerations below)Nb: This process must not fail under CI/CD, while any tests for its functionality or integrity can do so. Being able to set up and run this testing pipeline is a goal, and this includes being able to set it up under CI/CD. Doing so is not a goal, except for in service of OSCAL 1989 (and for demonstration).
Secondary goals
Considerations
This Issue is also in support of https://github.com/usnistgov/metaschema-xslt/issues/105; when that bug repair is done, the work done here should be able to confirm the correctness of its schema(s).
To do this in the XML stack we need XProc 3.0 for its JSON Schema support. This is a minus as well as a plus, as this will require a new processor (MorganaXProcIII). Documenting and scripting this is in scope for this Issue.