OSCAL on an XProc 3.0 platform
OSCAL is the Open Security Controls Assessment Language. It is a domain-specific language supporting the field of systems security, security assessment and risk management (RMF), expressed in XML, JSON and other formats. This site supports primarily but not exclusively XML-formatted OSCAL.
The XProc demonstrations and capabilities shown are not limited to OSCAL.
This project aims to make XML-based processing using commodity (open-source) tools more broadly available, useful, reliable and intelligible, especially to users and developers of OSCAL.
The many subordinate goals include:
Learn more about XProc in researching-xproc.md.
Process OSCAL (XML and JSON) easily on a standards-based, portable and open-source XML platform.
This repository serves as a platform for demonstration and development of capabilities in information and data processing in support of OSCAL, the Open Security Controls and Assessment Language, using community-standard XProc 3.0 technology.
This may include either OSCAL applications, or applications built to be used in OSCAL development. See below and project (folder) readme files for more details.
Enabling these lightweight, transparent and declarative "logical layers" written in XProc (pipeline step definitions), in XSLT (transformations) and in Schematron (query-based validations) are these excellent libraries (with their dependencies):
These are open-source projects in support of W3C- and ISO-standardized technologies. Helping to install, configure, and make these work seamlessly, so users do not have to notice, is a goal of this project.
If this software is as easy, securable and performant as we hope to show, it might be useful not only to XML-stack developers but also to others who wish to cross-check their OSCAL data or software supporting OSCAL by comparison with another stack.
Projects currently planned for deployment in this repository include:
batch-validate validate OSCAL in batches against schemas and schema emulatorsdata-convert - convert OSCAL XML into JSON and OSCAL JSON into XMLdisplay-render - convert OSCAL catalogs (including resolved profiles) into HTML and PDFindex-oscal - produce indexes to information encoded in OSCAL TODO: update this list READERS: anything to add?
Applications in this repository may occasionally have general use outside OSCAL; users who find any of its capabilities should be generalized and published separately please create a Github Issue.
lib, template, testing, icons and (hidden) .github folders are special; others represent projects.
The lib directory comes bare bones - it has only its readme, a configuration file and a couple of utility pipelines. This library is populated by the installation script, and (once the basic setup is done) by running the pipelines.
lib can be cleaned up, and restored, more or less with impunity, but if it disappears or its contents are renamed, rearranged or altered, things will cease working - see its readme for more information.
Next to lib, each project is kept in its own separate folder. There, the project will have its own readme.md. While projects may rely on the shared libraries, each one is considered to be discrete and independent from others unless and except where noted otherwise.
One such project is smoketest, devoted to testing the software installation both in its basic and fully configured forms.
The testing directory contains tests and logic applicable to the repository or its contents, such as Schematron governing usage of XProc or other formats - XML-based code introspection. As this is still in development, it can be expected to change and grow.
The template directory contains a blank project template. Copy and rename this folder for a quick start on XProc infrastructure for your project. It contains boilerlate documentation and logic ready for rewriting.
Finally, the icons directory holds SVG and Windows icon files that can be associated with scripts or file types.
The software in this repository is at varying levels of maturity. Many stylesheets are fairly mature, having been developed and tested in other contexts. Yet work here also includes job configuration (XProc itself) and testing, which is to say on tools to determine correctness and reliability, therefore (by implication) maturity. These may be experimental, not mature.
At the same time, the libraries we use (Morgana, Saxon and others) are themselves at various levels of maturity (Saxon in particular having been field-tested for over 20 years). And both particular initiatives and the code repository as a whole follow an incremental development model. Things left as good-enough-for-now are regarded as being good enough, until experience shows us it is no longer so. Punctuated equilibrium is normal.
Assume the worst, hope for the best, and test.
Cloning the repository is encouraged and taken as a sign of success. So is any participation in testing and development activities.
As of mid-2024, we believe some aspects of this initiative are innovative or at least unusual, even if they have been foreseen for a long time. Please let us know of relevant prior art, or independent invention, especially if it pre-empts our work here.
After a single script is run to download and unpack, all setup and configuration is done with pipelines. Like all XProc pipelines, these are available, enabling the developer or user to override or customize the setup easily, or to perform it manually.
After a single installation of a Java application to initialize (Morgana XProcIIIse), XProc itself is used to define and drive all installation, extraction, building, testing and deployment under continuous integration / continuous development (CI/CD).
This includes testing for all included capabilities and upstream dependendencies (XSLT, Schematron, XSpec, iXML etc.), protecting the ongoing integrity of the build and runtimes.
File lists are maintained in XProc 3.0 for batch processing. This includes files to be validated with a schema or Schematron, XSpec test files to be compiled and run, or any other processes to be run on sets of files. Lists are cross-checked against sources using Schematron to ensure link integrity.
Batch runtime results, including reports, can be aggregated using XProc and XSLT.
This repository supports running XSpec to test XSLT, Schematron and XQuery.
Because it runs under XProc, XSpec becomes available to any project as well as CI/CD.
Assuming it holds up under use and testing, we hope to be able to contribute this work into the XSpec repository (XSpec developers please take note), if not to retire it in favor of a better implementation.
When we discover bugs in dependent processes, we can not only report them as bugs (to the upstream project), but also mitigate them in place. Since XSLT transformations are in XML, we can use XProc and XSLT to modify any XSLT discovered to have bugs and lapses.
The code capturing such a modification provides a patch to upstream developers, who can refer to it easily to discover, analyze and evaluate our problem diagnosis and solution in its home context, at any later point.
And until this is done, we can run ahead, while proceeding to use the feature in question as modified.
Examples of this can be found in the (XSpec implementation pipeline](xspec/xspec-execute.xpl), which (at time of writing) offers patches to an XSLT provided in the XSpec repository.
Projects each have their own libraries and dependencies in addition to the common (platform) dependencies. Consequently the repository as a whole can easily be pared down to just a minimalistic implementation of one or a few functionalities or applications, for demonstration or further development.
This makes cloning and further development easier.
Note: if you already have Morgana XProc III installed, you should be able to use it, appropriately configured, to run any pipeline in the repository. But local installation is also easy and clean.
Platform requirements: Java, with a bash shell for automated installation. Only Java is required if you can install manually.
Developed on Windows and tested with WSL, Git bash, and Windows Powershell.
I. Install the XProc3 engine by running the script:
./setup.shThe setup-notes file provides a walkthrough of what this script does, if bash does not come through: the setup can be done by hand.
After setup, you should now be able to run bare-bones XProc using a pipeline processor (Morgana) with rudimentary capabilities. How to run any of these pipelines is detailed below.
If impatient to test, review Does It Work? below before proceeding. But expect errors with many pipelines if you forget to finish the installation.
The next steps both test the runtime, and provide Morgana with more power, namely XSLT for transformations (using Saxon), and Schematron for query-based validation (using SchXLST).
SchXSLT is written in XSLT and requires Saxon, so run (and test) the Saxon pipeline first.
II. To install Saxon: lib/GRAB-SAXON.xpl
To test Saxon: smoketest/SMOKETEST-XSLT.xpl
III. To install SchXSLT: lib/GRAB-SCHXSLT.xpl
To test SchXSLT: smoketest/SMOKETEST-SCHEMATRON.xpl
IV. To install XSpec: lib/GRAB-SCHXSLT.xpl
To test XSPec: smoketest/SMOKETEST-SCHEMATRON.xpl
If Morgana is installed with Saxon-HE you should be good to go running any pipeline. See project readme documents for details on each project.
TODO: keep this list up to date
Any XProc3 pipeline can be executed using the script xp3.sh (bash) or xp3.bat (Windows CMD). For example:
./xp3.sh LAUNCH.xplWill initiate an XProc 3 step (pipeline) defined in the file LAUNCH.xpl (or return an error when no pipeline is found by that name).
Note that a pipeline may run successfully without XSLT or Schematron support, if the pipeline itself does not depend on these capabilities.
Similarly, the Morgana installation provided by this project is not the only way to run XProc - these pipelines will work just as well in any other conformant XProc 3.0 engine or processing environment.
Testing is a central focus of this initiative. See TESTING.md.
Some repository-wide testing, not for functionality but for other requirements, is maintained in the testing directory.
This project is being maintained by Wendell Piez, w a p i e z @ n i s t . g o v of the National Institute of Standards and Technology, Information Technology Laboratory, Computer Security Division (NIST/ITL/CSD 773.03).
Piez, Wendell (2024), OSCAL-XProc3, US National Institute of Standards and Technology (NIST), https://github.com/usnistgov/oscal-xslt3.
TODO: secure a DOI by registering the project in MIDAS, and update accordingly
The code in this repository is all declarative, and depends on lower-level processors provided in libraries.
Currently these dependencies are:
Morgana and Saxon both require Java, as detailed on their support pages. SchXSLT requires XSLT, hence Saxon.
See THIRD_PARTY_LICENSES.md for more.
XProc 3.0 aims to be platform- and application-independent, so one use of this project will be to test and assess portability across environments supporting XProc.
With the authors of incorporated tooling, the many contributors to the XProc and XML stacks underlying this functionality are owed thanks and acknowledgement. These include Norman Walsh, Achim Berndzen and the developers of XProc versions 1.0 and 3.0; developers of embedded commodity parsers and processers such as Java Xerces, Trang, and Apache FOP (to mention only three); and all developers of XML, XSLT, and XQuery especially unencumbered and open-source. Only an open, dedicated and supportive community could prove capable of such a collective achievement.
This README was composed starting from the NIST Open Source Repository template as of April 24, 2024.