Convert OpenControl plain-texts to Markdown

[isimluk]

Problem statement

NIST published controls (source) are well structured and hierarchical. Our content here, however, is unstructured plain-text.

The plain-text looks nice in TXT editor, however rendering the controls elsewhere (web, word processor) leads into problems with padding, text overflows, alignment etc. Having the unstructured text in our yaml files leaves us with little options, but developing custom parser and re-formaters.

Proposed Solution

Parse NIST structured and hierarchical text to standardized text format.

I considered putting here HTML, Markdown (.md), restructured text (.rst). But ultimately, I believe Markdown is the best fit.

• First of all every modern ui framework nowdays offers methods to render markdown nicely
• It makes only minor changes to our texts (just adds bullets to the bullet lists)

[wendellpiez]

In the OSCAL repo at https://github.com/usnistgov/OSCAL/tree/master/src/utils/util/publish/XSLT we have XSLT transformations in various flavors and states, which produce HTML from OSCAL sources. They have not been touched in a few months so they might be a little behind. Another utility XSLT in there will convert from HTML to Markdown.

For a one-time conversion, those would be enough to get the job done (with some tuning). To run repeatedly (over for example, resolved OSCAL profile inputs, not just the NIST catalog), you have a choice between the XSLT dependency, and building it on another stack. 😎

Building out and supporting these better is on our (long) list of tasks. So is encouraging parallel implementations. I would be interested in seeing a spec for conversion into .rst. FTM, our XSLT could provide a starting point for an HTML or a Markdown mapping (both pretty simple).

[isimluk]

In the OSCAL repo at https://github.com/usnistgov/OSCAL/tree/master/src/utils/util/publish/XSLT we have XSLT transformations in various flavors and states, which produce HTML from OSCAL sources.

Thank You @wendellpiez for bringing my attention to OSCAL repo. Nevertheless, it seems that OSCAL repo contains XSL Transformations that expect XML in OSCAL format. (Noticing xmlns:oscal="http://csrc.nist.gov/ns/oscal/1.0").

While the source XML used by opencontrols, however, is https://nvd.nist.gov/static/feeds/xml/sp80053/rev4/800-53-controls.xml and that one does not seem to be in OSCAL format.

This means, that I would have to go through hurdle of amending the OSCAL XSLT to work with the source xml. In that case, howeve, we would loose the benefits of using OSCAL as we would diverge from it on day 0.

Thus, I am not sure I can see a viable path to adopt OSCAL to build opencontrols data at this point. It seems significantly more effort, than just adding two characters to the build script (this PR adds just 2 characters).

---

Nevertheless, the OpenControl data seems to be similar to OSCAL, OpenControl data is just much more simpler and easier. Since OSCAL repo contains various transformation to various forms and formats I think I may make a sense for OSCAL to provide transformation to OpenControl as well along all the other transformations. WDYT?

[isimluk]

If the existing YAML files are converted to Markdown, would/could that break downstream apps consuming this content? Would it be better to create nist-800053-latest.md files?

This PR edits nist-800-53-latest.yaml along the way of nist-800-53-rev4.yaml. Do you want me to drop edits of nist-800-53-rev4.yaml and keep edits of nist-800-53-latest.yaml only?

[shawndwells]

If the existing YAML files are converted to Markdown, would/could that break downstream apps consuming this content? Would it be better to create nist-800053-latest.md files?

This PR edits nist-800-53-latest.yaml along the way of nist-800-53-rev4.yaml. Do you want me to drop edits of nist-800-53-rev4.yaml and keep edits of nist-800-53-latest.yaml only?

To give the community a some time to review/comment, why don't we let this sit until Weds 27-NOV? Pending no objections lets merge then.

[wendellpiez]

@isimluk Apologies for my confusion: I had assumed since you referred to the "NIST controls" that you meant OSCAL -- I should have looked at the link and seen the NVD XML. (We have the same data set, with more granular markup and Annex A integrated, in the OSCAL repo.) Of course you are correct that the OSCAL XSLTs will not work satisfactorily, unmodified, starting from another data format.

Maintaining data in OSCAL and transforming "downhill" into simpler formats does make a lot of sense, if you need the complexity. One of the reasons for OSCAL is that NVD XML doesn't give quite enough 'semantic leverage' for the profiling functionality that OSCAL is intended to support.

[shawndwells]

Has sat for a few weeks with no comment/objections. Merging!

[isimluk]

In the OSCAL repo at https://github.com/usnistgov/OSCAL/tree/master/src/utils/util/publish/XSLT we have XSLT transformations in various flavors and states, which produce HTML from OSCAL sources. They have not been touched in a few months so they might be a little behind. Another utility XSLT in there will convert from HTML to Markdown.

For a one-time conversion, those would be enough to get the job done (with some tuning). To run repeatedly (over for example, resolved OSCAL profile inputs, not just the NIST catalog), you have a choice between the XSLT dependency, and building it on another stack. sunglasses

Building out and supporting these better is on our (long) list of tasks. So is encouraging parallel implementations. I would be interested in seeing a spec for conversion into .rst. FTM, our XSLT could provide a starting point for an HTML or a Markdown mapping (both pretty simple).

@wendellpiez, bumping this year long discussion. I have implemented conversion from OSCAL Catalog to OpenControl Standard the same way you mentioned here in the thread.

You can see the results at https://github.com/opencontrol/standards/pull/14 If you want to see the code, you can start at https://github.com/GoComply/oscalkit/blob/master/pkg/opencontrol/standard.go#L32