Improve CI/CD Diagrams and Documentation

[aj-stein-nist]

User Story

As an OSCAL tools developer, in order to better understand the impact of the CI/CD operations, I would like to see some more clarity, consistency, and more details for CI/CD and more general workflow diagramming for the NIST OSCAL repositories.

Specifically, this is to improve usnistgov/OSCAL#1165 and what was merged in at https://github.com/usnistgov/OSCAL/commit/35041015b055e23de77319cf747e999a46e823db.

Hat tip to @gregelin for this feedback.

Goals

• [x] Determine goals of documentation and contribution: quality or depth or number/frequency of contributions; use that to determine subsequent diagrams and other changes
• [x] Consider breaking out the different workflow requirements in to sub-diagrams, don't build everything only in one large diagram (represent the diagram for the appropriate level of the documentation the diagram is in, if there are repo submodules or other uses, be specific in diagrams in that repo and make a progression)
• ~Clarify mixup of intent, driver (tool, script, GHA workflow name) and artifact/output for each step with grey label versus purple action box~
• ~Add a legend to explain what the diamond, squares, labels and what they actually mean here~ Work needed in sprint:
• [x] Break it down for workflow
• [x] Clean up the Compton-NIST/Inspector workflow

Dependencies

• N/A at this time

Acceptance Criteria

• ~All OSCAL website and readme documentation affected by the changes in this issue have been updated. Changes to the OSCAL website can be made in the docs/content directory of your branch.~
• ~A Pull Request (PR) is submitted that fully addresses the goals of this User Story. This issue is referenced in the PR.~
• ~The CI-CD build process runs without any reported errors on the PR. This can be confirmed by reviewing that all checks have passed in the PR.~

[aj-stein-nist]

Scope to "what can we do in a day's work" and "go wide than deep" per triage discussion.

[aj-stein-nist]

Moving to Sprint 61.

[Compton-US]

@aj-stein-nist I'd be willing to take a crack at this, but I may need to scope it a bit.

[aj-stein-nist]

@aj-stein-nist I'd be willing to take a crack at this, but I may need to scope it a bit.

Let's talk about that in a minute then. :-)

[aj-stein-nist]

@Compton-NIST did I understand correctly that you prefer you'd like to come into this with a "fresh set of eyes" perspective and not directly work with me on the documentation piece? I am supportive of that and like that idea, if I understood you correctly. So I will unassign myself for the time being. Feel free to add me back or correct if I misunderstood.

[aj-stein-nist]

I am nominating @wendellpiez for review relevant code outputs and resulting process diagram, separate of AJ's reviews and others, to get a fresh perspective and dogfood these diagrams.

[Compton-US]

I have written a script that will automatically produce a) a markdown document that outlines each workflow in a repository, and b) an svg that contains a graphical representation of each workflow, step and action. The svg also displays the relationship between workflows.

This script can be run against multiple repositories. I am going to attach some of the svg files as comments under this one. We can discuss. A full zip of a run of this script that includes all markdown and graphics can be found here: https://github.com/Compton-NIST/Inspector/suites/10917151481/artifacts/551758004

[Compton-US]

The most complex one, from the OSCAL repository:

[Compton-US]

oscal-cli:

[Compton-US]

oscal-content:

[Compton-US]

Additional thoughts, and I'm done for a while:

I've looked through the specification for workflow files to see if there are elements we could use to fully self-document within the workflow, for example, to describe a job or step, so this can be used to add context to the markdown and diagrams. The only thing I see is slightly-abusing the env element that is available at most levels of a workflow. If a variable such as HELP_TEXT existed, the script could print that as a part of the box to provide additional information to the reader.

We should also name each step. I think many of the ones that just point to an action are missing names, so those are noted as unnamed. Cleaning this up would be helpful.

We might also consider how we name steps so that the name provides as much context as possible to the reader. This would not require hacking the env variables, and might be sufficient.

[wendellpiez]

Agreed on the names (both wanting them and needing them to be good). Liking the approach very much. I am seeing there is lots of (irreducible?) overhead to push out even apart from the artifact productions that are wrapped in the red boxes, is that more or less what is going on here?

In any case this is excellent, bravo graph dbs (perfect for something like this).

[aj-stein-nist]

Additional thoughts, and I'm done for a while

So how are you going to finalize this work re the diagrams already committed in repos and the goals/AC? Can we discuss that during the weekly status meeting today? Thanks.

[Compton-US]

@aj-stein-nist That sounds like a great idea.

[aj-stein-nist]

Work needed in sprint has been moved into the updated goals section of the issue.

Work needed for follow-on sprints:

• [ ] Configuring it submit with a commit/back to the following repos (depends on assistance)
  • [ ] usnistgov/OSCAL https://github.com/usnistgov/OSCAL/issues/1678
  • [ ] usnistgov/oscal-content
  • [ ] usnistgov/oscal-cli
• [ ] Add documentation metadata for each workflow step so it shows up in the diagram https://github.com/usnistgov/OSCAL/issues/1679
• [ ] Add identifiers each in each workflow (unnamed steps need to be named)
• [ ] Further diagramming the internals of shell scripts and/or transforms and/or shell utilities executed by the GHA runner

A.J. will review to the backlog for the latter list by end of week, or he shall be heckled.

[aj-stein-nist]

Chris and I discussed this during this week's status review meeting. AJ still has to make the follow-on work issues mentioned ☝. He is going to do so today, or else. Chris will follow on with updated works for this sprint in the AC section updated in the ticket from last week. He says still on track for completion before end of sprint/February.

[Compton-US]

I've recommended use of an automated method to produce the diagrams and documentation of workflows to maintain parity with the actual workflows as they change across repositories.

That solution currently resides here: https://github.com/Compton-NIST/Inspector

• It likely should be converted into a GitHub Action so that it can be attached to repositories.
• Currently it will run against a list of repositories, which is enough to get started.
• It produces an overview diagram of the entire repository workflows.
• It produces individual workflow diagrams.
• It produces a markdown file with the text of the workflow, as an alternative representation of the graphic, and includes all diagrams with the text.
• It executes a github workflow that will produce the content, and store the markdown and diagrams in a zip artifact.

If we are good with the proposed solution:

Goals

• [x] Determine goals of documentation and contribution: quality or depth or number/frequency of contributions; use that to determine subsequent diagrams and other changes

This is adjustable now and in the future across all repositories if we use the solution.

• [x] Clarify mixup of intent, driver (tool, script, GHA workflow name) and artifact/output for each step with grey label versus purple action box

In the diagrams, color and text is used to describe nodes. Text is present in the markdown that adds context without the need for color.

• [x] Add a legend to explain what the diamond, squares, labels and what they actually mean here.

There is not a legend, but let's see what we think about the current state. There may be enough context present between markdown and diagrams that this is not necessary.

• [x] Consider breaking out the different workflow requirements in to sub-diagrams, don't build everything only in one large diagram (represent the diagram for the appropriate level of the documentation the diagram is in, if there are repo submodules or other uses, be specific in diagrams in that repo and make a progression)

We do both. The only limitation may be that it wasn't designed to traverse submodules, and that is something we could do later.

Work needed in sprint:

• [x] Break it down for workflow

Done

• [x] Clean up the Compton-NIST/Inspector workflow

The main branch has been updated, and includes a README. Need to figure out what we want to do long term, but we can start here.

Dependencies

• N/A at this time

Acceptance Criteria

• [ ] All OSCAL website and readme documentation affected by the changes in this issue have been updated. Changes to the OSCAL website can be made in the docs/content directory of your branch.

Not addressing this yet.

• [ ] A Pull Request (PR) is submitted that fully addresses the goals of this User Story. This issue is referenced in the PR.

No PR based on the automation approach. We will need to remove old content, but I want to make sure we like the approach and are happy with it before we commit to integrating.

• [ ] The CI-CD build process runs without any reported errors on the PR. This can be confirmed by reviewing that all checks have passed in the PR.

Also not addressed due to the approach above.

[Compton-US]

A Sample:

A full artifact can be found here: https://github.com/Compton-NIST/Inspector/suites/11232430558/artifacts/574991343

Single Workflow Example

Overview Example

Visual Representation of an Entire Document

[gregelin]

@Compton-NIST @aj-stein-nist I think an automated generation of visual documentation would be very useful!

Please consider the following criteria as improvements:

• [ ] Generating a CHANGELOG specific to the documentation help people find and examine what is different (like a new relationship, or a new step)
• [ ] Archive previous version of generated documentation (or at version of generated documentation associated with a tag and/or release). Having previous versions will make it much easier for comparisons by everyone else to see changes

[gregelin]

BTW, @Compton-NIST What tool are using to generate the diagrams and flows?

[Compton-US]

@gregelin graphviz (via python library) is used for rendering.

[aj-stein-nist]

@Compton-NIST @aj-stein-nist I think an automated generation of visual documentation would be very useful!

Please consider the following criteria as improvements:

* [ ]  Generating a CHANGELOG specific to the documentation help people find and examine what is different (like a new relationship, or a new step)

* [ ]  Archive previous version of generated documentation (or at version of generated documentation associated with a tag and/or release). Having previous versions will make it much easier for comparisons by everyone else to see changes

Thanks for the feedback. I am trying to work this into the backlog. @gregelin, can you clarify what specifically you mean by previous version of the generated documentation? The reference information about the models or the whole website? We do have that in git in the nist-pages branch, but I do want to understand your point about tagging better.

[aj-stein-nist]

@Compton-NIST re the two pending action items on for "this sprint" (Sprint 63), are we good to close this and mark as complete? Let me know or I will put it on pause to revisit in Sprint 65. Thanks for your efforts on this.