[SPIKE] Approach to System Documentation for DevSecOps

[ndouglas]

For context, see:

• 14012

• 14298

Tolase is reviewing existing documentation for our DevSecOps systems in #14298 in the CMS github repo. The expected outcome for that precise ticket includes some system diagrams and business process diagrams. An additional outcome is that Tolase has reviewed the DevOps READMEs.

The timing should be right then for We Three DevOps™ to review the current state of DevSecOps system documentation and construct a strategy for tackling the deficits there, using Tolase's work in #14298 and fresh eyes as a starting point for this work.

With our team currently facing various challenges such as onboarding, upcoming leave, and a potential near-future total overhaul of our deployment architecture, we need to define an efficient and practical approach to documenting system components. This includes defining what to consider when tackling system documentation and diagramming with our limited resources and the changes on the horizon.

Tasks

• Review Tolase's work: Go through #14298 and understand the components Tolase is working on. Assess his feedback and the improvements he is making.

• Prioritize components: Identify the most critical components that need immediate documentation. In my mind, this is going to be high-level, practical stuff.

• Documentation methodology: Investigate effective ways of creating maintainable and easily understandable documentation.

• Documentation schedule: Propose a feasible timeline for documenting system components, taking into account the team's other commitments. If possible, include the impact of the potential architecture overhaul on this schedule.

Acceptance Criteria

• [ ] Review the github repo for the system documentation gaps.
• [ ] The team has met or asynchronously discussed this topic.
• [ ] Discrete, actionable followup tickets have been drafted or created.
• [ ] Components or tickets have been pre-prioritized roughly.

[ndouglas]

Nug Thoughts

• Review Tolase's work: I'm mostly interested in seeing what was confusing/incomprehensible/fragmentary/otherwise poorly documented. I don't think #14298 is going to lead to a substantial alteration to our documentation because understanding and documenting our system is just a very big thing.

• Prioritize components: From my perspective, the BRD system is the largest, most complex, most critical, and scariest 👻 Also, it involves other teams providing services to us and being required for reviews, etc, depending on specifics. Also, it has the largest chance of changing completely in the next few months, so I think it's important to fly high and move fast and not get bogged down.

  • Documentation/diagram of the product deployment pipeline. I created a table with some useful information in the deploy-oob doc.

  • Documentation/diagram of the DevOps repos, what concerns they contain, and who is needed for approval for actions in those repos. There's some stuff here that I have to look up every time I want to reference it, like terraform-vsp-infra-something-or-other. Then there's the devops repo, stuff managed within va.gov-cms, etc.

  • Documentation/diagram of Tugboat. I think there was an issue yesterday with someone going into the wrong place and building a preview, so it'd be nice to have a conceptual structure here explaining what each section is. I think our Tugboat documentation is overall okay, but I think we're lacking some technical information like about the VPC, etc that's kind of tribal knowledge and not (???) written down anywhere.

  • Documentation/diagram(s) of DDEV local environments. I think this is mostly good too, I don't think we have a lot of issues here. I'm mostly mentioning it because it's easy to forget that we probably have ownership of this.

• Documentation methodology: I'm kinda into Mermaid and Markdown at this point. I'm screaming internally at the thought of doing stuff with Draw.io and Confluence and so forth. I think that's going to be too effort-intensive and bog us down at a bad time.

• Documentation schedule: I'm thinking that we can split each of the components above into 1-3 or 1-5 discrete, actionable tickets. Thinking maybe 8-15 tickets total, each around 3 story points. I don't think we need to create completely new documentation, just maybe refactor/enhance existing documentation, flesh out FAQs, and add diagrams to help comprehension. Reasonable people may differ 🤷‍♂

[olivereri]

Just read through the documentation spike. One question I have is, and this might be obvious, but who would be the audience that reviews the documentation? Just us? Or is there an expectation that the teams that use our platform would need to make some sense of them?

Also, the last bit about Markdown + Mermaid. I think we ought to pare back Tolase's documentation ticket and focus on those simpler tools rather than using draw.io. The DSVA Platform team is already making more elaborate and colorful diagrams for ATO purposes, we don't have to recreate that same thing.

[productmike]

@olivereri Re: audience we should assume the audience is outside of our team. So anything that you'd ned to levelset with them on how our systems are set up and/or any other noteworthy items should be included. Helpful?