[SPIKE] Define and Communicate DevSecOps Component Ownership

[ndouglas]

The DevSecOps team has principal ownership of the larger team's infrastructural components, processes, and tools. "Ownership" is a bit vague, though, so we need to clarify what it means. We also depend on some other teams and organizations for aspects of this, e.g. DNS resolution, some pieces of the SSO infrastructure, etc, and this should be clarified. This also necessitates that we delineate responsibilities, points of contact, escalation procedures, etc.

Tasks

• Define "ownership".
• Define our responsibilities.
• Define interaction guidelines
  • Procedures/methods
  • Points of contact
  • Escalation procedures
• Describe and/or formalize our notification system/policies.
• Document ownership.

Acceptance Criteria

• [ ] The definition of DevSecOps "ownership".
• [ ] A catalog of resources/components/systems/whatever owned by DevSecOps, and our responsibilities across/within them as appropriate.
• [ ] Our interaction guidelines.
• [ ] Notification system/policies for certain contingencies.
• [ ] All of the above is actually incorporated into system documentation in an effective and meaningful way.

[ndouglas]

Nug Thoughts

• Define "ownership": Ownership. Ow-ner-ship. Own-er-ship. Weird word. Oh-no-ship.
• Define our responsibilities: This likely needs to be defined broadly across our associated projects (CMS, content-build, parts of the devops repo, a GitHub Action or two that we maintain in separate repositories, etc), and then broken down for each of the projects. Some projects we have total ownership, some projects are owned by others and we exist merely by courtesy, etc.
• Interaction Guidelines:
  • Define points of contact: This will change from turnover, leave, etc, so we should probably find some way of abstracting this that doesn't necessitate us continually updating documentation. Maybe "here's how you get the current schmuck on duty in Slack". We need to work on this alongside or incorporating outcomes from #14297 and #14319.
  • This needs to address: Create user-centered governance that outlines how to best interact (enhancements, defects and general feedback loops) with the component owners if I'm a(n) Editor, Veteran, Product team(s) consuming the CMS
  • Define escalation procedures: This works with points of contact. This is probably going to be broad but shallow, because we're generally the only people who understand these specific systems, but there are countless different failure modes that may each involve us or be completely beyond our control. In some cases (DNS resolution), the issue is above our paygrade.
  • Probably the best approach here is to identify a few common cases (content build failing, staging post-deploy tests failing, Tugboat not building PRs) and acknowledge that anything outside of these scenarios may need to be evaluated on a case-by-case basis. This is also for our reference, I guess, like "if this is breaking, don't bother trying to fix it because we have no control over it. Just open a ticket in #vfs-platform-support or #github-information and pray."
• Describe and/or formalize our notification system/policies: Notifications are mostly handled in Slack and with an opt-in fashion. It might be nice to formalize how we notify people if we encounter an incident, if we have issues that affect engineers, if the PO needs to be made aware of something, etc.
  • This needs to address: Development of methods your functional area will use to communicate CMS platform outages, issues, defects, releases and/or change management (e.g. light business continuity planning)
• Document ownership: I'm again leaning toward Markdown/Mermaid for this. I don't think this should be complicated. I think it's more important to be simple, concise, and readable.