Discovery: CMS Deployment Replacement

gracekretschmer-metrostar commented 6 months ago

User Story or Problem Statement

The CMS team needs to determine a path forward to replace it’s current system (BRD) for deploying code.

Description or Additional Context

BRD is the system that is used by CMS to deploy code and, although BRD is being maintained by DevOps CoP, their focus is only on the maintenance to keep BRD alive without doing any new development on the system. In addition, OCTO is moving away from utilizing Jenkins and BRD relies on Jenkins. Therefore, BRD is at risk of becoming an unusable system and, in turn, failure, as it continues to be only maintained without substantial development.

BRD is critical for CMS to meet one of its primary functions of delivering new or changed content to veterans. Therefore, the risk of BRD will become a risk to CMS’ core functions and, in turn, the CMS team needs to determine a path forward to replace BRD with a deployment system that healthier and more sustainable for the CMS team, both with system functionality and clear and robust ownership from the appropriate team within OCTO.

Relevant Links

BRD - Platform CMS Team - Confluence (atlassian.net)

Acceptance Criteria

[ ] PoC for Platform’s Argo/EKS solution
[ ] Create an evaluation framework to determine the deployment solution that best meets in the CMS team’s needs
[ ] Recommendation of best solution, using the evaluation framework
[ ] A plan of action for implementation is documented in Github and/or Confluence

Team

Please check the team(s) that will do this work.

[x] CMS Team
[ ] Public Websites
[ ] Facilities
[ ] Accelerated Publishing

### Tasks
- [ ] https://github.com/department-of-veterans-affairs/va.gov-cms/issues/17430
- [ ] https://github.com/department-of-veterans-affairs/va.gov-cms/issues/17988
- [ ] https://github.com/department-of-veterans-affairs/va.gov-cms/issues/18012

gracekretschmer-metrostar commented 6 months ago

@7hunderbird my knowledge on BRD is limited, so I am not feeling confident about the content of this epic, so feel free to make your own edits as well. My general headspace for the goal of this discovery work is: "I want to understand BRD in the current CMS infrastructure to inform my current support and future replacement of the system."

gracekretschmer-metrostar commented 6 months ago

@7hunderbird I am also including myself in this epic, to help support the roadmapping.

gracekretschmer-metrostar commented 6 months ago

Guidance from @little-oddball, Patrick Black would be a helpful resource in this discovery and research work.

7hunderbird commented 5 months ago

I wanted to comment on the Acceptance Criteria item in this ticket.

The DevOps team has a proposal for a system to replace BRD and a high-level roadmap for how to replace BRD.

This is at least it's own ticket, "replacing a system" is why.

The term "tech debt" is often used to crudely group together "problems to solve". But I'd like to propose the idea of "technical currency". I think that BRD is actually generating value for us because of it's general stability.

But There Was a Problem!

The main problem we solved did take a long time to fix and was painful for alot of people. And the more pain felt means that it's more valuable when it's working.

Yet BRD gave us feedback and prevented a broken prod.

BRD's Value Prop

Well Organized
- With a quick primer, you can understand where to go to get information about the status of the CI/CD workflow
We Have a Lower Environment for Dev/Testing
- There is a va.gov-cms-test repo and similar jobs in Jenkins that we can use to test changes to the BRD workflow
- We used the practices of "rollback" and "lower environments" to accelerate un-pausing BRD and restoring service
BRD rebuilds the Virtual Machine for Staging and Prod EVERY DAY
- This means we've got all the steps to build from scratch a virtual machine on Amazon to run the staging.cms.va.gov and prod.cms.va.gov, every. single. day.
- This is a huge value and would also be the largest value we'd need to replace by moving to a different system.

The largest value for me personally is:

When BRD is working, we can prioritize other work
- BRD has to be working for content-build to go out on time multiple times a day from 8am to 8pm EST

gracekretschmer-metrostar commented 5 months ago

Next steps: grab time with Erika to re-refine this ticket and determine if/how to possibly replace BRD.

7hunderbird commented 4 months ago

I have submitted this ticket to onboard CMS to the platform:

https://github.com/department-of-veterans-affairs/va.gov-team/issues/81691

I'm following up with the #plaform-cop-devops slack room to see if there's other information or steps to take to onboard the CMS application.

7hunderbird commented 4 months ago

Also we've created this ticket to track the Proof of Concept to onboard the CMS to the application.

https://github.com/department-of-veterans-affairs/va.gov-cms/issues/17988

gracekretschmer-metrostar commented 2 months ago

Pausing this work, so that there can be more focus and resources on the AP work.

gracekretschmer-metrostar commented 3 weeks ago

Tim: there's probably opportunity to explore moving the Jenkins jobs into Github actions.

department-of-veterans-affairs / va.gov-cms