Determination on best workflow tool & Implement

[cmaeng]

Description

As part of the migration off of BRD, there was a spike done in #9765 to inventory all CMS tasks that need to be moved.

Before starting any migrations, we need to determine the best workflow tool to leverage. We defaulted to GitHub Actions but have brainstormed multiple potential issues with that.

• Limitation on frequency of job runs
• GHA isn't great to see the output of tasks that run with a high frequency

We'll want to coordinate with the Platform Infrastructure team on their recommendations.

Acceptance Criteria

• [x] Research Solutions - Determine possible alternative solutions and document findings below
• [x] Decide on Possible Alternative - Decide on alternative and develop plan to finalize review & decision
• [ ] Develop & Document Implementation Plan - Develop plan (issues) to roll out changes and new platform

[ElijahLynn]

Slack discussion here > https://dsva.slack.com/archives/CJYRZK2HH/p1668112547807419

[cmaeng]

Hey team! Please add your planning poker estimate with Zenhub @ElijahLynn @olivereri @timcosgrove

[olivereri]

Starting a discussion with Github support to explore what our options are for running workflows on a sub-5-minute schedule: https://dsva.slack.com/archives/CU1E4CX9U/p1668549708262719

[olivereri]

Previous exploration of Lagoon as a solution: https://github.com/department-of-veterans-affairs/va.gov-cms/issues/6673

[olivereri]

Impetus

https://dsva.slack.com/archives/CT4GZBM8F/p1616531738323200 VSP-Ops is planning to sunset Jenkins later this year. they will announce in the platform newsletter with additional details and guidance on transitioning Jenkins jobs to GitHub actions March 32rd 2021

Options

Take Over Existing Jenkins

Pros

• Currently hosts all current jobs. Handles build, deployment, release and run time workflows.
• Most flexibility in job type and frequency.

Cons

• Lots of maintenance debt with this installation.
• Lots of security debt wit this installation.
• Plugin maintenance debt with limited team capacity.
• Infrastructure maintenance with limited team capacity.
• Underlying VPC will eventually be removed. Bypasses the Trusted Internet Connection (TIC) to reach the internet.
• Does not directly address modernizing hosting around containers on a container platform.

Provisional Determination

A larger better staffed team no longer wishes to support or maintain Jenkins. Our team has less capacity. There is mounting pressure to deprecate the underlying network.

Non-starter

Roll Our Own Jenkins in EKS

Pros

• One to one port. Would handle build, deployment, release and run time workflows.
• Most flexibility in job type and frequency.
• Clean slate for maintenance, plugin and security debt.
• Less infrastructure maintenance while running in EKS. Workload shared with Platform Crew.
• Would run on approved/supported network.
• Instance/configuration would be dedicated entirely to CMS purposes.

  Cons

• Limited team capacity would now include Jenkins Plugin management, general maintenance, and security.
• Limited team capacity would now include shared infrastructure support requirements.
• Diverges from Platform Crew model of supported/utilized workflow tools.
• Does not directly address modernizing hosting around containers on a container platform.

Provisional Determination

While this would technically meet the goal of migrating away from existing Jenkins, rolling our own Jenkins will likely not fit well into a robust long-term solution. With that in mind:

• Does it make sense to invest the time in setting it up and migrating jobs when that time could be spent on a more robust long-term solution?
• Will this buy us time to gain workflow feature parity then focus on the long-term solution?

At best an interim solution. Quick and Dirty.

Github Actions Worflows

Pros

• Self-hosted Workflow Runner executes Ansible tasks on current infrastructure.
  • Configuration code re-use.
• Almost all current jobs portable to GHA since it will be simply used to trigger Ansible configuration code.

  Cons

• Workflow run reliability issues:
  • No guarantee that jobs will run at the requested scheduled time.
  • Workflow runs could take as long as 20 minutes to begin running due to the processing queue.
  • Can't perform sub-5-minute workflow schedule.
• Does not directly address modernizing hosting around containers on a container platform.

Provisional Determination

Github Actions Workflows has general schedule reliability issues. Frequently run jobs will be more affected by this reliability issue but even for jobs with lower frequency, even a few inconsistently timed job runs will be annoying. This just doesn't seem suitable for important production tasks.

A note on using automation for sub-5-minute task runs. It is considered a rare or odd case to not use a cloud cron service, on server cron, or other app timer. The reason we do this is to expose the task definition to developers. Talk with Elijah about this because task definition code is exposed to devs, scheduling and method of execution is not. Devs shouldn't care about how tasks are executed, just what is executed. However, having a single task runner or other method of executing things is preferable to splitting it across multiple methods. I think.

Non-starter At best an interim solution

Lagoon

Pros

• Directly addresses modernizing hosting around containers on a container platform.
• Provides current Build/Deploy pipeline feature parity.
• Designed with Drupal in mind.
• Handles all Build and Deployment tasks.
• Provides Cron and Custom Task definitions in .lagoon.yaml for workflow related thing-doing.

  Cons

• Big system for a single site Drupal deployment. Currently no other apps would be hosted on Lagoon.

  Argo Workflows

  Pros

• Operates like Jenkins.

  Cons

• Unused by current Platform Operations team. CMS team will be learning on our own.
• Assumes adoption of Argo CD + Github Actions and engineering solutions to reach current Build/Deploy pipeline feature parity.

  Kubernetes CronJob

  Pros

  Cons

• Unused by current Platform Operations team. CMS team will be learning on our own.
• Assumes adoption of Argo CD + Github Actions and engineering solutions to reach current Build/Deploy pipeline feature parity.

[timcosgrove]

This is a large ticket and will roll over.

[mchelen-gov]

Can this be addressed with https://github.com/department-of-veterans-affairs/va.gov-cms/issues/8849 ?

[olivereri]

@mchelen-gov For the every-minute-job that sends Datadog metrics and content release queues, yes that would be appropriate. However, in the broader context of what we want to achieve, no we can't address this with the Advanced Queue runner in #8849.

Thinking about it while I type this out, using the Advanced Queue runner for this more difficult to migrate tasks would provide us more flexibility. But I'm hesitant to make a expedient solution decision when we can land on a more comprehensive and long-term solution.

[TheBoatyMcBoatFace]

Adding to "Epic" workflow tool

[TheBoatyMcBoatFace]

Removed Sprint 72 and Added Sprint 73 Label

Issue larger than anticipated. Converting to Epic

[TheBoatyMcBoatFace]

Shifting focus from #11133 to this issue. See #11133 for additional info.

[TheBoatyMcBoatFace]

The findings from this issue are transitioning to the Lagoon Implementation Epic.

• 11839

Lagoon Implementation Epics

• 11836

• 11837

• 11838