Automate EKS Cluster Deployment and Tear Down

[raywangoctova]

Problem Statement

Currently the EKS clusters that host vets-api and other applications are provisioned in manual, time intensive ways. This creates overhead when Platform teams are performing updates, maintenance, and in the overall management of these clusters. These tasks include:

• Creating new environments (both live and ephemeral)
• Performing blue/green upgrades to EKS clusters
• Monitoring and logging those clusters

In addition, developers do not have the ability to spin up and tear down clusters without risk of creating orphaned resources.

Our hypothesis is that we can solve these problems by implementing improved automation for the deployment, management, and decommissioning of these EKS clusters.

This would support the goals of zero downtime deploys using blue/green deployment, and facilitate the ongoing maintenance and updates of EKS control plane.

User Impact

• All teams that are building, maintaining, or deploying a Kubernetes (EKS) cluster on Platform
• In the event of an upgrade gone wrong, we have the previous cluster still running to [DNS] switch back to
  • Creates a more stable and reliable platform for our end users and engineers
• Provides a safe environment for building and testing new core Platform services (Ingress, Networking, etc.)

Where was this problem reported?

• Going through multiple in-place cluster upgrades where rolling back is not an option
• Disaster recovery exercises
• Developers and system users looking for a safe place to learn and work with the Platform's cluster setup

What do we not know about the problem space?

What areas do we need more information about so that we can accurately solve the problem? What discovery is needed?

• Using the 172.16.0.0/16 CIDR for cluster nodes
  • Platform's 10.0.0.0/8 CIDR is incredibly limited (in-place cluster upgrades had to wait for IPs at times)
  • Only the cluster load balancers require VA resolution on the 10.0.0.0/8 CIDR
• Outside of when the current EKS clusters were built, no other cluster has been built and deployed with our current set of tools and applications

What (if any) research or discovery has been done?

This has been a topic since mid to late 2022, and has undergone extensive research into use cases that today we cannot accommodate with our existing infrastructure

• https://vfs.atlassian.net/wiki/spaces/OT/pages/2522448886/Platform+v.+Next+Goals
• https://vfs.atlassian.net/wiki/spaces/OT/pages/2333802651/Kubernetes+Production-Ready+Checklist+EKS
• https://vfs.atlassian.net/wiki/spaces/OT/pages/1549041761/Infrastructure+Roadmap+WIP#Future-initiatives-(in-logical-priority)
• Through experience in maintaining the existing infrastructure

What is the acceptance criteria?

• [x] EKS clusters are deployed using Terraform
  • [x] No resources are shared between clusters
• [x] Every cluster has it's own unique vfs.va.gov URL (i.e., same way that existing clusters have traefik-{{env}}.vfs.va.gov)
• [ ] Clusters are deployed and torn down through automation
  • [x] Allow Platform engineers to spin up and manage their own ephemeral EKS cluster
  • [x] Environments (live clusters)
  • [x] Ephemeral
    • [ ] Scheduled reminders to tear down ephemeral clusters
  • [ ] Nightly (fresh build and teardown)
  • [ ] Nightly Upgrade (fresh deploy + cluster upgrade + teardown)
    • [ ] Nightly Next (fresh deploy of the next version of EKS)
• [ ] Deploy smoke test with each cluster [to verify that it can be deployed to as well as serve traffic]
• [ ] Develop a tech stack upgrade schedule/roadmap
• [ ] Documentation (Update existing, and/or create new)
  • [ ] Internal cluster architecture diagrams
  • [ ] Steps for a Blue/Green EKS environment
  • [ ] Include steps for switching back and forth
• [ ] The existing observability stack is deployed
• [ ] Successfully deploy EKS version 1.27 to the dev environment using A/B deployment

How should we measure success?

Time to deploy a new EKS cluster Time to teardown a new EKS cluster Time to perform blue/green deploy of EKS cluster of orphaned resources remaining after EKS cluster teardown of manual actions required to update EKS cluster control plane version

TODOs

• [x] Convert this issue to an epic

[mchelen-gov]

Which of these other issues would be covered by this?

• https://github.com/department-of-veterans-affairs/va.gov-team/issues/54633
• https://github.com/department-of-veterans-affairs/va.gov-team/issues/50682
• https://github.com/department-of-veterans-affairs/va.gov-team/issues/52910

[npeterson54]

Action items:

• [x] Clarify the user impact
• [x] Success metrics (how do we know we are successful)
• [x] Make a decision on the last 2 AC. If not, create new projects for them.
• [x] Clarify what self service means
• [x] Reconcile other issues Mike mentioned into this.

[npeterson54]

Random thoughts; mostly just reiterating what was said yesterday. To avoid scope creep, I think we should break the last two AC out into their own project. They make an awesome project and do not fit into the problem statement of the issue.

[EWashb]

@npeterson54 I'm reading through all the documentation so that I can understand the landscape here. I'm curious if yall have given more thought intro the metrics of success that we can track. I see above you mentioned # of failures per month, but would love to jam about your ideas!

[mchelen-gov]

Quick process note, @EWashb should be the only one adding the "PO Endorsed" label to this project. Erika as soon as you feel it is ready from a refinement perspective, please re-add. Thanks everyone for understanding!

[little-oddball]

If we are taking the PO Endorsed label off that indicates that no work should be being performed on the effort. This also means the effort should NOT be in the In Progress column. In a previous discussion w/ Bill and Mike C it was communicated that we SHOULD move forward. @EWashb , happy to chat about this and remove all the confusion, etc.

[EWashb]

This looks good to me. Thank you @npeterson54 for getting those metrics defined!

[EWashb]

@little-oddball thanks for you comments. Everything is fine from my perspective to start this work. Since I am now onboarded, hopefully that will not be a blocker or a source of confusion again.

[annekerr49]

Erika is this is your ticket could you please add it to the DE Procuts -In progress column ? https://github.com/orgs/department-of-veterans-affairs/projects/940/views/2

[EWashb]

@annekerr49 it has now been added to the DE board.

[EWashb]

@npeterson54 as we are completing tasks within this epic, can you be sure to check off the ACs listed so that I can follow along with the progress? Thank you.

[EWashb]

@AparnaNittalaUSDS going through our DE product board and I added the start date to this since I was still POing this team at that time. However, I don't know the target delivery date for this. Could you add it here?

[annekerr49]

Aparna is checking with Nate Peterson as to status.

[AparnaNittalaUSDS]

@annekerr49 Jan 2024 is the latest date for the completion of this activity. Confirmed with Nate

[AparnaNittalaUSDS]

Updating the target delivery date to February 28, 2025

[annekerr49]

What is the reason this project is targeting Feb 28, 2015 as end date?

[ph-One]

This work has been on hold for many months while all of Platform was being migrated to Amazon Linux 2 based images (Amazon Linux 1 is end of life). We are beginning to get back to this work.