Setup auto-scaling for Managed Node Groups (MNGs)

mogul commented 2 years ago

User Story

In order to avoid running out of compute resources underlying brokered SOLR instances, data.gov wants EC2 instances to be added on an as-needed basis to handle EKS workloads.

Acceptance Criteria

[ACs should be clearly demoable/verifiable whenever possible. Try specifying them using BDD.]

[x] GIVEN the an EKS cluster was brokered with a desired managed node group of 1 \ AND kubectl shows 1 node \ WHEN a SOLR cloud instance is brokered with 10 replicas and 12G mem (each) \ AND the appropriate amount of time has passed (20 min?) \ THEN kubectl shows the number of nodes is greater than 1

Background

AWS Managed Node Groups have autoscaling capabilitied built-in by default. However, for kubernetes to utlize the AWS capability, it needs to deploy an autoscaling mechanism that is aware of the AWS infrastructure. (The EKS module we're using also has an example of configuring cluster autoscaling.)

Security Considerations (required)

[Any security concerns that might be implicated in the change. "None" is OK, just be explicit here!] None.

Sketch

[ ] Familiarize yourself with the AWS Console, datagov-brokerpak-eks and datagov-brokerpak-solr
[x] Investigate whether to implement Cluster Autoscaler or Karpenter
[x] Pick up the Auto-Scaler PR to finish implementing a design.
[x] Add test to verify the ACs
- [x] Useful link: https://stackoverflow.com/questions/62146263/kubernetes-cluster-autoscaler-how-to-verify-autoscaling-is-working

mogul commented 2 years ago

I stumbled on a recent blog post describing how to use Karpenter with spot instances.