Spike: AWS EKS - MvP for Tools Cluster

[AntonyBishop]

Background:

We want to spike EKS and we also want to build a Tools cluster. We'll use this opportunity to build a Tools Cluster using EKS.

Questions:

We have some specific questions we want to answer:

• Questions (Just for Tools Cluster):
• Can we stop using KOPS?
• What do we lose from our control plane?
• How does authentication work?
• What does ingress look like?
• Impact on all components
• Admission controllers (OPA, Cert Manager, PSPs, External DNS, Mod Security)
• How easy to deploy/change
• Check limits on Nodes

Definition of done:

• [ ] We can answer our questions
• [ ] We have a Tools cluster
• [ ] Demo for the team
• [ ] We can make a decision to run this as our Tools Cluster on EKS

[AntonyBishop]

Questions (Just for Tools Cluster): Can we stop using KOPS? What do we lose from our control plane? How does authentication work? What does ingress look like? Impact on all components Admission controllers (OPA, Cert Manager, PSPs, External DNS, Mod Security) How easy to deploy/change Check limits on Nodes

[mogaal]

Almost all components work great except for a few tweaks I did. Comments bellow. To summarize the questions within the Spike I feel confident we can use EKS for the management cluster.

As a TODO it needs to be documented the components, but it will be similar to what you already have. Also needs to be demo to the team.

There were certain components that were not tested because they are not needed for the management cluster. They are: ecr-exporter, cloudwatch-exporter, kuberos, cluster-backup-checker.

PR is already created, if someone has some comments it would be useful so I can check it out when I come back from holidays :-).

Questions of the Spike:

• Can we stop using KOPS?: Yes, we can.
• What do we lose from our control plane?: In term of visibility, we know what is happening in the control plane because logs are shipped to CloudWatch. In term of custom configurations in the control-plane side is not possible as far as I looked in the documentation.
• How does authentication work? : Authentication works with IAM by default. Initially, the management cluster is going to be used by CP team so it might not be needed to set up Kuberos to integrate it with auth0. Maybe for the future, in case we want to replace live-1 with EKS we'll have to explore this
• What does ingress look like?: The same
• Impact on all components: Mostly all components works we use for kOps work. There is one exception, KIAM, which I couldn't make it work and I need to invest more time to figure it out. Not all components are needed inside the management cluster. Components included When I say all components I mean: cert-manager, external-dns, metrics-server, nginx-ingress, prometheus, eventrouter. It was also included an extra-component: cluster-autoscaller. Which allow us to scale up/down the cluster automatically.
• Admission controllers (OPA, Cert Manager, PSPs, External DNS, Mod Security): They look to be working, check the PR.
• How easy to deploy/change: Very easy, there are terraform resources to manage the cluster itself and every component of it, very similar approach taken for the kOps cluster. Here the terraform code