[ci.jenkins.io] Create private EKS cluster with "side" services (datadog, ACP, etc.)

[dduportal]

We need a private EKS cluster to run ci.jenkins.io container agents.

• No need for multiple AZs unless strictly necessary
• We want to use a private endpoint for ci.jenkins.io access to the cluster: https://docs.aws.amazon.com/eks/latest/userguide/cluster-endpoint.html
  • But we also need the public endpoint to be restricted to our VPN (for admin access) and infra.ci.jenkins.io outbound IP (for terraform and Kubernetes management)
  • The subnets can (and should) be private (see https://github.com/jenkins-infra/helpdesk/issues/4320)
• EKS networks considerations: https://docs.aws.amazon.com/eks/latest/userguide/network-reqs.html
• Our former Terraform code, using a module: https://github.com/jenkins-infra/aws/blob/ac02157bb6646918594827b293320f64eacffeb5/cik8s-cluster.tf
• Some addons are needed (ref. https://docs.aws.amazon.com/eks/latest/userguide/workloads-add-ons-available-eks.html):
  • Kube Proxy (of course)
  • CoreDNS (of course)
  • VPC CNI to get native VPC networking for pods and services
  • EBS CSI to ensure we can have volumes for Artifact Caching Proxy
• We need the autoscaler. Worth checking Karpenter.
• We also need the AWS LB Controller installed to allow creating a private LB for the Artifact Caching Proxy to ensure VM agents of ci.jenkins.io can reach the service
  • See https://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/access-container-applications-privately-on-amazon-eks-using-aws-privatelink-and-a-network-load-balancer.html for a guide on how to expose privately the ACP service with a private LB
• About Node pools:
  • "application" ARM64 Node pool to host ACP, datadog an any other "non Jenkins agent" pods. Check the current one in azure: https://github.com/jenkins-infra/azure/blob/e81b9697e4a9cedebf10e119b6a5a112e09b651f/ci.jenkins.io-kubernetes-agents.tf#L69
  • "agents" x86 Node pool to host usual Linux agents: https://github.com/jenkins-infra/azure/blob/e81b9697e4a9cedebf10e119b6a5a112e09b651f/ci.jenkins.io-kubernetes-agents.tf#L103
  • "bom agents" x86 Node pool to host BOm agents: https://github.com/jenkins-infra/azure/blob/e81b9697e4a9cedebf10e119b6a5a112e09b651f/ci.jenkins.io-kubernetes-agents.tf#L137
  • We should start with no Spot agents for the first months (we have credits and we want to avoid too much build retries
  • Note: see https://github.com/jenkins-infra/helpdesk/issues/4318 for upcoming Windows Node pool agents
• We'll have to add this cluster to jenkins-infra/kubernetes-management as usual:
  • An admin SVC account is needed like https://github.com/jenkins-infra/azure/blob/main/ci.jenkins.io-kubernetes-agents.tf#L180-L189
  • Then set up the kubeconfig credential once in SOPS secrts and create the cluster in kubernetes-management
  • Proceed to helmfile releases definitions
  • set up ACP to use a private LB and check access from the ci.jio controller or a VM agent through the private endpoint

[dduportal]

Discussed with @smerle33:

• https://github.com/jenkins-infra/helpdesk/issues/4320 reached a first milestone allowing starting work on the issue here
• Stephane prefers more the EKS topic than the VM topic: assigning EKS to @smerle33
• Scheduling for next milestone when he's back from holidays

[smerle33]

change of usage for the module since last time we used it https://github.com/terraform-aws-modules/terraform-aws-eks/blob/master/docs/UPGRADE-20.0.md

[smerle33]

We choose to deal with all the IAM usage within the private repository https://github.com/jenkins-infra/terraform-states/commit/cfd08c45dd4153d676c9223670f927d515585679 instead of giving the module user too much power.