Restrict acess to EKS Control Plane

[nickumia-reisys]

Related to https://github.com/GSA/data.gov/issues/3744

New Additions:

• Configure cluster_endpoint_public_access_cidrs to be a variable in the broker specification
• Grab the Host IP of the machine starting the broker to be able to run the tests on github actions
• Default cluster_endpoint_public_access_cidrs to be the cloud.gov egress ranges
• http provider to obtain the IP of the machine running the terraform code
• Add dependency for all kubernetes resources to be null_resource.cluster-functional
• Cleanup of terraform/modules/provision-aws/variables.tf structure

Cool References (hidden in commits):

• Terraform EKS Module Inputs - cluster_endpoint_public_access_cidrs
• Cloud.gov egress ranges
• Getting Public IP of terraform machine
• Terraform EKS Module
• Terraform VPC Module

[nickumia-reisys]

Okay, I think I figured out the real problem... hopefully the next test works.. it's based on the following output,

curl https://ip-ranges.amazonaws.com/ip-ranges.json | jq '.prefixes[] | select(.region | contains("us-west-2")) | select(.service | contains("EC2")) | .ip_prefix'

[nickumia-reisys]

Granted.. that would have worked, but it also wouldn't have been a good solution at all..

So the problem are the Load Balancer IPs. When the repo gets to the part where it's specifying kubernetes resources and communicating with the control plane to create those resources, it's communicating with the control plane through public methods, specifically by going through the Load Balancer. Because the Load Balancer is created after the EKS module is complete, the Load Balancer IPs can't be added to the list of allowed IPs because they are not known at that time.

I guess the real solution is re-routing those k8s calls through the private control plane endpoint.. will try and investigate...

[nickumia-reisys]

Okay, well, everything is setup for cluster node to control plane traffic to be private, but maybe something about how the Terraform code calls k8s resources forces it to originate from the Load Balancer as opposed to internal node IPs..

References:

• Enabling private access to control plane endpoint.
• How cluster endpoints are supposed to work.
• Proof that all of the settings are set properly,
  • default_vpc_enable_dns_support defaults to true.
  • enable_dns_hostnames is coded as true.
  • cluster_endpoint_private_access is set to true.
  • We are allowing all of the CIDRs that we have access to.

Two examples of the k8s resources originating from non-internal IPs:

• https://github.com/GSA/datagov-brokerpak-eks/runs/6396195252?check_suite_focus=true#step:6:164

  Error: Kubernetes cluster unreachable: Get "https://8DAE37A77B85DC4B0D001FDBCE602A4C.yl4.***.eks.amazonaws.com/version": dial tcp 44.225.15.140:443: i/o timeout with helm_release.zookeeper-operator, on crds.tf line 9, in resource "helm_release" "zookeeper-operator": 9: resource "helm_release" "zookeeper-operator" { Error: Post "https://8DAE37A77B85DC4B0D001FDBCE602A4C.yl4.***.eks.amazonaws.com/api/v1/namespaces": dial tcp 54.213.193.9:443: i/o timeout with kubernetes_namespace.starboard-system, on k8s-scanning.tf line 1, in resource "kubernetes_namespace" "starboard-system": 1: resource "kubernetes_namespace" "starboard-system" {

• https://github.com/GSA/datagov-brokerpak-eks/runs/6394179937?check_suite_focus=true#step:6:176

  Error: Kubernetes cluster unreachable: Get "https://417EBEE167649BBD2B92C09DB7367AA1.sk1.***.eks.amazonaws.com/version": dial tcp 35.84.236.168:443: i/o timeout with helm_release.zookeeper-operator, on crds.tf line 9, in resource "helm_release" "zookeeper-operator": 9: resource "helm_release" "zookeeper-operator" { Error: Post "https://417EBEE167649BBD2B92C09DB7367AA1.sk1.***.eks.amazonaws.com/apis/rbac.authorization.k8s.io/v1/clusterroles": dial tcp 34.218.191.92:443: i/o timeout with module.aws_load_balancer_controller.kubernetes_cluster_role.this, on .terraform/modules/aws_load_balancer_controller/main.tf line 332, in resource "kubernetes_cluster_role" "this": 332: resource "kubernetes_cluster_role" "this" {

Those IPs are regular EC2 node IPs,

Very much similar to the EC2 IP of the Network Interface attached to the Load Balancer,

The only solutions I can think of are:

• To figure out how to rework the whole Load Balancer Creation in order to include the Load Balancer IPs in the allowed public endpoint CIDRs
• Figure out why the terraform is trying to create the k8s resources using the Load Balancer IP as opposed to originating the requests from the worker nodes to the control plane.

@mogul Any ideas would be greatly appreciated..

[nickumia-reisys]

Reference on how to look up the EC2 IPs mapped to the Load Balancers.