Closed nickumia-reisys closed 2 years ago
Okay, I think I figured out the real problem... hopefully the next test works.. it's based on the following output,
curl https://ip-ranges.amazonaws.com/ip-ranges.json | jq '.prefixes[] | select(.region | contains("us-west-2")) | select(.service | contains("EC2")) | .ip_prefix'
Granted.. that would have worked, but it also wouldn't have been a good solution at all..
So the problem are the Load Balancer IPs. When the repo gets to the part where it's specifying kubernetes resources and communicating with the control plane to create those resources, it's communicating with the control plane through public methods, specifically by going through the Load Balancer. Because the Load Balancer is created after the EKS module is complete, the Load Balancer IPs can't be added to the list of allowed IPs because they are not known at that time.
I guess the real solution is re-routing those k8s calls through the private control plane endpoint.. will try and investigate...
Okay, well, everything is setup for cluster node to control plane traffic to be private, but maybe something about how the Terraform code calls k8s resources forces it to originate from the Load Balancer as opposed to internal node IPs..
References:
default_vpc_enable_dns_support
defaults to true
.enable_dns_hostnames
is coded as true
.cluster_endpoint_private_access
is set to true
.Two examples of the k8s resources originating from non-internal IPs:
Error: Kubernetes cluster unreachable: Get "https://8DAE37A77B85DC4B0D001FDBCE602A4C.yl4.***.eks.amazonaws.com/version": dial tcp 44.225.15.140:443: i/o timeout with helm_release.zookeeper-operator, on crds.tf line 9, in resource "helm_release" "zookeeper-operator": 9: resource "helm_release" "zookeeper-operator" { Error: Post "https://8DAE37A77B85DC4B0D001FDBCE602A4C.yl4.***.eks.amazonaws.com/api/v1/namespaces": dial tcp 54.213.193.9:443: i/o timeout with kubernetes_namespace.starboard-system, on k8s-scanning.tf line 1, in resource "kubernetes_namespace" "starboard-system": 1: resource "kubernetes_namespace" "starboard-system" {
Error: Kubernetes cluster unreachable: Get "https://417EBEE167649BBD2B92C09DB7367AA1.sk1.***.eks.amazonaws.com/version": dial tcp 35.84.236.168:443: i/o timeout with helm_release.zookeeper-operator, on crds.tf line 9, in resource "helm_release" "zookeeper-operator": 9: resource "helm_release" "zookeeper-operator" { Error: Post "https://417EBEE167649BBD2B92C09DB7367AA1.sk1.***.eks.amazonaws.com/apis/rbac.authorization.k8s.io/v1/clusterroles": dial tcp 34.218.191.92:443: i/o timeout with module.aws_load_balancer_controller.kubernetes_cluster_role.this, on .terraform/modules/aws_load_balancer_controller/main.tf line 332, in resource "kubernetes_cluster_role" "this": 332: resource "kubernetes_cluster_role" "this" {
Those IPs are regular EC2 node IPs,
Very much similar to the EC2 IP of the Network Interface attached to the Load Balancer,
The only solutions I can think of are:
@mogul Any ideas would be greatly appreciated..
Reference on how to look up the EC2 IPs mapped to the Load Balancers.
Related to https://github.com/GSA/data.gov/issues/3744
New Additions:
http
provider to obtain the IP of the machine running the terraform codenull_resource.cluster-functional
terraform/modules/provision-aws/variables.tf
structureCool References (hidden in commits):
cluster_endpoint_public_access_cidrs