search

kubernetes-sigs / aws-load-balancer-controller

A Kubernetes controller for Elastic Load Balancers
https://kubernetes-sigs.github.io/aws-load-balancer-controller/
Apache License 2.0
3.82k stars 1.41k forks source link

The port range Backend Security Group Rules not generated #3737

Closed nontster closed 3 weeks ago

nontster commented 4 weeks ago

Describe the bug The Port Range Backend Security Group Rules are not generated when creating ingress with ingressClassName: alb.

The frontend security group, k8s-traffic--, has been auto-generated and attached to the ALB and also -node- attached to the EC2 instance.

However, the port range Backend Security Group Rules (with description "elbv2.k8s.aws/targetGroupBinding=shared") in Security group -node- does not exist.

Lack of the port range backend security group rules, the LB target health becomes "Unhealthy" and I cannot access the service. When I add a SecurityGroup rule from the frontend LB to the Kubernetes node SecurityGroup manually, the health status becomes healthy and the service becomes accessible.

Here are the generated Backend Security Group Rules in Security Group -node-, but no port range rule (The one has description elbv2.k8s.aws/targetGroupBinding=shared).

Security group rule ID, Type, Protocol, Port range, Source, Description
sgr-07aeb2e56a71192ec, DNS (TCP), TCP   53, sg-0292e3d0726f7e77a / cicd-eks-dev-node-20240603084514546900000002,    Node to node CoreDNS
sgr-07af79604c3736ae4, Custom TCP,  TCP 9443,   sg-0a0566b426cf8006b / cicd-eks-dev-cluster-20240603084516488800000003, Cluster API to node 9443/tcp webhook
sgr-0f1fea45e2682398f, Custom TCP,  TCP 10250,  sg-0a0566b426cf8006b / cicd-eks-dev-cluster-20240603084516488800000003, Cluster API to node kubelets
sgr-03e08966c55d3b57f, Custom TCP,  TCP 1025 - 65535,   sg-0292e3d0726f7e77a / cicd-eks-dev-node-20240603084514546900000002,    Node to node ingress on ephemeral ports
sgr-087ea4cca9c431c6d, Custom TCP,  TCP 6443,   sg-0a0566b426cf8006b / cicd-eks-dev-cluster-20240603084516488800000003, Cluster API to node 6443/tcp webhook
sgr-0b8221fe2ca8b83da, Custom TCP,  TCP 4443,   sg-0a0566b426cf8006b / cicd-eks-dev-cluster-20240603084516488800000003, Cluster API to node 4443/tcp webhook
sgr-0b06cb8157f2c26a9, Custom TCP,  TCP 8443,   sg-0a0566b426cf8006b / cicd-eks-dev-cluster-20240603084516488800000003, Cluster API to node 8443/tcp webhook
sgr-0188ff71a51b337db, HTTPS TCP,   443,    sg-0a0566b426cf8006b / cicd-eks-dev-cluster-20240603084516488800000003, Cluster API to node groups
sgr-0d8375e3a42c3d671, DNS (UDP),   UDP 53, sg-0292e3d0726f7e77a / cicd-eks-dev-node-20240603084514546900000002,    Node to node CoreDNS UDP

Steps to reproduce

  1. Deploy EKS using AWS EKS Terraform module
  2. Deploy aws-load-balancer-controller using helm chart, https://aws.github.io/eks-charts, and Terraform
  3. Deploy application (SonarQube) using helm chart and Terraform

Expected outcome Application deployed with AWS ALB created and has target has health status 'Healthy'

Environment

Additional Context:

Ingress settings in Helm values,

ingress:
  enabled: true
  hosts:
  - name: sonar.example.net
    path: /*
  annotations:
    nginx.ingress.kubernetes.io/proxy-body-size: "64m"
    alb.ingress.kubernetes.io/load-balancer-name: "cicd-eks-alb-sonarqube"
    alb.ingress.kubernetes.io/backend-protocol: "HTTP"
    alb.ingress.kubernetes.io/scheme: "internal"
    alb.ingress.kubernetes.io/ssl-redirect: "443"
    alb.ingress.kubernetes.io/target-type: 'ip'
    alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:ap-southeast-1:xxxx:certificate/xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx

Log

kubectl logs -l app.kubernetes.io/name=aws-load-balancer-controller -n kube-system

{"level":"info","ts":"2024-06-03T09:01:07Z","logger":"controller-runtime.metrics","msg":"Starting metrics server"}
{"level":"info","ts":"2024-06-03T09:01:07Z","logger":"controller-runtime.metrics","msg":"Serving metrics server","bindAddress":":8080","secure":false}
{"level":"info","ts":"2024-06-03T09:01:07Z","msg":"starting server","name":"health probe","addr":"[::]:61779"}
{"level":"info","ts":"2024-06-03T09:01:07Z","logger":"controller-runtime.webhook","msg":"Starting webhook server"}
{"level":"info","ts":"2024-06-03T09:01:07Z","logger":"controller-runtime.certwatcher","msg":"Updated current TLS certificate"}
{"level":"info","ts":"2024-06-03T09:01:07Z","logger":"controller-runtime.webhook","msg":"Serving webhook server","host":"","port":9443}
{"level":"info","ts":"2024-06-03T09:01:07Z","logger":"controller-runtime.certwatcher","msg":"Starting certificate watcher"}
I0603 09:01:07.323934       1 leaderelection.go:250] attempting to acquire leader lease kube-system/aws-load-balancer-controller-leader...
2024/06/03 09:01:37 http: TLS handshake error from 100.64.3.201:49686: EOF
2024/06/03 09:01:37 http: TLS handshake error from 100.64.3.201:49690: EOF
{"level":"info","ts":"2024-06-03T10:14:13Z","logger":"controllers.ingress","msg":"created loadBalancer","stackID":"sonarqube/sonarqube-sonarqube","resourceID":"LoadBalancer","arn":"arn:aws:elasticloadbalancing:ap-southeast-1:xxxxxx:loadbalancer/app/cicd-eks-alb-sonarqube/xxxxxx"}
{"level":"info","ts":"2024-06-03T10:14:13Z","logger":"controllers.ingress","msg":"creating listener","stackID":"sonarqube/sonarqube-sonarqube","resourceID":"443"}
{"level":"info","ts":"2024-06-03T10:14:13Z","logger":"controllers.ingress","msg":"created listener","stackID":"sonarqube/sonarqube-sonarqube","resourceID":"443","arn":"arn:aws:elasticloadbalancing:ap-southeast-1:xxxxxx:listener/app/cicd-eks-alb-sonarqube/xxxxxx/xxxxxx"}
{"level":"info","ts":"2024-06-03T10:14:13Z","logger":"controllers.ingress","msg":"creating listener rule","stackID":"sonarqube/sonarqube-sonarqube","resourceID":"443:1"}
{"level":"info","ts":"2024-06-03T10:14:14Z","logger":"controllers.ingress","msg":"created listener rule","stackID":"sonarqube/sonarqube-sonarqube","resourceID":"443:1","arn":"arn:aws:elasticloadbalancing:ap-southeast-1:xxxxxx:listener-rule/app/cicd-eks-alb-sonarqube/xxxxxx/xxxxxx/xxxxxx"}
{"level":"info","ts":"2024-06-03T10:14:14Z","logger":"controllers.ingress","msg":"creating targetGroupBinding","stackID":"sonarqube/sonarqube-sonarqube","resourceID":"sonarqube/sonarqube-sonarqube-sonarqube-sonarqube:9000"}
{"level":"info","ts":"2024-06-03T10:14:14Z","logger":"controllers.ingress","msg":"created targetGroupBinding","stackID":"sonarqube/sonarqube-sonarqube","resourceID":"sonarqube/sonarqube-sonarqube-sonarqube-sonarqube:9000","targetGroupBinding":{"name":"k8s-sonarqub-sonarqub-07de3f6bf7","namespace":"sonarqube"}}
{"level":"info","ts":"2024-06-03T10:14:14Z","logger":"controllers.ingress","msg":"successfully deployed model","ingressGroup":"sonarqube/sonarqube-sonarqube"}
{"level":"info","ts":"2024-06-03T10:17:30Z","msg":"registering targets","arn":"arn:aws:elasticloadbalancing:ap-southeast-1:xxxxxx:targetgroup/k8s-sonarqub-sonarqub-xxxxxx/xxxxxx","targets":[{"AvailabilityZone":null,"Id":"100.64.29.209","Port":9000}]}
{"level":"info","ts":"2024-06-03T10:17:30Z","msg":"registered targets","arn":"arn:aws:elasticloadbalancing:ap-southeast-1:xxxxxx:targetgroup/k8s-sonarqub-sonarqub-xxxxxx/xxxxxx"}
kubectl get events -A --field-selector type!=Normal
NAMESPACE   LAST SEEN   TYPE      REASON                   OBJECT                                                MESSAGE
sonarqube   11m         Warning   FailedNetworkReconcile   targetgroupbinding/k8s-sonarqub-sonarqub-07de3f6bf7   expected exactly one securityGroup tagged with kubernetes.io/cluster/cicd-eks-dev for eni eni-066f0709ddc6e0811, got: [sg-0292e3d0726f7e77a sg-0609012206beefad0] (clusterName: cicd-eks-dev)
nontster commented 3 weeks ago

The problem solved by adding create_cluster_primary_security_group_tags = false to terraform eks module