NLB TargetGroup attribute change causes downtime due to SG rule reconciliation

[bnutt]

Describe the bug Modifying the target group attributes of a loadbalancer provisioned by the LBC will result in the controller briefly revoking security group rules on the backend security group. In the time that the rules are revoked and when the rules are added back as part of reconciliation, traffic is dropped to external users. Right now this only looks to occur when using IP targets. In testing, there has been up to a minute gap between the revoke and add of the security group rules.

Steps to reproduce Provision NLB using a k8s service object. Configure the NLB with client IP preservation enabled.

apiVersion: v1
kind: Service
metadata:
  name: echoserver
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
    service.beta.kubernetes.io/aws-load-balancer-target-group-attributes: preserve_client_ip.enabled=true
spec:
  selector:
    app: echoserver
  ports:
    - port: 80
      targetPort: 8080
      protocol: TCP
  type: LoadBalancer
  loadBalancerClass: service.k8s.aws/nlb

Allow the controller to provision the loadbalancer.

apiVersion: v1
kind: Service
metadata:
  name: echoserver
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
    service.beta.kubernetes.io/aws-load-balancer-target-group-attributes: preserve_client_ip.enabled=false
spec:
  selector:
    app: echoserver
  ports:
    - port: 80
      targetPort: 8080
      protocol: TCP
  type: LoadBalancer
  loadBalancerClass: service.k8s.aws/nlb

Expected outcome Client IP is disabled without downtime. If I understand the docs correctly, this may not be happening because the controller modifies the rules on what IP ranges are allowed depending whether the flag is set or not. If this is correct, is the only safe way to disable client IP preservation without downtime to provision a new loadbalancer and migrate traffic to it?

Environment

• AWS Load Balancer controller version: 2.8.1
• Kubernetes version: v1.27.13
• Using EKS (yes/no), if so version? Yes, v1.27.13. Plat version: eks.17

Additional Context:

[bnutt]

I think I have confirmed this is due to how the LB controller handles disabling client IP preservation.

1. When preserveClientIP is true, the lbc will use what is specified in loadBalancerSourceRanges and by default allow 0.0.0.0/0 for public loadbalancers.
2. When preserveClientIP is false, the lbc will use the loadbalancer subnet CIDR blocks in the security group rules, this allows only the loadbalancer ips to reach the worker node hosting the workload.
3. Since we're flipping the flag, this results in a diff the generated security group rules list. Which would be fine if the controller authorized rules before revoking them, but because it doesn't, it causes an ingress outage. Referencing https://github.com/kubernetes-sigs/aws-load-balancer-controller/issues/3639 for the revoke before authorize problem

[M00nF1sh]

@bnutt even if the controller handles the sg rules in correct order, there could still be gaps of downtimes due to the eventually consistency nature of AWS APIs, and the underlying implementation of preserve_client_ip in ELB. If you indeed need to change the preserve_client_ip, which is a big change to LBs, to avoid downtimes, i'd suggest use a new service with weighted DNS based migration.