search

kubernetes-sigs / aws-load-balancer-controller

A Kubernetes controller for Elastic Load Balancers
https://kubernetes-sigs.github.io/aws-load-balancer-controller/
Apache License 2.0
3.93k stars 1.46k forks source link

Delay of 6m15s between building a model and applying it #2070

Closed ecrousseau closed 3 years ago

ecrousseau commented 3 years ago

I am using aws-load-balancer-controller v2.2.0 in an EKS cluster and seeing a delay that I don't understand between when I update a Kubernetes service and when the controller begins to perform actions via the AWS API.

After noticing the controller seemed slow, I reduced the deployment to a single replica, added debug logging, and performed two tests - changing the port of an existing service, and creating a new service. In both cases there was a delay of exactly 6m15s between the log entry for "successfully built model" and starting to make changes in AWS.

What is the controller doing/waiting for when we change a service? Is there anything we can do to make it faster?

In contrast - when a pod is deleted, the controller starts deregistering the target immediately - same for registering newly started pods.

Log entries for changing a port:

aws-load-balancer-controller-5899c598f7-rll8b aws-load-balancer-controller {"level":"info","ts":1623387509.3880403,"logger":"controllers.service","msg":"successfully built model","model":"{\"id\":\"istio-internal/istio-ingressgateway\",\"resources\":{\"AWS::ElasticLoadBalancingV2::Listener\":{\"15021\":{\"spec\":{\"loadBalancerARN\":{\"$ref\":\"#/resources/AWS::ElasticLoadBalancingV2::LoadBalancer/LoadBalancer/status/loadBalancerARN\"},\"port\":15021,\"protocol\":\"TCP\",\"defaultActions\":[{\"type\":\"forward\",\"forwardConfig\":{\"targetGroups\":[{\"targetGroupARN\":{\"$ref\":\"#/resources/AWS::ElasticLoadBalancingV2::TargetGroup/istio-internal/istio-ingressgateway:15021/status/targetGroupARN\"}}]}}]}},\"15443\":{\"spec\":{\"loadBalancerARN\":{\"$ref\":\"#/resources/AWS::ElasticLoadBalancingV2::LoadBalancer/LoadBalancer/status/loadBalancerARN\"},\"port\":15443,\"protocol\":\"TCP\",\"defaultActions\":[{\"type\":\"forward\",\"forwardConfig\":{\"targetGroups\":[{\"targetGroupARN\":{\"$ref\":\"#/resources/AWS::ElasticLoadBalancingV2::TargetGroup/istio-internal/istio-ingressgateway:15443/status/targetGroupARN\"}}]}}]}},\"443\":{\"spec\":{\"loadBalancerARN\":{\"$ref\":\"#/resources/AWS::ElasticLoadBalancingV2::LoadBalancer/LoadBalancer/status/loadBalancerARN\"},\"port\":443,\"protocol\":\"TCP\",\"defaultActions\":[{\"type\":\"forward\",\"forwardConfig\":{\"targetGroups\":[{\"targetGroupARN\":{\"$ref\":\"#/resources/AWS::ElasticLoadBalancingV2::TargetGroup/istio-internal/istio-ingressgateway:443/status/targetGroupARN\"}}]}}]}},\"8080\":{\"spec\":{\"loadBalancerARN\":{\"$ref\":\"#/resources/AWS::ElasticLoadBalancingV2::LoadBalancer/LoadBalancer/status/loadBalancerARN\"},\"port\":8080,\"protocol\":\"TCP\",\"defaultActions\":[{\"type\":\"forward\",\"forwardConfig\":{\"targetGroups\":[{\"targetGroupARN\":{\"$ref\":\"#/resources/AWS::ElasticLoadBalancingV2::TargetGroup/istio-internal/istio-ingressgateway:8080/status/targetGroupARN\"}}]}}]}}},\"AWS::ElasticLoadBalancingV2::LoadBalancer\":{\"LoadBalancer\":{\"spec\":{\"name\":\"k8s-istioint-istioing-1ae93523c4\",\"type\":\"network\",\"scheme\":\"internal\",\"ipAddressType\":\"ipv4\",\"subnetMapping\":[{\"subnetID\":\"subnet-02b023da39b461794\"},{\"subnetID\":\"subnet-06b0977af45d22943\"},{\"subnetID\":\"subnet-0d9c496c221892d5d\"}],\"loadBalancerAttributes\":[{\"key\":\"access_logs.s3.enabled\",\"value\":\"false\"},{\"key\":\"access_logs.s3.bucket\",\"value\":\"\"},{\"key\":\"access_logs.s3.prefix\",\"value\":\"\"},{\"key\":\"load_balancing.cross_zone.enabled\",\"value\":\"true\"}]}}},\"AWS::ElasticLoadBalancingV2::TargetGroup\":{\"istio-internal/istio-ingressgateway:15021\":{\"spec\":{\"name\":\"k8s-istioint-istioing-371c36e83a\",\"targetType\":\"ip\",\"port\":15021,\"protocol\":\"TCP\",\"healthCheckConfig\":{\"port\":\"traffic-port\",\"protocol\":\"TCP\",\"intervalSeconds\":10,\"healthyThresholdCount\":3,\"unhealthyThresholdCount\":3},\"targetGroupAttributes\":[{\"key\":\"proxy_protocol_v2.enabled\",\"value\":\"false\"}]}},\"istio-internal/istio-ingressgateway:15443\":{\"spec\":{\"name\":\"k8s-istioint-istioing-94a49b7570\",\"targetType\":\"ip\",\"port\":15443,\"protocol\":\"TCP\",\"healthCheckConfig\":{\"port\":\"traffic-port\",\"protocol\":\"TCP\",\"intervalSeconds\":10,\"healthyThresholdCount\":3,\"unhealthyThresholdCount\":3},\"targetGroupAttributes\":[{\"key\":\"proxy_protocol_v2.enabled\",\"value\":\"false\"}]}},\"istio-internal/istio-ingressgateway:443\":{\"spec\":{\"name\":\"k8s-istioint-istioing-77f4ff3ffc\",\"targetType\":\"ip\",\"port\":8443,\"protocol\":\"TCP\",\"healthCheckConfig\":{\"port\":\"traffic-port\",\"protocol\":\"TCP\",\"intervalSeconds\":10,\"healthyThresholdCount\":3,\"unhealthyThresholdCount\":3},\"targetGroupAttributes\":[{\"key\":\"proxy_protocol_v2.enabled\",\"value\":\"false\"}]}},\"istio-internal/istio-ingressgateway:8080\":{\"spec\":{\"name\":\"k8s-istioint-istioing-e82d819816\",\"targetType\":\"ip\",\"port\":8080,\"protocol\":\"TCP\",\"healthCheckConfig\":{\"port\":\"traffic-port\",\"protocol\":\"TCP\",\"intervalSeconds\":10,\"healthyThresholdCount\":3,\"unhealthyThresholdCount\":3},\"targetGroupAttributes\":[{\"key\":\"proxy_protocol_v2.enabled\",\"value\":\"false\"}]}}},\"K8S::ElasticLoadBalancingV2::TargetGroupBinding\":{\"istio-internal/istio-ingressgateway:15021\":{\"spec\":{\"template\":{\"metadata\":{\"name\":\"k8s-istioint-istioing-371c36e83a\",\"namespace\":\"istio-internal\",\"creationTimestamp\":null},\"spec\":{\"targetGroupARN\":{\"$ref\":\"#/resources/AWS::ElasticLoadBalancingV2::TargetGroup/istio-internal/istio-ingressgateway:15021/status/targetGroupARN\"},\"targetType\":\"ip\",\"serviceRef\":{\"name\":\"istio-ingressgateway\",\"port\":15021},\"networking\":{\"ingress\":[{\"from\":[{\"ipBlock\":{\"cidr\":\"REDACTED\"}},{\"ipBlock\":{\"cidr\":\"REDACTED\"}},{\"ipBlock\":{\"cidr\":\"REDACTED\"}}],\"ports\":[{\"protocol\":\"TCP\",\"port\":15021}]}]}}}}},\"istio-internal/istio-ingressgateway:15443\":{\"spec\":{\"template\":{\"metadata\":{\"name\":\"k8s-istioint-istioing-94a49b7570\",\"namespace\":\"istio-internal\",\"creationTimestamp\":null},\"spec\":{\"targetGroupARN\":{\"$ref\":\"#/resources/AWS::ElasticLoadBalancingV2::TargetGroup/istio-internal/istio-ingressgateway:15443/status/targetGroupARN\"},\"targetType\":\"ip\",\"serviceRef\":{\"name\":\"istio-ingressgateway\",\"port\":15443},\"networking\":{\"ingress\":[{\"from\":[{\"ipBlock\":{\"cidr\":\"REDACTED\"}},{\"ipBlock\":{\"cidr\":\"REDACTED\"}},{\"ipBlock\":{\"cidr\":\"REDACTED\"}}],\"ports\":[{\"protocol\":\"TCP\",\"port\":15443}]}]}}}}},\"istio-internal/istio-ingressgateway:443\":{\"spec\":{\"template\":{\"metadata\":{\"name\":\"k8s-istioint-istioing-77f4ff3ffc\",\"namespace\":\"istio-internal\",\"creationTimestamp\":null},\"spec\":{\"targetGroupARN\":{\"$ref\":\"#/resources/AWS::ElasticLoadBalancingV2::TargetGroup/istio-internal/istio-ingressgateway:443/status/targetGroupARN\"},\"targetType\":\"ip\",\"serviceRef\":{\"name\":\"istio-ingressgateway\",\"port\":443},\"networking\":{\"ingress\":[{\"from\":[{\"ipBlock\":{\"cidr\":\"REDACTED\"}},{\"ipBlock\":{\"cidr\":\"REDACTED\"}},{\"ipBlock\":{\"cidr\":\"REDACTED\"}}],\"ports\":[{\"protocol\":\"TCP\",\"port\":8443}]}]}}}}},\"istio-internal/istio-ingressgateway:8080\":{\"spec\":{\"template\":{\"metadata\":{\"name\":\"k8s-istioint-istioing-e82d819816\",\"namespace\":\"istio-internal\",\"creationTimestamp\":null},\"spec\":{\"targetGroupARN\":{\"$ref\":\"#/resources/AWS::ElasticLoadBalancingV2::TargetGroup/istio-internal/istio-ingressgateway:8080/status/targetGroupARN\"},\"targetType\":\"ip\",\"serviceRef\":{\"name\":\"istio-ingressgateway\",\"port\":8080},\"networking\":{\"ingress\":[{\"from\":[{\"ipBlock\":{\"cidr\":\"REDACTED\"}},{\"ipBlock\":{\"cidr\":\"REDACTED\"}},{\"ipBlock\":{\"cidr\":\"REDACTED\"}}],\"ports\":[{\"protocol\":\"TCP\",\"port\":8080}]}]}}}}}}}}"}
aws-load-balancer-controller-5899c598f7-rll8b aws-load-balancer-controller {"level":"info","ts":1623387884.226536,"logger":"controllers.service","msg":"creating targetGroup","stackID":"istio-internal/istio-ingressgateway","resourceID":"istio-internal/istio-ingressgateway:8080"}

Log entries for creating a new service:

aws-load-balancer-controller-5899c598f7-rll8b aws-load-balancer-controller {"level":"info","ts":1623389441.1386073,"logger":"controllers.service","msg":"successfully built model","model":"{\"id\":\"istio-internal/test\",\"resources\":{\"AWS::ElasticLoadBalancingV2::Listener\":{\"15021\":{\"spec\":{\"loadBalancerARN\":{\"$ref\":\"#/resources/AWS::ElasticLoadBalancingV2::LoadBalancer/LoadBalancer/status/loadBalancerARN\"},\"port\":15021,\"protocol\":\"TCP\",\"defaultActions\":[{\"type\":\"forward\",\"forwardConfig\":{\"targetGroups\":[{\"targetGroupARN\":{\"$ref\":\"#/resources/AWS::ElasticLoadBalancingV2::TargetGroup/istio-internal/test:15021/status/targetGroupARN\"}}]}}]}}},\"AWS::ElasticLoadBalancingV2::LoadBalancer\":{\"LoadBalancer\":{\"spec\":{\"name\":\"k8s-istioint-test-e7dd476790\",\"type\":\"network\",\"scheme\":\"internal\",\"ipAddressType\":\"ipv4\",\"subnetMapping\":[{\"subnetID\":\"subnet-02b023da39b461794\"},{\"subnetID\":\"subnet-06b0977af45d22943\"},{\"subnetID\":\"subnet-0d9c496c221892d5d\"}],\"loadBalancerAttributes\":[{\"key\":\"access_logs.s3.enabled\",\"value\":\"false\"},{\"key\":\"access_logs.s3.bucket\",\"value\":\"\"},{\"key\":\"access_logs.s3.prefix\",\"value\":\"\"},{\"key\":\"load_balancing.cross_zone.enabled\",\"value\":\"true\"}]}}},\"AWS::ElasticLoadBalancingV2::TargetGroup\":{\"istio-internal/test:15021\":{\"spec\":{\"name\":\"k8s-istioint-test-8758dcb619\",\"targetType\":\"ip\",\"port\":15021,\"protocol\":\"TCP\",\"healthCheckConfig\":{\"port\":\"traffic-port\",\"protocol\":\"TCP\",\"intervalSeconds\":10,\"healthyThresholdCount\":3,\"unhealthyThresholdCount\":3},\"targetGroupAttributes\":[{\"key\":\"proxy_protocol_v2.enabled\",\"value\":\"false\"}]}}},\"K8S::ElasticLoadBalancingV2::TargetGroupBinding\":{\"istio-internal/test:15021\":{\"spec\":{\"template\":{\"metadata\":{\"name\":\"k8s-istioint-test-8758dcb619\",\"namespace\":\"istio-internal\",\"creationTimestamp\":null},\"spec\":{\"targetGroupARN\":{\"$ref\":\"#/resources/AWS::ElasticLoadBalancingV2::TargetGroup/istio-internal/test:15021/status/targetGroupARN\"},\"targetType\":\"ip\",\"serviceRef\":{\"name\":\"test\",\"port\":15021},\"networking\":{\"ingress\":[{\"from\":[{\"ipBlock\":{\"cidr\":\"REDACTED\"}},{\"ipBlock\":{\"cidr\":\"REDACTED\"}},{\"ipBlock\":{\"cidr\":\"REDACTED\"}}],\"ports\":[{\"protocol\":\"TCP\",\"port\":15021}]}]}}}}}}}}"}
aws-load-balancer-controller-5899c598f7-rll8b aws-load-balancer-controller {"level":"info","ts":1623389815.428173,"logger":"controllers.service","msg":"creating targetGroup","stackID":"istio-internal/test","resourceID":"istio-internal/test:15021"}

Full log (--log-level=debug): aws-load-balancer-controller-5899c598f7-rll8b.txt

M00nF1sh commented 3 years ago

@ecrousseau Hi, is your cluster runnings in some environment that is fully-private? The controller by default will reach-out to AWSShield API to check whether shieldService is enabled or not. It might be causing the delay if your network environment cannot access the shield API. If so, would you mind try add a flag --enable-shield=false to the controller?

ecrousseau commented 3 years ago

The cluster is in a fully private network, yes - access to certain AWS APIs is enabled via VPC endpoints. I'll give that option a try and report back.

ecrousseau commented 3 years ago

Yep - much faster. (For those who come across this later: I also disabled WAF and WAFv2). Thankyou @M00nF1sh!