Closed ecrousseau closed 3 years ago
@ecrousseau
Hi, is your cluster runnings in some environment that is fully-private?
The controller by default will reach-out to AWSShield API to check whether shieldService is enabled or not. It might be causing the delay if your network environment cannot access the shield API.
If so, would you mind try add a flag --enable-shield=false
to the controller?
The cluster is in a fully private network, yes - access to certain AWS APIs is enabled via VPC endpoints. I'll give that option a try and report back.
Yep - much faster. (For those who come across this later: I also disabled WAF and WAFv2). Thankyou @M00nF1sh!
I am using aws-load-balancer-controller v2.2.0 in an EKS cluster and seeing a delay that I don't understand between when I update a Kubernetes service and when the controller begins to perform actions via the AWS API.
After noticing the controller seemed slow, I reduced the deployment to a single replica, added debug logging, and performed two tests - changing the port of an existing service, and creating a new service. In both cases there was a delay of exactly 6m15s between the log entry for "successfully built model" and starting to make changes in AWS.
What is the controller doing/waiting for when we change a service? Is there anything we can do to make it faster?
In contrast - when a pod is deleted, the controller starts deregistering the target immediately - same for registering newly started pods.
Log entries for changing a port:
Log entries for creating a new service:
Full log (
--log-level=debug
): aws-load-balancer-controller-5899c598f7-rll8b.txt