Throttling: Rate exceeded error when reconcile ALB

[lmbaschiera]

Describe the bug We're currently upgrading our alb ingress controller to v2.4.4 from v1.2.0-alpha.1 in preparation for a kubernetes upgrade. Hundreds of DescribeTargetHealth, DescribeTargetGroups, DescribeTags and RegisterTargets events are making us hit the API rate limits impacting all clusters - even the ones not upgraded to the new controller version.

How can I fix this issue? I've asked for an API rate limit increase but that's just a band-aid for this issue.

{"level":"error","ts":1682003556.0227876,"logger":"controller-runtime.manager.controller.ingress","msg":"Reconciler error","name":"release-8611","namespace":"","error":"Throttling: Rate exceeded\n\tstatus code: 400, request id: a450ce32-cf10-48a4-9736-aa5de377aa93"}

We're having issue creating and destroying ingresses because we hit the limits even when destroying the load balancers, leaving a lot of Terminating resources in our clusters.

Environment

• AWS Load Balancer controller version v2.4.4
• Kubernetes version v1.21.14
• Using EKS (yes/no), if so version? yes

Additional Context: We use these clusters to provision whole sets of applications in isolated environments for testing purposes, so we create and destroy dozens of environments per day, and each environment has between 20 and 30 ingress resources.

[lmbaschiera]

This seems to be the same issue as #1871

[oliviassss]

@lmbaschiera The fix has been shipped with the v2.5.2 release, I'm closing this issue as of now. Please feel free to reach out if you're still having this issue.