kubernetes-sigs / aws-load-balancer-controller

A Kubernetes controller for Elastic Load Balancers
https://kubernetes-sigs.github.io/aws-load-balancer-controller/
Apache License 2.0
3.93k stars 1.46k forks source link

Issue with WAFv2 webACL association #2022

Closed Leankit-Stuart-Will closed 3 years ago

Leankit-Stuart-Will commented 3 years ago

After the controller successfully builds the model, it tries to associate the ALB with the WAFv2 webACL but fails due to WAF not being able to find the ALB.

I'm doing this by setting the annotation: alb.ingress.kubernetes.io/wafv2-acl-arn: arn:aws:wafv2:us-west-2:*******:regional/webacl/*****-waf/******

Errors from the controller logs are: {"level":"info","ts":1621449454.0985126,"logger":"controllers.ingress","msg":"associating WAFv2 webACL","resourceARN":"arn:aws:elasticloadbalancing:us-west-2:******:loadbalancer/app/*****/*****","webACLARN":"arn:aws:wafv2:us-west-2:*******:regional/webacl/**********"}

{"level":"error","ts":1621449454.133283,"logger":"controller","msg":"Reconciler error","controller":"ingress","name":"******","namespace":"*****","error":"failed to create WAFv2 webACL association on LoadBalancer: WAFNonexistentItemException: AWS WAF couldn’t perform the operation because your resource doesn’t exist."}

Presumable this happens because the controller tries to associate the webACL with the ALB before the ALB has finished provisioning, and it gives up on retries before that happens.

I've created the WAFv2 webACL beforehand, and I'm able to associate the two manually after waiting a minute or two for the provisioning to finish.

M00nF1sh commented 3 years ago

@Leankit-Stuart-Will we haven't observed such case in the past. But the controller have automatically retry, didn't the retry recover from this error automatically?

Leankit-Stuart-Will commented 3 years ago

@M00nF1sh Terribly sorry, ended up being a mistake on my part.

We're using flux to manage updates, and I forgot to commit my local changes of the webACL ARN annotation to the main branch. Once I did that the controller was able to rebuild the ALB and successfully associate it with the webACL.