Closed chancez closed 3 years ago
Also, in these circumstances, failures are not surfaced to events (this is the only event, nothing about listeners/rules/missing service backends):
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal CREATE 5m37s alb-ingress-controller LoadBalancer 665168b2-exampledev-examplegateway-1f77 created, ARN: arn:aws:elasticloadbalancing:us-west-2:117923182973:loadbalancer/app/665168b2-exampledev-examplegateway-1f77/b2c7649aa487c76a
Seems to be the same as https://github.com/kubernetes-sigs/aws-alb-ingress-controller/issues/1304
Hi, I think we should add events to the Ingress object about such errors. However, I don't think we should ignore rules when the service is missing. The rules can have dependencies, and the Ingress should work as a whole. e.g.
/auth -> service-missing
/* -> service-exists
If we ignore the first "/auth" rule, it changes the application behavior completely. /auth
will be routed to service-exists
@M00nF1sh You don't have to do it like that though. I'm not saying to ignore the rule, but to ignore the missing service. You can still define the route rule in the ALB, you just have to point it at something like a default backend or even just point it at a non-existent nodePort, or something.
We encountered a similar issue today that had the same end result but for a different reason.
We had a bad certificate configuration:
alb.ingress.kubernetes.io/certificate-arn: ${ssl_certificate_alb},${secondary_certificate}
It could not reconcile properly since the secondary_certificate was just a bad variable and was not deployed in the region where this was running. Once you encounter this error it stops and does not apply any of the rules.
I0807 02:08:35.666873 1 listener.go:185] example-namespace/example-app: adding certificate arn:aws:acm:us-east-1:blabla:certificate/guid to listener arn:aws:elasticloadbalancing:ap-southeast-2:blabla:listener/app/myawesomeapp
E0807 02:08:35.749797 1 controller.go:217] kubebuilder/controller "msg"="Reconciler error" "error"="failed to reconcile listeners due to failed to reconcile extra certificates on listener arn:aws:elasticloadbalancing:ap-southeast-2:blabla:listener/app/myawesomeapp: CertificateNotFound: Certificate 'arn:aws:acm:us-east-1:blabla:certificate/guid' not found\n\tstatus code: 400, request id: guid" "controller"="alb-ingress-controller" "request"={"Namespace":"example-namespace","Name":"example-app"}
You could have potato.awesomecorp.com -> /potato
and banana.awesomecorp.com -> /banana
.
If one of the certificate is invalid you are essentially breaking everything even if both are most likely unrelated.
Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale
.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close
.
Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale
.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close
.
Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle stale
Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten
.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close
.
Send feedback to sig-contributor-experience at kubernetes/community. /lifecycle rotten
Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen
.
Mark the issue as fresh with /remove-lifecycle rotten
.
Send feedback to sig-contributor-experience at kubernetes/community. /close
@fejta-bot: Closing this issue.
/reopen
@EdwinPhilip: You can't reopen an issue/PR unless you authored it or you are a collaborator.
look like the problem still exist. Should I assume it is expected behavior?
If you create an ingress resource with a mix of existing, and non-existing services, then it will stop the reconciliation after the first failure to lookup the non-existent service, causing the ingress to be partially configured.
This means if you have an ingress with a dozen or so rules, and the first rule references a service that doesn't exist, then you will have a loadbalancer with no listeners and no rules.
Ideally, it would skip over the missing service, and create a route to the default backend instead, and continue configuring the rest of the listeners and rules for the loadbalancer.
Currently as it is a misconfiguration or a backend service being deleted (for any reason) could lead to an ingress ignoring all updates to the rules, which could lead to unexpected results. When using a single ingress with multiple services, this is very undesirable.
For example, the following is what I'm seeing for one of my ingresses: