search

kubernetes-sigs / aws-load-balancer-controller

A Kubernetes controller for Elastic Load Balancers
https://kubernetes-sigs.github.io/aws-load-balancer-controller/
Apache License 2.0
3.93k stars 1.46k forks source link

Controller doesn't process request if stuck elsewhere (OIDC annotation missing secret) #2669

Closed theintz closed 1 year ago

theintz commented 2 years ago

Describe the bug When an ingress change request can't be fulfilled by the controller, it logs the error and then stops processing all other requests, even those that could be fulfilled.

Steps to reproduce Create two deployments with ingresses, one that will deploy without errors, another one that doesn't (for example by requiring external resources such as a secret that aren't available). If the faulty one is requested first, the controller will fail to deploy it and log errors repeatedly. When the second request is issued, it will not be processed, even though it is not connected to the faulty request and could be fulfilled.

More concretely, we are using the alb.ingress.kubernetes.io/auth-idp-oidc annotation, to set up the LB to do auth over OIDC. It requires passing a parameter called secretName containing the name of the k8s secret that holds the clientSecret as required by the Oauth2 standard. If this secret does not exist, the controller exhibits the mentioned behavior.

Expected outcome Valid ingress change requests are processed, even if the controller is stuck on a faulty request elsewhere.

Environment

kishorj commented 2 years ago

@theintz, in case of errors, controller will not reconcile the ingress or the group further. If you use ingress group feature, fatal errors encountered during processing of one ingress affects the entire group. Other ingresses not part of the group should still continue to work as expected.

Controller tolerates certain errors, for example if backend service doesn't exist, it configures a 404 response but doesn't block the reconciliation.

nabadger commented 2 years ago

Is there a more general solution/roadmap in this area to allow valid ingresses within the same group to reconcile if 1 or more are in an error state?

Seems somewhat similar to https://github.com/kubernetes-sigs/aws-load-balancer-controller/issues/2042

k8s-triage-robot commented 1 year ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

theintz commented 1 year ago

/remove-lifecycle stale

theintz commented 1 year ago

@kishorj Thanks for the explanation and sorry for the late reply. I don't understand the motivation behind this behavior. Even if a number of ingresses share the same group, why would the reconciliation be blocked if one of them fails? Shouldn't the controller try to reconcile as many ingresses as possible?

k8s-triage-robot commented 1 year ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot commented 1 year ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

k8s-triage-robot commented 1 year ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

k8s-ci-robot commented 1 year ago

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to [this](https://github.com/kubernetes-sigs/aws-load-balancer-controller/issues/2669#issuecomment-1576862449): >The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. > >This bot triages issues according to the following rules: >- After 90d of inactivity, `lifecycle/stale` is applied >- After 30d of inactivity since `lifecycle/stale` was applied, `lifecycle/rotten` is applied >- After 30d of inactivity since `lifecycle/rotten` was applied, the issue is closed > >You can: >- Reopen this issue with `/reopen` >- Mark this issue as fresh with `/remove-lifecycle rotten` >- Offer to help out with [Issue Triage][1] > >Please send feedback to sig-contributor-experience at [kubernetes/community](https://github.com/kubernetes/community). > >/close not-planned > >[1]: https://www.kubernetes.dev/docs/guide/issue-triage/ Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
ivanfoo commented 3 weeks ago

This is still problematic as it can disrupt a whole environment's service. Are you planning to make any changes related to this? If not, what's the reasoning behind it? I don't see why the controller cannot keep reconciling the valid Ingresses while ignoring the invalid ones.

Thanks!

/reopen

k8s-ci-robot commented 3 weeks ago

@ivanfoo: You can't reopen an issue/PR unless you authored it or you are a collaborator.

In response to [this](https://github.com/kubernetes-sigs/aws-load-balancer-controller/issues/2669#issuecomment-2383066422): >This is still problematic as it can disrupt a whole environment's service. Are you planning to make any changes related to this? If not, what's the reasoning behind it? I don't see why the controller cannot keep reconciling the valid Ingresses while ignoring the invalid ones. > >Thanks! > >/reopen Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.