search

kubernetes / ingress-nginx

Ingress NGINX Controller for Kubernetes
https://kubernetes.github.io/ingress-nginx/
Apache License 2.0
17.34k stars 8.22k forks source link

Configure sane default value for grpc_next_upstream_tries #12090

Open mattb18 opened 3 days ago

mattb18 commented 3 days ago

I'd like ingress-nginx to set a sane default value for grpc_next_upstream_tries. There is currently no value set for this directive, meaning that the default nginx value is used, which in this case is zero. Ideally, this value should also be configurable via an annotation.

I've recently experienced this issue, which resulted in an infinite retry (every 5 seconds due to timeouts) to an upstream that was removed mid request. To make it worse, this turned into an infinite retry with no timeout once the IP was used by another pod which was not listening on the corresponding port, resulting in millions of requests attempts a minute.

As per the ticket, configuring grpc_next_upstream_tries to a sane value (3 in my case) resolves this issue, as the request is tried only 3 times before giving up, rather than retrying infinitely.

A similar change was made for proxy_next_upstream_tries in https://github.com/kubernetes/ingress-nginx/pull/6553, to resolve https://github.com/kubernetes/ingress-nginx/issues/5425.

Would it make sense to also set grpc_next_upstream_tries to 3 or another similar value?

Alternatively, it would be good to set the default to 0, and allow configuring via an annotation (to prevent any potential breaking changes). This way we can avoid needing to configure server snippets annotations.

I'm happy to create a PR if the consensus for this requests is positive.

k8s-ci-robot commented 3 days ago

This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.