search

kubernetes / ingress-nginx

Ingress NGINX Controller for Kubernetes
https://kubernetes.github.io/ingress-nginx/
Apache License 2.0
17.6k stars 8.27k forks source link

Too many redirects when using ingress-nginx on DigitalOcean #6846

Closed ianwalter closed 3 years ago

ianwalter commented 3 years ago

NGINX Ingress controller version:

v0.43.0

Kubernetes version (use kubectl version):

Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.3", GitCommit:"1e11e4a2108024935ecfcb2912226cedeafd99df", GitTreeState:"clean", BuildDate:"2020-10-14T12:41:49Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"linux/amd64"}

Environment:

What happened:

  1. I installed nginx-ingress using: https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v0.43.0/deploy/static/provider/do/deploy.yaml
  2. I created a Deployment and Ingress for a backend Next.js app
  3. cert-manager provisioned a cert for the host in the Ingress
  4. I created a DNS record in Cloudflare for the domain pointing to the load balancer IP
  5. I went to the URL and received a "Too many redirects" error from Safari

What you expected to happen:

I expected the request to be served successfully by the backend.

I'm not sure what went wrong but adding use-forwarded-headers: "true" to the nginx-ingress ConfigMap properly serves the request. The request is also served properly if I remove use-proxy-protocol: "true" from the nginx-ingress ConfigMap and manually disable proxy protocol from the load balancer through the DigitalOcean UI.

I'm wondering if proxy protocol should not be enabled by default or if there are other solutions to make this work out-of-the-box.

How to reproduce it:

  1. Create a Kubernetes cluster on DigitalOcean through their UI with a node pool with 2 nodes.
  2. Setup cert-manager by applying: https://github.com/jetstack/cert-manager/releases/download/v1.1.0/cert-manager.yaml
  3. Setup a ClusterIssuer using dns01 verification with Cloudflare
  4. Perform the steps above in the "What happened" section

Anything else we need to know:

During the too many redirects issue, the logs show:

162.xxx.xxx.xxx - - [04/Feb/2021:16:09:28 +0000] "GET / HTTP/1.1" 308 164 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0.3 Safari/605.1.15" 618 0.000 [prod-backend-8802] [] - - - - b0f584cb1ee5211bc80cd86e581fb3b

/kind bug

ianwalter commented 3 years ago

I originally brought up this issue in #6845 where @aledbf replied

@ianwalter when you enable the proxy-protocol you are trusting that as the source of the truth, not the header. This change contradicts the configuration.

This makes me wonder if there is a discrepancy between how DigitalOcean implements the proxy protocol. In this blog post it says:

DigitalOcean Load Balancers implement Proxy Protocol version 1, which simply prepends a human-readable header containing client information to the data sent to your Droplet.

So doesn't this mean that you do need to use use-forwarded-headers: 'true' so that those headers aren't ignored?

aledbf commented 3 years ago

@ianwalter are you using ingress-nginx behind cloudflare?

If that's the case, then you don't need use-proxy-protocol: "true". Enabling this setting makes no sense in this scenario. That said, you need something the next settings in the configuration configmap to use the headers from Cloudflare:

# https://www.cloudflare.com/ips
proxy-real-ip-cidr: "173.245.48.0/20,103.21.244.0/22,103.22.200.0/22,103.31.4.0/22,141.101.64.0/18,108.162.192.0/18,190.93.240.0/20,188.114.96.0/20,197.234.240.0/22,198.41.128.0/17,162.158.0.0/15,104.16.0.0/12,172.64.0.0/13,131.0.72.0/22,2400:cb00::/32,2606:4700::/32,2803:f800::/32,2405:b500::/32,2405:8100::/32,2a06:98c0::/29,2c0f:f248::/32,10.0.0.0/8"
use-forwarded-headers: "true"
forwarded-for-header: "CF-Connecting-IP"
ianwalter commented 3 years ago

@aledbf Yes, that was the problem. Turning off proxying in Cloudflare seems to have fixed the issue. Thank you for your help!