client IP is incorrect when using CloudFront -> ELB -> Nginx and proxy protocol

noqcks commented 4 years ago

NGINX Ingress controller version: 0.30.0

Kubernetes version (use kubectl version): 1.15.9

Environment: Kops on AWS

Cloud provider or hardware configuration: AWS
OS (e.g. from /etc/os-release): Debian 9
Kernel (e.g. uname -a): Linux ip-172-20-1-51 4.9.0-11-amd64 #1 SMP Debian 4.9.189-3+deb9u1 (2019-09-20) x86_64 GNU/Linux
Install tools: Helm
Others: N/A

What happened:

We are attempting to setup a our ingress so that we have traffic like so

CloudFront -> ELB -> Nginx Ingress

Previously it was ELB -> Nginx Ingress and everything worked. However, once we switched to add CloudFront our IP whitelisting no longer works. The reason for this I believe is that the real_ip is being set incorrectly.

This is what our logs look like


Mar 10 12:45:23 v2-nginx-ingress-controller-7884c7cd8b-hsngz nginx-ingress-controller { 
"@timestamp": "2020-03-10T16:45:23+00:00", "remote_addr": "70.132.33.156", "x-forward-for": 
"18.232.45.xxx", "request_id": "20db865d9855b088de361d652cec83f7", "remote_user": "-", 
"bytes_sent": 428, "request_time": 0.000, "status": 403, "vhost": "REDACTED", "request_proto": 
"HTTP/1.1", "path": "/api", "request_query": "-", "request_length": 1019, "duration": 0.000, "method": "GET", 
"http_referrer": "-", "http_user_agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_3) 
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.132 Safari/537.36" }

My Client/Computer IP = 18.232.45.xxx

The IP that's listed under remote_addr (70.132.33.156) is using $proxy_protocol_addr and is from the CloudFront IP address range. This is also used by real_ip module here

configmap (the proxy-real-ip-cidr list contains all the CloudFront IP addresses as well as our VPC CIDR (172.20.0.0/16))

  compute-full-forwarded-for: "true"
  enable-vts-status: "true"
  limit-req-status-code: "429"
  log-format-upstream: '{ "@timestamp": "$time_iso8601", "remote_addr": "$proxy_protocol_addr",
    "x-forward-for": "$http_x_forwarded_for", "request_id": "$request_id", "remote_user":
    "$remote_user", "bytes_sent": $bytes_sent, "request_time": $request_time, "status":
    $status, "vhost": "$host", "request_proto": "$server_protocol", "path": "$uri",
    "request_query": "$args", "request_length": $request_length, "duration": $request_time,
    "method": "$request_method", "http_referrer": "$http_referer", "http_user_agent":
    "$http_user_agent" }'
  proxy-body-size: 64m
  proxy-next-upstream: error timeout invalid_header http_502 http_503 http_504
  proxy-real-ip-cidr: 172.20.0.0/16,144.220.0.0/16,52.124.128.0/17,54.230.0.0/16,54.239.128.0/18,52.82.128.0/19,99.84.0.0/16,204.246.172.0/24,54.239.192.0/19,70.132.0.0/18,13.32.0.0/15,205.251.208.0/20,13.224.0.0/14,13.35.0.0/16,204.246.164.0/22,204.246.168.0/22,71.152.0.0/17,216.137.32.0/19,205.251.249.0/24,99.86.0.0/16,52.46.0.0/18,52.84.0.0/15,204.246.173.0/24,130.176.0.0/16,205.251.200.0/21,204.246.174.0/23,64.252.128.0/18,205.251.254.0/24,143.204.0.0/16,205.251.252.0/23,204.246.176.0/20,13.249.0.0/16,54.240.128.0/18,205.251.250.0/23,52.222.128.0/17,54.182.0.0/16,54.192.0.0/16,13.124.199.0/24,34.226.14.0/24,52.15.127.128/26,35.158.136.0/24,52.57.254.0/24,18.216.170.128/25,13.52.204.0/23,13.54.63.128/26,13.59.250.0/26,13.210.67.128/26,35.167.191.128/26,52.47.139.0/24,52.199.127.192/26,52.212.248.0/26,52.66.194.128/26,13.113.203.0/24,99.79.168.0/23,34.195.252.0/24,35.162.63.192/26,34.223.12.224/27,52.56.127.0/25,34.223.80.192/26,13.228.69.0/24,34.216.51.0/25,3.231.2.0/25,54.233.255.128/26,18.200.212.0/23,64.252.64.0/18,52.52.191.128/26,3.234.232.224/27,52.78.247.128/26,52.220.191.0/26,34.232.163.208/29
  server-tokens: "False"
  use-forwarded-headers: "true"
  use-proxy-protocol: "true"

And a simple ingress i'm using to test whitelisting

---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: test
  namespace: default
  annotations:
    kubernetes.io/ingress.class: "nginx-ingress-v2"
    nginx.ingress.kubernetes.io/rewrite-target: /$1
    nginx.ingress.kubernetes.io/whitelist-source-range: 18.232.45.xxx
spec:
  tls:
    - secretName: tls-secret
      hosts:
        - "test.<redacted>"
  rules:
    - host: "test.<redacted>"
      http:
        paths:
          - path: /(.*)
            backend:
              serviceName: api
              servicePort: 80

What you expected to happen:

I expected whitelisting to work correctly based on the correct IP (18.232.45.xxx), but it seems to checking whitelist against the CloudFront IP (70.132.33.156) instead. If I update my upstream whitelist to allow 70.132.33.156 I no longer get a 403 error.

How to reproduce it:

Setup a CloudFront Origin pointing to an ELB. See screenshots below for exact settings.
Setup a new ingress that has IP whitelisting for your IP address.

Supplementary Links:

Similar Issues:

/kind bug

noqcks commented 4 years ago

/assign @aledbf

aledbf commented 4 years ago

use-forwarded-headers: "true" use-proxy-protocol: "true"

These two annotations don't work together. If you enable the two, only the use-proxy-protocol will be used.

Please add the annotation

nginx.ingress.kubernetes.io/server-snippet: |
  real_ip_header X-Forwarded-For;

aledbf commented 4 years ago

@noqcks it works?

noqcks commented 4 years ago

Yes! It worked! Thank you!

Tried on 0.21.0 (an older version we're running). If you had any idea what might make it work also on 0.21.0 that would be very helpful too.

aledbf commented 4 years ago

If you had any idea what might make it work also on 0.21.0 that would be very helpful too.

Unfortunately, in versions older than 0.26.0 is not that easy. You should try to update to a more recent version

noqcks commented 4 years ago

Yep, I figured that was the case. Thank you.

kubernetes / ingress-nginx

client IP is incorrect when using CloudFront -> ELB -> Nginx and proxy protocol #5233