Load Balancer and Forwarded For

[nathansamson]

I've implemented a HAproxy LB according to option 2 (https://github.com/deis/router#option-2) I am using workflow v2.0.0

My setup heavily uses private network (the kubernetes cluster should not expose anything on the public IP's -- except for the host SSH).

In front of these I use a custom setup HAProxy Load Balancer, that will proxy to all k8s worker nodes in my system over the private network.

The port 80 forward looks like this

backend www_pool
  balance roundrobin
  mode tcp

  option httpchk GET /healthz
  http-check expect status 200

  option forwardfor

  server eu1-cloud-k8s-worker-01 10.133.**.**:80 check port 9090
  server eu1-cloud-k8s-worker-02 10.133.**.**:80 check port 9090
  server eu1-cloud-k8s-worker-03 10.133.**.**:80 check port 9090

As you can see I am using the private network to forward traffic to the deis-router

The problem I am having now is that I believe the deis routers are actually dropping the X-Forwarded-For headers..

Eg. see at this example app: http://blurry-pachinko.eu1-cloud.beeple.eu/

It will render something like

10.2.83.12

---
10.2.83.1

The first line is rails request.remote_ip, the second line is a printout of request.headers['X-Forwarded-For']

As you can see the only IP's visible are the flanneld/kubernetes network IP's and not even the DigitalOcean private network (which should be 10.133.0.0/16).

Any idea what might be wrong?

[krancour]

You should probably look at this setting if you haven't already.

[nathansamson]

@krancour I actually did already. But this did not help.

I configured it as follow: router.deis.io/nginx.proxyRealIpCidrs: "10.0.0.0/8,188.166.0.178/32,188.166.0.178/32,188.166.0.178/32"

which resulted in the following nginx config

outer@deis-router-xfz13:/$ cat opt/router/conf/nginx.conf | grep real
    set_real_ip_from 10.0.0.0/8;
    set_real_ip_from 188.166.0.178/32;
    set_real_ip_from 188.166.0.246/32;
    set_real_ip_from 188.166.203.127/32;
    real_ip_recursive on;
    real_ip_header X-Forwarded-For;
router@deis-router-xfz13:/$ 

so that seems to be correct.

[krancour]

@nathansamson, I am no HAProxy expert by any stretch, but taking another glance at your configuration, there's something there that doesn't look right to me. It seem you are using mode tcp, which would mean the proxy is forwarding traffic at the TCP/IP layers and not fussing with higher layer protocols like HTTP. I believe that invalidates your use of option forwardfor. i.e. If the proxy isn't speaking HTTP, it cannot set an HTTP header. So the request is arriving at the router without your HAProxy having set X-Forwarded-For.

Never fear... there's an option for dealing with this-- in fact it's a better option.

I'm not sure of the exact HAProxy config that's going to get you there, but what you need to do is enable support for PROXY protocol on both the HAProxy end and the router end (see router.deis.io/nginx.useProxyProtocol. This will result in the end-user's IP being relayed through a TCP header instead of an an HTTP header... and the router can handle that no problem.

[nathansamson]

@krancour

I feel stupid.

My initial thought was that in mode tcp the X-Forwarded-For would indeed not work but actually would not be needed because mode tcp was taking care of that. (Somehow i was believing the haproxy would transparantly forward the tcp connection as if the proxy was not there. Note to self: that is not how tcp works ;) )

I enabled proxy protocol and stuff worked. For reference this is my final haproxy config if anyone needs this (warning: as I was having other issues with V2 in the mean time I decided to go back to a V1 cluster but actually with a proper LB so this config is actually tested on a V1 cluster; but the health checks adapted for V2)

frontend www
  bind    *:80
  default_backend www_pool

frontend wwws
  bind    *:443
  default_backend wwws_pool

frontend git
  bind    *:2222
  default_backend git_pool

backend www_pool
  balance roundrobin
  mode tcp

  option httpchk GET /healthz
  http-check expect status 200

  server eu1-cloud-worker-01 10.133.a.b:80 check port 9090 send-proxy check-send-proxy
  server eu1-cloud-worker-02 10.133.a.b:80 check port 9090 send-proxy check-send-proxy
  server eu1-cloud-worker-03 10.133.a.b:80 check port 9090 send-proxy check-send-proxy

backend wwws_pool
  balance roundrobin
  mode tcp

  option httpchk GET /healthz
  http-check expect status 200

  server eu1-cloud-worker-01 10.133.a.b:443 check port 9090 send-proxy check-send-proxy
  server eu1-cloud-worker-02 10.133.a.b:443 check port 9090 send-proxy check-send-proxy
  server eu1-cloud-worker-03 10.133.a.b:443 check port 9090 send-proxy check-send-proxy

backend git_pool
  balance roundrobin
  mode tcp

  option httpchk GET /healthz
  http-check expect status 200

  server eu1-cloud-worker-01 10.133.a.b:2222 check port 9090 check-send-proxy
  server eu1-cloud-worker-02 10.133.a.b:2222 check port 9090 check-send-proxy
  server eu1-cloud-worker-03 10.133.a.b:2222 check port 9090 check-send-proxy

[krancour]

@nathansamson no need to feel stupid. This is sometimes a dark art.