Docker flow proxy behind AWS ELB

[akilnagati]

Description

We are using Docker for AWS that comes with the AWS Classic ELB.

To forward the traffic to services based on the domain, and to manage certs, we are using Docker flow proxy.

Issue:

The scenarios: USER (HTTPS: 443) ==> Classic ELB (TCP: 80) => DFP (Frontend HTTPS Mode / Backend HTTP Mode) => Service (HTTP: 80)

Th issue: I'm not able to get the USER's IP.

Tests

To fix this:

• I've published DFP using the Host Mode, so docker doesn't change the source IP.
• Configured Appache to get the IP from X-Forwarded-For added by DFP.

By these two steps I was able to get the correct user's IP, but only when I connect directly to the node.

When I use the ELB, I'm getting the ELB IP. This is an expected behavior as the ELB is changing the source IP.

Solution

By looking on ELB & HAProxy documentation, I've found that ELB is implimenting Proxy Protocol that add the client IP to the TCP header.

I've activated it using these steps: https://docs.aws.amazon.com/elasticloadbalancing/latest/classic/enable-proxy-protocol.html

The only missing thing is enabling "accept-proxy" on HAProxy so it can catch the TCP header and set the correct IP on X-Forwarded-For header.

This can be done by adding accept-proxy to the HAproxy configuration.

http://cbonte.github.io/haproxy-dconv/1.9/configuration.html#5.1-accept-proxy

Could you add this feature please ?

By checking the DFP code, I think the missing tag should be added to: haproxy.cfg

frontend dummy-fe
    bind *:80 accept-proxy
    bind *:443 accept-proxy

The only issue is by doing it this way, It will be static and I will not be able to disable it without building the Docker Image again.

[vfarcic]

I don't think that'll work. The problem is not in HAProxy but in Overlay network used by Swarm. The moment a request enters the cluster (before it reaches DFP) it looses the IP.

[akilnagati]

Yes, I've got this issue, but when I deploy the DFP using global mode and published the port using HOST mode, docker keeps the source IP.

When I connect directly to the node, without going through ELB, My service is getting the right IP, thanks to X-Forwarded-For.

The only issue is when I use the AWS ELB. I think this can be solved using Proxy protocol.

[vfarcic]

That's true. When you publish the port as HOST mode, you keep the IP. However, that's because that mode does not use Overlay which is the requirement for DFP and, in general, for inter-node communication.

It would be interesting if you could make DFP communicate with all other services without Overlay. The last time I tried, I failed. But, maybe the tech changed in the meantime. Would you like to try it out and send a PR?

[akilnagati]

DFP is communicating with the external network using HOST mode, so the DFP/HAProxy can see the right IP.

Then, it forwards the request to the service using an overlay network, while passing the IP using the X-Forwarded-For header.

I tested this and it works fine:

Configuration: DFP deployed globally with HOST mode and listening on PORT 80. DFP is connected to a Apache service using an overlay network. Apache service just print the user's IP from X-Forwarded-For header.

When I connect to the swarm using a node public IP, I'm getting my correct IP in X-Forwarded-For header. When I connect to the swarm using ELB, I'm getting ELB IP in X-Forwarded-For header.

[akilnagati]

I've found a solution without the need to edit the code:

1/ Deployed the default DF Stack (Without Host mode) 2/ Enabled Proxy Protocol on the AWS ELB: https://docs.aws.amazon.com/elasticloadbalancing/latest/classic/enable-proxy-protocol.html 3/ Used DEFAULT_PORTS="80 accept-proxy,443 accept-proxy:ssl" to enable the Proxy Protocol on the HAProxy frontend 4/ Configured my service (Apache) to set the IP from X-Forwarded-For

Now I'm able to get the correct source IP.