search

mastercactapus / caddy2-proxyprotocol

Apache License 2.0
59 stars 14 forks source link

Caddy behind AWS Network Load Balancer with proxyprotocol enabled leads to tls errors #8

Open kellertobias opened 2 years ago

kellertobias commented 2 years ago

We are trying to pass the client IP through our load balancing stack. The stack is NLB => Caddy (on Docker/Fargate) => ALB.

My Understanding based on the documentation of your caddy plugin, caddies documentations and AWS's documentation of the Network Load Balancer (NLB) (https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-target-groups.html#proxy-protocol), I need to enable proxy protocol v2 in the NLB and install your plugin in caddy as well as configure it. I've written about my configuration in the Caddy Forum: https://caddy.community/t/caddy-behind-nlb/13988

when I enable all the stuff mentioned above, I get the following lines in caddy if I set the log level to debug:

`{"level":"debug","ts":1634749313.4648879,"logger":"http.stdlib","msg":"http: TLS handshake error from 10.20.201.113:12064: invalid length"}``

where the 10.20.201.113 is the private IP of our NLB.

What can I do to further debug this issue? Might there be a problem in your plugin when working with the NLB?

ermiaqasemi commented 2 years ago

Having the same issue!

ermiaqasemi commented 2 years ago

@kellertobias I read your details in Caddy Forum, I am using the same design as yours but the difference is that we deployed Caddy as an ingress in Kubernetes and I just succeed to get the client IP by enabling preserve Client IP address at NLB level, then in order to get the Client IP you can get it from {http.request.remote.host} placeholder and the pass this as X-forward-for header to ALB and then reset will be handle by AWS. hope this helps you!

toreym commented 2 years ago

@kellertobias, curious if you've found a resolution to this or if you changed your setup to work around it. Thanks!

kellertobias commented 2 years ago

@toreym not yet. we kind of just ignored the problem for now

toreym commented 2 years ago

Will update you if I find a solution.

ChrisLahaye commented 2 years ago

Did anyone manage to get this working?

toreym commented 2 years ago

No, sorry @ChrisLahaye. We're not that familiar with this protocol so it's been difficult to debug.

toreym commented 2 years ago

@ChrisLahaye @kellertobias We were able to create a listener that works. As long as you are using JSON to configure your server this will work. I assume you could point xcaddy build to our module (I'm not a go person so don't know how their package management works).

xcaddy build --with github.com/rocketreferrals/caddy-go-proxyproto

And the json config would look like this:

"listener_wrappers": [{"wrapper": "go_proxyproto", "timeout": "5s"},{"wrapper": "tls"}]

At the end of the day this is just a simple wrapper for https://github.com/pires/go-proxyproto which is a different lib, but it seems to work better with AWS ELBs.

Hope this helps!

ChrisLahaye commented 2 years ago

@toreym We applied your module behind an NLB with proxy protocol enabled, and didn't encounter any request errors. Unfortunately we found out that IP preservation does not work when using Global Accelerator with NLB, and therefore, have no need for proxy protocol. Either way, I would like to thank you for your efforts as your module resolved the errors we were originally encountering. You should try to get your module listed in the official documentation as I am sure it will be helpful to others. I really like the simplicity of it.

Thank you.

kellertobias commented 2 years ago

@toreym sounds good, will give it a try; we are building caddy ourselves too at the moment, so adding another module should be no problem and the configuration is generated at startup of the container also as a json file.

Do you know if IP Preservation works if we are not using Global Accelerator in the NLB - as this is the main reason for us as well to use proxy protocol.

ChrisLahaye commented 2 years ago

@toreym sounds good, will give it a try; we are building caddy ourselves too at the moment, so adding another module should be no problem and the configuration is generated at startup of the container also as a json file.

Do you know if IP Preservation works if we are not using Global Accelerator in the NLB - as this is the main reason for us as well to use proxy protocol.

I am not completely certain but when preservation is enabled and registering the targets by instance ID, then targets receive the client's IP address, else, i.e. if targets are registered by IP address, targets receive the NLB's IP address and you can get the client's IP address from the proxy protocol.

toreym commented 2 years ago

@kellertobias We're not using Global Accelerator and do not have IP Preservation turned on. So, that's why we need to get the IP from the Proxy Protocol. We've had this setup running in production for a couple of weeks now and have not had any issues. I don't imagine you'd have any problem running the module with IP Preservation on though.