Question: Reverse tunneling to caddy behind a NAT

[crabdancing]

My setup is as follows:

1. a VPS (ingress point)
2. A tunnel to move incoming connections from my VPS to my (NAT) business network
3. Caddy (as a unified TLS frontend & reverse proxy)
4. local services on my business network

One of the constraints here is that I can't have the TLS encryption happen on the VPS, as that would in principle allow the VPS to spy on unencrypted traffic.

The problem is that every tunnel technology I've tried resets all IP address information about the connection. This is unacceptable, as the backend service can't block bruteforcing attempts or otherwise punish misbehavior based on IP address.

How do I set up a backend on the server such that clients can connect to the VPS as if the services were hosted there, and have them actually connect to a service on my business network, while still having a useful X-Forwarded-For / X-Real-IP that the services can use to identify the incoming connection? Can I use caddy-l4 on both VPS & backend?

[mohammed90]

You need to use the PROXY Protocol. Both, the HTTP server in standard Caddy and this module support the PROXY protocol. If you just need HTTP, then use the proxy_protocol listener_wrapper in the HTTP app. Otherwise, you can enable the use of the proxy_protocol option in the proxy handler of the layer4 app.

[crabdancing]

I think what I want to do is:

Is this correct? Sorry, I'm kind of new to Caddy.

I've been trying to figure out the listener_wrapper thing for awhile, but I can't really find any good examples. I think I've figured out how to get Caddy to take an 'upstream' connection via PROXY, but not a 'downstream' one.

[mohammed90]

You're very close. You don't need 2 Caddy instances on the trusted server. The Caddy with layer4 will listen on port 443 and only proxies to the other Caddy instance using the proxy_protocol. The backend Caddy instance needs to have proxy_protocol in its listener_wrappers part of the config. The listener_wrappers manipulate the connection data before being processed by the HTTP app of Caddy. In this case, using the proxy_protocol listener wrapper removes the bits added by the Proxy Protocol to the HTTPS request, turning them back into regular HTTPS request.

Your config may end up being something like this --

• caddy-l4 config:

{
    layer4 {
        0.0.0.0:443 {
            route {
                proxy {
                    proxy_protocol v2
                    upstream 192.168.1.200:443
                }
            }
        }
    }
}

• Backend Caddy:

{
    servers {
        listener_wrappers {
            proxy_protocol {
                allow 192.168.86.1/24 # replace this with the IP address of the server of caddy-layer4
                fallback_policy
            }
            tls
        }
    }
}
example.com {
    reverse_proxy backend-1:80 backend-2:80
}

[crabdancing]

@mohammed90 Oh, thanks so much. :) I really appreciate the help!

I'm starting to understand better, but the one thing that confuses me is the direction of the connection. In this setup, does it require port 443? Am I correct in inferring that 192.168.1.200 is the IP of the backend in this example?

If that's the case, does that mean that the caddy-l4 VPS server opens the connection to the backend, and not vice versa? If it is behind a NAT, it may or may not be feasible to port forward, so I was hoping to have the connection come from behind the NAT.

[mohammed90]

In this setup, does it require port 443?

The layer4 instance, yes, because this is what the public ACME servers require to validate the certificate requests. For the inner Caddy, you can tell Caddy to expect HTTPS on a different port by using the https_port global option.

Am I correct in inferring that 192.168.1.200 is the IP of the backend in this example?

Yes.

If that's the case, does that mean that the caddy-l4 VPS server opens the connection to the backend, and not vice versa?

Correct

If it is behind a NAT, it may or may not be feasible to port forward, so I was hoping to have the connection come from behind the NAT.

This is not something Caddy can help you with. You can run an SSH reverse-tunnel or VPN between the nodes.

[crabdancing]

Oof. I was using SSH, but I think FRP might be more performant, so I'll try setting that up. Thank you for all your help. :)

[crabdancing]

@mohammed90

Edit: I've got the HTTPS connection to go from VPS to remote server over caddy+L4 -> FRPS -> FRPC -> Caddy (backend server).

This was achieved with a config on the backend server, of the form:

{
    servers {
        listener_wrappers {
            proxy_protocol {
                allow 127.0.0.1/24
            }
            tls
        }
    }
}

example.com, example.com:4443 {
  ...
}

The main confusion I had previously is that I didn't understand that it was expecting explicit protocol varients for all the domains. Francis on the Discord was very helpful in explaining that. ^w^

The main problem now is that somehow, it still doesn't correctly set the true IP in the headers -- backend services like Authelia erroneously believe the origin IP is the local machine.

journalctl -xef -u caddy | rg '(X-Real-IP|X-Forwarded-For)' with logging set to DEBUG shows that the backend-side Caddy instance is in fact reporting X-Forwarded-For as the local IP of the backend server (10.0.1.1). However, local intranet connections (via split DNS) correctly report the client's local IP. It appears that X-Forwarded-For is in lockstep with client_ip in the logs.

I thought maybe it was because the proxy is not trusted, but I've tried doing things like:

servers :4443 {
    trusted_proxies static 127.0.0.1/24 10.0.1.1/24
    client_ip_headers X-Forwarded-For X-Real-IP
    listener_wrappers {
        proxy_protocol {
            allow 127.0.0.1/24
        }
        tls
    }
}

Which don't seem to change the outcome.

[mohammed90]

This isn't right

          allow 127.0.0.1/24

Why are you telling the proxy_protocol to only trust the localhost?! Does frp appear like localhost? I doubt that very much. With that, you need to use the fallback_policy with reject to avoid accepting direct connections.

Also, you might want to check Authelia's docs for trusting X-Forwarded-* headers: https://www.authelia.com/integration/proxies/forwarded-headers/

[crabdancing]

Oh, fair enough. That was silly of me. I think I was confused because you have to specify something ilke 'allowed/trusted proxies' in multiple places, so it was easier to overlook that line when sleepy. I imagine this is because of the modulerized structure of caddy's internals -- caddy-l4 has its own source IP permissions separate from the whole of the application.

Also, you might want to check Authelia's docs for trusting X-Forwarded-* headers: https://www.authelia.com/integration/proxies/forwarded-headers/

No worries, I already understand the security implications of proxy headers, so I'm good on that front. :) Thank you for the warning about fallback_policy. I will be sure to add that back.

Thank you for your help. It works as it should now.