forwarding client_ip with proxy and logs config

[doums]

Hi, not a bug but rather I have a few questions. First off, this is my caddy config:

{
    email m@m.com
    dynamic_dns […]
    crowdsec […]
    order crowdsec first

    servers {
        metrics
    }

    # default logger
    log default {
        output stdout
        format console
    }

    # write http access logs to 2 different outputs
    # see https://caddyserver.com/docs/caddyfile/directives/log#examples

    # HTTP logger - console
    log access-console {
        include http.log.access.http-logger
        output stdout
        format console
    }

    # HTTP logger - file JSON
    log access-json {
        include http.log.access.http-logger
        output file /log/access.log
        format json
    }

    # l4 logger
    log l4 {
        include layer4.l4
        output stdout
        format console
    }

    layer4 {
                 log l4 # tried this

        0.0.0.0:8054 {
                         log l4 # tried this

            @foo tls sni sub.mydomain.com
            route @foo {
                tls
                proxy service:1234
            }
        }
    }
}

*.mydomain.com {
    tls  […]

    log http-logger

    […] some reverse proxy blocks
}

1. How to configure access logs correctly for layer4? Especially to logs the requests going through the tls matcher (:8054)? Right now I don't get any logs but in the other side of the proxy (service:1234), I can see the requests received as expected.
2. Is there a way to forward the initial client_ip to the proxied service in layer4?

[francislavoie]

There's no such thing as access logs for l4 right now.

Is there a way to forward the initial client_ip to the proxied service in layer4?

You can use PROXY procotol to do that.

[mholt]

Well, this module does log connections (it shows you how many bytes are read/written), but they're not like HTTP access logs where you can see the decoded request. We can improve logging for sure though.

[doums]

The issue is I don't even see those connections log you mentioned. Are they emitted at a lower level of info? Also, I'm not sure where to put the log directive inside the layer4 app?

[mholt]

Oh, they might be at DEBUG level come to think of it. You don't need to enable them (because they're at a more verbose level.)

We can rework this I suppose, I just whipped up the logging quickly.

[doums]

Ok cool. Yea the 'logic' would be to have some trace logged (at INFO level so) whenever some bytes hit the tls handler. So that we're able to see the activities at least.

[doums]

You can use PROXY procotol to do that.

@francislavoie thx for the suggestion. But I'm still failing to get this working. Also note that the target service is a blocky instance. And the purpose is to tap into its dns endpoint, raw no tls, since tls is terminated at l4 proxy level.

I tried to change the handler to this but without success

                         route @foo {
                tls
                proxy {
                    proxy_protocol v1 # also tried v2
                    upstream blocky:53
                }
            }

[francislavoie]

Your upstream would need to be ready to handle PROXY protocol as well. Basically it prepends some bytes to the start of the connection which includes the original client IP. Looks like it doesn't support it yet https://github.com/0xERR0R/blocky/issues/320

[alexander-potemkin]

Oh, they might be at DEBUG level come to think of it. You don't need to enable them (because they're at a more verbose level.)

We can rework this I suppose, I just whipped up the logging quickly.

Thank you! I'm trying to get at least some logs as well - to troubleshoot my config. Could you please, advice how to enable logging / debugging?

My current Caddyfile looks like that:

{
    layer4 {
        0.0.0.0:443 {
            @s5 socks5
            route @s5 {
                proxy IP:443
            }
            @secure tls
            route @secure {
                proxy some_site:443
            }
        }
    }
}

I've tried all kinds of log directive, and searched for an answer in the source code, but it didn't work...

[mholt]

Put the word debug in your global options: https://caddyserver.com/docs/caddyfile/options#debug