caddyserver / caddy

Fast and extensible multi-platform HTTP/1-2-3 web server with automatic HTTPS
https://caddyserver.com
Apache License 2.0
58.11k stars 4.03k forks source link

with per_host in metrics, how about compatible with old version #6604

Closed ta2013 closed 1 week ago

ta2013 commented 4 weeks ago

current caddyfile is using:

{
    servers {
        metrics 
    }
}

if i switch to caddy@master (with per_host support in metrics), then above config will cause panic!! because missing :port (ie servers :80, servers :443 )

the full log:

panic: duplicate metrics collector registration attempted

goroutine 1 [running]:
github.com/prometheus/client_golang/prometheus.(*Registry).MustRegister(0xc00068ddb0, {0xc000812f20?, 0x1d1de87?, 0x4?})
        github.com/prometheus/client_golang@v1.19.1/prometheus/registry.go:405 +0x66
github.com/prometheus/client_golang/prometheus/promauto.Factory.NewGaugeVec({{0x22edd30?, 0xc00068ddb0?}}, {{0x1d1fa3f, 0x5}, {0x1d1de87, 0x4}, {0x1d51a67, 0x12}, {0x1da476e, 0x34}, ...}, ...)
        github.com/prometheus/client_golang@v1.19.1/prometheus/promauto/auto.go:308 +0x163
github.com/caddyserver/caddy/v2/modules/caddyhttp.initHTTPMetrics({{0x22f2478, 0xc00080ee70}, 0xc0004ae360, 0xc0000bed20, {0xc000645150, 0x1, 0x1}, {0x0, 0x0, 0x0}, ...}, ...)
        github.com/caddyserver/caddy/v2@v2.8.4/modules/caddyhttp/metrics.go:46 +0x130
github.com/caddyserver/caddy/v2/modules/caddyhttp.newMetricsInstrumentedHandler.func1()
        github.com/caddyserver/caddy/v2@v2.8.4/modules/caddyhttp/metrics.go:121 +0x38
sync.(*Once).doSlow(0x1?, 0x3?)
        sync/once.go:76 +0xb4
sync.(*Once).Do(...)
        sync/once.go:67
github.com/caddyserver/caddy/v2/modules/caddyhttp.newMetricsInstrumentedHandler({{0x22f2478, 0xc00080ee70}, 0xc0004ae360, 0xc0000bed20, {0xc000645150, 0x1, 0x1}, {0x0, 0x0, 0x0}, ...}, ...)
        github.com/caddyserver/caddy/v2@v2.8.4/modules/caddyhttp/metrics.go:120 +0xb3
github.com/caddyserver/caddy/v2/modules/caddyhttp.wrapMiddleware({{0x22f2478, 0xc00080ee70}, 0xc0004ae360, 0xc0000bed20, {0xc000645150, 0x1, 0x1}, {0x0, 0x0, 0x0}, ...}, ...)
        github.com/caddyserver/caddy/v2@v2.8.4/modules/caddyhttp/routes.go:321 +0x99
github.com/caddyserver/caddy/v2/modules/caddyhttp.(*Route).ProvisionHandlers(0xc000978a88, {{0x22f2478, 0xc00080ee70}, 0xc0004ae360, 0xc0000bed20, {0xc000645150, 0x1, 0x1}, {0x0, 0x0, ...}, ...}, ...)
        github.com/caddyserver/caddy/v2@v2.8.4/modules/caddyhttp/routes.go:167 +0x2e5
github.com/caddyserver/caddy/v2/modules/caddyhttp.RouteList.ProvisionHandlers({0xc000978a88, 0xb, 0xc000975190?}, {{0x22f2478, 0xc00080ee70}, 0xc0004ae360, 0xc0000bed20, {0xc000645150, 0x1, 0x1}, ...}, ...)
        github.com/caddyserver/caddy/v2@v2.8.4/modules/caddyhttp/routes.go:212 +0xa9
github.com/caddyserver/caddy/v2/modules/caddyhttp.(*App).Provision(0xc0006f3680, {{0x22f2478, 0xc00080ee70}, 0xc0004ae360, 0xc0000bed20, {0xc000645150, 0x1, 0x1}, {0x0, 0x0, ...}, ...})
        github.com/caddyserver/caddy/v2@v2.8.4/modules/caddyhttp/app.go:354 +0xe1e
github.com/caddyserver/caddy/v2.Context.LoadModuleByID({{0x22f24b0, 0xc00068de00}, 0xc0004ae360, 0xc0000bed20, {0xc000645150, 0x1, 0x1}, {0x0, 0x0, 0x0}, ...}, ...)
        github.com/caddyserver/caddy/v2@v2.8.4/context.go:389 +0x695
github.com/caddyserver/caddy/v2.Context.App({{0x22f24b0, 0xc00068de00}, 0xc0004ae360, 0xc0000bed20, {0x0, 0x0, 0x0}, {0x0, 0x0, 0x0}, ...}, ...)
        github.com/caddyserver/caddy/v2@v2.8.4/context.go:465 +0xee
github.com/caddyserver/caddy/v2.provisionContext.func3(...)
        github.com/caddyserver/caddy/v2@v2.8.4/caddy.go:545
github.com/caddyserver/caddy/v2.provisionContext(0x1?, 0x0)
        github.com/caddyserver/caddy/v2@v2.8.4/caddy.go:550 +0x53c
github.com/caddyserver/caddy/v2.run(0x1942f40?, 0x0)
        github.com/caddyserver/caddy/v2@v2.8.4/caddy.go:400 +0x78
github.com/caddyserver/caddy/v2.Validate(0xc0000bed20)
        github.com/caddyserver/caddy/v2@v2.8.4/caddy.go:714 +0x5a
github.com/caddyserver/caddy/v2/cmd.cmdValidateConfig({0x0?})
        github.com/caddyserver/caddy/v2@v2.8.4/cmd/commandfuncs.go:551 +0x155
github.com/caddyserver/caddy/v2/cmd.init.1.func8.WrapCommandFuncForCobra.1(0xc000924908, {0x1d1de5b?, 0x4?, 0x1d1ddc7?})
        github.com/caddyserver/caddy/v2@v2.8.4/cmd/cobra.go:141 +0x2f
github.com/spf13/cobra.(*Command).execute(0xc000924908, {0xc0006a3660, 0x2, 0x2})
        github.com/spf13/cobra@v1.8.1/command.go:985 +0xaaa
github.com/spf13/cobra.(*Command).ExecuteC(0xc00098a608)
        github.com/spf13/cobra@v1.8.1/command.go:1117 +0x3ff
github.com/spf13/cobra.(*Command).Execute(...)
        github.com/spf13/cobra@v1.8.1/command.go:1041
github.com/caddyserver/caddy/v2/cmd.Main()
        github.com/caddyserver/caddy/v2@v2.8.4/cmd/main.go:75 +0x1dd
main.main()
        caddy/main.go:26 +0xf
mohammed90 commented 4 weeks ago

because missing :port (ie servers :80, servers :443 )

I don't think that's related. Can you share your full config file? I need to replicate it, because I don't see the same issue on my side.

ta2013 commented 3 weeks ago

I don't think that's related. Can you share your full config file? I need to replicate it, because I don't see the same issue on my side.

Here is the configuration file that can be replicated.

{
        servers {
                metrics
        }

}

*.example.com {
        respond "fallback"
}

http://nossl.example.com {
        respond "nossl"
}
steffenbusch commented 3 weeks ago

[!NOTE]
I have the same panic while trying to test v2.9.0-beta.2 with my existing Caddyfile from v.2.8.4.

panic: duplicate metrics collector registration attempted
goroutine 1 [running]:
github.com/prometheus/client_golang/prometheus.(*Registry).MustRegister(0xc0003d39f0, {0xc0008d36f0?, 0x1b21117?, 0x4?})
        github.com/prometheus/client_golang@v1.19.1/prometheus/registry.go:405 +0x66
github.com/prometheus/client_golang/prometheus/promauto.Factory.NewGaugeVec({{0x1ffd3c0?, 0xc0003d39f0?}}, {{0x1b229f7, 0x5}, {0x1b21117, 0x4}, {0x1b450a6, 0x12}, {0x1b9>
        github.com/prometheus/client_golang@v1.19.1/prometheus/promauto/auto.go:308 +0x163
github.com/caddyserver/caddy/v2/modules/caddyhttp.initHTTPMetrics({{0x2001aa8, 0xc000643bc0}, 0xc0004e0810, 0xc000284840, {0xc00073c1f0, 0x1, 0x1}, {0x0, 0x0, 0x0}, ...}>
        github.com/caddyserver/caddy/v2@v2.9.0-beta.2/modules/caddyhttp/metrics.go:46 +0x130
github.com/caddyserver/caddy/v2/modules/caddyhttp.newMetricsInstrumentedHandler.func1()
        github.com/caddyserver/caddy/v2@v2.9.0-beta.2/modules/caddyhttp/metrics.go:121 +0x38
sync.(*Once).doSlow(0x1?, 0x3?)
        sync/once.go:76 +0xb4
sync.(*Once).Do(...)
        sync/once.go:67
github.com/caddyserver/caddy/v2/modules/caddyhttp.newMetricsInstrumentedHandler({{0x2001aa8, 0xc000643bc0}, 0xc0004e0810, 0xc000284840, {0xc00073c1f0, 0x1, 0x1}, {0x0, 0>
        github.com/caddyserver/caddy/v2@v2.9.0-beta.2/modules/caddyhttp/metrics.go:120 +0xb3
github.com/caddyserver/caddy/v2/modules/caddyhttp.wrapMiddleware({{0x2001aa8, 0xc000643bc0}, 0xc0004e0810, 0xc000284840, {0xc00073c1f0, 0x1, 0x1}, {0x0, 0x0, 0x0}, ...},>
        github.com/caddyserver/caddy/v2@v2.9.0-beta.2/modules/caddyhttp/routes.go:321 +0x99
github.com/caddyserver/caddy/v2/modules/caddyhttp.(*Route).ProvisionHandlers(0xc00062a288, {{0x2001aa8, 0xc000643bc0}, 0xc0004e0810, 0xc000284840, {0xc00073c1f0, 0x1, 0x>
...
github.com/spf13/cobra.(*Command).execute(0xc000980608, {0xc0008fee70, 0x3, 0x3})
        github.com/spf13/cobra@v1.8.0/command.go:983 +0xaaa
github.com/spf13/cobra.(*Command).ExecuteC(0xc000980008)
        github.com/spf13/cobra@v1.8.0/command.go:1115 +0x3ff
github.com/spf13/cobra.(*Command).Execute(...)
        github.com/spf13/cobra@v1.8.0/command.go:1039
github.com/caddyserver/caddy/v2/cmd.Main()
        github.com/caddyserver/caddy/v2@v2.9.0-beta.2/cmd/main.go:75 +0x1dd
main.main()
        caddy/main.go:20 +0xf
mohammed90 commented 3 weeks ago

Okay, I figured the root cause. This is hairy. The shared caddy.Context applies to the entire process, while the metrics key may exist in multiple servers within the same configuration object, so they both try to register themselves and end up registering the same collectors.

I need to control the registration flow within the caddy context. I wonder if it's better to swap the control and have the context register the collectors instead of the registry being given to the HTTP app.

mohammed90 commented 3 weeks ago

@ta2013 @steffenbusch, can you test the CI artifacts of #6606 (here) to validate it works? I've tested it locally, and the panic is gone.

ta2013 commented 3 weeks ago

Appreciate the swift fix; everything's running smoothly now. 👍

steffenbusch commented 3 weeks ago

I have used xcaddy with the branch from #6606 (--with github.com/caddyserver/caddy/v2=github.com/caddyserver/caddy/v2@promote-metrics) and all other plugins I use. The panic is gone 👍

ta2013 commented 3 weeks ago

@mohammed90

I observed that r.Host="example.com" differs from r.Host="exAmple.com". Shouldn't the metrics for these labels be treated as identical?

mohammed90 commented 3 weeks ago

Shouldn't the metrics for these labels be treated as identical?

Yep. I've pushed a commit to normalize them.

ta2013 commented 3 weeks ago

Shouldn't the metrics for these labels be treated as identical?

Yep. I've pushed a commit to normalize them.

Thanks. It works!