Trendyol / go-dcp-kafka

The Go implementation of the Couchbase to Kafka with DCP.
MIT License
68 stars 15 forks source link

Connector cannot restart after closed #62

Closed EmreKumas closed 1 year ago

EmreKumas commented 1 year ago

Describe the bug We want to implement a dynamic configuration system in which after updating the config, the connector needs to be closed and started again. But we're getting "duplicate metrics collector registration attempted" error when the second connector is registering its prometheus metric connector.

To Reproduce Create a connector, start it, close it. Then create a new connector and start it.

Expected behavior While closing the connector, it should unregister all prometheus collectors which causes the duplicate error.

Stack Trace

> panic: duplicate metrics collector registration attempted
goroutine 4671 [running]:
[github.com/prometheus/client_golang/prometheus.(*Registry).MustRegister(0x140003f4a50](http://github.com/prometheus/client_golang/prometheus.(*Registry).MustRegister(0x140003f4a50), {0x14026b26f00, 0x1, 0x1})
        /Users/firat.feroglu/go/pkg/mod/github.com/prometheus/client_golang@v1.16.0/prometheus/registry.go:405 +0xf4
[github.com/Trendyol/go-dcp/api.NewMetricMiddleware(0x1401982f200](http://github.com/Trendyol/go-dcp/api.NewMetricMiddleware(0x1401982f200), 0x1400032c908, {0x1024f5d40, 0x1401ff720f0}, {0x1024f9ea0, 0x14019c49da0}, {0x1024e4908, 0x1401ff7e5a0}, {0x1401ffc4a10, 0x1, ...})
        /Users/firat.feroglu/go/pkg/mod/github.com/!trendyol/go-dcp@v1.0.0/api/metric.go:345 +0xfc
[github.com/Trendyol/go-dcp/api.NewAPI(0x1400032c908](http://github.com/Trendyol/go-dcp/api.NewAPI(0x1400032c908), {0x1024f9ea0, 0x14019c49da0}, {0x1024f5d40, 0x1401ff720f0}, {0x0, 0x0}, {0x1024e4908, 0x1401ff7e5a0}, {0x1401ffc4a10, ...})
        /Users/firat.feroglu/go/pkg/mod/github.com/!trendyol/go-dcp@v1.0.0/api/api.go:97 +0x21c
[github.com/Trendyol/go-dcp.(*dcp).Start.func1()](http://github.com/Trendyol/go-dcp.(*dcp).Start.func1())
        /Users/firat.feroglu/go/pkg/mod/github.com/!trendyol/go-dcp@v1.0.0/dcp.go:150 +0x138
created by [github.com/Trendyol/go-dcp.(*dcp).Start](http://github.com/Trendyol/go-dcp.(*dcp).Start) in goroutine 4862
        /Users/firat.feroglu/go/pkg/mod/github.com/!trendyol/go-dcp@v1.0.0/dcp.go:144 +0x774
Exiting.
oguzyildirim commented 1 year ago

hi @EmreKumas sorry to hear that and thank you for your contribution.

we will publish a fix πŸ‘

firatferoglu commented 1 year ago

Thank you for your support, we’re waiting this fix πŸ™ btw we tried to unregister all prometheus collectors but after that we faced fiber prometheus metric problem also fyi.

oguzyildirim commented 1 year ago

Hello @firatferoglu we've implemented a fix but I got confused about what you guys tried. how did you try to unregister collectors? is that similar to what we did below?

https://github.com/Trendyol/go-dcp/pull/61/files

firatferoglu commented 1 year ago

Yes, it looks similar. We edited the code in the vendor folder and unregistered prometheus collectors.

I also try this branch locally but, same error still exists :(

I changed unregisterMetricCollectors() method as below.

func (s *dcp) unregisterMetricCollectors() {
    for i := range s.metricCollectors {
        result := prometheus.DefaultRegisterer.Unregister(s.metricCollectors[i])
        fmt.Println(result)
    }

    s.metricCollectors = []prometheus.Collector{}
}

And observed that the metric is created in api.NewMetricCollector() method couldn't be unregistered.

oguzyildirim commented 1 year ago

Thank you for the information. We should investigate the library and find a proper solution. I will update here if I find the reason and solution.

oguzyildirim commented 1 year ago

Hello @firatferoglu we have released a new version v1.0.3. Thanks for help πŸ™

firatferoglu commented 1 year ago

Brilliant, thank you for your support πŸ™