SigNoz / charts

Helm Charts for SigNoz
MIT License
68 stars 73 forks source link

clickhouse: Nil pointer dereference in metrics exporter #253

Open prashant-shahi opened 1 year ago

prashant-shahi commented 1 year ago

After upgrading to ClickHouse Operator v0.21.2, the following error occurs at the start of the installation. Though it seems to self-heal after few restarts.

Error Log:

I0707 05:05:52.512308       1 rest_server.go:37] Starting metrics exporter at ':8888/metrics'
I0707 05:05:52.529129       1 exporter.go:294] Add explicitly found CHI default/signoz-6hjc6q1prm-clickhouse
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x198 pc=0x1370d8e]

goroutine 1 [running]:
github.com/altinity/clickhouse-operator/pkg/apis/metrics.(*Exporter).DiscoveryWatchedCHIs(0x16c6661?, {0x18f6338, 0xc00022dce0}, 0xc0004d60c0)
    /clickhouse-operator/pkg/apis/metrics/exporter.go:295 +0x3ee
github.com/altinity/clickhouse-operator/cmd/metrics_exporter/app.Run()
    /clickhouse-operator/cmd/metrics_exporter/app/metrics_exporter.go:108 +0x36b
main.main()
    /clickhouse-operator/cmd/metrics_exporter/main.go:23 +0x17

Ref: https://github.com/Altinity/clickhouse-operator/issues/1187

spenczar commented 1 year ago

I hit this in my first install of signoz from charts as well.

prashant-shahi commented 1 year ago

@spenczar This has been fixed in the default branch of the clickhouse-operator, and will be part of >=v0.21.3 release.

However, we wouldn't be bumping up operator version anytime soon, since it would involve migration steps for CRDs once again.