search

oliver006 / redis_exporter

Prometheus Exporter for ValKey & Redis Metrics. Supports ValKey and Redis 2.x, 3.x, 4.x, 5.x, 6.x, and 7.x
https://github.com/oliver006/redis_exporter
MIT License
3.04k stars 860 forks source link

Redis Exporter causing CPU spikes #802

Closed NirmalP29 closed 1 year ago

NirmalP29 commented 1 year ago

Redis server has zero keys. The vm in screenshot has 4 CPUs I have tested this in multiple azure vms and most of them shows cpu spikes due to redis exporter.

TOP command output

image

The top panel shows CPU total usage from node exporter The middle panel shows "process_cpu_seconds_total" from redis exporter. It also shows "redis_cpu_sys_seconds_total" and "redis_cpu_user_seconds_total" but both are flat (ie zero) The bottom panel shows max single cpu utilization from node exporter.

image

INFO[0000] Redis Metrics Exporter v1.43.0 build date: 2022-06-18-11:52:03 sha1: 980a9741e77c4e065faba80805f93d4694d71d67 Go: go1.18.3 GOOS: linux GOARCH: amd64

Can someone help me debug why redis exporter is causing cpu spikes?

bjosv commented 1 year ago

We had the same problem a couple of month ago. After a lot of investigations we found an issue with timers in the Go runtime, see https://github.com/golang/go/issues/51654 You are likely affected by the same problem since you run Go 1.18.3

The problem is solved and merged in Go 1.19 and backported to 1.17.13 and 1.18.5

oliver006 commented 1 year ago

INFO[0000] Redis Metrics Exporter v1.43.0 build date: 2022-06-18-11:52:03 sha1: https://github.com/oliver006/redis_exporter/commit/980a9741e77c4e065faba80805f93d4694d71d67 Go: go1.18.3 GOOS: linux GOARCH: amd64

@NirmalP29 - You're running a pretty old version of the exporter (that, according to @bjosv, was build with the buggy 1.18.3 Go runtime) - can you try upgrading and see if the issue is fixed?

NirmalP29 commented 1 year ago

Thank you @oliver006 and @bjosv 😀. The issue is resolved after upgrading go and redis_exporter.