search

oliver006 / redis_exporter

Prometheus Exporter for ValKey & Redis Metrics. Supports ValKey and Redis 2.x, 3.x, 4.x, 5.x, 6.x, and 7.x
https://github.com/oliver006/redis_exporter
MIT License
3.04k stars 860 forks source link

Connection Reset by Peer upon a "Cluster Refresh" #790

Closed DGRFL closed 2 months ago

DGRFL commented 1 year ago

I have a redis 6.2.6. cluster over 3 VMs, with 2 nodes per VM (ports 7005, 7006) Redis exporter is running with following command

./redis_exporter  --redis.addr=redis://git-kfkrds01:7005 --skip-tls-verification --tls-ca-cert-file=/apps/Testing/redistest/ca/ca.crt --tls-client-key-file=/apps/Testing/redistest/ca/redis.key --tls-client-cert-file=/apps/Testing/redistest/ca/redis.crt --set-client-name=true --is-cluster=true

Prometheus extracts info with

  - job_name: 'redis_exporter_targets'
    static_configs:
      - targets:
        - rediss://git-kfkrds01:7005
        - rediss://git-kfkrds01:7006
        - rediss://git-kfkrds02:7005
        - rediss://git-kfkrds02:7006
        - rediss://git-kfkrds03:7005
        - rediss://git-kfkrds03:7006
    metrics_path: /scrape
    scheme: http
    tls_config:
      insecure_skip_verify: true
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: git-kfkrds01:9121

  ## config for scraping the exporter itself
  - job_name: 'redis_exporter'
    static_configs:
      - targets:
        - git-kfkrds01:9121

Grafana dashboards show actual exported metrics.

However the redis-exporter output contains clauses like 

ERRO[2880] Couldn't set client name, err: read tcp 10.x.x.93:46436->10.x.x.93:7005: read: connection reset by peer
ERRO[2880] Redis INFO err: set tcp 10.x.x.93:46436: use of closed network connection
ERRO[2880] Cluster refresh failed: redisc: all nodes failed
read tcp 10.x.x.93:33556->10.x.x.95:7006: read: connection reset by peer
ERRO[2883] Cluster refresh failed: redisc: all nodes failed
read tcp 10.x.x.93:46576->10.x.x.95:7005: read: connection reset by peer
ERRO[2885] Cluster refresh failed: redisc: all nodes failed
read tcp 10.x.x.93:52606->10.x.x.94:7005: read: connection reset by peer
ERRO[2888] Cluster refresh failed: redisc: all nodes failed
read tcp 10.x.x.93:55820->10.x.x.93:7006: read: connection reset by peer
ERRO[2889] Cluster refresh failed: redisc: all nodes failed
read tcp 10.x.x.93:37098->10.x.x.94:7006: read: connection reset by peer
ERRO[2892] Cluster refresh failed: redisc: all nodes failed
read tcp 10.x.x.93:47072->10.x.x.93:7005: read: connection reset by peer

When enabled --debug, I get following additional info.

DEBU[0001] c.Do() - running command: LATENCY [LATEST]
DEBU[0001] c.Do() - done
DEBU[0001] Load key pair: /apps/Testing/redistest/ca/redis.crt /apps/Testing/redistest/ca/redis.key
DEBU[0001] Load CA cert file: /apps/Testing/redistest/ca/ca.crt
DEBU[0001] Creating cluster object
DEBU[0001] Running refresh on cluster object
ERRO[0001] Cluster refresh failed: redisc: all nodes failed
read tcp 10.244.226.93:44172->10.244.226.94:7006: read: connection reset by peer
DEBU[0001] Creating redis connection object
DEBU[0001] parseKeyArg(): Got empty key arguments, parsing skipped
DEBU[0001] keys: []exporter.dbKeyPair(nil)
DEBU[0001] parseKeyArg(): Got empty key arguments, parsing skipped
DEBU[0001] e.singleKeys: []exporter.dbKeyPair(nil)
DEBU[0001] e.keys: []exporter.dbKeyPair(nil)

How can I avoid such error logging?