search

grafana / loki

Like Prometheus, but for logs.
https://grafana.com/loki
GNU Affero General Public License v3.0
23.35k stars 3.38k forks source link

MOVED/EXECABORT/NOAUTH errors from redis #3786

Closed boniek83 closed 3 years ago

boniek83 commented 3 years ago

Describe the bug There are Redis errors in my log. It's not immediately obvious if MOVED are redirected by client or not. I have not idea where EXECABORT comes from. I specify password, so why NOAUTH errors from sentinel? For some reason passing ${redis-password} variable name to password field causes error but ${redis_password} does not. Value is set to correct password in both cases of course.

To Reproduce Steps to reproduce the behavior:

  1. Configure redis cluster as caching layer (below fragment of loki configuration): 
      chunk_cache_config:
    redis:
      endpoint: "redis-cluster-headless:6379"
      master_name: ""
      timeout: 1000ms
      pool_size: 1000
      password: ${redis_password}
  write_dedupe_cache_config:
    redis:
      endpoint: "loki-redis-headless:26379"
      master_name: "mymaster"
      timeout: 1000ms
      db: 1
      pool_size: 1000
      password: ${redis_password}

Expected behavior No errors during normal operation.

Environment:

Screenshots, Promtail config, or terminal output

redis: 2021/06/02 09:14:24 sentinel.go:471: sentinel: GetMasterAddrByName master="mymaster" failed: NOAUTH Authentication required.
level=error ts=2021-06-02T09:14:24.956985223Z caller=redis_cache.go:37 msg="failed to get from redis" name=store.index-cache-read.redis err="redis: all sentinels are unreachable"
level=info ts=2021-06-02T01:23:25.443611013Z caller=main.go:130 msg="Starting Loki" version="(version=2.2.1, branch=HEAD, revision=babea82ef)"
level=info ts=2021-06-02T01:23:25.446638191Z caller=server.go:229 http=[::]:3100 grpc=[::]:9095 msg="server listening on addresses"
level=warn ts=2021-06-02T01:23:25.447203406Z caller=experimental.go:19 msg="experimental feature in use" feature="In-memory (FIFO) cache"
level=warn ts=2021-06-02T01:23:25.44751322Z caller=experimental.go:19 msg="experimental feature in use" feature="Redis cache"
level=info ts=2021-06-02T01:23:26.076483117Z caller=table_manager.go:241 msg="loading table index_18779"
level=info ts=2021-06-02T01:23:26.237754537Z caller=table_manager.go:241 msg="loading table index_18780"
level=info ts=2021-06-02T01:23:27.340988017Z caller=shipper_index_client.go:100 msg="starting boltdb shipper in 2 mode"
level=info ts=2021-06-02T01:23:27.341085985Z caller=table_manager.go:171 msg="uploading tables"
level=info ts=2021-06-02T01:23:27.343284016Z caller=table.go:336 msg="uploading table index_18780"
level=info ts=2021-06-02T01:23:27.390812874Z caller=memberlist_client.go:380 msg="Using memberlist cluster node name" name=loki-distributed-ingester-2-ada3deaf
level=info ts=2021-06-02T01:23:27.39148899Z caller=module_service.go:59 msg=initialising module=memberlist-kv
level=info ts=2021-06-02T01:23:27.391498731Z caller=module_service.go:59 msg=initialising module=store
level=info ts=2021-06-02T01:23:27.391548031Z caller=module_service.go:59 msg=initialising module=server
level=info ts=2021-06-02T01:23:27.391630238Z caller=module_service.go:59 msg=initialising module=ingester
level=info ts=2021-06-02T01:23:27.39170175Z caller=ingester.go:241 msg="recovering from checkpoint"
level=info ts=2021-06-02T01:23:27.416943945Z caller=memberlist_client.go:492 msg="joined memberlist cluster" reached_nodes=7
level=info ts=2021-06-02T01:23:27.536829962Z caller=table.go:363 msg="finished uploading table index_18780"
level=info ts=2021-06-02T01:23:27.536931721Z caller=table.go:421 msg="cleaning up unwanted dbs from table index_18780"
level=info ts=2021-06-02T01:23:28.168469481Z caller=ingester.go:257 msg="recovered WAL checkpoint recovery finished" elapsed=776.764645ms errors=false
level=info ts=2021-06-02T01:23:28.168596236Z caller=ingester.go:263 msg="recovering from WAL"
level=info ts=2021-06-02T01:23:28.829212897Z caller=ingester.go:279 msg="WAL segment recovery finished" elapsed=1.437507887s errors=false
level=info ts=2021-06-02T01:23:28.830056087Z caller=ingester.go:287 msg="recovery finished" time=1.438352178s
level=info ts=2021-06-02T01:23:28.830125119Z caller=wal.go:156 msg=started component=wal
level=info ts=2021-06-02T01:23:28.830192749Z caller=lifecycler.go:521 msg="not loading tokens from file, tokens file path is empty"
level=info ts=2021-06-02T01:23:28.830371607Z caller=lifecycler.go:550 msg="instance not found in ring, adding with no tokens" ring=ingester
level=info ts=2021-06-02T01:23:28.830642112Z caller=lifecycler.go:397 msg="auto-joining cluster after timeout" ring=ingester
level=info ts=2021-06-02T01:23:28.831847152Z caller=loki.go:248 msg="Loki started"
level=error ts=2021-06-02T01:23:58.83649127Z caller=redis_cache.go:37 msg="failed to get from redis" name=chunksredis err="MOVED 8298 10.34.190.139:6379"
level=error ts=2021-06-02T01:23:58.836810935Z caller=redis_cache.go:37 msg="failed to get from redis" name=chunksredis err="MOVED 8609 10.34.190.139:6379"
level=error ts=2021-06-02T01:23:58.837762161Z caller=redis_cache.go:37 msg="failed to get from redis" name=chunksredis err="MOVED 7371 10.34.190.139:6379"
level=error ts=2021-06-02T01:23:58.838192171Z caller=redis_cache.go:37 msg="failed to get from redis" name=chunksredis err="MOVED 11205 10.34.191.9:6379"
level=error ts=2021-06-02T01:23:58.839678103Z caller=redis_cache.go:37 msg="failed to get from redis" name=chunksredis err="MOVED 16345 10.34.191.9:6379"
level=error ts=2021-06-02T01:23:58.838823108Z caller=redis_cache.go:37 msg="failed to get from redis" name=chunksredis err="MOVED 12032 10.34.191.9:6379"
level=error ts=2021-06-02T01:23:58.845750157Z caller=redis_cache.go:37 msg="failed to get from redis" name=chunksredis err="MOVED 9988 10.34.190.139:6379"
level=error ts=2021-06-02T01:23:58.845968053Z caller=redis_cache.go:37 msg="failed to get from redis" name=chunksredis err="MOVED 9238 10.34.190.139:6379"
level=error ts=2021-06-02T01:23:58.84604478Z caller=redis_cache.go:37 msg="failed to get from redis" name=chunksredis err="MOVED 11169 10.34.191.9:6379"
level=error ts=2021-06-02T01:23:58.846029143Z caller=redis_cache.go:37 msg="failed to get from redis" name=chunksredis err="MOVED 12840 10.34.191.9:6379"
level=error ts=2021-06-02T01:23:58.846244842Z caller=redis_cache.go:37 msg="failed to get from redis" name=chunksredis err="MOVED 8689 10.34.190.139:6379"
level=error ts=2021-06-02T01:23:59.087794736Z caller=redis_cache.go:57 msg="failed to put to redis" name=chunksredis err="EXECABORT Transaction discarded because of previous errors."
level=error ts=2021-06-02T01:23:59.091019579Z caller=redis_cache.go:57 msg="failed to put to redis" name=chunksredis err="EXECABORT Transaction discarded because of previous errors."
level=error ts=2021-06-02T01:23:59.091142773Z caller=redis_cache.go:37 msg="failed to get from redis" name=chunksredis err="MOVED 12126 10.34.191.9:6379"
level=error ts=2021-06-02T01:23:59.092478078Z caller=redis_cache.go:37 msg="failed to get from redis" name=chunksredis err="MOVED 7297 10.34.190.139:6379"
level=error ts=2021-06-02T01:23:59.092893276Z caller=redis_cache.go:57 msg="failed to put to redis" name=chunksredis err="EXECABORT Transaction discarded because of previous errors."
level=error ts=2021-06-02T01:23:59.095404484Z caller=redis_cache.go:37 msg="failed to get from redis" name=chunksredis err="MOVED 9511 10.34.190.139:6379"
level=error ts=2021-06-02T01:23:59.09543319Z caller=redis_cache.go:57 msg="failed to put to redis" name=chunksredis err="EXECABORT Transaction discarded because of previous errors."
level=error ts=2021-06-02T01:23:59.096801787Z caller=redis_cache.go:37 msg="failed to get from redis" name=chunksredis err="MOVED 9250 10.34.190.139:6379"
level=error ts=2021-06-02T01:23:59.096837621Z caller=redis_cache.go:57 msg="failed to put to redis" name=chunksredis err="EXECABORT Transaction discarded because of previous errors."
level=error ts=2021-06-02T01:23:59.098363754Z caller=redis_cache.go:57 msg="failed to put to redis" name=chunksredis err="EXECABORT Transaction discarded because of previous errors."
level=error ts=2021-06-02T01:23:59.100994825Z caller=redis_cache.go:37 msg="failed to get from redis" name=chunksredis err="MOVED 8726 10.34.190.139:6379"
level=error ts=2021-06-02T01:23:59.103693047Z caller=redis_cache.go:57 msg="failed to put to redis" name=chunksredis err="EXECABORT Transaction discarded because of previous errors."
level=error ts=2021-06-02T01:23:59.105369738Z caller=redis_cache.go:37 msg="failed to get from redis" name=chunksredis err="MOVED 6136 10.34.190.139:6379"
level=error ts=2021-06-02T01:23:59.105790566Z caller=redis_cache.go:57 msg="failed to put to redis" name=chunksredis err="EXECABORT Transaction discarded because of previous errors."
level=error ts=2021-06-02T01:23:59.107268657Z caller=redis_cache.go:37 msg="failed to get from redis" name=chunksredis err="MOVED 9310 10.34.190.139:6379"
level=error ts=2021-06-02T01:23:59.112213178Z caller=redis_cache.go:37 msg="failed to get from redis" name=chunksredis err="MOVED 9687 10.34.190.139:6379"
dannykopping commented 3 years ago

Hey @boniek83

I think all of the errors are arising because Loki cannot authenticate against your redis installation; in other words I think the MOVED/EXECABORT errors can be ignored.

That said, we try to keep GitHub issues strictly for bug reports and feature requests.

You may submit questions and support requests in any of the following ways:

I'm closing this issue, but please feel free to reach out in any of the channels listed above.