[loki-distributed] Errors with redis-cluster (cache)

[felrivero]

Hi everyone! I use loki-distributed chart (version 0.55.7) and setup redis-cluster for cache (redis version 7.0.4, 3 Primary and 3 Replica nodes). But I get errors in querier pod:

level=error ts=2022-08-23T13:39:57.238530229Z caller=redis_cache.go:58 msg="failed to put to redis" name=store.index-cache-read.redis err="CROSSSLOT Keys in request don't hash to the same slot"
level=error ts=2022-08-23T13:39:57.281428544Z caller=redis_cache.go:58 msg="failed to put to redis" name=store.index-cache-read.redis err="CROSSSLOT Keys in request don't hash to the same slot"
level=error ts=2022-08-23T13:39:57.285443083Z caller=redis_cache.go:38 msg="failed to get from redis" name=store.index-cache-read.redis err="CROSSSLOT Keys in request don't hash to the same slot"
level=error ts=2022-08-23T13:39:57.285513083Z caller=redis_cache.go:38 msg="failed to get from redis" name=store.index-cache-read.redis err="CROSSSLOT Keys in request don't hash to the same slot"
level=error ts=2022-08-23T13:39:57.290890635Z caller=redis_cache.go:58 msg="failed to put to redis" name=store.index-cache-read.redis err="CROSSSLOT Keys in request don't hash to the same slot"
level=error ts=2022-08-23T13:39:57.29111892Z caller=redis_cache.go:58 msg="failed to put to redis" name=store.index-cache-read.redis err="CROSSSLOT Keys in request don't hash to the same slot"

I try to change redis endpoint from loki-redis-headless:6379 to primary1:6379,primary2:6379,primary3:6379 but it didn't help

loki config.yaml

```yaml chunk_store_config: chunk_cache_config: redis: endpoint: loki-redis-headless:6379 max_look_back_period: 0s compactor: shared_store: s3 distributor: ring: kvstore: store: memberlist frontend: compress_responses: true log_queries_longer_than: 5s tail_proxy_url: http://loki-distributed-querier:3100 frontend_worker: frontend_address: loki-distributed-query-frontend:9095 ingester: chunk_block_size: 262144 chunk_encoding: snappy chunk_idle_period: 1h chunk_retain_period: 1m chunk_target_size: 1536000 lifecycler: ring: kvstore: store: memberlist replication_factor: 1 max_chunk_age: 1h max_transfer_retries: 0 wal: dir: /var/loki/wal limits_config: enforce_metric_name: false ingestion_burst_size_mb: 150 ingestion_rate_mb: 300 max_cache_freshness_per_query: 50m max_concurrent_tail_requests: 2048 max_entries_limit_per_query: 15000 max_global_streams_per_user: 0 max_query_parallelism: 100 per_stream_rate_limit: 50MB per_stream_rate_limit_burst: 100MB reject_old_samples: false reject_old_samples_max_age: 168h retention_period: 2160h split_queries_by_interval: 15m memberlist: join_members: - loki-distributed-memberlist query_range: align_queries_with_step: true cache_results: true max_retries: 5 results_cache: cache: enable_fifocache: true fifocache: max_size_items: 1024 ttl: 24h redis: endpoint: loki-redis-headless:6379 ruler: alertmanager_url: https://alertmanager.xx external_url: https://alertmanager.xx ring: kvstore: store: memberlist rule_path: /tmp/loki/scratch storage: local: directory: /etc/loki/rules type: local schema_config: configs: - from: "2020-09-07" index: period: 24h prefix: loki_index_ object_store: aws schema: v11 store: boltdb-shipper server: http_listen_port: 3100 storage_config: aws: bucketnames: my-s3 s3: s3://eu-north-1 boltdb_shipper: active_index_directory: /var/loki/index cache_location: /var/loki/cache cache_ttl: 168h index_gateway_client: server_address: dns:///loki-distributed-index-gateway:9095 shared_store: s3 filesystem: directory: /var/loki/chunks index_queries_cache_config: redis: endpoint: loki-redis-headless:6379 table_manager: retention_deletes_enabled: false retention_period: 0s ```

[sureshgoli25]

I am also looking for a way to use redis instead of memcache. It seems, memcache is failing when the larger objects are trying to store in the cache.

[Moep90]

@felrivero @sureshgoli25 did someone found a solution on this?

[salareinavipour]

We're also eager to know if Grafana Labs will add support for Redis as an alternative to Memcache. However, it seems they're not in a rush! We tried adding configs of this section: https://grafana.com/docs/loki/latest/configuration/#cache_config To the config in: https://github.com/grafana/helm-charts/blob/main/charts/loki-distributed/values.yaml Checking if it works like the other option, but we weren't successful! So, for now, Memcache is the only standard supported option in this chart.