search

bitnami / charts

Bitnami Helm Charts
https://bitnami.com
Other
9.03k stars 9.22k forks source link

REDIS SENTINEL - Stuck on 15.3.3; slave connections not working #8614

Closed aeimer closed 2 years ago

aeimer commented 2 years ago

Which chart: The name (and version) of the affected chart

bitnami/redis:15.4.0

Describe the bug A clear and concise description of what the bug is.

After upgrading from 15.3.3 to 15.4.0 our java redis clients were not able to connect to the slaves anymore.

Its seem to be a problem with this PR: https://github.com/bitnami/charts/pull/7461

{"timestamp":"2021-12-07T13:51:37.729Z","level":"INFO","thread":"SimpleAsyncTaskExecutor-27","logger":"org.apache.catalina.core.StandardService","message":"Starting service [Tomcat[]"}
{"timestamp":"2021-12-07T13:51:37.730Z","level":"INFO","thread":"SimpleAsyncTaskExecutor-27","logger":"org.apache.catalina.core.StandardEngine","message":"Starting Servlet engine: [Apache Tomcat/9.0.55]"}
{"timestamp":"2021-12-07T13:51:37.738Z","level":"INFO","thread":"SimpleAsyncTaskExecutor-27","logger":"org.apache.catalina.core.ContainerBase.[Tomcat-1[].[localhost[].[/]","message":"Initializing Spring embedded WebApplicationContext"}
{"timestamp":"2021-12-07T13:51:37.738Z","level":"INFO","thread":"SimpleAsyncTaskExecutor-27","logger":"org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext","message":"Root WebApplicationContext: initialization completed in 1504 ms"}
{"timestamp":"2021-12-07T13:51:37.930Z","level":"INFO","thread":"SimpleAsyncTaskExecutor-27","logger":"org.springframework.boot.actuate.endpoint.web.EndpointLinksResolver","message":"Exposing 3 endpoint(s) beneath base path '/actuator'"}
{"timestamp":"2021-12-07T13:51:38.430Z","level":"INFO","thread":"SimpleAsyncTaskExecutor-27","logger":"org.apache.coyote.http11.Http11NioProtocol","message":"Starting ProtocolHandler [\"http-nio-8081\"]"}
{"timestamp":"2021-12-07T13:51:38.433Z","level":"INFO","thread":"SimpleAsyncTaskExecutor-27","logger":"org.springframework.boot.web.embedded.tomcat.TomcatWebServer","message":"Tomcat started on port(s): 8081 (http) with context path ''"}
{"timestamp":"2021-12-07T13:51:44.132Z","level":"INFO","thread":"http-nio-8081-exec-1","logger":"org.apache.catalina.core.ContainerBase.[Tomcat-1[].[localhost[].[/]","message":"Initializing Spring DispatcherServlet 'dispatcherServlet'"}
{"timestamp":"2021-12-07T13:51:44.133Z","level":"INFO","thread":"http-nio-8081-exec-1","logger":"org.springframework.web.servlet.DispatcherServlet","message":"Initializing Servlet 'dispatcherServlet'"}
{"timestamp":"2021-12-07T13:51:44.134Z","level":"INFO","thread":"http-nio-8081-exec-1","logger":"org.springframework.web.servlet.DispatcherServlet","message":"Completed initialization in 1 ms"}
{"timestamp":"2021-12-07T13:52:02.258Z","level":"WARN","thread":"redisson-netty-2-13","logger":"org.redisson.connection.SentinelConnectionManager","message":"slave: redis://10.0.16.188:6379 is down"}
{"timestamp":"2021-12-07T13:52:04.637Z","level":"INFO","thread":"redisson-netty-2-20","logger":"org.redisson.connection.SentinelConnectionManager","message":"slave: redis://10.0.16.188:6379 is up"}
{"timestamp":"2021-12-07T13:52:04.638Z","level":"INFO","thread":"redisson-netty-2-15","logger":"org.redisson.connection.pool.PubSubConnectionPool","message":"1 connections initialized for 10.0.16.188/10.0.16.188:6379"}
{"timestamp":"2021-12-07T13:52:04.741Z","level":"INFO","thread":"redisson-netty-2-28","logger":"org.redisson.connection.pool.SlaveConnectionPool","message":"24 connections initialized for 10.0.16.188/10.0.16.188:6379"}
{"timestamp":"2021-12-07T13:52:12.666Z","level":"WARN","thread":"redisson-netty-2-12","logger":"org.redisson.connection.SentinelConnectionManager","message":"slave: redis://10.0.16.188:6379 is down"}
{"timestamp":"2021-12-07T13:52:13.668Z","level":"INFO","thread":"redisson-netty-2-32","logger":"org.redisson.connection.SentinelConnectionManager","message":"slave: redis://10.0.16.188:6379 is up"}
{"timestamp":"2021-12-07T13:52:13.673Z","level":"INFO","thread":"redisson-netty-2-26","logger":"org.redisson.connection.pool.PubSubConnectionPool","message":"1 connections initialized for 10.0.16.188/10.0.16.188:6379"}
{"timestamp":"2021-12-07T13:52:13.677Z","level":"INFO","thread":"redisson-netty-2-28","logger":"org.redisson.connection.pool.SlaveConnectionPool","message":"24 connections initialized for 10.0.16.188/10.0.16.188:6379"}
{"timestamp":"2021-12-07T13:52:26.896Z","level":"WARN","thread":"redisson-netty-2-14","logger":"org.redisson.connection.SentinelConnectionManager","message":"slave: redis://10.0.16.188:6379 is down"}
{"timestamp":"2021-12-07T13:52:27.897Z","level":"INFO","thread":"redisson-netty-2-20","logger":"org.redisson.connection.SentinelConnectionManager","message":"slave: redis://10.0.16.188:6379 is up"}
{"timestamp":"2021-12-07T13:52:27.902Z","level":"INFO","thread":"redisson-netty-2-3","logger":"org.redisson.connection.pool.PubSubConnectionPool","message":"1 connections initialized for 10.0.16.188/10.0.16.188:6379"}
{"timestamp":"2021-12-07T13:52:27.903Z","level":"INFO","thread":"redisson-netty-2-13","logger":"org.redisson.connection.pool.SlaveConnectionPool","message":"24 connections initialized for 10.0.16.188/10.0.16.188:6379"}
{"timestamp":"2021-12-07T13:52:54.965Z","level":"WARN","thread":"redisson-netty-2-9","logger":"org.redisson.connection.SentinelConnectionManager","message":"slave: redis://10.0.16.188:6379 is down"}
[...]

Config of redis:

---

# We don't want password protection for now
auth:
  enabled: false
  sentinel: false

architecture: replication

pdb:
  create: true
  minAvailable: 2

sentinel:
  enabled: true
  resources:
    limits:
      cpu: 200m
      memory: 256Mi
  quorum: 2
  downAfterMilliseconds: 10000
  failoverTimeout: 15000

serviceAccount:
  create: true

replica:
  affinity:
    podAntiAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
        - podAffinityTerm:
            labelSelector:
              matchLabels:
                app.kubernetes.io/name: redis
                app.kubernetes.io/instance: redis
            topologyKey: kubernetes.io/hostname
          weight: 100
  replicaCount: 3
  persistence:
    size: 8Gi
  resources:
    limits:
      cpu: 600m
      memory: 4Gi
    requests:
      cpu: 400m
  livenessProbe:
    initialDelaySeconds: 60
  readinessProbe:
    # This will give the existing pods enough time to find a new master on roll-over
    initialDelaySeconds: 20

configmap: |-
  # <<<BEGIN FROM default values.yaml
  # Enable AOF https://redis.io/topics/persistence#append-only-file
  appendonly yes
  # Disable RDB persistence, AOF persistence already enabled.
  save ""
  # >>>END FROM default values.yaml

  # Setup redis as cache
  # memory limit minus about 95% => 256mb
  maxmemory 3840mb
  # Evict using approximated LFU among the keys with an expire set
  maxmemory-policy volatile-lfu

To Reproduce Steps to reproduce the behavior:

  1. Setup redis sentinel
  2. Try to use redis with redisson
  3. See logs in java pods

Expected behavior A clear and concise description of what you expected to happen.

After the non-breaking update the setup should work as before. Or there should be some kind of info on how to approach this new version.

We are now stuck on 15.3.3, also the latest versions were tried, but the problem persists.

Version of Helm and Kubernetes:

(paste your output here)
(paste your output here)

Additional context Add any other context about the problem here.

Interactions till now:

javsalgar commented 2 years ago

Hi,

Could you provide more details of the Redis logs?

aeimer commented 2 years ago

Hi @javsalgar , sure!

I just installed the latest version 15.7.2, coming from 15.3.3.

These are the logs of the java pods:

{"timestamp":"2022-01-11T12:11:59.71Z","level":"INFO","thread":"redisson-netty-2-8","logger":"org.redisson.connection.SentinelConnectionManager","message":"sentinel: redis://10.0.18.79:26379 added"}
{"timestamp":"2022-01-11T12:11:59.223Z","level":"INFO","thread":"redisson-netty-2-13","mdc":{"instana.trace.id":"d13c89208fe60f4f"},"logger":"org.redisson.connection.pool.PubSubConnectionPool","message":"1 connections initialized for 10.0.18.79/10.0.18.79:6379"}
{"timestamp":"2022-01-11T12:11:59.225Z","level":"INFO","thread":"redisson-netty-2-8","mdc":{"instana.trace.id":"d13c89208fe60f4f"},"logger":"org.redisson.connection.SentinelConnectionManager","message":"slave: redis://10.0.18.79:6379 added"}
{"timestamp":"2022-01-11T12:11:59.225Z","level":"INFO","thread":"redisson-netty-2-8","mdc":{"instana.trace.id":"d13c89208fe60f4f"},"logger":"org.redisson.connection.pool.SlaveConnectionPool","message":"24 connections initialized for 10.0.18.79/10.0.18.79:6379"}
{"timestamp":"2022-01-11T12:12:00.222Z","level":"WARN","thread":"redisson-netty-2-11","mdc":{"instana.trace.id":"d13c89208fe60f4f"},"logger":"org.redisson.connection.SentinelConnectionManager","message":"slave: redis://10.0.18.79:6379 is down"}
{"timestamp":"2022-01-11T12:12:01.224Z","level":"INFO","thread":"redisson-netty-2-21","mdc":{"instana.trace.id":"d13c89208fe60f4f"},"logger":"org.redisson.connection.SentinelConnectionManager","message":"slave: redis://10.0.18.79:6379 is up"}
{"timestamp":"2022-01-11T12:12:01.226Z","level":"INFO","thread":"redisson-netty-2-22","mdc":{"instana.trace.id":"d13c89208fe60f4f"},"logger":"org.redisson.connection.pool.PubSubConnectionPool","message":"1 connections initialized for 10.0.18.79/10.0.18.79:6379"}
{"timestamp":"2022-01-11T12:12:01.227Z","level":"INFO","thread":"redisson-netty-2-11","mdc":{"instana.trace.id":"d13c89208fe60f4f"},"logger":"org.redisson.connection.pool.SlaveConnectionPool","message":"24 connections initialized for 10.0.18.79/10.0.18.79:6379"}
{"timestamp":"2022-01-11T12:12:02.227Z","level":"WARN","thread":"redisson-netty-2-13","mdc":{"instana.trace.id":"d13c89208fe60f4f"},"logger":"org.redisson.connection.SentinelConnectionManager","message":"slave: redis://10.0.18.79:6379 is down"}
{"timestamp":"2022-01-11T12:12:03.229Z","level":"INFO","thread":"redisson-netty-2-24","mdc":{"instana.trace.id":"d13c89208fe60f4f"},"logger":"org.redisson.connection.SentinelConnectionManager","message":"slave: redis://10.0.18.79:6379 is up"}
{"timestamp":"2022-01-11T12:12:03.231Z","level":"INFO","thread":"redisson-netty-2-27","mdc":{"instana.trace.id":"d13c89208fe60f4f"},"logger":"org.redisson.connection.pool.PubSubConnectionPool","message":"1 connections initialized for 10.0.18.79/10.0.18.79:6379"}
{"timestamp":"2022-01-11T12:12:03.233Z","level":"INFO","thread":"redisson-netty-2-13","mdc":{"instana.trace.id":"d13c89208fe60f4f"},"logger":"org.redisson.connection.pool.SlaveConnectionPool","message":"24 connections initialized for 10.0.18.79/10.0.18.79:6379"}
{"timestamp":"2022-01-11T12:12:05.233Z","level":"WARN","thread":"redisson-netty-2-25","mdc":{"instana.trace.id":"d13c89208fe60f4f"},"logger":"org.redisson.connection.SentinelConnectionManager","message":"slave: redis://10.0.18.79:6379 is down"}
{"timestamp":"2022-01-11T12:12:07.237Z","level":"INFO","thread":"redisson-netty-2-15","mdc":{"instana.trace.id":"d13c89208fe60f4f"},"logger":"org.redisson.connection.SentinelConnectionManager","message":"slave: redis://10.0.18.79:6379 is up"}
{"timestamp":"2022-01-11T12:12:07.239Z","level":"INFO","thread":"redisson-netty-2-9","mdc":{"instana.trace.id":"d13c89208fe60f4f"},"logger":"org.redisson.connection.pool.PubSubConnectionPool","message":"1 connections initialized for 10.0.18.79/10.0.18.79:6379"}
{"timestamp":"2022-01-11T12:12:07.241Z","level":"INFO","thread":"redisson-netty-2-7","mdc":{"instana.trace.id":"d13c89208fe60f4f"},"logger":"org.redisson.connection.pool.SlaveConnectionPool","message":"24 connections initialized for 10.0.18.79/10.0.18.79:6379"}
{"timestamp":"2022-01-11T12:12:09.241Z","level":"WARN","thread":"redisson-netty-2-18","mdc":{"instana.trace.id":"d13c89208fe60f4f"},"logger":"org.redisson.connection.SentinelConnectionManager","message":"slave: redis://10.0.18.79:6379 is down"}
{"timestamp":"2022-01-11T12:12:10.244Z","level":"INFO","thread":"redisson-netty-2-2","mdc":{"instana.trace.id":"d13c89208fe60f4f"},"logger":"org.redisson.connection.SentinelConnectionManager","message":"slave: redis://10.0.18.79:6379 is up"}
{"timestamp":"2022-01-11T12:12:10.245Z","level":"INFO","thread":"redisson-netty-2-4","mdc":{"instana.trace.id":"d13c89208fe60f4f"},"logger":"org.redisson.connection.pool.PubSubConnectionPool","message":"1 connections initialized for 10.0.18.79/10.0.18.79:6379"}
{"timestamp":"2022-01-11T12:12:10.247Z","level":"INFO","thread":"redisson-netty-2-18","mdc":{"instana.trace.id":"d13c89208fe60f4f"},"logger":"org.redisson.connection.pool.SlaveConnectionPool","message":"24 connections initialized for 10.0.18.79/10.0.18.79:6379"}
{"timestamp":"2022-01-11T12:12:12.247Z","level":"WARN","thread":"redisson-netty-2-15","mdc":{"instana.trace.id":"d13c89208fe60f4f"},"logger":"org.redisson.connection.SentinelConnectionManager","message":"slave: redis://10.0.18.79:6379 is down"}
{"timestamp":"2022-01-11T12:12:21.262Z","level":"INFO","thread":"redisson-netty-2-30","mdc":{"instana.trace.id":"d13c89208fe60f4f"},"logger":"org.redisson.connection.SentinelConnectionManager","message":"slave: redis://10.0.18.79:6379 is up"}
{"timestamp":"2022-01-11T12:12:21.264Z","level":"INFO","thread":"redisson-netty-2-29","mdc":{"instana.trace.id":"d13c89208fe60f4f"},"logger":"org.redisson.connection.pool.PubSubConnectionPool","message":"1 connections initialized for 10.0.18.79/10.0.18.79:6379"}
{"timestamp":"2022-01-11T12:12:21.267Z","level":"INFO","thread":"redisson-netty-2-1","mdc":{"instana.trace.id":"d13c89208fe60f4f"},"logger":"org.redisson.connection.pool.SlaveConnectionPool","message":"24 connections initialized for 10.0.18.79/10.0.18.79:6379"}
{"timestamp":"2022-01-11T12:12:22.266Z","level":"WARN","thread":"redisson-netty-2-12","mdc":{"instana.trace.id":"d13c89208fe60f4f"},"logger":"org.redisson.connection.SentinelConnectionManager","message":"slave: redis://10.0.18.79:6379 is down"}
{"timestamp":"2022-01-11T12:12:27.275Z","level":"INFO","thread":"redisson-netty-2-19","mdc":{"instana.trace.id":"d13c89208fe60f4f"},"logger":"org.redisson.connection.SentinelConnectionManager","message":"slave: redis://10.0.18.79:6379 is up"}
{"timestamp":"2022-01-11T12:12:27.277Z","level":"INFO","thread":"redisson-netty-2-17","mdc":{"instana.trace.id":"d13c89208fe60f4f"},"logger":"org.redisson.connection.pool.PubSubConnectionPool","message":"1 connections initialized for 10.0.18.79/10.0.18.79:6379"}
{"timestamp":"2022-01-11T12:12:27.279Z","level":"INFO","thread":"redisson-netty-2-9","mdc":{"instana.trace.id":"d13c89208fe60f4f"},"logger":"org.redisson.connection.pool.SlaveConnectionPool","message":"24 connections initialized for 10.0.18.79/10.0.18.79:6379"}
{"timestamp":"2022-01-11T12:12:28.278Z","level":"WARN","thread":"redisson-netty-2-31","mdc":{"instana.trace.id":"d13c89208fe60f4f"},"logger":"org.redisson.connection.SentinelConnectionManager","message":"slave: redis://10.0.18.79:6379 is down"}
{"timestamp":"2022-01-11T12:12:30.281Z","level":"INFO","thread":"redisson-netty-2-17","mdc":{"instana.trace.id":"d13c89208fe60f4f"},"logger":"org.redisson.connection.SentinelConnectionManager","message":"slave: redis://10.0.18.79:6379 is up"}
{"timestamp":"2022-01-11T12:12:30.283Z","level":"INFO","thread":"redisson-netty-2-20","mdc":{"instana.trace.id":"d13c89208fe60f4f"},"logger":"org.redisson.connection.pool.PubSubConnectionPool","message":"1 connections initialized for 10.0.18.79/10.0.18.79:6379"}
{"timestamp":"2022-01-11T12:12:30.296Z","level":"INFO","thread":"redisson-netty-2-30","mdc":{"instana.trace.id":"d13c89208fe60f4f"},"logger":"org.redisson.connection.pool.SlaveConnectionPool","message":"24 connections initialized for 10.0.18.79/10.0.18.79:6379"}
{"timestamp":"2022-01-11T12:12:32.287Z","level":"WARN","thread":"redisson-netty-2-21","mdc":{"instana.trace.id":"d13c89208fe60f4f"},"logger":"org.redisson.connection.SentinelConnectionManager","message":"slave: redis://10.0.18.79:6379 is down"}
{"timestamp":"2022-01-11T12:12:34.290Z","level":"INFO","thread":"redisson-netty-2-11","mdc":{"instana.trace.id":"d13c89208fe60f4f"},"logger":"org.redisson.connection.SentinelConnectionManager","message":"slave: redis://10.0.18.79:6379 is up"}
{"timestamp":"2022-01-11T12:12:34.293Z","level":"INFO","thread":"redisson-netty-2-14","mdc":{"instana.trace.id":"d13c89208fe60f4f"},"logger":"org.redisson.connection.pool.PubSubConnectionPool","message":"1 connections initialized for 10.0.18.79/10.0.18.79:6379"}
{"timestamp":"2022-01-11T12:12:34.294Z","level":"INFO","thread":"redisson-netty-2-3","mdc":{"instana.trace.id":"d13c89208fe60f4f"},"logger":"org.redisson.connection.pool.SlaveConnectionPool","message":"24 connections initialized for 10.0.18.79/10.0.18.79:6379"}
{"timestamp":"2022-01-11T12:12:35.292Z","level":"WARN","thread":"redisson-netty-2-5","mdc":{"instana.trace.id":"d13c89208fe60f4f"},"logger":"org.redisson.connection.SentinelConnectionManager","message":"slave: redis://10.0.18.79:6379 is down"}
{"timestamp":"2022-01-11T12:12:39.299Z","level":"INFO","thread":"redisson-netty-2-9","mdc":{"instana.trace.id":"d13c89208fe60f4f"},"logger":"org.redisson.connection.SentinelConnectionManager","message":"slave: redis://10.0.18.79:6379 is up"}
{"timestamp":"2022-01-11T12:12:39.300Z","level":"INFO","thread":"redisson-netty-2-7","mdc":{"instana.trace.id":"d13c89208fe60f4f"},"logger":"org.redisson.connection.pool.PubSubConnectionPool","message":"1 connections initialized for 10.0.18.79/10.0.18.79:6379"}
{"timestamp":"2022-01-11T12:12:39.304Z","level":"INFO","thread":"redisson-netty-2-12","mdc":{"instana.trace.id":"d13c89208fe60f4f"},"logger":"org.redisson.connection.pool.SlaveConnectionPool","message":"24 connections initialized for 10.0.18.79/10.0.18.79:6379"}
{"timestamp":"2022-01-11T12:12:40.306Z","level":"WARN","thread":"redisson-netty-2-5","mdc":{"instana.trace.id":"d13c89208fe60f4f"},"logger":"org.redisson.connection.SentinelConnectionManager","message":"slave: redis://10.0.18.79:6379 is down"}
{"timestamp":"2022-01-11T12:12:42.310Z","level":"INFO","thread":"redisson-netty-2-21","mdc":{"instana.trace.id":"d13c89208fe60f4f"},"logger":"org.redisson.connection.SentinelConnectionManager","message":"slave: redis://10.0.18.79:6379 is up"}
{"timestamp":"2022-01-11T12:12:42.312Z","level":"INFO","thread":"redisson-netty-2-18","mdc":{"instana.trace.id":"d13c89208fe60f4f"},"logger":"org.redisson.connection.pool.PubSubConnectionPool","message":"1 connections initialized for 10.0.18.79/10.0.18.79:6379"}
{"timestamp":"2022-01-11T12:12:42.314Z","level":"INFO","thread":"redisson-netty-2-11","mdc":{"instana.trace.id":"d13c89208fe60f4f"},"logger":"org.redisson.connection.pool.SlaveConnectionPool","message":"24 connections initialized for 10.0.18.79/10.0.18.79:6379"}
{"timestamp":"2022-01-11T12:12:43.311Z","level":"WARN","thread":"redisson-netty-2-16","mdc":{"instana.trace.id":"d13c89208fe60f4f"},"logger":"org.redisson.connection.SentinelConnectionManager","message":"slave: redis://10.0.18.79:6379 is down"}
{"timestamp":"2022-01-11T12:12:48.320Z","level":"INFO","thread":"redisson-netty-2-16","mdc":{"instana.trace.id":"d13c89208fe60f4f"},"logger":"org.redisson.connection.SentinelConnectionManager","message":"slave: redis://10.0.18.79:6379 is up"}
{"timestamp":"2022-01-11T12:12:48.322Z","level":"INFO","thread":"redisson-netty-2-10","mdc":{"instana.trace.id":"d13c89208fe60f4f"},"logger":"org.redisson.connection.pool.PubSubConnectionPool","message":"1 connections initialized for 10.0.18.79/10.0.18.79:6379"}
{"timestamp":"2022-01-11T12:12:48.328Z","level":"INFO","thread":"redisson-netty-2-30","mdc":{"instana.trace.id":"d13c89208fe60f4f"},"logger":"org.redisson.connection.pool.SlaveConnectionPool","message":"24 connections initialized for 10.0.18.79/10.0.18.79:6379"}
{"timestamp":"2022-01-11T12:12:49.322Z","level":"WARN","thread":"redisson-netty-2-14","mdc":{"instana.trace.id":"d13c89208fe60f4f"},"logger":"org.redisson.connection.SentinelConnectionManager","message":"slave: redis://10.0.18.79:6379 is down"}
{"timestamp":"2022-01-11T12:12:50.324Z","level":"INFO","thread":"redisson-netty-2-26","mdc":{"instana.trace.id":"d13c89208fe60f4f"},"logger":"org.redisson.connection.SentinelConnectionManager","message":"slave: redis://10.0.18.79:6379 is up"}
{"timestamp":"2022-01-11T12:12:50.326Z","level":"INFO","thread":"redisson-netty-2-23","mdc":{"instana.trace.id":"d13c89208fe60f4f"},"logger":"org.redisson.connection.pool.PubSubConnectionPool","message":"1 connections initialized for 10.0.18.79/10.0.18.79:6379"}
{"timestamp":"2022-01-11T12:12:50.332Z","level":"INFO","thread":"redisson-netty-2-15","mdc":{"instana.trace.id":"d13c89208fe60f4f"},"logger":"org.redisson.connection.pool.SlaveConnectionPool","message":"24 connections initialized for 10.0.18.79/10.0.18.79:6379"}
{"timestamp":"2022-01-11T12:12:51.326Z","level":"WARN","thread":"redisson-netty-2-18","mdc":{"instana.trace.id":"d13c89208fe60f4f"},"logger":"org.redisson.connection.SentinelConnectionManager","message":"slave: redis://10.0.18.79:6379 is down"}
{"timestamp":"2022-01-11T12:12:52.328Z","level":"INFO","thread":"redisson-netty-2-2","mdc":{"instana.trace.id":"d13c89208fe60f4f"},"logger":"org.redisson.connection.SentinelConnectionManager","message":"slave: redis://10.0.18.79:6379 is up"}
{"timestamp":"2022-01-11T12:12:52.330Z","level":"INFO","thread":"redisson-netty-2-2","mdc":{"instana.trace.id":"d13c89208fe60f4f"},"logger":"org.redisson.connection.pool.PubSubConnectionPool","message":"1 connections initialized for 10.0.18.79/10.0.18.79:6379"}
{"timestamp":"2022-01-11T12:12:52.332Z","level":"INFO","thread":"redisson-netty-2-18","mdc":{"instana.trace.id":"d13c89208fe60f4f"},"logger":"org.redisson.connection.pool.SlaveConnectionPool","message":"24 connections initialized for 10.0.18.79/10.0.18.79:6379"}
{"timestamp":"2022-01-11T12:12:54.332Z","level":"WARN","thread":"redisson-netty-2-2","mdc":{"instana.trace.id":"d13c89208fe60f4f"},"logger":"org.redisson.connection.SentinelConnectionManager","message":"slave: redis://10.0.18.79:6379 is down"}
{"timestamp":"2022-01-11T12:12:55.335Z","level":"INFO","thread":"redisson-netty-2-10","mdc":{"instana.trace.id":"d13c89208fe60f4f"},"logger":"org.redisson.connection.SentinelConnectionManager","message":"slave: redis://10.0.18.79:6379 is up"}
{"timestamp":"2022-01-11T12:12:55.337Z","level":"INFO","thread":"redisson-netty-2-8","mdc":{"instana.trace.id":"d13c89208fe60f4f"},"logger":"org.redisson.connection.pool.PubSubConnectionPool","message":"1 connections initialized for 10.0.18.79/10.0.18.79:6379"}
{"timestamp":"2022-01-11T12:12:55.340Z","level":"INFO","thread":"redisson-netty-2-12","mdc":{"instana.trace.id":"d13c89208fe60f4f"},"logger":"org.redisson.connection.pool.SlaveConnectionPool","message":"24 connections initialized for 10.0.18.79/10.0.18.79:6379"}
{"timestamp":"2022-01-11T12:12:56.336Z","level":"WARN","thread":"redisson-netty-2-5","mdc":{"instana.trace.id":"d13c89208fe60f4f"},"logger":"org.redisson.connection.SentinelConnectionManager","message":"slave: redis://10.0.18.79:6379 is down"}
{"timestamp":"2022-01-11T12:13:00.344Z","level":"INFO","thread":"redisson-netty-2-9","mdc":{"instana.trace.id":"d13c89208fe60f4f"},"logger":"org.redisson.connection.SentinelConnectionManager","message":"slave: redis://10.0.18.79:6379 is up"}
{"timestamp":"2022-01-11T12:13:00.345Z","level":"INFO","thread":"redisson-netty-2-12","mdc":{"instana.trace.id":"d13c89208fe60f4f"},"logger":"org.redisson.connection.pool.PubSubConnectionPool","message":"1 connections initialized for 10.0.18.79/10.0.18.79:6379"}
{"timestamp":"2022-01-11T12:13:00.349Z","level":"INFO","thread":"redisson-netty-2-1","mdc":{"instana.trace.id":"d13c89208fe60f4f"},"logger":"org.redisson.connection.pool.SlaveConnectionPool","message":"24 connections initialized for 10.0.18.79/10.0.18.79:6379"}
{"timestamp":"2022-01-11T12:13:02.348Z","level":"WARN","thread":"redisson-netty-2-11","mdc":{"instana.trace.id":"d13c89208fe60f4f"},"logger":"org.redisson.connection.SentinelConnectionManager","message":"slave: redis://10.0.18.79:6379 is down"}
{"timestamp":"2022-01-11T12:13:04.351Z","level":"INFO","thread":"redisson-netty-2-5","mdc":{"instana.trace.id":"d13c89208fe60f4f"},"logger":"org.redisson.connection.SentinelConnectionManager","message":"slave: redis://10.0.18.79:6379 is up"}
{"timestamp":"2022-01-11T12:13:04.353Z","level":"INFO","thread":"redisson-netty-2-24","mdc":{"instana.trace.id":"d13c89208fe60f4f"},"logger":"org.redisson.connection.pool.PubSubConnectionPool","message":"1 connections initialized for 10.0.18.79/10.0.18.79:6379"}
{"timestamp":"2022-01-11T12:13:04.355Z","level":"INFO","thread":"redisson-netty-2-22","mdc":{"instana.trace.id":"d13c89208fe60f4f"},"logger":"org.redisson.connection.pool.SlaveConnectionPool","message":"24 connections initialized for 10.0.18.79/10.0.18.79:6379"}
{"timestamp":"2022-01-11T12:13:05.353Z","level":"WARN","thread":"redisson-netty-2-1","mdc":{"instana.trace.id":"d13c89208fe60f4f"},"logger":"org.redisson.connection.SentinelConnectionManager","message":"slave: redis://10.0.18.79:6379 is down"}
{"timestamp":"2022-01-11T12:13:07.357Z","level":"INFO","thread":"redisson-netty-2-5","mdc":{"instana.trace.id":"d13c89208fe60f4f"},"logger":"org.redisson.connection.SentinelConnectionManager","message":"slave: redis://10.0.18.79:6379 is up"}
{"timestamp":"2022-01-11T12:13:07.359Z","level":"INFO","thread":"redisson-netty-2-2","mdc":{"instana.trace.id":"d13c89208fe60f4f"},"logger":"org.redisson.connection.pool.PubSubConnectionPool","message":"1 connections initialized for 10.0.18.79/10.0.18.79:6379"}
{"timestamp":"2022-01-11T12:13:07.362Z","level":"INFO","thread":"redisson-netty-2-22","mdc":{"instana.trace.id":"d13c89208fe60f4f"},"logger":"org.redisson.connection.pool.SlaveConnectionPool","message":"24 connections initialized for 10.0.18.79/10.0.18.79:6379"}
{"timestamp":"2022-01-11T12:13:08.360Z","level":"WARN","thread":"redisson-netty-2-26","mdc":{"instana.trace.id":"d13c89208fe60f4f"},"logger":"org.redisson.connection.SentinelConnectionManager","message":"slave: redis://10.0.18.79:6379 is down"}
{"timestamp":"2022-01-11T12:13:09.361Z","level":"INFO","thread":"redisson-netty-2-30","mdc":{"instana.trace.id":"d13c89208fe60f4f"},"logger":"org.redisson.connection.SentinelConnectionManager","message":"slave: redis://10.0.18.79:6379 is up"}
{"timestamp":"2022-01-11T12:13:09.362Z","level":"INFO","thread":"redisson-netty-2-4","mdc":{"instana.trace.id":"d13c89208fe60f4f"},"logger":"org.redisson.connection.pool.PubSubConnectionPool","message":"1 connections initialized for 10.0.18.79/10.0.18.79:6379"}
{"timestamp":"2022-01-11T12:13:09.366Z","level":"INFO","thread":"redisson-netty-2-27","mdc":{"instana.trace.id":"d13c89208fe60f4f"},"logger":"org.redisson.connection.pool.SlaveConnectionPool","message":"24 connections initialized for 10.0.18.79/10.0.18.79:6379"}
{"timestamp":"2022-01-11T12:13:10.364Z","level":"WARN","thread":"redisson-netty-2-12","mdc":{"instana.trace.id":"d13c89208fe60f4f"},"logger":"org.redisson.connection.SentinelConnectionManager","message":"slave: redis://10.0.18.79:6379 is down"}
{"timestamp":"2022-01-11T12:13:12.368Z","level":"INFO","thread":"redisson-netty-2-17","mdc":{"instana.trace.id":"d13c89208fe60f4f"},"logger":"org.redisson.connection.SentinelConnectionManager","message":"slave: redis://10.0.18.79:6379 is up"}
{"timestamp":"2022-01-11T12:13:12.369Z","level":"INFO","thread":"redisson-netty-2-21","mdc":{"instana.trace.id":"d13c89208fe60f4f"},"logger":"org.redisson.connection.pool.PubSubConnectionPool","message":"1 connections initialized for 10.0.18.79/10.0.18.79:6379"}
{"timestamp":"2022-01-11T12:13:12.371Z","level":"INFO","thread":"redisson-netty-2-8","mdc":{"instana.trace.id":"d13c89208fe60f4f"},"logger":"org.redisson.connection.pool.SlaveConnectionPool","message":"24 connections initialized for 10.0.18.79/10.0.18.79:6379"}
{"timestamp":"2022-01-11T12:13:15.373Z","level":"WARN","thread":"redisson-netty-2-27","mdc":{"instana.trace.id":"d13c89208fe60f4f"},"logger":"org.redisson.connection.SentinelConnectionManager","message":"slave: redis://10.0.18.79:6379 is down"}
{"timestamp":"2022-01-11T12:13:17.376Z","level":"INFO","thread":"redisson-netty-2-14","mdc":{"instana.trace.id":"d13c89208fe60f4f"},"logger":"org.redisson.connection.SentinelConnectionManager","message":"slave: redis://10.0.18.79:6379 is up"}
{"timestamp":"2022-01-11T12:13:17.378Z","level":"INFO","thread":"redisson-netty-2-16","mdc":{"instana.trace.id":"d13c89208fe60f4f"},"logger":"org.redisson.connection.pool.PubSubConnectionPool","message":"1 connections initialized for 10.0.18.79/10.0.18.79:6379"}
{"timestamp":"2022-01-11T12:13:17.381Z","level":"INFO","thread":"redisson-netty-2-29","mdc":{"instana.trace.id":"d13c89208fe60f4f"},"logger":"org.redisson.connection.pool.SlaveConnectionPool","message":"24 connections initialized for 10.0.18.79/10.0.18.79:6379"}
{"timestamp":"2022-01-11T12:13:18.379Z","level":"WARN","thread":"redisson-netty-2-8","mdc":{"instana.trace.id":"d13c89208fe60f4f"},"logger":"org.redisson.connection.SentinelConnectionManager","message":"slave: redis://10.0.18.79:6379 is down"}
{"timestamp":"2022-01-11T12:13:20.383Z","level":"INFO","thread":"redisson-netty-2-3","mdc":{"instana.trace.id":"d13c89208fe60f4f"},"logger":"org.redisson.connection.SentinelConnectionManager","message":"slave: redis://10.0.18.79:6379 is up"}
{"timestamp":"2022-01-11T12:13:20.393Z","level":"INFO","thread":"redisson-netty-2-17","mdc":{"instana.trace.id":"d13c89208fe60f4f"},"logger":"org.redisson.connection.pool.SlaveConnectionPool","message":"24 connections initialized for 10.0.18.79/10.0.18.79:6379"}
{"timestamp":"2022-01-11T12:13:20.393Z","level":"INFO","thread":"redisson-netty-2-20","mdc":{"instana.trace.id":"d13c89208fe60f4f"},"logger":"org.redisson.connection.pool.PubSubConnectionPool","message":"1 connections initialized for 10.0.18.79/10.0.18.79:6379"}
{"timestamp":"2022-01-11T12:13:21.385Z","level":"WARN","thread":"redisson-netty-2-24","mdc":{"instana.trace.id":"d13c89208fe60f4f"},"logger":"org.redisson.connection.SentinelConnectionManager","message":"slave: redis://10.0.18.79:6379 is down"}
{"timestamp":"2022-01-11T12:13:22.388Z","level":"INFO","thread":"redisson-netty-2-6","mdc":{"instana.trace.id":"d13c89208fe60f4f"},"logger":"org.redisson.connection.SentinelConnectionManager","message":"slave: redis://10.0.18.79:6379 is up"}
{"timestamp":"2022-01-11T12:13:22.389Z","level":"INFO","thread":"redisson-netty-2-27","mdc":{"instana.trace.id":"d13c89208fe60f4f"},"logger":"org.redisson.connection.pool.PubSubConnectionPool","message":"1 connections initialized for 10.0.18.79/10.0.18.79:6379"}
{"timestamp":"2022-01-11T12:13:22.406Z","level":"INFO","thread":"redisson-netty-2-24","mdc":{"instana.trace.id":"d13c89208fe60f4f"},"logger":"org.redisson.connection.pool.SlaveConnectionPool","message":"24 connections initialized for 10.0.18.79/10.0.18.79:6379"}
{"timestamp":"2022-01-11T12:13:24.390Z","level":"WARN","thread":"redisson-netty-2-17","mdc":{"instana.trace.id":"d13c89208fe60f4f"},"logger":"org.redisson.connection.SentinelConnectionManager","message":"slave: redis://10.0.18.79:6379 is down"}
{"timestamp":"2022-01-11T12:13:28.397Z","level":"INFO","thread":"redisson-netty-2-3","mdc":{"instana.trace.id":"d13c89208fe60f4f"},"logger":"org.redisson.connection.SentinelConnectionManager","message":"slave: redis://10.0.18.79:6379 is up"}
{"timestamp":"2022-01-11T12:13:28.399Z","level":"INFO","thread":"redisson-netty-2-16","mdc":{"instana.trace.id":"d13c89208fe60f4f"},"logger":"org.redisson.connection.pool.PubSubConnectionPool","message":"1 connections initialized for 10.0.18.79/10.0.18.79:6379"}
{"timestamp":"2022-01-11T12:13:28.401Z","level":"INFO","thread":"redisson-netty-2-19","mdc":{"instana.trace.id":"d13c89208fe60f4f"},"logger":"org.redisson.connection.pool.SlaveConnectionPool","message":"24 connections initialized for 10.0.18.79/10.0.18.79:6379"}
{"timestamp":"2022-01-11T12:13:29.402Z","level":"WARN","thread":"redisson-netty-2-24","mdc":{"instana.trace.id":"d13c89208fe60f4f"},"logger":"org.redisson.connection.SentinelConnectionManager","message":"slave: redis://10.0.18.79:6379 is down"}
{"timestamp":"2022-01-11T12:13:32.404Z","level":"INFO","thread":"redisson-netty-2-21","mdc":{"instana.trace.id":"d13c89208fe60f4f"},"logger":"org.redisson.connection.SentinelConnectionManager","message":"slave: redis://10.0.18.79:6379 is up"}
{"timestamp":"2022-01-11T12:13:32.406Z","level":"INFO","thread":"redisson-netty-2-17","mdc":{"instana.trace.id":"d13c89208fe60f4f"},"logger":"org.redisson.connection.pool.PubSubConnectionPool","message":"1 connections initialized for 10.0.18.79/10.0.18.79:6379"}
{"timestamp":"2022-01-11T12:13:32.409Z","level":"INFO","thread":"redisson-netty-2-29","mdc":{"instana.trace.id":"d13c89208fe60f4f"},"logger":"org.redisson.connection.pool.SlaveConnectionPool","message":"24 connections initialized for 10.0.18.79/10.0.18.79:6379"}
{"timestamp":"2022-01-11T12:13:34.408Z","level":"WARN","thread":"redisson-netty-2-21","mdc":{"instana.trace.id":"d13c89208fe60f4f"},"logger":"org.redisson.connection.SentinelConnectionManager","message":"slave: redis://10.0.18.79:6379 is down"}
{"timestamp":"2022-01-11T12:13:38.415Z","level":"INFO","thread":"redisson-netty-2-3","mdc":{"instana.trace.id":"d13c89208fe60f4f"},"logger":"org.redisson.connection.SentinelConnectionManager","message":"slave: redis://10.0.18.79:6379 is up"}
{"timestamp":"2022-01-11T12:13:38.417Z","level":"INFO","thread":"redisson-netty-2-1","mdc":{"instana.trace.id":"d13c89208fe60f4f"},"logger":"org.redisson.connection.pool.PubSubConnectionPool","message":"1 connections initialized for 10.0.18.79/10.0.18.79:6379"}
{"timestamp":"2022-01-11T12:13:38.419Z","level":"INFO","thread":"redisson-netty-2-20","mdc":{"instana.trace.id":"d13c89208fe60f4f"},"logger":"org.redisson.connection.pool.SlaveConnectionPool","message":"24 connections initialized for 10.0.18.79/10.0.18.79:6379"}
{"timestamp":"2022-01-11T12:13:39.417Z","level":"WARN","thread":"redisson-netty-2-24","mdc":{"instana.trace.id":"d13c89208fe60f4f"},"logger":"org.redisson.connection.SentinelConnectionManager","message":"slave: redis://10.0.18.79:6379 is down"}
{"timestamp":"2022-01-11T12:13:45.426Z","level":"INFO","thread":"redisson-netty-2-8","mdc":{"instana.trace.id":"d13c89208fe60f4f"},"logger":"org.redisson.connection.SentinelConnectionManager","message":"slave: redis://10.0.18.79:6379 is up"}
{"timestamp":"2022-01-11T12:13:45.428Z","level":"INFO","thread":"redisson-netty-2-9","mdc":{"instana.trace.id":"d13c89208fe60f4f"},"logger":"org.redisson.connection.pool.PubSubConnectionPool","message":"1 connections initialized for 10.0.18.79/10.0.18.79:6379"}
{"timestamp":"2022-01-11T12:13:45.431Z","level":"INFO","thread":"redisson-netty-2-1","mdc":{"instana.trace.id":"d13c89208fe60f4f"},"logger":"org.redisson.connection.pool.SlaveConnectionPool","message":"24 connections initialized for 10.0.18.79/10.0.18.79:6379"}
{"timestamp":"2022-01-11T12:13:48.434Z","level":"WARN","thread":"redisson-netty-2-15","mdc":{"instana.trace.id":"d13c89208fe60f4f"},"logger":"org.redisson.connection.SentinelConnectionManager","message":"slave: redis://10.0.18.79:6379 is down"}
{"timestamp":"2022-01-11T12:13:50.437Z","level":"INFO","thread":"redisson-netty-2-29","mdc":{"instana.trace.id":"d13c89208fe60f4f"},"logger":"org.redisson.connection.SentinelConnectionManager","message":"slave: redis://10.0.18.79:6379 is up"}
{"timestamp":"2022-01-11T12:13:50.438Z","level":"INFO","thread":"redisson-netty-2-28","mdc":{"instana.trace.id":"d13c89208fe60f4f"},"logger":"org.redisson.connection.pool.PubSubConnectionPool","message":"1 connections initialized for 10.0.18.79/10.0.18.79:6379"}
{"timestamp":"2022-01-11T12:13:50.440Z","level":"INFO","thread":"redisson-netty-2-25","mdc":{"instana.trace.id":"d13c89208fe60f4f"},"logger":"org.redisson.connection.pool.SlaveConnectionPool","message":"24 connections initialized for 10.0.18.79/10.0.18.79:6379"}

These are the logs of the redis master

############### REDIS

1:S 11 Jan 2022 12:10:10.018 * Residual parent diff successfully flushed to the rewritten AOF (0.00 MB)
1:S 11 Jan 2022 12:10:10.018 * Background AOF rewrite finished successfully
1:S 11 Jan 2022 12:10:14.230 # Connection with master lost.
1:S 11 Jan 2022 12:10:14.230 * Caching the disconnected master state.
1:S 11 Jan 2022 12:10:14.230 * Reconnecting to MASTER 10.0.16.208:6379
1:S 11 Jan 2022 12:10:14.231 * MASTER <-> REPLICA sync started
1:S 11 Jan 2022 12:10:14.232 * Non blocking connect for SYNC fired the event.
1:S 11 Jan 2022 12:10:14.242 * Master replied to PING, replication can continue...
1:S 11 Jan 2022 12:10:14.242 * Trying a partial resynchronization (request 296188a43284827261ed669624337a0163f8e3c7:968336907).
1:S 11 Jan 2022 12:10:14.242 * Master is currently unable to PSYNC but should be in the future: -NOMASTERLINK Can't SYNC while not connected with my master
1:S 11 Jan 2022 12:10:15.043 * Connecting to MASTER 10.0.16.208:6379
1:S 11 Jan 2022 12:10:15.043 * MASTER <-> REPLICA sync started
1:S 11 Jan 2022 12:10:15.044 * Non blocking connect for SYNC fired the event.
1:S 11 Jan 2022 12:10:15.045 * Master replied to PING, replication can continue...
1:S 11 Jan 2022 12:10:15.045 * Trying a partial resynchronization (request 296188a43284827261ed669624337a0163f8e3c7:968336907).
1:S 11 Jan 2022 12:10:15.045 * Master is currently unable to PSYNC but should be in the future: -NOMASTERLINK Can't SYNC while not connected with my master
1:S 11 Jan 2022 12:10:16.046 * Connecting to MASTER 10.0.16.208:6379
1:S 11 Jan 2022 12:10:16.046 * MASTER <-> REPLICA sync started
1:S 11 Jan 2022 12:10:16.047 * Non blocking connect for SYNC fired the event.
1:S 11 Jan 2022 12:10:16.047 * Master replied to PING, replication can continue...
1:S 11 Jan 2022 12:10:16.048 * Trying a partial resynchronization (request 296188a43284827261ed669624337a0163f8e3c7:968336907).
1:S 11 Jan 2022 12:10:16.048 * Master is currently unable to PSYNC but should be in the future: -NOMASTERLINK Can't SYNC while not connected with my master
1:S 11 Jan 2022 12:10:17.052 * Connecting to MASTER 10.0.16.208:6379
1:S 11 Jan 2022 12:10:17.052 * MASTER <-> REPLICA sync started
1:S 11 Jan 2022 12:10:17.053 * Non blocking connect for SYNC fired the event.
1:S 11 Jan 2022 12:10:17.053 * Master replied to PING, replication can continue...
1:S 11 Jan 2022 12:10:17.054 * Trying a partial resynchronization (request 296188a43284827261ed669624337a0163f8e3c7:968336907).
1:S 11 Jan 2022 12:10:17.054 * Master is currently unable to PSYNC but should be in the future: -NOMASTERLINK Can't SYNC while not connected with my master
1:S 11 Jan 2022 12:10:18.057 * Connecting to MASTER 10.0.16.208:6379
1:S 11 Jan 2022 12:10:18.057 * MASTER <-> REPLICA sync started
1:S 11 Jan 2022 12:10:18.058 * Non blocking connect for SYNC fired the event.
1:S 11 Jan 2022 12:10:18.058 * Master replied to PING, replication can continue...
1:S 11 Jan 2022 12:10:18.058 * Trying a partial resynchronization (request 296188a43284827261ed669624337a0163f8e3c7:968336907).
1:S 11 Jan 2022 12:10:18.059 * Master is currently unable to PSYNC but should be in the future: -NOMASTERLINK Can't SYNC while not connected with my master
1:S 11 Jan 2022 12:10:19.063 * Connecting to MASTER 10.0.16.208:6379
1:S 11 Jan 2022 12:10:19.063 * MASTER <-> REPLICA sync started
1:S 11 Jan 2022 12:10:19.064 * Non blocking connect for SYNC fired the event.
1:S 11 Jan 2022 12:10:19.064 * Master replied to PING, replication can continue...
1:S 11 Jan 2022 12:10:19.064 * Trying a partial resynchronization (request 296188a43284827261ed669624337a0163f8e3c7:968336907).
1:S 11 Jan 2022 12:10:19.065 * Master is currently unable to PSYNC but should be in the future: -NOMASTERLINK Can't SYNC while not connected with my master
1:S 11 Jan 2022 12:10:20.068 * Connecting to MASTER 10.0.16.208:6379
1:S 11 Jan 2022 12:10:20.075 * MASTER <-> REPLICA sync started
1:S 11 Jan 2022 12:10:20.076 * Non blocking connect for SYNC fired the event.
1:S 11 Jan 2022 12:10:20.077 * Master replied to PING, replication can continue...
1:S 11 Jan 2022 12:10:20.077 * Trying a partial resynchronization (request 296188a43284827261ed669624337a0163f8e3c7:968336907).
1:S 11 Jan 2022 12:10:20.077 * Master is currently unable to PSYNC but should be in the future: -NOMASTERLINK Can't SYNC while not connected with my master
1:S 11 Jan 2022 12:10:21.080 * Connecting to MASTER 10.0.16.208:6379
1:S 11 Jan 2022 12:10:21.080 * MASTER <-> REPLICA sync started
1:S 11 Jan 2022 12:10:21.082 * Non blocking connect for SYNC fired the event.
1:S 11 Jan 2022 12:10:21.082 * Master replied to PING, replication can continue...
1:S 11 Jan 2022 12:10:21.082 * Trying a partial resynchronization (request 296188a43284827261ed669624337a0163f8e3c7:968336907).
1:S 11 Jan 2022 12:10:21.082 * Master is currently unable to PSYNC but should be in the future: -NOMASTERLINK Can't SYNC while not connected with my master
1:S 11 Jan 2022 12:10:22.084 * Connecting to MASTER 10.0.16.208:6379
1:S 11 Jan 2022 12:10:22.084 * MASTER <-> REPLICA sync started
1:S 11 Jan 2022 12:10:22.086 * Non blocking connect for SYNC fired the event.
1:S 11 Jan 2022 12:10:22.086 * Master replied to PING, replication can continue...
1:S 11 Jan 2022 12:10:22.086 * Trying a partial resynchronization (request 296188a43284827261ed669624337a0163f8e3c7:968336907).
1:S 11 Jan 2022 12:10:22.086 * Master is currently unable to PSYNC but should be in the future: -NOMASTERLINK Can't SYNC while not connected with my master
1:S 11 Jan 2022 12:10:23.088 * Connecting to MASTER 10.0.16.208:6379
1:S 11 Jan 2022 12:10:23.088 * MASTER <-> REPLICA sync started
1:S 11 Jan 2022 12:10:23.090 * Non blocking connect for SYNC fired the event.
1:S 11 Jan 2022 12:10:23.090 * Master replied to PING, replication can continue...
1:S 11 Jan 2022 12:10:23.090 * Trying a partial resynchronization (request 296188a43284827261ed669624337a0163f8e3c7:968336907).
1:S 11 Jan 2022 12:10:23.090 * Master is currently unable to PSYNC but should be in the future: -NOMASTERLINK Can't SYNC while not connected with my master
1:S 11 Jan 2022 12:10:24.093 * Connecting to MASTER 10.0.16.208:6379
1:S 11 Jan 2022 12:10:24.093 * MASTER <-> REPLICA sync started
1:S 11 Jan 2022 12:10:24.095 * Non blocking connect for SYNC fired the event.
1:S 11 Jan 2022 12:10:24.095 * Master replied to PING, replication can continue...
1:S 11 Jan 2022 12:10:24.095 * Trying a partial resynchronization (request 296188a43284827261ed669624337a0163f8e3c7:968336907).
1:S 11 Jan 2022 12:10:24.095 * Master is currently unable to PSYNC but should be in the future: -NOMASTERLINK Can't SYNC while not connected with my master
1:S 11 Jan 2022 12:10:25.097 * Connecting to MASTER 10.0.16.208:6379
1:S 11 Jan 2022 12:10:25.097 * MASTER <-> REPLICA sync started
1:S 11 Jan 2022 12:10:25.098 * Non blocking connect for SYNC fired the event.
1:S 11 Jan 2022 12:10:25.098 * Master replied to PING, replication can continue...
1:S 11 Jan 2022 12:10:25.099 * Trying a partial resynchronization (request 296188a43284827261ed669624337a0163f8e3c7:968336907).
1:S 11 Jan 2022 12:10:25.099 * Master is currently unable to PSYNC but should be in the future: -NOMASTERLINK Can't SYNC while not connected with my master
1:S 11 Jan 2022 12:10:26.101 * Connecting to MASTER 10.0.16.208:6379
1:S 11 Jan 2022 12:10:26.101 * MASTER <-> REPLICA sync started
1:M 11 Jan 2022 12:10:36.553 * Discarding previously cached master state.
1:M 11 Jan 2022 12:10:36.553 # Setting secondary replication ID to 296188a43284827261ed669624337a0163f8e3c7, valid up to offset: 968336907. New replication ID is 6c278ab479412fe8192f7d174a9e875791d1c481
1:M 11 Jan 2022 12:10:36.553 * MASTER MODE enabled (user request from 'id=3 addr=10.0.16.106:38134 laddr=10.0.16.106:6379 fd=11 name=sentinel-9fe32540-cmd age=32 idle=0 flags=x db=0 sub=0 psub=0 multi=4 qbuf=202 qbuf-free=40752 argv-mem=4 obl=45 oll=0 omem=0 tot-mem=61468 events=r cmd=exec user=default redir=-1')
1:M 11 Jan 2022 12:10:37.231 * Replica 10.0.17.255:6379 asks for synchronization
1:M 11 Jan 2022 12:10:37.231 * Partial resynchronization request from 10.0.17.255:6379 accepted. Sending 479 bytes of backlog starting from offset 968336907.
1:M 11 Jan 2022 12:11:15.385 * Replica redis-node-1.redis-headless.redis-sentinel-main.svc.cluster.local:6379 asks for synchronization
1:M 11 Jan 2022 12:11:15.385 * Full resync requested by replica redis-node-1.redis-headless.redis-sentinel-main.svc.cluster.local:6379
1:M 11 Jan 2022 12:11:15.385 * Starting BGSAVE for SYNC with target: disk
1:M 11 Jan 2022 12:11:15.390 * Background saving started by pid 165
165:C 11 Jan 2022 12:11:17.943 * DB saved on disk
165:C 11 Jan 2022 12:11:17.949 * RDB: 7 MB of memory used by copy-on-write
1:M 11 Jan 2022 12:11:18.014 * Background saving terminated with success
1:M 11 Jan 2022 12:11:18.864 * Synchronization with replica redis-node-1.redis-headless.redis-sentinel-main.svc.cluster.local:6379 succeeded
1:M 11 Jan 2022 12:11:39.900 # Connection with replica 10.0.17.255:6379 lost.
1:M 11 Jan 2022 12:11:51.851 * Replica redis-node-0.redis-headless.redis-sentinel-main.svc.cluster.local:6379 asks for synchronization
1:M 11 Jan 2022 12:11:51.851 * Full resync requested by replica redis-node-0.redis-headless.redis-sentinel-main.svc.cluster.local:6379
1:M 11 Jan 2022 12:11:51.851 * Starting BGSAVE for SYNC with target: disk
1:M 11 Jan 2022 12:11:51.855 * Background saving started by pid 285
285:C 11 Jan 2022 12:11:54.350 * DB saved on disk
285:C 11 Jan 2022 12:11:54.358 * RDB: 5 MB of memory used by copy-on-write
1:M 11 Jan 2022 12:11:54.405 * Background saving terminated with success
1:M 11 Jan 2022 12:11:55.366 * Synchronization with replica redis-node-0.redis-headless.redis-sentinel-main.svc.cluster.local:6379 succeeded

############### SENTINEL

 12:10:03.79 INFO  ==> about to run the command: timeout 5 redis-cli -h redis.redis-sentinel-main.svc.cluster.local -p 26379 sentinel get-master-addr-by-name redis-master-main
10.0.16.208
6379
 12:10:03.81 INFO  ==> about to run the command: timeout 5 redis-cli -h redis.redis-sentinel-main.svc.cluster.local -p 26379 sentinel get-master-addr-by-name redis-master-main
 12:10:03.89 INFO  ==> printing REDIS_SENTINEL_INFO=(10.0.16.208,6379)
1:X 11 Jan 2022 12:10:04.207 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:X 11 Jan 2022 12:10:04.207 # Redis version=6.2.6, bits=64, commit=00000000, modified=0, pid=1, just started
1:X 11 Jan 2022 12:10:04.207 # Configuration loaded
1:X 11 Jan 2022 12:10:04.208 * monotonic clock: POSIX clock_gettime
1:X 11 Jan 2022 12:10:04.208 * Running mode=sentinel, port=26379.
1:X 11 Jan 2022 12:10:04.209 # Sentinel ID is 9fe32540b27937ed9f341b0f610a0d8df405bb63
1:X 11 Jan 2022 12:10:04.209 # +monitor master redis-master-main 10.0.16.208 6379 quorum 2
1:X 11 Jan 2022 12:10:04.211 * +slave slave 10.0.17.255:6379 10.0.17.255 6379 @ redis-master-main 10.0.16.208 6379
1:X 11 Jan 2022 12:10:04.295 # +new-epoch 6
1:X 11 Jan 2022 12:10:14.230 * +convert-to-slave slave redis-node-1.redis-headless.redis-sentinel-main.svc.cluster.local:6379 redis-node-1.redis-headless.redis-sentinel-main.svc.cluster.local 6379 @ redis-master-main 10.0.16.208 6379
1:X 11 Jan 2022 12:10:35.174 # +sdown master redis-master-main 10.0.16.208 6379
1:X 11 Jan 2022 12:10:35.174 # +sdown slave redis-node-1.redis-headless.redis-sentinel-main.svc.cluster.local:6379 redis-node-1.redis-headless.redis-sentinel-main.svc.cluster.local 6379 @ redis-master-main 10.0.16.208 6379
1:X 11 Jan 2022 12:10:35.265 # +sdown sentinel 33535e4e17bf8f9f9ff9ce8f9ddf609e558ff4f2 redis-node-1.redis-headless.redis-sentinel-main.svc.cluster.local 26379 @ redis-master-main 10.0.16.208 6379
1:X 11 Jan 2022 12:10:36.307 # +odown master redis-master-main 10.0.16.208 6379 #quorum 2/2
1:X 11 Jan 2022 12:10:36.307 # +new-epoch 7
1:X 11 Jan 2022 12:10:36.307 # +try-failover master redis-master-main 10.0.16.208 6379
1:X 11 Jan 2022 12:10:36.349 # +vote-for-leader 9fe32540b27937ed9f341b0f610a0d8df405bb63 7
1:X 11 Jan 2022 12:10:36.391 # 2a09ba7abbb41ee71e79087310d75f9809c3c815 voted for 9fe32540b27937ed9f341b0f610a0d8df405bb63 7
1:X 11 Jan 2022 12:10:36.426 # +elected-leader master redis-master-main 10.0.16.208 6379
1:X 11 Jan 2022 12:10:36.426 # +failover-state-select-slave master redis-master-main 10.0.16.208 6379
1:X 11 Jan 2022 12:10:36.497 # +selected-slave slave redis-node-2.redis-headless.redis-sentinel-main.svc.cluster.local:6379 redis-node-2.redis-headless.redis-sentinel-main.svc.cluster.local 6379 @ redis-master-main 10.0.16.208 6379
1:X 11 Jan 2022 12:10:36.497 * +failover-state-send-slaveof-noone slave redis-node-2.redis-headless.redis-sentinel-main.svc.cluster.local:6379 redis-node-2.redis-headless.redis-sentinel-main.svc.cluster.local 6379 @ redis-master-main 10.0.16.208 6379
1:X 11 Jan 2022 12:10:36.553 * +failover-state-wait-promotion slave redis-node-2.redis-headless.redis-sentinel-main.svc.cluster.local:6379 redis-node-2.redis-headless.redis-sentinel-main.svc.cluster.local 6379 @ redis-master-main 10.0.16.208 6379
1:X 11 Jan 2022 12:10:37.184 # +promoted-slave slave redis-node-2.redis-headless.redis-sentinel-main.svc.cluster.local:6379 redis-node-2.redis-headless.redis-sentinel-main.svc.cluster.local 6379 @ redis-master-main 10.0.16.208 6379
1:X 11 Jan 2022 12:10:37.184 # +failover-state-reconf-slaves master redis-master-main 10.0.16.208 6379
1:X 11 Jan 2022 12:10:37.222 * +slave-reconf-sent slave redis-node-0.redis-headless.redis-sentinel-main.svc.cluster.local:6379 redis-node-0.redis-headless.redis-sentinel-main.svc.cluster.local 6379 @ redis-master-main 10.0.16.208 6379
1:X 11 Jan 2022 12:10:37.657 * +slave-reconf-inprog slave redis-node-0.redis-headless.redis-sentinel-main.svc.cluster.local:6379 redis-node-0.redis-headless.redis-sentinel-main.svc.cluster.local 6379 @ redis-master-main 10.0.16.208 6379
1:X 11 Jan 2022 12:10:37.657 * +slave-reconf-done slave redis-node-0.redis-headless.redis-sentinel-main.svc.cluster.local:6379 redis-node-0.redis-headless.redis-sentinel-main.svc.cluster.local 6379 @ redis-master-main 10.0.16.208 6379
1:X 11 Jan 2022 12:10:37.747 * +slave-reconf-sent slave 10.0.17.255:6379 10.0.17.255 6379 @ redis-master-main 10.0.16.208 6379
1:X 11 Jan 2022 12:10:38.316 * +slave-reconf-inprog slave 10.0.17.255:6379 10.0.17.255 6379 @ redis-master-main 10.0.16.208 6379
1:X 11 Jan 2022 12:10:38.316 * +slave-reconf-done slave 10.0.17.255:6379 10.0.17.255 6379 @ redis-master-main 10.0.16.208 6379
1:X 11 Jan 2022 12:10:38.369 * +slave-reconf-sent slave redis-node-1.redis-headless.redis-sentinel-main.svc.cluster.local:6379 redis-node-1.redis-headless.redis-sentinel-main.svc.cluster.local 6379 @ redis-master-main 10.0.16.208 6379
1:X 11 Jan 2022 12:10:38.369 # +failover-end master redis-master-main 10.0.16.208 6379
1:X 11 Jan 2022 12:10:38.369 # +switch-master redis-master-main 10.0.16.208 6379 redis-node-2.redis-headless.redis-sentinel-main.svc.cluster.local 6379
1:X 11 Jan 2022 12:10:38.370 * +slave slave redis-node-0.redis-headless.redis-sentinel-main.svc.cluster.local:6379 redis-node-0.redis-headless.redis-sentinel-main.svc.cluster.local 6379 @ redis-master-main redis-node-2.redis-headless.redis-sentinel-main.svc.cluster.local 6379
1:X 11 Jan 2022 12:10:38.370 * +slave slave 10.0.17.255:6379 10.0.17.255 6379 @ redis-master-main redis-node-2.redis-headless.redis-sentinel-main.svc.cluster.local 6379
1:X 11 Jan 2022 12:10:38.383 * +slave slave 10.0.16.208:6379 10.0.16.208 6379 @ redis-master-main redis-node-2.redis-headless.redis-sentinel-main.svc.cluster.local 6379
1:X 11 Jan 2022 12:10:48.392 # +sdown slave 10.0.16.208:6379 10.0.16.208 6379 @ redis-master-main redis-node-2.redis-headless.redis-sentinel-main.svc.cluster.local 6379
1:X 11 Jan 2022 12:11:18.451 * +sentinel-address-switch master redis-master-main redis-node-2.redis-headless.redis-sentinel-main.svc.cluster.local 6379 ip redis-node-1.redis-headless.redis-sentinel-main.svc.cluster.local port 26379 for 33535e4e17bf8f9f9ff9ce8f9ddf609e558ff4f2
1:X 11 Jan 2022 12:11:18.554 * +slave slave redis-node-1.redis-headless.redis-sentinel-main.svc.cluster.local:6379 redis-node-1.redis-headless.redis-sentinel-main.svc.cluster.local 6379 @ redis-master-main redis-node-2.redis-headless.redis-sentinel-main.svc.cluster.local 6379
1:X 11 Jan 2022 12:11:49.888 # +sdown sentinel 2a09ba7abbb41ee71e79087310d75f9809c3c815 redis-node-0.redis-headless.redis-sentinel-main.svc.cluster.local 26379 @ redis-master-main redis-node-2.redis-headless.redis-sentinel-main.svc.cluster.local 6379
1:X 11 Jan 2022 12:11:49.972 # +sdown slave redis-node-0.redis-headless.redis-sentinel-main.svc.cluster.local:6379 redis-node-0.redis-headless.redis-sentinel-main.svc.cluster.local 6379 @ redis-master-main redis-node-2.redis-headless.redis-sentinel-main.svc.cluster.local 6379
1:X 11 Jan 2022 12:11:49.972 # +sdown slave 10.0.17.255:6379 10.0.17.255 6379 @ redis-master-main redis-node-2.redis-headless.redis-sentinel-main.svc.cluster.local 6379
1:X 11 Jan 2022 12:11:55.901 * +sentinel-address-switch master redis-master-main redis-node-2.redis-headless.redis-sentinel-main.svc.cluster.local 6379 ip redis-node-0.redis-headless.redis-sentinel-main.svc.cluster.local port 26379 for 2a09ba7abbb41ee71e79087310d75f9809c3c815
1:X 11 Jan 2022 12:11:59.980 * +sentinel sentinel 2a09ba7abbb41ee71e79087310d75f9809c3c815 redis-node-0.redis-headless.redis-sentinel-main.svc.cluster.local 26379 @ redis-master-main redis-node-2.redis-headless.redis-sentinel-main.svc.cluster.local 6379

These are the logs of the redis slave 1

############### REDIS

 12:11:49.80 INFO  ==> about to run the command: redis-cli -h redis.redis-sentinel-main.svc.cluster.local -p 26379 sentinel get-master-addr-by-name redis-master-main
redis-node-2.redis-headless.redis-sentinel-main.svc.cluster.local
6379
 12:11:49.82 INFO  ==> about to run the command: redis-cli -h redis.redis-sentinel-main.svc.cluster.local -p 26379 sentinel get-master-addr-by-name redis-master-main
 12:11:49.89 INFO  ==> printing REDIS_SENTINEL_INFO=(redis-node-2.redis-headless.redis-sentinel-main.svc.cluster.local,6379)
1:C 11 Jan 2022 12:11:49.902 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:C 11 Jan 2022 12:11:49.902 # Redis version=6.2.6, bits=64, commit=00000000, modified=0, pid=1, just started
1:C 11 Jan 2022 12:11:49.902 # Configuration loaded
1:S 11 Jan 2022 12:11:49.903 * monotonic clock: POSIX clock_gettime
1:S 11 Jan 2022 12:11:49.903 * Running mode=standalone, port=6379.
1:S 11 Jan 2022 12:11:49.903 # Server initialized
1:S 11 Jan 2022 12:11:49.904 * Reading RDB preamble from AOF file...
1:S 11 Jan 2022 12:11:49.904 * Loading RDB produced by version 6.2.5
1:S 11 Jan 2022 12:11:49.904 * RDB age 3185 seconds
1:S 11 Jan 2022 12:11:49.904 * RDB memory usage when created 275.45 Mb
1:S 11 Jan 2022 12:11:49.904 * RDB has an AOF tail
1:S 11 Jan 2022 12:11:51.234 # Done loading RDB, keys loaded: 354, keys expired: 0.
1:S 11 Jan 2022 12:11:51.234 * Reading the remaining AOF tail...
1:S 11 Jan 2022 12:11:51.244 * DB loaded from append only file: 1.340 seconds
1:S 11 Jan 2022 12:11:51.244 * Ready to accept connections
1:S 11 Jan 2022 12:11:51.845 * Connecting to MASTER redis-node-2.redis-headless.redis-sentinel-main.svc.cluster.local:6379
1:S 11 Jan 2022 12:11:51.848 * MASTER <-> REPLICA sync started
1:S 11 Jan 2022 12:11:51.851 * Non blocking connect for SYNC fired the event.
1:S 11 Jan 2022 12:11:51.851 * Master replied to PING, replication can continue...
1:S 11 Jan 2022 12:11:51.851 * Partial resynchronization not possible (no cached master)
1:S 11 Jan 2022 12:11:51.856 * Full resync from master: 6c278ab479412fe8192f7d174a9e875791d1c481:968365337
1:S 11 Jan 2022 12:11:54.405 * MASTER <-> REPLICA sync: receiving 156284330 bytes from master to disk
1:S 11 Jan 2022 12:11:55.371 * MASTER <-> REPLICA sync: Flushing old data
1:S 11 Jan 2022 12:11:55.465 * MASTER <-> REPLICA sync: Loading DB in memory
1:S 11 Jan 2022 12:11:55.476 * Loading RDB produced by version 6.2.6
1:S 11 Jan 2022 12:11:55.476 * RDB age 4 seconds
1:S 11 Jan 2022 12:11:55.476 * RDB memory usage when created 285.54 Mb
1:S 11 Jan 2022 12:11:56.561 # Done loading RDB, keys loaded: 378, keys expired: 0.
1:S 11 Jan 2022 12:11:56.561 * MASTER <-> REPLICA sync: Finished with success
1:S 11 Jan 2022 12:11:56.567 * Background append only file rewriting started by pid 29
1:S 11 Jan 2022 12:11:58.987 * AOF rewrite child asks to stop sending diffs.
29:C 11 Jan 2022 12:11:58.987 * Parent agreed to stop sending diffs. Finalizing AOF...
29:C 11 Jan 2022 12:11:58.987 * Concatenating 0.01 MB of AOF diff received from parent.
29:C 11 Jan 2022 12:11:58.996 * SYNC append only file rewrite performed
29:C 11 Jan 2022 12:11:59.002 * AOF rewrite: 5 MB of memory used by copy-on-write
1:S 11 Jan 2022 12:11:59.076 * Background AOF rewrite terminated with success
1:S 11 Jan 2022 12:11:59.076 * Residual parent diff successfully flushed to the rewritten AOF (0.00 MB)
1:S 11 Jan 2022 12:11:59.076 * Background AOF rewrite finished successfully

############### SENTINEL

 12:11:53.34 INFO  ==> about to run the command: timeout 5 redis-cli -h redis.redis-sentinel-main.svc.cluster.local -p 26379 sentinel get-master-addr-by-name redis-master-main
redis-node-2.redis-headless.redis-sentinel-main.svc.cluster.local
6379
 12:11:53.44 INFO  ==> about to run the command: timeout 5 redis-cli -h redis.redis-sentinel-main.svc.cluster.local -p 26379 sentinel get-master-addr-by-name redis-master-main
 12:11:53.48 INFO  ==> printing REDIS_SENTINEL_INFO=(redis-node-2.redis-headless.redis-sentinel-main.svc.cluster.local,6379)
1:X 11 Jan 2022 12:11:53.838 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:X 11 Jan 2022 12:11:53.838 # Redis version=6.2.6, bits=64, commit=00000000, modified=0, pid=1, just started
1:X 11 Jan 2022 12:11:53.838 # Configuration loaded
1:X 11 Jan 2022 12:11:53.839 * monotonic clock: POSIX clock_gettime
1:X 11 Jan 2022 12:11:53.839 * Running mode=sentinel, port=26379.
1:X 11 Jan 2022 12:11:53.840 # Sentinel ID is 2a09ba7abbb41ee71e79087310d75f9809c3c815
1:X 11 Jan 2022 12:11:53.840 # +monitor master redis-master-main redis-node-2.redis-headless.redis-sentinel-main.svc.cluster.local 6379 quorum 2
1:X 11 Jan 2022 12:11:54.172 # +new-epoch 7

These are the logs of the redis slave 2

############### REDIS
 12:11:12.43 INFO  ==> about to run the command: redis-cli -h redis.redis-sentinel-main.svc.cluster.local -p 26379 sentinel get-master-addr-by-name redis-master-main
redis-node-2.redis-headless.redis-sentinel-main.svc.cluster.local
6379
 12:11:12.45 INFO  ==> about to run the command: redis-cli -h redis.redis-sentinel-main.svc.cluster.local -p 26379 sentinel get-master-addr-by-name redis-master-main
 12:11:12.46 INFO  ==> printing REDIS_SENTINEL_INFO=(redis-node-2.redis-headless.redis-sentinel-main.svc.cluster.local,6379)
1:C 11 Jan 2022 12:11:12.475 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:C 11 Jan 2022 12:11:12.475 # Redis version=6.2.6, bits=64, commit=00000000, modified=0, pid=1, just started
1:C 11 Jan 2022 12:11:12.475 # Configuration loaded
1:S 11 Jan 2022 12:11:12.475 * monotonic clock: POSIX clock_gettime
1:S 11 Jan 2022 12:11:12.476 * Running mode=standalone, port=6379.
1:S 11 Jan 2022 12:11:12.476 # Server initialized
1:S 11 Jan 2022 12:11:12.479 * Reading RDB preamble from AOF file...
1:S 11 Jan 2022 12:11:12.479 * Loading RDB produced by version 6.2.5
1:S 11 Jan 2022 12:11:12.479 * RDB age 3963 seconds
1:S 11 Jan 2022 12:11:12.479 * RDB memory usage when created 274.58 Mb
1:S 11 Jan 2022 12:11:12.479 * RDB has an AOF tail
1:S 11 Jan 2022 12:11:14.718 # Done loading RDB, keys loaded: 246, keys expired: 0.
1:S 11 Jan 2022 12:11:14.718 * Reading the remaining AOF tail...
1:S 11 Jan 2022 12:11:14.780 * DB loaded from append only file: 2.305 seconds
1:S 11 Jan 2022 12:11:14.780 * Ready to accept connections
1:S 11 Jan 2022 12:11:15.382 * Connecting to MASTER redis-node-2.redis-headless.redis-sentinel-main.svc.cluster.local:6379
1:S 11 Jan 2022 12:11:15.384 * MASTER <-> REPLICA sync started
1:S 11 Jan 2022 12:11:15.385 * Non blocking connect for SYNC fired the event.
1:S 11 Jan 2022 12:11:15.385 * Master replied to PING, replication can continue...
1:S 11 Jan 2022 12:11:15.385 * Partial resynchronization not possible (no cached master)
1:S 11 Jan 2022 12:11:15.390 * Full resync from master: 6c278ab479412fe8192f7d174a9e875791d1c481:968355187
1:S 11 Jan 2022 12:11:18.014 * MASTER <-> REPLICA sync: receiving 156287487 bytes from master to disk
1:S 11 Jan 2022 12:11:18.962 * MASTER <-> REPLICA sync: Flushing old data
1:S 11 Jan 2022 12:11:19.067 * MASTER <-> REPLICA sync: Loading DB in memory
1:S 11 Jan 2022 12:11:19.085 * Loading RDB produced by version 6.2.6
1:S 11 Jan 2022 12:11:19.085 * RDB age 4 seconds
1:S 11 Jan 2022 12:11:19.085 * RDB memory usage when created 289.53 Mb
1:S 11 Jan 2022 12:11:20.369 # Done loading RDB, keys loaded: 389, keys expired: 0.
1:S 11 Jan 2022 12:11:20.370 * MASTER <-> REPLICA sync: Finished with success
1:S 11 Jan 2022 12:11:20.375 * Background append only file rewriting started by pid 29
1:S 11 Jan 2022 12:11:22.885 * AOF rewrite child asks to stop sending diffs.
29:C 11 Jan 2022 12:11:22.885 * Parent agreed to stop sending diffs. Finalizing AOF...
29:C 11 Jan 2022 12:11:22.885 * Concatenating 0.00 MB of AOF diff received from parent.
29:C 11 Jan 2022 12:11:22.886 * SYNC append only file rewrite performed
29:C 11 Jan 2022 12:11:22.893 * AOF rewrite: 5 MB of memory used by copy-on-write
1:S 11 Jan 2022 12:11:22.931 * Background AOF rewrite terminated with success
1:S 11 Jan 2022 12:11:22.931 * Residual parent diff successfully flushed to the rewritten AOF (0.00 MB)
1:S 11 Jan 2022 12:11:22.931 * Background AOF rewrite finished successfully

############### SENTINEL

 12:11:16.03 INFO  ==> about to run the command: timeout 5 redis-cli -h redis.redis-sentinel-main.svc.cluster.local -p 26379 sentinel get-master-addr-by-name redis-master-main
10.0.16.208
6379
 12:11:16.12 INFO  ==> about to run the command: timeout 5 redis-cli -h redis.redis-sentinel-main.svc.cluster.local -p 26379 sentinel get-master-addr-by-name redis-master-main
 12:11:16.14 INFO  ==> printing REDIS_SENTINEL_INFO=(redis-node-2.redis-headless.redis-sentinel-main.svc.cluster.local,6379)
1:X 11 Jan 2022 12:11:16.427 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:X 11 Jan 2022 12:11:16.427 # Redis version=6.2.6, bits=64, commit=00000000, modified=0, pid=1, just started
1:X 11 Jan 2022 12:11:16.427 # Configuration loaded
1:X 11 Jan 2022 12:11:16.428 * monotonic clock: POSIX clock_gettime
1:X 11 Jan 2022 12:11:16.428 * Running mode=sentinel, port=26379.
1:X 11 Jan 2022 12:11:16.429 # Sentinel ID is 33535e4e17bf8f9f9ff9ce8f9ddf609e558ff4f2
1:X 11 Jan 2022 12:11:16.429 # +monitor master redis-master-main redis-node-2.redis-headless.redis-sentinel-main.svc.cluster.local 6379 quorum 2
1:X 11 Jan 2022 12:11:16.443 * +slave slave 10.0.17.255:6379 10.0.17.255 6379 @ redis-master-main redis-node-2.redis-headless.redis-sentinel-main.svc.cluster.local 6379
1:X 11 Jan 2022 12:11:16.971 # +new-epoch 7
1:X 11 Jan 2022 12:11:49.877 # +sdown sentinel 2a09ba7abbb41ee71e79087310d75f9809c3c815 redis-node-0.redis-headless.redis-sentinel-main.svc.cluster.local 26379 @ redis-master-main redis-node-2.redis-headless.redis-sentinel-main.svc.cluster.local 6379
1:X 11 Jan 2022 12:11:49.954 # +sdown slave redis-node-0.redis-headless.redis-sentinel-main.svc.cluster.local:6379 redis-node-0.redis-headless.redis-sentinel-main.svc.cluster.local 6379 @ redis-master-main redis-node-2.redis-headless.redis-sentinel-main.svc.cluster.local 6379
1:X 11 Jan 2022 12:11:49.954 # +sdown slave 10.0.17.255:6379 10.0.17.255 6379 @ redis-master-main redis-node-2.redis-headless.redis-sentinel-main.svc.cluster.local 6379
1:X 11 Jan 2022 12:11:55.895 * +sentinel-address-switch master redis-master-main redis-node-2.redis-headless.redis-sentinel-main.svc.cluster.local 6379 ip redis-node-0.redis-headless.redis-sentinel-main.svc.cluster.local port 26379 for 2a09ba7abbb41ee71e79087310d75f9809c3c815
1:X 11 Jan 2022 12:11:59.979 * +sentinel sentinel 2a09ba7abbb41ee71e79087310d75f9809c3c815 redis-node-0.redis-headless.redis-sentinel-main.svc.cluster.local 26379 @ redis-master-main redis-node-2.redis-headless.redis-sentinel-main.svc.cluster.local 6379
alemorcuq commented 2 years ago

We have a task in which we are investigating sentinel issues. We will update the ticket when we have more information.

aeimer commented 2 years ago

@alemorcuq can we support you somehow? As long as this breaking change stays, we are stuck with that version :/

aeimer commented 2 years ago

I just checked the state with 16.4.0, still persisting.

aeimer commented 2 years ago

It is probably related to this ticket/fix https://github.com/redisson/redisson/issues/4003

We are testing it now.

aeimer commented 2 years ago

@alemorcuq we were able to resolve the issue and uprade to 16.4.0, we had to use the redison lib with version 3.16.7+ and do the following steps to resolve the issue:

Which leads to the question what could be saved in the PVs that the sentinel/client makes go crazy?

alemorcuq commented 2 years ago

No idea, @aeimer, but I'm glad you were able to solve your issues. I'll reflect this on our task on sentinel issues.

aeimer commented 2 years ago

@alemorcuq alright. Then Ill close the ticket, as the problem is solved for me.

Thank you for the help! Alex