[bitnami/redis] Redis sentinel enters tilt mode and dies after a time

Name and Version

bitnami/redis 16.8.2

What steps will reproduce the bug?

Install bitnami/redis in sentinel mode.

Wait a while (It typically happens within an hour) for sentinel containers to start restarting for no reason.

Are you using any custom parameters or values?

---
image:
  debug: true
auth:
  password: "password"
replica:
  replicaCount: 3
  persistence:
    enabled: false
  resources:
    requests:
      cpu: "100m"
      memory: "256Mi"
  limits:
    requests:
      cpu: "1000m"
      memory: "512Mi"
sentinel:
  enabled: true
  service:
    type: NodePort
  image:
    debug: true
  quorum: 2
  downAfterMilliseconds: 2000
  failoverTimeout: 1000
  livenessProbe:
    enabled: true
    initialDelaySeconds: 40
  readinessProbe:
    enabled: true
    initialDelaySeconds: 40
metrics:
  enabled: true
  image:
    tag: latest
  serviceMonitor:
    enabled: true
sysctlImage:
  enabled: true
  mountHostSys: true
  command:
    - /bin/sh
    - -c
    - |-
      install_packages procps
      sysctl -w net.core.somaxconn=1024
      echo never > /host-sys/kernel/mm/transparent_hugepage/enabled

What is the expected behavior?

Expected behavior is sentinel to not restart for no seemingly reason.

What do you see instead?

After some period of time, sentinel containers enter tilt mode on and off again, and then just restart.

Entire log of sentinel container up till restart

23:39:39.04 INFO ==> about to run the command: redis-cli -h redis1.redis1.svc.cluster.local -p 30014 sentinel get-master-addr-by-name mymaster
--
Mon, Apr 4 2022 5:39:39 pm | redis1-node-1.redis1
Mon, Apr 4 2022 5:39:39 pm | 30019
Mon, Apr 4 2022 5:39:39 pm | 23:39:39.20 INFO ==> about to run the command: redis-cli -h redis1.redis1.svc.cluster.local -p 30014 sentinel get-master-addr-by-name mymaster
Mon, Apr 4 2022 5:39:39 pm | 23:39:39.24 INFO ==> printing REDIS_SENTINEL_INFO=(redis1-node-1.redis1,30019)
Mon, Apr 4 2022 5:39:39 pm | 1:X 04 Apr 2022 23:39:39.565 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
Mon, Apr 4 2022 5:39:39 pm | 1:X 04 Apr 2022 23:39:39.565 # Redis version=6.2.6, bits=64, commit=00000000, modified=0, pid=1, just started
Mon, Apr 4 2022 5:39:39 pm | 1:X 04 Apr 2022 23:39:39.565 # Configuration loaded
Mon, Apr 4 2022 5:39:39 pm | 1:X 04 Apr 2022 23:39:39.566 * monotonic clock: POSIX clock_gettime
Mon, Apr 4 2022 5:39:39 pm | 1:X 04 Apr 2022 23:39:39.567 * Running mode=sentinel, port=26379.
Mon, Apr 4 2022 5:39:39 pm | 1:X 04 Apr 2022 23:39:39.567 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
Mon, Apr 4 2022 5:39:39 pm | 1:X 04 Apr 2022 23:39:39.568 # Sentinel ID is 93060445d3bd07e1b6b21a89638966d80990170a
Mon, Apr 4 2022 5:39:39 pm | 1:X 04 Apr 2022 23:39:39.568 # +monitor master mymaster redis1-node-1.redis1 30019 quorum 2
Mon, Apr 4 2022 5:39:39 pm | 1:X 04 Apr 2022 23:39:39.650 # +new-epoch 1
Mon, Apr 4 2022 5:39:41 pm | 1:X 04 Apr 2022 23:39:41.572 # +sdown slave redis1-node-0.redis1:30017 redis1-node-0.redis1 30017 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 5:39:59 pm | 1:X 04 Apr 2022 23:39:59.496 # +sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 5:40:02 pm | 1:X 04 Apr 2022 23:40:02.335 # -sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 5:40:23 pm | 1:X 04 Apr 2022 23:40:23.001 # -sdown slave redis1-node-0.redis1:30017 redis1-node-0.redis1 30017 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 5:54:12 pm | 1:X 04 Apr 2022 23:54:12.637 # +sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 5:54:15 pm | 1:X 04 Apr 2022 23:54:15.470 # -sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 5:57:01 pm | 1:X 04 Apr 2022 23:57:01.362 # +sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 5:57:05 pm | 1:X 04 Apr 2022 23:57:05.261 # -sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 6:00:23 pm | 1:X 05 Apr 2022 00:00:23.440 # +sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 6:00:27 pm | 1:X 05 Apr 2022 00:00:27.440 # -sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 6:01:09 pm | 1:X 05 Apr 2022 00:01:09.066 # +sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 6:01:10 pm | 1:X 05 Apr 2022 00:01:10.220 # +sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 6:01:12 pm | 1:X 05 Apr 2022 00:01:12.266 # -sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 6:01:14 pm | 1:X 05 Apr 2022 00:01:14.009 # -sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 6:01:33 pm | 1:X 05 Apr 2022 00:01:33.982 # +sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 6:01:38 pm | 1:X 05 Apr 2022 00:01:38.199 # -sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 6:01:43 pm | 1:X 05 Apr 2022 00:01:43.563 # +sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 6:01:47 pm | 1:X 05 Apr 2022 00:01:47.225 # -sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 6:01:51 pm | 1:X 05 Apr 2022 00:01:51.749 # +sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 6:02:00 pm | 1:X 05 Apr 2022 00:02:00.576 # +tilt #tilt mode entered
Mon, Apr 4 2022 6:02:30 pm | 1:X 05 Apr 2022 00:02:30.596 # -tilt #tilt mode exited
Mon, Apr 4 2022 6:02:30 pm | 1:X 05 Apr 2022 00:02:30.596 # -sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 6:03:02 pm | 1:X 05 Apr 2022 00:03:02.905 # +sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 6:03:08 pm | 1:X 05 Apr 2022 00:03:08.583 # +sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 6:03:17 pm | 1:X 05 Apr 2022 00:03:17.052 # +tilt #tilt mode entered
Mon, Apr 4 2022 6:03:47 pm | 1:X 05 Apr 2022 00:03:47.079 # -tilt #tilt mode exited
Mon, Apr 4 2022 6:03:47 pm | 1:X 05 Apr 2022 00:03:47.079 # -sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 6:03:51 pm | 1:X 05 Apr 2022 00:03:51.501 # +sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 6:03:57 pm | 1:X 05 Apr 2022 00:03:57.303 # +tilt #tilt mode entered
Mon, Apr 4 2022 6:04:27 pm | 1:X 05 Apr 2022 00:04:27.330 # -tilt #tilt mode exited
Mon, Apr 4 2022 6:04:27 pm | 1:X 05 Apr 2022 00:04:27.330 # -sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 6:04:57 pm | 1:X 05 Apr 2022 00:04:57.720 # -sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 6:06:39 pm | 1:X 05 Apr 2022 00:06:39.105 # +sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 6:06:42 pm | 1:X 05 Apr 2022 00:06:42.825 # -sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 6:08:11 pm | 1:X 05 Apr 2022 00:08:11.048 # +sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 6:08:19 pm | 1:X 05 Apr 2022 00:08:19.397 # +tilt #tilt mode entered
Mon, Apr 4 2022 6:08:49 pm | 1:X 05 Apr 2022 00:08:49.412 # -tilt #tilt mode exited
Mon, Apr 4 2022 6:08:49 pm | 1:X 05 Apr 2022 00:08:49.412 # -sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 6:10:05 pm | 1:X 05 Apr 2022 00:10:05.427 # +sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 6:10:08 pm | 1:X 05 Apr 2022 00:10:08.266 # -sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 6:10:26 pm | 1:X 05 Apr 2022 00:10:26.889 # +sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 6:10:26 pm | 1:X 05 Apr 2022 00:10:26.890 # +sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 6:10:44 pm | 1:X 05 Apr 2022 00:10:44.726 # +tilt #tilt mode entered
Mon, Apr 4 2022 6:10:49 pm | 1:X 05 Apr 2022 00:10:49.884 # +tilt #tilt mode entered
Mon, Apr 4 2022 6:11:19 pm | 1:X 05 Apr 2022 00:11:19.936 # -tilt #tilt mode exited
Mon, Apr 4 2022 6:11:19 pm | 1:X 05 Apr 2022 00:11:19.936 # -sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 6:11:36 pm | 1:X 05 Apr 2022 00:11:36.123 # -sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 6:12:33 pm | 1:X 05 Apr 2022 00:12:33.381 # +sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 6:12:33 pm | 1:X 05 Apr 2022 00:12:33.914 # +sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 6:12:36 pm | 1:X 05 Apr 2022 00:12:36.709 # -sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 6:12:36 pm | 1:X 05 Apr 2022 00:12:36.931 # -sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 6:12:47 pm | 1:X 05 Apr 2022 00:12:47.992 # +tilt #tilt mode entered
Mon, Apr 4 2022 6:13:18 pm | 1:X 05 Apr 2022 00:13:18.012 # -tilt #tilt mode exited
Mon, Apr 4 2022 6:15:16 pm | 1:X 05 Apr 2022 00:15:16.327 # +sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 6:15:19 pm | 1:X 05 Apr 2022 00:15:19.653 # -sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 6:15:20 pm | 1:X 05 Apr 2022 00:15:20.319 # +sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 6:15:23 pm | 1:X 05 Apr 2022 00:15:23.662 # -sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 6:26:31 pm | 1:X 05 Apr 2022 00:26:31.059 # +sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 6:26:34 pm | 1:X 05 Apr 2022 00:26:34.609 # -sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 6:31:08 pm | 1:X 05 Apr 2022 00:31:08.298 # +tilt #tilt mode entered
Mon, Apr 4 2022 6:31:38 pm | 1:X 05 Apr 2022 00:31:38.376 # -tilt #tilt mode exited
Mon, Apr 4 2022 6:35:41 pm | 1:X 05 Apr 2022 00:35:41.686 # +tilt #tilt mode entered
Mon, Apr 4 2022 6:35:47 pm | 1:X 05 Apr 2022 00:35:47.182 # +tilt #tilt mode entered
Mon, Apr 4 2022 6:36:17 pm | 1:X 05 Apr 2022 00:36:17.193 # -tilt #tilt mode exited
Mon, Apr 4 2022 6:40:22 pm | 1:X 05 Apr 2022 00:40:22.826 # +tilt #tilt mode entered
Mon, Apr 4 2022 6:40:28 pm | 1:X 05 Apr 2022 00:40:28.539 # +tilt #tilt mode entered
Mon, Apr 4 2022 6:40:41 pm | 1:X 05 Apr 2022 00:40:41.700 # +tilt #tilt mode entered
Mon, Apr 4 2022 6:40:53 pm | 1:X 05 Apr 2022 00:40:53.560 # +tilt #tilt mode entered
Mon, Apr 4 2022 6:41:01 pm | 1:X 05 Apr 2022 00:41:01.778 # +tilt #tilt mode entered
Mon, Apr 4 2022 6:41:31 pm | 1:X 05 Apr 2022 00:41:31.841 # -tilt #tilt mode exited
Mon, Apr 4 2022 6:42:14 pm | 1:X 05 Apr 2022 00:42:14.368 # +sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 6:42:14 pm | 1:X 05 Apr 2022 00:42:14.498 # +sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 6:42:17 pm | 1:X 05 Apr 2022 00:42:17.347 # -sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 6:42:17 pm | 1:X 05 Apr 2022 00:42:17.856 # -sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 6:42:26 pm | 1:X 05 Apr 2022 00:42:26.578 # +tilt #tilt mode entered
Mon, Apr 4 2022 6:42:56 pm | 1:X 05 Apr 2022 00:42:56.588 # -tilt #tilt mode exited
Mon, Apr 4 2022 6:43:00 pm | 1:X 05 Apr 2022 00:43:00.092 # +sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 6:43:03 pm | 1:X 05 Apr 2022 00:43:03.483 # -sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 7:04:18 pm | 1:X 05 Apr 2022 01:04:18.538 # +tilt #tilt mode entered
Mon, Apr 4 2022 7:04:48 pm | 1:X 05 Apr 2022 01:04:48.576 # -tilt #tilt mode exited
Mon, Apr 4 2022 7:05:04 pm | 1:X 05 Apr 2022 01:05:04.876 # +sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 7:05:08 pm | 1:X 05 Apr 2022 01:05:08.921 # -sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 7:05:48 pm | 1:X 05 Apr 2022 01:05:48.718 # +sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 7:05:57 pm | 1:X 05 Apr 2022 01:05:57.477 # -sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 7:06:33 pm | 1:X 05 Apr 2022 01:06:33.731 # +sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 7:06:36 pm | 1:X 05 Apr 2022 01:06:36.460 # -sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 7:06:43 pm | 1:X 05 Apr 2022 01:06:43.575 # +tilt #tilt mode entered
Mon, Apr 4 2022 7:07:13 pm | 1:X 05 Apr 2022 01:07:13.625 # -tilt #tilt mode exited
Mon, Apr 4 2022 7:08:04 pm | 1:X 05 Apr 2022 01:08:04.225 # +tilt #tilt mode entered
Mon, Apr 4 2022 7:08:34 pm | 1:X 05 Apr 2022 01:08:34.266 # -tilt #tilt mode exited
Mon, Apr 4 2022 7:08:35 pm | 1:X 05 Apr 2022 01:08:35.095 # +sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 7:08:38 pm | 1:X 05 Apr 2022 01:08:38.395 # +sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 7:08:41 pm | 1:X 05 Apr 2022 01:08:41.623 # -sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 7:08:42 pm | 1:X 05 Apr 2022 01:08:42.940 # -sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 7:08:50 pm | 1:X 05 Apr 2022 01:08:50.476 # +sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 7:08:53 pm | 1:X 05 Apr 2022 01:08:53.914 # -sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 7:09:43 pm | 1:X 05 Apr 2022 01:09:43.447 # +sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 7:09:47 pm | 1:X 05 Apr 2022 01:09:47.018 # -sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 7:10:04 pm | 1:X 05 Apr 2022 01:10:04.790 # +sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 7:10:07 pm | 1:X 05 Apr 2022 01:10:07.356 # -sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 7:10:26 pm | 1:X 05 Apr 2022 01:10:26.712 # +sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 7:10:29 pm | 1:X 05 Apr 2022 01:10:29.417 # -sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 7:10:50 pm | 1:X 05 Apr 2022 01:10:50.378 # +sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 7:10:53 pm | 1:X 05 Apr 2022 01:10:53.472 # -sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 7:11:53 pm | 1:X 05 Apr 2022 01:11:53.579 # +sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 7:11:57 pm | 1:X 05 Apr 2022 01:11:57.069 # -sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 7:15:32 pm | 1:X 05 Apr 2022 01:15:32.047 # +tilt #tilt mode entered
Mon, Apr 4 2022 7:16:02 pm | 1:X 05 Apr 2022 01:16:02.062 # -tilt #tilt mode exited
Mon, Apr 4 2022 7:36:33 pm | 1:X 05 Apr 2022 01:36:33.739 # +sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 7:36:37 pm | 1:X 05 Apr 2022 01:36:37.526 # -sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 7:37:29 pm | 1:X 05 Apr 2022 01:37:29.027 # +tilt #tilt mode entered
Mon, Apr 4 2022 7:37:59 pm | 1:X 05 Apr 2022 01:37:59.049 # -tilt #tilt mode exited
Mon, Apr 4 2022 7:39:10 pm | 1:X 05 Apr 2022 01:39:10.739 # +sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 7:39:14 pm | 1:X 05 Apr 2022 01:39:14.236 # -sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 7:39:14 pm | 1:X 05 Apr 2022 01:39:14.934 # +sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 7:39:18 pm | 1:X 05 Apr 2022 01:39:18.204 # -sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 7:40:23 pm | 1:X 05 Apr 2022 01:40:23.961 # +sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 7:40:27 pm | 1:X 05 Apr 2022 01:40:27.390 # -sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 7:40:33 pm | 1:X 05 Apr 2022 01:40:33.185 # +tilt #tilt mode entered
Mon, Apr 4 2022 7:41:03 pm | 1:X 05 Apr 2022 01:41:03.250 # -tilt #tilt mode exited
Mon, Apr 4 2022 7:41:03 pm | 1:X 05 Apr 2022 01:41:03.250 # +sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 7:41:12 pm | 1:X 05 Apr 2022 01:41:12.354 # -sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 7:41:25 pm | 1:X 05 Apr 2022 01:41:25.881 # +sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 7:41:28 pm | 1:X 05 Apr 2022 01:41:28.917 # -sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 7:42:18 pm | 1:X 05 Apr 2022 01:42:18.768 # +tilt #tilt mode entered
Mon, Apr 4 2022 7:42:24 pm | 1:X 05 Apr 2022 01:42:24.740 # +tilt #tilt mode entered
Mon, Apr 4 2022 7:42:54 pm | 1:X 05 Apr 2022 01:42:54.762 # -tilt #tilt mode exited
Mon, Apr 4 2022 7:43:04 pm | 1:X 05 Apr 2022 01:43:04.522 # +sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 7:43:07 pm | 1:X 05 Apr 2022 01:43:07.787 # -sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 8:20:11 pm | 1:X 05 Apr 2022 02:20:11.954 # +sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 8:20:11 pm | 1:X 05 Apr 2022 02:20:11.954 # +sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 8:20:14 pm | 1:X 05 Apr 2022 02:20:14.629 # -sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 8:20:14 pm | 1:X 05 Apr 2022 02:20:14.629 # -sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 8:20:28 pm | 1:X 05 Apr 2022 02:20:28.214 # +sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 8:20:31 pm | 1:X 05 Apr 2022 02:20:31.501 # -sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 8:24:09 pm | 1:X 05 Apr 2022 02:24:09.933 # +sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 8:24:12 pm | 1:X 05 Apr 2022 02:24:12.907 # -sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 8:28:47 pm | 1:X 05 Apr 2022 02:28:47.699 # +tilt #tilt mode entered
Mon, Apr 4 2022 8:29:17 pm | 1:X 05 Apr 2022 02:29:17.735 # -tilt #tilt mode exited
Mon, Apr 4 2022 8:32:48 pm | 1:X 05 Apr 2022 02:32:48.816 # +sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 8:32:53 pm | 1:X 05 Apr 2022 02:32:53.076 # -sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 8:33:52 pm | 1:X 05 Apr 2022 02:33:52.541 # +sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 8:33:55 pm | 1:X 05 Apr 2022 02:33:55.602 # +sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 8:33:56 pm | 1:X 05 Apr 2022 02:33:55.998 # -sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 8:34:03 pm | 1:X 05 Apr 2022 02:34:03.954 # -sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 8:58:46 pm | 1:X 05 Apr 2022 02:58:46.179 # +sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 8:58:49 pm | 1:X 05 Apr 2022 02:58:49.037 # -sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 9:12:54 pm | 1:X 05 Apr 2022 03:12:54.818 # +sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 9:12:57 pm | 1:X 05 Apr 2022 03:12:57.783 # -sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 9:18:10 pm | 1:X 05 Apr 2022 03:18:10.185 # +sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 9:18:12 pm | 1:X 05 Apr 2022 03:18:12.841 # -sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 9:18:25 pm | 1:X 05 Apr 2022 03:18:25.447 # +tilt #tilt mode entered
Mon, Apr 4 2022 9:18:55 pm | 1:X 05 Apr 2022 03:18:55.469 # -tilt #tilt mode exited
Mon, Apr 4 2022 9:19:04 pm | 1:X 05 Apr 2022 03:19:04.369 # +tilt #tilt mode entered
Mon, Apr 4 2022 9:19:34 pm | 1:X 05 Apr 2022 03:19:34.398 # -tilt #tilt mode exited
Mon, Apr 4 2022 9:30:50 pm | 1:X 05 Apr 2022 03:30:50.867 # +tilt #tilt mode entered
Mon, Apr 4 2022 9:31:09 pm | 1:X 05 Apr 2022 03:31:09.762 # +tilt #tilt mode entered
Mon, Apr 4 2022 9:31:39 pm | 1:X 05 Apr 2022 03:31:39.839 # -tilt #tilt mode exited
Mon, Apr 4 2022 9:32:48 pm | 1:X 05 Apr 2022 03:32:48.719 # +sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 9:32:52 pm | 1:X 05 Apr 2022 03:32:52.115 # -sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 9:33:49 pm | 1:X 05 Apr 2022 03:33:49.500 # +sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 9:33:53 pm | 1:X 05 Apr 2022 03:33:53.015 # -sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 9:34:14 pm | 1:X 05 Apr 2022 03:34:14.849 # +sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 9:34:17 pm | 1:X 05 Apr 2022 03:34:17.819 # -sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 9:34:32 pm | 1:X 05 Apr 2022 03:34:32.451 # +sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 9:34:36 pm | 1:X 05 Apr 2022 03:34:36.613 # -sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 9:35:12 pm | 1:X 05 Apr 2022 03:35:12.113 # +sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 9:35:15 pm | 1:X 05 Apr 2022 03:35:15.396 # -sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 9:37:22 pm | 1:X 05 Apr 2022 03:37:22.709 # +tilt #tilt mode entered
Mon, Apr 4 2022 9:37:52 pm | 1:X 05 Apr 2022 03:37:52.796 # -tilt #tilt mode exited
Mon, Apr 4 2022 9:39:49 pm | 1:X 05 Apr 2022 03:39:49.641 # +tilt #tilt mode entered
Mon, Apr 4 2022 9:40:19 pm | 1:X 05 Apr 2022 03:40:19.698 # -tilt #tilt mode exited
Mon, Apr 4 2022 9:47:31 pm | 1:X 05 Apr 2022 03:47:31.739 # +tilt #tilt mode entered
Mon, Apr 4 2022 9:48:01 pm | 1:X 05 Apr 2022 03:48:01.793 # -tilt #tilt mode exited
Mon, Apr 4 2022 9:48:18 pm | 1:X 05 Apr 2022 03:48:18.242 # +sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 9:48:21 pm | 1:X 05 Apr 2022 03:48:21.916 # -sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 9:49:21 pm | 1:X 05 Apr 2022 03:49:21.832 # +sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 9:49:35 pm | 1:X 05 Apr 2022 03:49:35.659 # -sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 9:49:47 pm | 1:X 05 Apr 2022 03:49:47.207 # +tilt #tilt mode entered
Mon, Apr 4 2022 9:50:03 pm | 1:X 05 Apr 2022 03:50:03.126 # +tilt #tilt mode entered
Mon, Apr 4 2022 9:50:14 pm | 1:X 05 Apr 2022 03:50:14.364 # +tilt #tilt mode entered
Mon, Apr 4 2022 9:50:26 pm | 1:X 05 Apr 2022 03:50:26.260 # +tilt #tilt mode entered
Mon, Apr 4 2022 9:50:36 pm | 1:X 05 Apr 2022 03:50:36.989 # +tilt #tilt mode entered
Mon, Apr 4 2022 9:50:37 pm | 1:signal-handler (1649130637) Received SIGTERM scheduling shutdown...
Mon, Apr 4 2022 9:50:38 pm | 1:X 05 Apr 2022 03:50:38.052 # User requested shutdown...
Mon, Apr 4 2022 9:50:38 pm | 1:X 05 Apr 2022 03:50:38.052 # Sentinel is now ready to exit, bye bye...

Redis container log.

20:47:48.10 INFO ==> about to run the command: timeout 220 redis-cli -h redis1.redis1.svc.cluster.local -p 30014 sentinel get-master-addr-by-name mymaster
--
Mon, Apr 4 2022 2:47:48 pm | 20:47:48.33 INFO ==> about to run the command: timeout 220 redis-cli -h redis1.redis1.svc.cluster.local -p 30014 sentinel get-master-addr-by-name mymaster
Mon, Apr 4 2022 2:47:48 pm | 20:47:48.39 INFO ==> Current master: REDIS_SENTINEL_INFO=(redis1-node-1.redis1,30019)
Mon, Apr 4 2022 2:47:48 pm | 20:47:48.39 INFO ==> Configuring the node as replica
Mon, Apr 4 2022 2:47:48 pm | 1:C 04 Apr 2022 20:47:48.465 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
Mon, Apr 4 2022 2:47:48 pm | 1:C 04 Apr 2022 20:47:48.465 # Redis version=6.2.6, bits=64, commit=00000000, modified=0, pid=1, just started
Mon, Apr 4 2022 2:47:48 pm | 1:C 04 Apr 2022 20:47:48.465 # Configuration loaded
Mon, Apr 4 2022 2:47:48 pm | 1:S 04 Apr 2022 20:47:48.466 * monotonic clock: POSIX clock_gettime
Mon, Apr 4 2022 2:47:48 pm | 1:S 04 Apr 2022 20:47:48.467 * Running mode=standalone, port=6379.
Mon, Apr 4 2022 2:47:48 pm | 1:S 04 Apr 2022 20:47:48.467 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
Mon, Apr 4 2022 2:47:48 pm | 1:S 04 Apr 2022 20:47:48.467 # Server initialized
Mon, Apr 4 2022 2:47:48 pm | 1:S 04 Apr 2022 20:47:48.468 * Ready to accept connections
Mon, Apr 4 2022 2:47:48 pm | 1:S 04 Apr 2022 20:47:48.468 * Connecting to MASTER redis1-node-1.redis1:30019
Mon, Apr 4 2022 2:47:48 pm | 1:S 04 Apr 2022 20:47:48.475 * MASTER <-> REPLICA sync started
Mon, Apr 4 2022 2:47:48 pm | 1:S 04 Apr 2022 20:47:48.476 * Non blocking connect for SYNC fired the event.
Mon, Apr 4 2022 2:47:48 pm | 1:S 04 Apr 2022 20:47:48.477 * Master replied to PING, replication can continue...
Mon, Apr 4 2022 2:47:48 pm | 1:S 04 Apr 2022 20:47:48.479 * Partial resynchronization not possible (no cached master)
Mon, Apr 4 2022 2:47:48 pm | 1:S 04 Apr 2022 20:47:48.480 * Full resync from master: a596e3271aa403b5cfb29c7c2f8126c412b5fe59:3429539
Mon, Apr 4 2022 2:47:48 pm | 1:S 04 Apr 2022 20:47:48.549 * MASTER <-> REPLICA sync: receiving 178 bytes from master to disk
Mon, Apr 4 2022 2:47:48 pm | 1:S 04 Apr 2022 20:47:48.549 * MASTER <-> REPLICA sync: Flushing old data
Mon, Apr 4 2022 2:47:48 pm | 1:S 04 Apr 2022 20:47:48.561 * MASTER <-> REPLICA sync: Loading DB in memory
Mon, Apr 4 2022 2:47:48 pm | 1:S 04 Apr 2022 20:47:48.601 * Loading RDB produced by version 6.2.6
Mon, Apr 4 2022 2:47:48 pm | 1:S 04 Apr 2022 20:47:48.601 * RDB age 0 seconds
Mon, Apr 4 2022 2:47:48 pm | 1:S 04 Apr 2022 20:47:48.601 * RDB memory usage when created 1.93 Mb
Mon, Apr 4 2022 2:47:48 pm | 1:S 04 Apr 2022 20:47:48.601 # Done loading RDB, keys loaded: 0, keys expired: 0.
Mon, Apr 4 2022 2:47:48 pm | 1:S 04 Apr 2022 20:47:48.601 * MASTER <-> REPLICA sync: Finished with success
Mon, Apr 4 2022 2:47:48 pm | 1:S 04 Apr 2022 20:47:48.602 * Background append only file rewriting started by pid 35
Mon, Apr 4 2022 2:47:48 pm | 1:S 04 Apr 2022 20:47:48.661 * AOF rewrite child asks to stop sending diffs.
Mon, Apr 4 2022 2:47:48 pm | 35:C 04 Apr 2022 20:47:48.661 * Parent agreed to stop sending diffs. Finalizing AOF...
Mon, Apr 4 2022 2:47:48 pm | 35:C 04 Apr 2022 20:47:48.661 * Concatenating 0.00 MB of AOF diff received from parent.
Mon, Apr 4 2022 2:47:48 pm | 35:C 04 Apr 2022 20:47:48.664 * SYNC append only file rewrite performed
Mon, Apr 4 2022 2:47:48 pm | 35:C 04 Apr 2022 20:47:48.665 * AOF rewrite: 0 MB of memory used by copy-on-write
Mon, Apr 4 2022 2:47:48 pm | 1:S 04 Apr 2022 20:47:48.703 * Background AOF rewrite terminated with success
Mon, Apr 4 2022 2:47:48 pm | 1:S 04 Apr 2022 20:47:48.704 * Residual parent diff successfully flushed to the rewritten AOF (0.00 MB)
Mon, Apr 4 2022 2:47:48 pm | 1:S 04 Apr 2022 20:47:48.704 * Background AOF rewrite finished successfully

Additional information

Seems like it may be the probes killing the container? Screenshot_20220405_103651 Screenshot_20220405_103718

This has also been happening for a while, we just have not noticed cause the redis container is not restarting, so all data is fine. Screenshot_20220405_103805

I think it might be host related somehow. The only reason I am thinking this is because we were testing in a multi-cluster environment. Part on-prem and the other part in a remote managed k8s. We see this behavior on-prem, and not in remote k8s.

I have been testing on a new cluster with Rocky Linux

Container Runtime Version | docker://20.10.13
Image | Rocky Linux 8.5 (Green Obsidian)
Kernel Version | 4.18.0-348.20.1.el8_5.x86_64
Kube Proxy Version | v1.21.9
Kubelet Version | v1.21.9

But this is also happening on the older deployment from the image above

Container Runtime Version | docker://20.10.8
Image | CentOS Linux 8
Kernel Version | 4.18.0-305.17.1.el8_4.x86_64
Kube Proxy Version | v1.20.6
Kubelet Version | v1.20.6

I am still testing, I am pairing down the values file more and more until I see the behavior stop. This is taking a while as it take a random long amount of time until sentinel enters tilt mode repeatedly and dies.

We have many things deployed onto these clusters, and have no other observed issues.

I have been poking around the hosts and trying to see if there was anything happening that would line up with the behavior explained here: https://redis.io/docs/manual/sentinel/#tilt-mode and I have not found anything yet. Metrics for the vms are all reasonable. No monitoring or alarming is going of for any of these clusters/workers.

I am unsure what is happening. If anyone reading this has any insight on what is happening, input would be greatly appreciated.

Thank you!

Finished retested with smaller values. Exact same results. Redoing again with more things stripped, will post in a bit.

---
image:
  debug: true
auth:
  password: "password"
replica:
  replicaCount: 3
  persistence:
    enabled: false
sentinel:
  enabled: true
  image:
    debug: true
  quorum: 2
  downAfterMilliseconds: 2000
  failoverTimeout: 1000
  livenessProbe:
    enabled: true
    initialDelaySeconds: 40
  readinessProbe:
    enabled: true
    initialDelaySeconds: 40

16:10:58.27 INFO ==> about to run the command: REDISCLI_AUTH=$REDIS_PASSWORD redis-cli -h redis2.redis2.svc.cluster.local -p 26379 sentinel get-master-addr-by-name mymaster
--
Tue, Apr 5 2022 10:10:59 am | Could not connect to Redis at redis2.redis2.svc.cluster.local:26379: Connection refused
Tue, Apr 5 2022 10:10:59 am | 1:X 05 Apr 2022 16:10:59.598 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
Tue, Apr 5 2022 10:10:59 am | 1:X 05 Apr 2022 16:10:59.599 # Redis version=6.2.6, bits=64, commit=00000000, modified=0, pid=1, just started
Tue, Apr 5 2022 10:10:59 am | 1:X 05 Apr 2022 16:10:59.599 # Configuration loaded
Tue, Apr 5 2022 10:10:59 am | 1:X 05 Apr 2022 16:10:59.599 * monotonic clock: POSIX clock_gettime
Tue, Apr 5 2022 10:10:59 am | 1:X 05 Apr 2022 16:10:59.600 * Running mode=sentinel, port=26379.
Tue, Apr 5 2022 10:10:59 am | 1:X 05 Apr 2022 16:10:59.600 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
Tue, Apr 5 2022 10:10:59 am | 1:X 05 Apr 2022 16:10:59.601 # Sentinel ID is d37bb3ee651ae251d15e4917d9e8571b71e35013
Tue, Apr 5 2022 10:10:59 am | 1:X 05 Apr 2022 16:10:59.601 # +monitor master mymaster redis2-node-0.redis2-headless.redis2.svc.cluster.local 6379 quorum 2
Tue, Apr 5 2022 10:11:46 am | 1:X 05 Apr 2022 16:11:46.232 * +sentinel sentinel 00de773dc2a8f87fb07d3446fdbc0f99e0dbefe1 redis2-node-1.redis2-headless.redis2.svc.cluster.local 26379 @ mymaster redis2-node-0.redis2-headless.redis2.svc.cluster.local 6379
Tue, Apr 5 2022 10:11:49 am | 1:X 05 Apr 2022 16:11:49.768 * +slave slave redis2-node-1.redis2-headless.redis2.svc.cluster.local:6379 redis2-node-1.redis2-headless.redis2.svc.cluster.local 6379 @ mymaster redis2-node-0.redis2-headless.redis2.svc.cluster.local 6379
Tue, Apr 5 2022 10:12:29 am | 1:X 05 Apr 2022 16:12:29.943 * +slave slave redis2-node-2.redis2-headless.redis2.svc.cluster.local:6379 redis2-node-2.redis2-headless.redis2.svc.cluster.local 6379 @ mymaster redis2-node-0.redis2-headless.redis2.svc.cluster.local 6379
Tue, Apr 5 2022 10:12:31 am | 1:X 05 Apr 2022 16:12:31.043 * +sentinel sentinel 6cfe9063dd168605a311b2eda37ca1e4fbbca27b redis2-node-2.redis2-headless.redis2.svc.cluster.local 26379 @ mymaster redis2-node-0.redis2-headless.redis2.svc.cluster.local 6379
Tue, Apr 5 2022 10:41:25 am | 1:X 05 Apr 2022 16:41:25.779 # +sdown sentinel 00de773dc2a8f87fb07d3446fdbc0f99e0dbefe1 redis2-node-1.redis2-headless.redis2.svc.cluster.local 26379 @ mymaster redis2-node-0.redis2-headless.redis2.svc.cluster.local 6379
Tue, Apr 5 2022 10:41:41 am | 1:X 05 Apr 2022 16:41:41.524 # +sdown sentinel 6cfe9063dd168605a311b2eda37ca1e4fbbca27b redis2-node-2.redis2-headless.redis2.svc.cluster.local 26379 @ mymaster redis2-node-0.redis2-headless.redis2.svc.cluster.local 6379
Tue, Apr 5 2022 10:41:44 am | 1:X 05 Apr 2022 16:41:44.441 # -sdown sentinel 6cfe9063dd168605a311b2eda37ca1e4fbbca27b redis2-node-2.redis2-headless.redis2.svc.cluster.local 26379 @ mymaster redis2-node-0.redis2-headless.redis2.svc.cluster.local 6379
Tue, Apr 5 2022 10:41:44 am | 1:X 05 Apr 2022 16:41:44.760 # -sdown sentinel 00de773dc2a8f87fb07d3446fdbc0f99e0dbefe1 redis2-node-1.redis2-headless.redis2.svc.cluster.local 26379 @ mymaster redis2-node-0.redis2-headless.redis2.svc.cluster.local 6379
Tue, Apr 5 2022 10:42:34 am | 1:X 05 Apr 2022 16:42:34.327 # +sdown sentinel 6cfe9063dd168605a311b2eda37ca1e4fbbca27b redis2-node-2.redis2-headless.redis2.svc.cluster.local 26379 @ mymaster redis2-node-0.redis2-headless.redis2.svc.cluster.local 6379
Tue, Apr 5 2022 10:42:34 am | 1:X 05 Apr 2022 16:42:34.428 # +sdown sentinel 00de773dc2a8f87fb07d3446fdbc0f99e0dbefe1 redis2-node-1.redis2-headless.redis2.svc.cluster.local 26379 @ mymaster redis2-node-0.redis2-headless.redis2.svc.cluster.local 6379
Tue, Apr 5 2022 10:42:37 am | 1:X 05 Apr 2022 16:42:37.245 # -sdown sentinel 00de773dc2a8f87fb07d3446fdbc0f99e0dbefe1 redis2-node-1.redis2-headless.redis2.svc.cluster.local 26379 @ mymaster redis2-node-0.redis2-headless.redis2.svc.cluster.local 6379
Tue, Apr 5 2022 10:42:41 am | 1:X 05 Apr 2022 16:42:41.924 # -sdown sentinel 6cfe9063dd168605a311b2eda37ca1e4fbbca27b redis2-node-2.redis2-headless.redis2.svc.cluster.local 26379 @ mymaster redis2-node-0.redis2-headless.redis2.svc.cluster.local 6379
Tue, Apr 5 2022 10:48:50 am | 1:X 05 Apr 2022 16:48:50.333 # +tilt #tilt mode entered
Tue, Apr 5 2022 10:48:55 am | 1:X 05 Apr 2022 16:48:55.384 # +tilt #tilt mode entered
Tue, Apr 5 2022 10:49:27 am | 1:X 05 Apr 2022 16:49:27.169 # +tilt #tilt mode entered
Tue, Apr 5 2022 10:49:57 am | 1:X 05 Apr 2022 16:49:57.201 # -tilt #tilt mode exited
Tue, Apr 5 2022 10:49:57 am | 1:X 05 Apr 2022 16:49:57.201 # +sdown sentinel 00de773dc2a8f87fb07d3446fdbc0f99e0dbefe1 redis2-node-1.redis2-headless.redis2.svc.cluster.local 26379 @ mymaster redis2-node-0.redis2-headless.redis2.svc.cluster.local 6379
Tue, Apr 5 2022 10:50:03 am | 1:X 05 Apr 2022 16:50:03.034 # -sdown sentinel 00de773dc2a8f87fb07d3446fdbc0f99e0dbefe1 redis2-node-1.redis2-headless.redis2.svc.cluster.local 26379 @ mymaster redis2-node-0.redis2-headless.redis2.svc.cluster.local 6379
Tue, Apr 5 2022 10:50:19 am | 1:X 05 Apr 2022 16:50:19.079 # +tilt #tilt mode entered
Tue, Apr 5 2022 10:50:49 am | 1:X 05 Apr 2022 16:50:49.083 # -tilt #tilt mode exited
Tue, Apr 5 2022 10:56:10 am | 1:X 05 Apr 2022 16:56:10.437 # +sdown sentinel 00de773dc2a8f87fb07d3446fdbc0f99e0dbefe1 redis2-node-1.redis2-headless.redis2.svc.cluster.local 26379 @ mymaster redis2-node-0.redis2-headless.redis2.svc.cluster.local 6379
Tue, Apr 5 2022 10:56:22 am | 1:X 05 Apr 2022 16:56:22.918 # +tilt #tilt mode entered
Tue, Apr 5 2022 10:56:53 am | 1:X 05 Apr 2022 16:56:53.034 # +tilt #tilt mode entered
Tue, Apr 5 2022 10:56:58 am | 1:X 05 Apr 2022 16:56:58.227 # +tilt #tilt mode entered
Tue, Apr 5 2022 10:57:28 am | 1:X 05 Apr 2022 16:57:28.238 # -tilt #tilt mode exited
Tue, Apr 5 2022 10:57:37 am | 1:X 05 Apr 2022 16:57:37.314 # -sdown sentinel 00de773dc2a8f87fb07d3446fdbc0f99e0dbefe1 redis2-node-1.redis2-headless.redis2.svc.cluster.local 26379 @ mymaster redis2-node-0.redis2-headless.redis2.svc.cluster.local 6379
Tue, Apr 5 2022 10:57:48 am | 1:X 05 Apr 2022 16:57:48.382 # +sdown sentinel 6cfe9063dd168605a311b2eda37ca1e4fbbca27b redis2-node-2.redis2-headless.redis2.svc.cluster.local 26379 @ mymaster redis2-node-0.redis2-headless.redis2.svc.cluster.local 6379
Tue, Apr 5 2022 10:57:50 am | 1:X 05 Apr 2022 16:57:50.939 # -sdown sentinel 6cfe9063dd168605a311b2eda37ca1e4fbbca27b redis2-node-2.redis2-headless.redis2.svc.cluster.local 26379 @ mymaster redis2-node-0.redis2-headless.redis2.svc.cluster.local 6379
Tue, Apr 5 2022 10:58:18 am | 1:X 05 Apr 2022 16:58:18.785 # +sdown sentinel 6cfe9063dd168605a311b2eda37ca1e4fbbca27b redis2-node-2.redis2-headless.redis2.svc.cluster.local 26379 @ mymaster redis2-node-0.redis2-headless.redis2.svc.cluster.local 6379
Tue, Apr 5 2022 10:58:21 am | 1:X 05 Apr 2022 16:58:21.638 # -sdown sentinel 6cfe9063dd168605a311b2eda37ca1e4fbbca27b redis2-node-2.redis2-headless.redis2.svc.cluster.local 26379 @ mymaster redis2-node-0.redis2-headless.redis2.svc.cluster.local 6379
Tue, Apr 5 2022 11:00:35 am | 1:X 05 Apr 2022 17:00:35.895 # +sdown sentinel 6cfe9063dd168605a311b2eda37ca1e4fbbca27b redis2-node-2.redis2-headless.redis2.svc.cluster.local 26379 @ mymaster redis2-node-0.redis2-headless.redis2.svc.cluster.local 6379
Tue, Apr 5 2022 11:00:39 am | 1:X 05 Apr 2022 17:00:39.014 # +sdown sentinel 00de773dc2a8f87fb07d3446fdbc0f99e0dbefe1 redis2-node-1.redis2-headless.redis2.svc.cluster.local 26379 @ mymaster redis2-node-0.redis2-headless.redis2.svc.cluster.local 6379
Tue, Apr 5 2022 11:00:47 am | 1:X 05 Apr 2022 17:00:47.210 # -sdown sentinel 00de773dc2a8f87fb07d3446fdbc0f99e0dbefe1 redis2-node-1.redis2-headless.redis2.svc.cluster.local 26379 @ mymaster redis2-node-0.redis2-headless.redis2.svc.cluster.local 6379
Tue, Apr 5 2022 11:00:48 am | 1:X 05 Apr 2022 17:00:48.973 # -sdown sentinel 6cfe9063dd168605a311b2eda37ca1e4fbbca27b redis2-node-2.redis2-headless.redis2.svc.cluster.local 26379 @ mymaster redis2-node-0.redis2-headless.redis2.svc.cluster.local 6379
Tue, Apr 5 2022 11:01:05 am | 1:X 05 Apr 2022 17:01:05.186 # +tilt #tilt mode entered
Tue, Apr 5 2022 11:01:10 am | 1:X 05 Apr 2022 17:01:10.583 # +tilt #tilt mode entered
Tue, Apr 5 2022 11:01:40 am | 1:X 05 Apr 2022 17:01:40.632 # -tilt #tilt mode exited
Tue, Apr 5 2022 11:01:48 am | 1:X 05 Apr 2022 17:01:48.130 # +sdown sentinel 6cfe9063dd168605a311b2eda37ca1e4fbbca27b redis2-node-2.redis2-headless.redis2.svc.cluster.local 26379 @ mymaster redis2-node-0.redis2-headless.redis2.svc.cluster.local 6379
Tue, Apr 5 2022 11:01:51 am | 1:X 05 Apr 2022 17:01:51.090 # -sdown sentinel 6cfe9063dd168605a311b2eda37ca1e4fbbca27b redis2-node-2.redis2-headless.redis2.svc.cluster.local 26379 @ mymaster redis2-node-0.redis2-headless.redis2.svc.cluster.local 6379
Tue, Apr 5 2022 11:01:57 am | 1:X 05 Apr 2022 17:01:57.288 # +sdown sentinel 6cfe9063dd168605a311b2eda37ca1e4fbbca27b redis2-node-2.redis2-headless.redis2.svc.cluster.local 26379 @ mymaster redis2-node-0.redis2-headless.redis2.svc.cluster.local 6379
Tue, Apr 5 2022 11:02:00 am | 1:X 05 Apr 2022 17:02:00.505 # -sdown sentinel 6cfe9063dd168605a311b2eda37ca1e4fbbca27b redis2-node-2.redis2-headless.redis2.svc.cluster.local 26379 @ mymaster redis2-node-0.redis2-headless.redis2.svc.cluster.local 6379
Tue, Apr 5 2022 11:11:50 am | 1:X 05 Apr 2022 17:11:50.006 # +tilt #tilt mode entered
Tue, Apr 5 2022 11:12:20 am | 1:X 05 Apr 2022 17:12:20.012 # -tilt #tilt mode exited
Tue, Apr 5 2022 11:13:19 am | 1:X 05 Apr 2022 17:13:19.666 # +tilt #tilt mode entered
Tue, Apr 5 2022 11:13:29 am | 1:X 05 Apr 2022 17:13:29.830 # +tilt #tilt mode entered
Tue, Apr 5 2022 11:13:49 am | 1:X 05 Apr 2022 17:13:49.668 # +tilt #tilt mode entered
Tue, Apr 5 2022 11:14:10 am | 1:X 05 Apr 2022 17:14:10.757 # +tilt #tilt mode entered
Tue, Apr 5 2022 11:14:40 am | 1:X 05 Apr 2022 17:14:40.814 # -tilt #tilt mode exited
Tue, Apr 5 2022 11:14:40 am | 1:X 05 Apr 2022 17:14:40.814 # +sdown sentinel 00de773dc2a8f87fb07d3446fdbc0f99e0dbefe1 redis2-node-1.redis2-headless.redis2.svc.cluster.local 26379 @ mymaster redis2-node-0.redis2-headless.redis2.svc.cluster.local 6379
Tue, Apr 5 2022 11:14:46 am | 1:X 05 Apr 2022 17:14:46.799 # -sdown sentinel 00de773dc2a8f87fb07d3446fdbc0f99e0dbefe1 redis2-node-1.redis2-headless.redis2.svc.cluster.local 26379 @ mymaster redis2-node-0.redis2-headless.redis2.svc.cluster.local 6379
Tue, Apr 5 2022 11:16:49 am | 1:X 05 Apr 2022 17:16:49.080 # +tilt #tilt mode entered
Tue, Apr 5 2022 11:17:19 am | 1:X 05 Apr 2022 17:17:19.082 # -tilt #tilt mode exited
Tue, Apr 5 2022 11:19:56 am | 1:X 05 Apr 2022 17:19:56.158 # +tilt #tilt mode entered
Tue, Apr 5 2022 11:20:01 am | 1:X 05 Apr 2022 17:20:01.331 # +tilt #tilt mode entered
Tue, Apr 5 2022 11:20:31 am | 1:X 05 Apr 2022 17:20:31.425 # -tilt #tilt mode exited
Tue, Apr 5 2022 11:20:40 am | 1:X 05 Apr 2022 17:20:40.730 # +sdown sentinel 6cfe9063dd168605a311b2eda37ca1e4fbbca27b redis2-node-2.redis2-headless.redis2.svc.cluster.local 26379 @ mymaster redis2-node-0.redis2-headless.redis2.svc.cluster.local 6379
Tue, Apr 5 2022 11:20:44 am | 1:X 05 Apr 2022 17:20:44.250 # -sdown sentinel 6cfe9063dd168605a311b2eda37ca1e4fbbca27b redis2-node-2.redis2-headless.redis2.svc.cluster.local 26379 @ mymaster redis2-node-0.redis2-headless.redis2.svc.cluster.local 6379
Tue, Apr 5 2022 11:40:07 am | 1:X 05 Apr 2022 17:40:07.161 # +sdown sentinel 00de773dc2a8f87fb07d3446fdbc0f99e0dbefe1 redis2-node-1.redis2-headless.redis2.svc.cluster.local 26379 @ mymaster redis2-node-0.redis2-headless.redis2.svc.cluster.local 6379
Tue, Apr 5 2022 11:40:09 am | 1:X 05 Apr 2022 17:40:09.151 # +sdown sentinel 6cfe9063dd168605a311b2eda37ca1e4fbbca27b redis2-node-2.redis2-headless.redis2.svc.cluster.local 26379 @ mymaster redis2-node-0.redis2-headless.redis2.svc.cluster.local 6379
Tue, Apr 5 2022 11:40:26 am | 1:X 05 Apr 2022 17:40:26.615 # +tilt #tilt mode entered
Tue, Apr 5 2022 11:40:56 am | 1:X 05 Apr 2022 17:40:56.655 # -tilt #tilt mode exited
Tue, Apr 5 2022 11:40:56 am | 1:X 05 Apr 2022 17:40:56.655 # -sdown sentinel 00de773dc2a8f87fb07d3446fdbc0f99e0dbefe1 redis2-node-1.redis2-headless.redis2.svc.cluster.local 26379 @ mymaster redis2-node-0.redis2-headless.redis2.svc.cluster.local 6379
Tue, Apr 5 2022 11:40:56 am | 1:X 05 Apr 2022 17:40:56.655 # -sdown sentinel 6cfe9063dd168605a311b2eda37ca1e4fbbca27b redis2-node-2.redis2-headless.redis2.svc.cluster.local 26379 @ mymaster redis2-node-0.redis2-headless.redis2.svc.cluster.local 6379
Tue, Apr 5 2022 11:41:03 am | 1:X 05 Apr 2022 17:41:03.956 # +sdown sentinel 00de773dc2a8f87fb07d3446fdbc0f99e0dbefe1 redis2-node-1.redis2-headless.redis2.svc.cluster.local 26379 @ mymaster redis2-node-0.redis2-headless.redis2.svc.cluster.local 6379
Tue, Apr 5 2022 11:41:13 am | 1:X 05 Apr 2022 17:41:13.376 # +tilt #tilt mode entered
Tue, Apr 5 2022 11:41:33 am | 1:X 05 Apr 2022 17:41:33.677 # +tilt #tilt mode entered
Tue, Apr 5 2022 11:41:43 am | 1:X 05 Apr 2022 17:41:43.997 # +new-epoch 1
Tue, Apr 5 2022 11:41:44 am | 1:X 05 Apr 2022 17:41:44.002 # +tilt #tilt mode entered
Tue, Apr 5 2022 11:42:04 am | 1:signal-handler (1649180524) Received SIGTERM scheduling shutdown...
Tue, Apr 5 2022 11:42:04 am | 1:X 05 Apr 2022 17:42:04.660 # User requested shutdown...
Tue, Apr 5 2022 11:42:04 am | 1:X 05 Apr 2022 17:42:04.660 # Sentinel is now ready to exit, bye bye...

Finished another test, this time with a more basic values file. Exact same behavior. I do not believe I can reduce the values more than this.

---
image:
  debug: true
auth:
  password: "password"
replica:
  replicaCount: 3
  persistence:
    enabled: false
sentinel:
  enabled: true
  image:
    debug: true
  quorum: 2

sentinel log

17:58:20.77 INFO ==> about to run the command: REDISCLI_AUTH=$REDIS_PASSWORD redis-cli -h redis2.redis2.svc.cluster.local -p 26379 sentinel get-master-addr-by-name mymaster
--
Tue, Apr 5 2022 11:58:21 am | Could not connect to Redis at redis2.redis2.svc.cluster.local:26379: Connection refused
Tue, Apr 5 2022 11:58:22 am | 1:X 05 Apr 2022 17:58:22.112 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
Tue, Apr 5 2022 11:58:22 am | 1:X 05 Apr 2022 17:58:22.112 # Redis version=6.2.6, bits=64, commit=00000000, modified=0, pid=1, just started
Tue, Apr 5 2022 11:58:22 am | 1:X 05 Apr 2022 17:58:22.112 # Configuration loaded
Tue, Apr 5 2022 11:58:22 am | 1:X 05 Apr 2022 17:58:22.113 * monotonic clock: POSIX clock_gettime
Tue, Apr 5 2022 11:58:22 am | 1:X 05 Apr 2022 17:58:22.113 * Running mode=sentinel, port=26379.
Tue, Apr 5 2022 11:58:22 am | 1:X 05 Apr 2022 17:58:22.113 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
Tue, Apr 5 2022 11:58:22 am | 1:X 05 Apr 2022 17:58:22.114 # Sentinel ID is d37bb3ee651ae251d15e4917d9e8571b71e35013
Tue, Apr 5 2022 11:58:22 am | 1:X 05 Apr 2022 17:58:22.114 # +monitor master mymaster redis2-node-0.redis2-headless.redis2.svc.cluster.local 6379 quorum 2
Tue, Apr 5 2022 11:58:48 am | 1:X 05 Apr 2022 17:58:48.550 * +sentinel sentinel 00de773dc2a8f87fb07d3446fdbc0f99e0dbefe1 redis2-node-1.redis2-headless.redis2.svc.cluster.local 26379 @ mymaster redis2-node-0.redis2-headless.redis2.svc.cluster.local 6379
Tue, Apr 5 2022 11:58:52 am | 1:X 05 Apr 2022 17:58:52.144 * +slave slave redis2-node-1.redis2-headless.redis2.svc.cluster.local:6379 redis2-node-1.redis2-headless.redis2.svc.cluster.local 6379 @ mymaster redis2-node-0.redis2-headless.redis2.svc.cluster.local 6379
Tue, Apr 5 2022 11:59:13 am | 1:X 05 Apr 2022 17:59:13.899 * +sentinel sentinel 6cfe9063dd168605a311b2eda37ca1e4fbbca27b redis2-node-2.redis2-headless.redis2.svc.cluster.local 26379 @ mymaster redis2-node-0.redis2-headless.redis2.svc.cluster.local 6379
Tue, Apr 5 2022 11:59:13 am | 1:X 05 Apr 2022 17:59:13.916 * +sentinel-address-switch master mymaster redis2-node-0.redis2-headless.redis2.svc.cluster.local 6379 ip redis2-node-2.redis2-headless.redis2.svc.cluster.local port 26379 for 6cfe9063dd168605a311b2eda37ca1e4fbbca27b
Tue, Apr 5 2022 11:59:15 am | 1:X 05 Apr 2022 17:59:15.983 * +sentinel sentinel 6cfe9063dd168605a311b2eda37ca1e4fbbca27b redis2-node-2.redis2-headless.redis2.svc.cluster.local 26379 @ mymaster redis2-node-0.redis2-headless.redis2.svc.cluster.local 6379
Tue, Apr 5 2022 11:59:16 am | 1:X 05 Apr 2022 17:59:16.004 * +sentinel-address-switch master mymaster redis2-node-0.redis2-headless.redis2.svc.cluster.local 6379 ip redis2-node-2.redis2-headless.redis2.svc.cluster.local port 26379 for 6cfe9063dd168605a311b2eda37ca1e4fbbca27b
Tue, Apr 5 2022 11:59:16 am | 1:X 05 Apr 2022 17:59:16.035 * +sentinel sentinel 6cfe9063dd168605a311b2eda37ca1e4fbbca27b redis2-node-2.redis2-headless.redis2.svc.cluster.local 26379 @ mymaster redis2-node-0.redis2-headless.redis2.svc.cluster.local 6379
Tue, Apr 5 2022 11:59:22 am | 1:X 05 Apr 2022 17:59:22.298 * +slave slave redis2-node-2.redis2-headless.redis2.svc.cluster.local:6379 redis2-node-2.redis2-headless.redis2.svc.cluster.local 6379 @ mymaster redis2-node-0.redis2-headless.redis2.svc.cluster.local 6379
Tue, Apr 5 2022 12:00:31 pm | 1:X 05 Apr 2022 18:00:31.909 # +tilt #tilt mode entered
Tue, Apr 5 2022 12:01:01 pm | 1:X 05 Apr 2022 18:01:01.946 # -tilt #tilt mode exited
Tue, Apr 5 2022 12:42:08 pm | 1:X 05 Apr 2022 18:42:08.900 # +tilt #tilt mode entered
Tue, Apr 5 2022 12:42:38 pm | 1:X 05 Apr 2022 18:42:38.906 # -tilt #tilt mode exited
Tue, Apr 5 2022 12:49:05 pm | 1:X 05 Apr 2022 18:49:05.948 # +tilt #tilt mode entered
Tue, Apr 5 2022 12:49:35 pm | 1:X 05 Apr 2022 18:49:35.954 # -tilt #tilt mode exited
Tue, Apr 5 2022 12:50:08 pm | 1:X 05 Apr 2022 18:50:08.378 # +tilt #tilt mode entered
Tue, Apr 5 2022 12:50:38 pm | 1:X 05 Apr 2022 18:50:38.391 # -tilt #tilt mode exited
Tue, Apr 5 2022 12:54:41 pm | 1:X 05 Apr 2022 18:54:41.707 # +tilt #tilt mode entered
Tue, Apr 5 2022 12:55:11 pm | 1:X 05 Apr 2022 18:55:11.738 # -tilt #tilt mode exited
Tue, Apr 5 2022 12:55:32 pm | 1:X 05 Apr 2022 18:55:32.159 # +tilt #tilt mode entered
Tue, Apr 5 2022 12:55:47 pm | 1:X 05 Apr 2022 18:55:47.903 # +tilt #tilt mode entered
Tue, Apr 5 2022 12:56:17 pm | 1:X 05 Apr 2022 18:56:17.947 # -tilt #tilt mode exited
Tue, Apr 5 2022 12:59:57 pm | 1:X 05 Apr 2022 18:59:57.456 # +tilt #tilt mode entered
Tue, Apr 5 2022 1:00:27 pm | 1:X 05 Apr 2022 19:00:27.494 # -tilt #tilt mode exited
Tue, Apr 5 2022 1:03:35 pm | 1:X 05 Apr 2022 19:03:35.305 # +tilt #tilt mode entered
Tue, Apr 5 2022 1:03:40 pm | 1:X 05 Apr 2022 19:03:40.660 # +tilt #tilt mode entered
Tue, Apr 5 2022 1:03:50 pm | 1:X 05 Apr 2022 19:03:50.853 # +tilt #tilt mode entered
Tue, Apr 5 2022 1:04:06 pm | 1:X 05 Apr 2022 19:04:06.261 # +tilt #tilt mode entered
Tue, Apr 5 2022 1:04:06 pm | 1:X 05 Apr 2022 19:04:06.305 # +new-epoch 1
Tue, Apr 5 2022 1:04:11 pm | 1:X 05 Apr 2022 19:04:11.550 # +tilt #tilt mode entered
Tue, Apr 5 2022 1:04:22 pm | 1:X 05 Apr 2022 19:04:22.434 # +config-update-from sentinel 6cfe9063dd168605a311b2eda37ca1e4fbbca27b redis2-node-2.redis2-headless.redis2.svc.cluster.local 26379 @ mymaster redis2-node-0.redis2-headless.redis2.svc.cluster.local 6379
Tue, Apr 5 2022 1:04:22 pm | 1:X 05 Apr 2022 19:04:22.434 # +switch-master mymaster redis2-node-0.redis2-headless.redis2.svc.cluster.local 6379 redis2-node-2.redis2-headless.redis2.svc.cluster.local 6379
Tue, Apr 5 2022 1:04:22 pm | 1:X 05 Apr 2022 19:04:22.441 * +slave slave redis2-node-1.redis2-headless.redis2.svc.cluster.local:6379 redis2-node-1.redis2-headless.redis2.svc.cluster.local 6379 @ mymaster redis2-node-2.redis2-headless.redis2.svc.cluster.local 6379
Tue, Apr 5 2022 1:04:22 pm | 1:X 05 Apr 2022 19:04:22.441 * +slave slave redis2-node-0.redis2-headless.redis2.svc.cluster.local:6379 redis2-node-0.redis2-headless.redis2.svc.cluster.local 6379 @ mymaster redis2-node-2.redis2-headless.redis2.svc.cluster.local 6379
Tue, Apr 5 2022 1:04:22 pm | 1:signal-handler (1649185462) Received SIGTERM scheduling shutdown...
Tue, Apr 5 2022 1:04:22 pm | 1:X 05 Apr 2022 19:04:22.937 # User requested shutdown...
Tue, Apr 5 2022 1:04:22 pm | 1:X 05 Apr 2022 19:04:22.937 # Sentinel is now ready to exit, bye bye...

For sanity I reconfigured chrony on the hosts and configured them all to the same server. This did not help either.

Hi! After some attempts trying to reproduce the issue, it seems it is not an issue related to the Bitnami Redis Helm chart but about how the application or environment is being used/configured.

For information regarding the application itself, customization of the content within the application, or questions about the use of technology or infrastructure; we highly recommend checking forums and user guides made available by the project behind the application or the technology.

That said, I will keep this ticket open until the stale bot closes it just in case someone from the community adds some valuable info.

~~Yeah I did not think it was a problem with the chart.~~

I am just completely baffled by this behavior, and if anyone has any ideas as to what could be causing this, I would be very grateful.

https://github.com/redis/redis/issues/10547#issuecomment-1092120764

@carrodher, is Redis Sentinel 7.0+ on the roadmap? I have not seen any reference nor do I see anything in https://hub.docker.com/r/bitnami/redis-sentinel

Thanks!

Hi, at this moment, Redis 7.0 is a "Release Candidate" version, once released this new major version as a stable version by the upstream Redis maintainer, we will include it in our catalog as usual. The policy at this moment is to support as many branches as supported by the upstream projects.

Hello @carrodher,

At this point I think I have it narrowed it down to Rancher RKE https://github.com/redis/redis/issues/10547#issuecomment-1098197509

I am wondering if you or someone else can test and can confirm?

I had same issue with bitnami images. Looks like default resource limits are to low. I was able to get reed of the issue once provided more resources:

resources:
      requests:
        cpu: "2"
        memory: 1024Mi
      limits:
        cpu: "2"
        memory: 1024Mi

also I have memory limit in configuration that is equal to limits.

Unfortunately no,

The chart does not have default. https://github.com/bitnami/charts/blob/master/bitnami/redis/values.yaml#L1061

And setting that does not help. I tried it out thinking maybe RKE does something weird, but nope. Screenshot_20220414_133154 Even when I increase the limit to an absurd amount Screenshot_20220414_133748 Screenshot_20220414_133622

The tilting can take a while to happen, but it eventually will.

EDIT: No difference on RKE2 cluster either Screenshot_20220414_134131

@jonathon2nd yep sorry for that. I was able to detect issue after 8h under load, however it's become more rare case

What is your setup @OxCom? Are you recreating issue on prem or in cloud?

@jonathon2nd on prem. Also I tried keep data in memory or save to disk, but still same issue in logs. monitoring shows only 500k keys and no load by memory and CPU.

@OxCom, Do you use Rancher? If so is your downstream k8s cluster RKE?

@jonathon2nd nope, we rolled out in hard way. only heml charts and custom yamls.

Hi @carrodher, I noticed the reference to custom images here: https://github.com/bitnami/charts/blob/master/bitnami/redis/values.yaml#L159 Is there any way to fork and modify the bitnami/redis image?

Yes, the code is present at https://github.com/bitnami/bitnami-docker-redis, you can fork this repo and do any modification in the container logic.

Then, you can use your own image by using it in the image: section https://github.com/bitnami/charts/blob/master/bitnami/redis/values.yaml#L78

I looked and it seems like the actual redis executables are downloaded from here https://github.com/bitnami/bitnami-docker-redis/blob/master/6.2/debian-10/prebuildfs/opt/bitnami/scripts/libcomponent.sh, use vars from here https://github.com/bitnami/bitnami-docker-redis/blob/master/6.2/debian-10/Dockerfile

To build a url, example: https://downloads.bitnami.com/files/stacksmith/redis-6.2.6-12-linux-amd64-debian-10.tar.gz

I see that bitnami has 12 iterations of 6.2.6, where as the official redis image only has one. We also see that the different 6.2.6 tars that bitnami is hosting are different. Is there a way to see the pipeline bitnami is using to build these executables?

That part is not hosted on GitHub. The main versioning is following the upstream releases, I mean, 6.2.6 and other versions will follow the same cadence as the Redis project. The revision (-12) is an internal version to track changes in our compilation recipes

Thanks! Turns out it does not really make a difference.

We are still trying to figure out what is happening. All tests still lead to it being some interaction between binami/redis and Rancher RKE. At least that is the best conclusion that matches everything so far.

We were thinking it was due to some change bitnami makes to the redis exec, because it does not match official redis at all.

Redis:6.2.6

root@redis-0:/usr/local/bin# ls -al
total 27460
drwxr-xr-x 1 root root       34 Mar 29 16:17 .
drwxr-xr-x 1 root root       17 Mar 28 00:00 ..
-rwxrwxr-x 1 root root      374 Mar 29 16:16 docker-entrypoint.sh
-rwxr-xr-x 1 root root  2289664 Dec  7 05:48 gosu
-rwxr-xr-x 1 root root  6783368 Mar 29 16:17 redis-benchmark
lrwxrwxrwx 1 root root       12 Mar 29 16:17 redis-check-aof -> redis-server
lrwxrwxrwx 1 root root       12 Mar 29 16:17 redis-check-rdb -> redis-server
-rwxr-xr-x 1 root root  6684952 Mar 29 16:17 redis-cli
lrwxrwxrwx 1 root root       12 Mar 29 16:17 redis-sentinel -> redis-server
-rwxr-xr-x 1 root root 12346120 Mar 29 16:17 redis-server

bitnami/redis:6.2.6-debian-10-r192

I have no name!@redis-node-0:/opt/bitnami/redis/bin$ ls -al
total 4688
drwxrwxr-x. 1 root root      66 Mar  2 13:47 .
drwxrwxr-x. 1 root root      25 Apr 28 16:03 ..
-rwxrwxr-x. 1 root root 1290320 Mar  2 11:30 redis-benchmark
lrwxrwxrwx. 1 root root      12 Mar  2 11:30 redis-check-aof -> redis-server
lrwxrwxrwx. 1 root root      12 Mar  2 11:30 redis-check-rdb -> redis-server
-rwxrwxr-x. 1 root root  877040 Mar  2 11:30 redis-cli
lrwxrwxrwx. 1 root root      12 Mar  2 11:30 redis-sentinel -> redis-server
-rwxrwxr-x. 1 root root 2622152 Mar  2 11:30 redis-server

I then noticed that there are different tars for redis-sentinel entirely https://downloads.bitnami.com/files/stacksmith/redis-6.2.6-9-linux-amd64-debian-10.tar.gz vs https://downloads.bitnami.com/files/stacksmith/redis-sentinel-6.2.6-9-linux-amd64-debian-10.tar.gz

So I started comparing. And there are differences between versions compare redis 2.6.2-12 to sentinel 2.6.2-9

Compare redis 2.6.2-9 to sentinel 2.6.2-9

Digging into it more, it looks like the redis and redis-sentinel tars are mostly the same, looks like some header or info change bitnami redis

:~/Downloads/redis-6.2.6-9-linux-amd64-debian-10/redis-6.2.6-linux-amd64-debian-10/files/redis/bin$ ls -al *
-rwxr-xr-x 1 jonathon jonathon 1290320 Feb 15 13:46 redis-benchmark
lrwxrwxrwx 1 jonathon jonathon      12 Apr 28 11:23 redis-check-aof -> redis-server
lrwxrwxrwx 1 jonathon jonathon      12 Apr 28 11:23 redis-check-rdb -> redis-server
-rwxr-xr-x 1 jonathon jonathon  877040 Feb 15 13:46 redis-cli
lrwxrwxrwx 1 jonathon jonathon      12 Apr 28 11:23 redis-sentinel -> redis-server
-rwxr-xr-x 1 jonathon jonathon 2112688 Feb 15 13:46 redis-server
:~/Downloads/redis-6.2.6-9-linux-amd64-debian-10/redis-6.2.6-linux-amd64-debian-10/files/redis/bin$ sha1sum redis-server 
f05c7fe54a9ef19d66632157a5a32264d1798847  redis-server

bitnami sentinel

:~/Downloads/redis-sentinel-6.2.6-9-linux-amd64-debian-10/redis-sentinel-6.2.6-linux-amd64-debian-10/files/redis-sentinel/bin$ ls -al *
-rwxr-xr-x 1 jonathon jonathon 1290320 Mar  1 07:04 redis-benchmark
lrwxrwxrwx 1 jonathon jonathon      12 Apr 28 11:21 redis-check-aof -> redis-server
lrwxrwxrwx 1 jonathon jonathon      12 Apr 28 11:21 redis-check-rdb -> redis-server
-rwxr-xr-x 1 jonathon jonathon  877040 Mar  1 07:04 redis-cli
lrwxrwxrwx 1 jonathon jonathon      12 Apr 28 11:21 redis-sentinel -> redis-server
-rwxr-xr-x 1 jonathon jonathon 2112688 Mar  1 07:04 redis-server
:~/Downloads/redis-sentinel-6.2.6-9-linux-amd64-debian-10/redis-sentinel-6.2.6-linux-amd64-debian-10/files/redis-sentinel/bin$ sha1sum redis-server
b5ca066c6043ddf38c2907ed393d7f91491ded1b  redis-server

So to test, we reinstall the chart in diagnostic mode, and installed redis from source. And replacing it in sentinel start script. Unfortunately, redis sentinel still tilted. This is most of the log from my redis sentinel container. It captures most of what I did, shows that the version number is different meaning it is using the newly built executable. manual-offical-redis-build-exec-with bitanmi-stack.log

The reason we tried this is because official redis sentinel never tilts. Even 17 days later. So we are trying to figure out what the difference is. Screenshot_20220428_164152 The reason the redis version in the log is 6.2.6 is because I tried it to see if 7.0 just fixed the problem, but no. Screenshot_20220428_164849

Something we tried yesterday was separating the redis and sentinel containers onto different pods, sentinel still tilts like crazy.

We got a couple more ideas we would like to try.

Have done some more tests.

Since changing out the bitnami built executable for redis source did not work, we quickly tried out changing the base image to debian:bullseye-slim and built our own fork, this resulted in no change.

We also tried changing the vm clock source from xen to hpet, this did not change anything either

Looking at the single manual official redis I setup (https://github.com/redis/redis/issues/10547#issuecomment-1093259568), I needed to retest a single bitnami/redis node. I had thought I had done this and documented it, but I can not find anything I recorded. So redid the test, it has been about three hours and no tilting yet Screenshot_20220429_162652 And for the manual redis we are at 18 days with no tilting still Screenshot_20220429_162932

So this got us thinking that even though the logic of tilting should just be dictated by the vm the container is running on (either by cpu/io issues or clock issues), something must be happening either network/controller wise, or the sentinels communicating that is somehow triggering issue.

To test this, I deployed a bitnami/redis replica count of 3 to the same vm. Not only did it still tilt, only two nodes have tilted so far, while one node did not tilt. Screenshot_20220429_163601 Screenshot_20220429_163613 Screenshot_20220429_163626 If it was VM related, I would have expected them all to tilt at the same time, but that did not happen.

So now to eliminate that official redis will not tilt with more than one node, I wrote up new yamls. manual-offical-redis.txt I did this twice, with redis:7.0 and redis:6.2.6 just to be safe This is not a perfect setup. But the sentinels are talking which should be all that matters. The sentinels talking should not matter for tilting, but at this point testing anything/everything.

And so far, those have not tilted This could mean a couple of things, too soon to tell going to let everything run over the weekend. If it never tilts while other tests continue to tilt, we can be affirmed that it somehow has something to do with bitnami config/scripts/something. If however it does tilt (which it has not so far), We can be sure that it is somehow a base redis problem, and has nothing to do with bitnami. Regardless this is caused by some strange interaction with our environment, but still trying to figure out exactly what is happening. Screenshot_20220429_164618 All three logs are the same, so not including.

Can confirm I'm also facing the same title issue with the sentinel container in redis-node-2. Tried bumping the resource limits but all in vain. I tried @jonathon2nd 's yml file and it seems to work for me too. Is bitnami/redis helm chart broken?

My org is facing this issue too. Would love to find a solution.

Any update on this? Thanks.

We experience this as well. Seems like the charts are not production ready. Using rancher k8s. Any update?

Hi all, note that we build Redis in the following way:

CFLAGS="-g0 -fno-omit-frame-pointer" make BUILD_TLS=yes
make install PREFIX=/opt/bitnami/redis BUILD_TLS=yes

Apart from that we don't do anything special. If you are finding issues using bitnami/redis you will most probably find them on your side as well if you build Redis from source on your side.

We know it's not the actual build, since we've changed the binary (as previously documented) and it didn't solve the issue.

However, testing it using an entirely different redis container image seems to fix the issue when deploying a similar redis sentinel cluster. If you look above at https://github.com/bitnami/charts/issues/9689#issuecomment-1113846711 you will see that I created a redis sentinel cluster manually with official redis images, and it does not have the same problem.

It appears from all my testing and other user reports (ex: https://github.com/bitnami/charts/issues/9689#issuecomment-1129650807) that it is some interactiong between bitnami/redis script or config and Rancher RKE. Is it possible for you to attempt to replicate @marcosbc or @carrodher ?

Hi @jonathon2nd, we appreciate the detailed investigation on your side.

It appears from all my testing and other user reports (ex: https://github.com/bitnami/charts/issues/9689#issuecomment-1129650807) that it is some interactiong between bitnami/redis script or config and Rancher RKE.

Have you considered deploying a Bitnami chart with the "official" image configuration and see if that helps? But using similar YAML files to the ones you described above. I don't think it is related to any scripts, since in the case of Redis Sentinel, they only set configuration entries, and when you mount a configuration file, these steps are skipped.

Is it possible for you to attempt to replicate @marcosbc or @carrodher ?

I've deployed a Redis and left it running about 12 hours, but no tilting yet... Should it happen in a specific node (e.g. node 0) or is it common to the entire cluster? I will leave it running for a few more hours/days if possible, in order to see if we are able to get it.

Hi @marcosbc

Have you considered deploying a Bitnami chart with the "official" image configuration and see if that helps

So the images are not innerchangable. What we did do was install the chart in diagnostic mode. We manually build redis and changed out the usage of the binary. Here I just updated the script, but my lead actually moved sym links on hist pod that he was testing. Both still tilted. https://github.com/bitnami/charts/issues/9689#issuecomment-1112721115

I've deployed a Redis and left it running about 12 hours

Is this spesicly an RKE cluster? If you want I could help out with setup on your side. But if no one in your circle has a rancher management cluster, you can setup a test instance of rancher with the following https://rancher.com/docs/rancher/v2.6/en/quick-start-guide/deployment/quickstart-manual-setup/

I have not used this, so I can not speak to it's ease of use. Our Rancher management cluster is setup on an RKE cluster, but I do not think it will matter what rancher is running on. As long as you can then create an RKE managed k8s cluster to deploy bitnami/redis on.

It will happen with all nodes, any order, all at once or one at a time. It may also take some time. We have seen periods of no tilting that have lasted many hours.

You would think this would be host related, but no. We have setup remote public cloud instances and they have had the exact same tilting behavior.

I hope this is able to help.

What is really strange is that a single node of either official redis or bitnami/redis does not tilt. From the doc and a redis dev response https://github.com/redis/redis/issues/10547#issuecomment-1094194736 a single node should be able to tilt.

It is when the replica is scaled up to 3 that bitnami/redis will start to tilt, while official redis does not. Because of that, and the testing with public cloud vm's, somehow it has to be related to some sort of networking or communications between the sentinels.

Hi all, we were able to reproduce the tilt issue, but it was only once in various days (i.e. a single "tilt mode entered" and "tilt mode exited"), so we're unsure if it could be unrelated to this or some networking issue of our cluster.

I will create an internal task for investigating this further, however, I'm unable to provide any ETA for when we will look into this. In the meantime, if anyone is able to find more information about the issue, or even its cause feel free to update this issue as it would be very appreciated. PRs are welcome as well.

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

We also face the same issue on multiple RKE systems

Same applies to chart version 16.9.3. After some time Sentinel stops responding to liveness and readiness probes.

Bumping up net.core.somaxconn (initially I thought it was an issue with connection limit) also did not help.

Did you guys found the core of the problem here? We are using bitnami redis helm chart 16.8.0 in GKE and sentinel is tilting as crazy. Nodes are Ubuntu based.

Tilt mode is entered pretty often in case of time traveling. Sentinel is very sensitive to time changes as described in https://redis.io/docs/manual/sentinel/#tilt-mode.

First thing to fix: stop using ntpd, switch to chronyd.

Sadly the tilt mode time is not configurable, it should be shortened but it's not possible without recompiling. The best would be actually to have possibility to disable tilt mode.

I know about time sensitivity, but since this is a pod in kubernetes being deployed from bitnami redis image what can I do about it?

I know about time sensitivity, but since this is a pod in kubernetes being deployed from bitnami redis image what can I do about it?

Bitnami images has not a lot to do here, it's sentinel that's messed up and nobody knows how to solve this properly.

What's used for syncing the time on your nodes?

Nodes are managed GKE with Ubuntu and Docker but not able to check or change the time sync daemon.

First thing to fix: stop using ntpd, switch to chronyd.

...

I know about time sensitivity, but since this is a pod in kubernetes being deployed from bitnami redis image what can I do about it?

Bitnami images has not a lot to do here, it's sentinel that's messed up and nobody knows how to solve this properly.

What's used for syncing the time on your nodes?

As I have documented before, chrony does not fix this issue. Even if manually setting all nodes to use the same time server.

Somehow this is bitnami related. We have manually deployed redis sentinel cluster using official redis images and our own yamls, completely ignoring bitnami helm. And there is no issues with sentinel tilting.

We have tested on brand new servers from our venders. All up to date hypervisors and hosts, deployed with brand new rke2 and nothing else on these AMD EPYC 7402 24-Core Processor servers. And it still tilts.

We do not understand what is happening, and this is obviously happening to other people deploying this chart.

Somehow https://github.com/bitnami/charts/compare/master...maxfilov:charts:master?diff=split made our redis/sentinel stable. From tilting every ~5m to days of uptime.

This patch makes the redis and sentinels announce themselves using their pod ip instead of their domain name. There might be some strange interactions between sentinel and the way rke2 handle DNS names resolution

@Melchior01 I was excited to try out your fix, and unfortunately it did not help.

I did notice that the containers are not restarting which is interesting, but the tilting is still occurring. Image: docker.io/bitnami/redis-sentinel:7.0.4-debian-11-r8 helm.sh/chart: redis-17.0.10

Can see here that it is using ip

1:X 16 Aug 2022 19:25:29.991 # -sdown master mymaster redis-ip-node-0.redis-ip 30011
--
Tue, Aug 16 2022 1:25:40 pm | 1:X 16 Aug 2022 19:25:40.035 * +slave slave 10.42.124.192:30015 10.42.124.192 30015 @ mymaster redis-ip-node-0.redis-ip 30011
Tue, Aug 16 2022 1:25:40 pm | 1:X 16 Aug 2022 19:25:40.037 * Sentinel new configuration saved on disk
Tue, Aug 16 2022 1:25:40 pm | 1:X 16 Aug 2022 19:25:40.037 * +slave slave 10.42.36.128:30013 10.42.36.128 30013 @ mymaster redis-ip-node-0.redis-ip 30011
Tue, Aug 16 2022 1:25:40 pm | 1:X 16 Aug 2022 19:25:40.038 * Sentinel new configuration saved on disk
Tue, Aug 16 2022 1:26:01 pm | 1:X 16 Aug 2022 19:26:01.117 # +tilt #tilt mode entered
Tue, Aug 16 2022 1:26:01 pm | 1:X 16 Aug 2022 19:26:01.189 # waitpid() returned a pid (202) we can't find in our scripts execution queue!
Tue, Aug 16 2022 1:26:31 pm | 1:X 16 Aug 2022 19:26:31.120 # -tilt #tilt mode exited
Tue, Aug 16 2022 1:43:51 pm | 1:X 16 Aug 2022 19:43:51.051 # +tilt #tilt mode entered

redis-ip-node-0_sentinel.log

Unfortunately, this issue was created a long time ago and although there is an internal task to fix it, it was not prioritized as something to address in the short/mid term. It's not a technical reason but something related to the capacity since we're a small team.

Being said that, contributions via PRs are more than welcome in both repositories (containers and charts). Just in case you would like to contribute.

During this time, there are several releases of this asset and it's possible the issue has gone as part of other changes. If that's not the case and you are still experiencing this issue, please feel free to reopen it and we will re-evaluate it.

@carrodher @marcosbc @jonathon2nd did you find any solution of it . i am also facing same problem

Same here, we are having problems with the tilt mode from time-to-time using the bitnami chart. In that case all applications can not connect to the sentinel:

Jan 27, 2023 @ 10:07:15.186 sentinel    cluster1-node-1     # +sdown sentinel 1kz6pq0ik2kus5785me13vvi3w5em9x60q9wjq6k cluster1-node-0.cluster1.example.com 26379 @ cluster1 cluster1-node-2.cluster1.example.com 6379
Jan 27, 2023 @ 10:07:15.186 sentinel    cluster1-node-1     # +sdown sentinel pq00pz0zfgpm42n45mdds3183gvvnb9fgeuoicww cluster1-node-2.cluster1.example.com 26379 @ cluster1 cluster1-node-2.cluster1.example.com 6379
Jan 27, 2023 @ 10:07:17.503 sentinel    cluster1-node-2     # +tilt #tilt mode entered
Jan 27, 2023 @ 10:07:17.503 sentinel    cluster1-node-2     # waitpid() returned a pid (2794392) we can't find in our scripts execution queue!
Jan 27, 2023 @ 10:07:17.503 sentinel    cluster1-node-2     # waitpid() returned a pid (2794401) we can't find in our scripts execution queue!
Jan 27, 2023 @ 10:07:17.503 sentinel    cluster1-node-2     # waitpid() returned a pid (2794410) we can't find in our scripts execution queue!
Jan 27, 2023 @ 10:07:17.703 sentinel    cluster1-node-1     # -sdown sentinel pq00pz0zfgpm42n45mdds3183gvvnb9fgeuoicww cluster1-node-2.cluster1.example.com 26379 @ cluster1 cluster1-node-2.cluster1.example.com 6379
Jan 27, 2023 @ 10:07:17.717 sentinel    cluster1-node-0     # +tilt #tilt mode entered
Jan 27, 2023 @ 10:07:17.979 sentinel    cluster1-node-1     # -sdown sentinel 1kz6pq0ik2kus5785me13vvi3w5em9x60q9wjq6k cluster1-node-0.cluster1.example.com 26379 @ cluster1 cluster1-node-2.cluster1.example.com 6379
Jan 27, 2023 @ 10:07:47.525 sentinel    cluster1-node-2     # -tilt #tilt mode exited
Jan 27, 2023 @ 10:07:47.749 sentinel    cluster1-node-0     # -tilt #tilt mode exited

When using that values.yml:

commonConfiguration: |
  client-output-buffer-limit replica 2000000kb 1000000kb 180
  databases 24
  maxmemory 15625mb
  maxmemory-policy volatile-lru
  maxmemory-samples 10
  notify-keyspace-events "gxE"
  rdb-save-incremental-fsync yes
  repl-backlog-size 500mb
  repl-diskless-sync no
  repl-timeout 180
  stream-node-max-bytes 4kb
  tcp-keepalive 60
  timeout 600
image:
  tag: 6.2.6
master:
  podAntiAffinityPreset: hard
replica:
  podAntiAffinityPreset: hard
  replicaCount: 3
  resources:
    limits:
      memory: 3Gi
    requests:
      cpu: "1"
      memory: 3Gi  
sentinel:
  automateClusterRecovery: true
  downAfterMilliseconds: 2000
  enabled: true
  failoverTimeout: 6000
  getMasterTimeout: 220
  image:
    tag: 6.2.6
  masterSet: cluster1
  readinessProbe:
    initialDelaySeconds: 120
  resources:
    limits:
      memory: 3Gi
    requests:
      cpu: "1"
      memory: 3Gi
useExternalDNS:
  additionalAnnotations:
    ttl: 5
  enabled: true
  suffix: example.com

Would be very happy if somebody can drop a hint. :pray:

we have same problem :) with automateClusterRecovery set to false and the sentinel container entering tilt mode often and reset the sentinel container!

Based on @Melchior01's comment above, I created this PR to disable the usage of the hostnames when configuring the deployment

https://github.com/bitnami/charts/pull/14599

I noticed that other Redis Chart solutions were using the IP instead of the hostnames when configuring the replication and the issue may be related to networking issues. You will need to set the useHostnames variable to false to make it work the new way. I tested the changes during the weekend and didn't get any tilt-related message. Could you please try the changes and provide feedback?

Thanks

@jotamartos if we use ips instead of hostanames like earlier we were gettting a headless service when we fetch sentinel masters . and in our application if we want to connect to it we can connnect with headless service , but now sentinel masters command is returning ip of pod which can change , which is not possible to use in applications using redis .

Hi @abhishekgupta2205,

As you mentioned, if you set useHostnames to false, the deployment will start using the IPs instead of the hostnames. I tested the changes and confirmed everything works as expected with that configuration. I also deleted secondary and primary nodes and everything was reconfigured as expected so you shouldn't find issues when using that configuration. You can get more information about the changes I tested in the PR. If you find any issue, please provide us with specific information about how to reproduce it.

Thanks

bitnami / charts