search

bitnami / charts

Bitnami Helm Charts
https://bitnami.com
Other
8.86k stars 9.14k forks source link

[bitnami/redis] Redis sentinel enters tilt mode and dies after a time #9689

Closed jonathon2nd closed 1 year ago

jonathon2nd commented 2 years ago

Name and Version

bitnami/redis 16.8.2

What steps will reproduce the bug?

Install bitnami/redis in sentinel mode.

Wait a while (It typically happens within an hour) for sentinel containers to start restarting for no reason.

Are you using any custom parameters or values?

---
image:
  debug: true
auth:
  password: "password"
replica:
  replicaCount: 3
  persistence:
    enabled: false
  resources:
    requests:
      cpu: "100m"
      memory: "256Mi"
  limits:
    requests:
      cpu: "1000m"
      memory: "512Mi"
sentinel:
  enabled: true
  service:
    type: NodePort
  image:
    debug: true
  quorum: 2
  downAfterMilliseconds: 2000
  failoverTimeout: 1000
  livenessProbe:
    enabled: true
    initialDelaySeconds: 40
  readinessProbe:
    enabled: true
    initialDelaySeconds: 40
metrics:
  enabled: true
  image:
    tag: latest
  serviceMonitor:
    enabled: true
sysctlImage:
  enabled: true
  mountHostSys: true
  command:
    - /bin/sh
    - -c
    - |-
      install_packages procps
      sysctl -w net.core.somaxconn=1024
      echo never > /host-sys/kernel/mm/transparent_hugepage/enabled

What is the expected behavior?

Expected behavior is sentinel to not restart for no seemingly reason.

What do you see instead?

After some period of time, sentinel containers enter tilt mode on and off again, and then just restart.

Entire log of sentinel container up till restart

23:39:39.04 INFO ==> about to run the command: redis-cli -h redis1.redis1.svc.cluster.local -p 30014 sentinel get-master-addr-by-name mymaster
--
Mon, Apr 4 2022 5:39:39 pm | redis1-node-1.redis1
Mon, Apr 4 2022 5:39:39 pm | 30019
Mon, Apr 4 2022 5:39:39 pm | 23:39:39.20 INFO ==> about to run the command: redis-cli -h redis1.redis1.svc.cluster.local -p 30014 sentinel get-master-addr-by-name mymaster
Mon, Apr 4 2022 5:39:39 pm | 23:39:39.24 INFO ==> printing REDIS_SENTINEL_INFO=(redis1-node-1.redis1,30019)
Mon, Apr 4 2022 5:39:39 pm | 1:X 04 Apr 2022 23:39:39.565 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
Mon, Apr 4 2022 5:39:39 pm | 1:X 04 Apr 2022 23:39:39.565 # Redis version=6.2.6, bits=64, commit=00000000, modified=0, pid=1, just started
Mon, Apr 4 2022 5:39:39 pm | 1:X 04 Apr 2022 23:39:39.565 # Configuration loaded
Mon, Apr 4 2022 5:39:39 pm | 1:X 04 Apr 2022 23:39:39.566 * monotonic clock: POSIX clock_gettime
Mon, Apr 4 2022 5:39:39 pm | 1:X 04 Apr 2022 23:39:39.567 * Running mode=sentinel, port=26379.
Mon, Apr 4 2022 5:39:39 pm | 1:X 04 Apr 2022 23:39:39.567 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
Mon, Apr 4 2022 5:39:39 pm | 1:X 04 Apr 2022 23:39:39.568 # Sentinel ID is 93060445d3bd07e1b6b21a89638966d80990170a
Mon, Apr 4 2022 5:39:39 pm | 1:X 04 Apr 2022 23:39:39.568 # +monitor master mymaster redis1-node-1.redis1 30019 quorum 2
Mon, Apr 4 2022 5:39:39 pm | 1:X 04 Apr 2022 23:39:39.650 # +new-epoch 1
Mon, Apr 4 2022 5:39:41 pm | 1:X 04 Apr 2022 23:39:41.572 # +sdown slave redis1-node-0.redis1:30017 redis1-node-0.redis1 30017 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 5:39:59 pm | 1:X 04 Apr 2022 23:39:59.496 # +sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 5:40:02 pm | 1:X 04 Apr 2022 23:40:02.335 # -sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 5:40:23 pm | 1:X 04 Apr 2022 23:40:23.001 # -sdown slave redis1-node-0.redis1:30017 redis1-node-0.redis1 30017 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 5:54:12 pm | 1:X 04 Apr 2022 23:54:12.637 # +sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 5:54:15 pm | 1:X 04 Apr 2022 23:54:15.470 # -sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 5:57:01 pm | 1:X 04 Apr 2022 23:57:01.362 # +sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 5:57:05 pm | 1:X 04 Apr 2022 23:57:05.261 # -sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 6:00:23 pm | 1:X 05 Apr 2022 00:00:23.440 # +sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 6:00:27 pm | 1:X 05 Apr 2022 00:00:27.440 # -sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 6:01:09 pm | 1:X 05 Apr 2022 00:01:09.066 # +sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 6:01:10 pm | 1:X 05 Apr 2022 00:01:10.220 # +sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 6:01:12 pm | 1:X 05 Apr 2022 00:01:12.266 # -sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 6:01:14 pm | 1:X 05 Apr 2022 00:01:14.009 # -sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 6:01:33 pm | 1:X 05 Apr 2022 00:01:33.982 # +sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 6:01:38 pm | 1:X 05 Apr 2022 00:01:38.199 # -sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 6:01:43 pm | 1:X 05 Apr 2022 00:01:43.563 # +sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 6:01:47 pm | 1:X 05 Apr 2022 00:01:47.225 # -sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 6:01:51 pm | 1:X 05 Apr 2022 00:01:51.749 # +sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 6:02:00 pm | 1:X 05 Apr 2022 00:02:00.576 # +tilt #tilt mode entered
Mon, Apr 4 2022 6:02:30 pm | 1:X 05 Apr 2022 00:02:30.596 # -tilt #tilt mode exited
Mon, Apr 4 2022 6:02:30 pm | 1:X 05 Apr 2022 00:02:30.596 # -sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 6:03:02 pm | 1:X 05 Apr 2022 00:03:02.905 # +sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 6:03:08 pm | 1:X 05 Apr 2022 00:03:08.583 # +sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 6:03:17 pm | 1:X 05 Apr 2022 00:03:17.052 # +tilt #tilt mode entered
Mon, Apr 4 2022 6:03:47 pm | 1:X 05 Apr 2022 00:03:47.079 # -tilt #tilt mode exited
Mon, Apr 4 2022 6:03:47 pm | 1:X 05 Apr 2022 00:03:47.079 # -sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 6:03:51 pm | 1:X 05 Apr 2022 00:03:51.501 # +sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 6:03:57 pm | 1:X 05 Apr 2022 00:03:57.303 # +tilt #tilt mode entered
Mon, Apr 4 2022 6:04:27 pm | 1:X 05 Apr 2022 00:04:27.330 # -tilt #tilt mode exited
Mon, Apr 4 2022 6:04:27 pm | 1:X 05 Apr 2022 00:04:27.330 # -sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 6:04:57 pm | 1:X 05 Apr 2022 00:04:57.720 # -sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 6:06:39 pm | 1:X 05 Apr 2022 00:06:39.105 # +sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 6:06:42 pm | 1:X 05 Apr 2022 00:06:42.825 # -sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 6:08:11 pm | 1:X 05 Apr 2022 00:08:11.048 # +sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 6:08:19 pm | 1:X 05 Apr 2022 00:08:19.397 # +tilt #tilt mode entered
Mon, Apr 4 2022 6:08:49 pm | 1:X 05 Apr 2022 00:08:49.412 # -tilt #tilt mode exited
Mon, Apr 4 2022 6:08:49 pm | 1:X 05 Apr 2022 00:08:49.412 # -sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 6:10:05 pm | 1:X 05 Apr 2022 00:10:05.427 # +sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 6:10:08 pm | 1:X 05 Apr 2022 00:10:08.266 # -sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 6:10:26 pm | 1:X 05 Apr 2022 00:10:26.889 # +sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 6:10:26 pm | 1:X 05 Apr 2022 00:10:26.890 # +sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 6:10:44 pm | 1:X 05 Apr 2022 00:10:44.726 # +tilt #tilt mode entered
Mon, Apr 4 2022 6:10:49 pm | 1:X 05 Apr 2022 00:10:49.884 # +tilt #tilt mode entered
Mon, Apr 4 2022 6:11:19 pm | 1:X 05 Apr 2022 00:11:19.936 # -tilt #tilt mode exited
Mon, Apr 4 2022 6:11:19 pm | 1:X 05 Apr 2022 00:11:19.936 # -sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 6:11:36 pm | 1:X 05 Apr 2022 00:11:36.123 # -sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 6:12:33 pm | 1:X 05 Apr 2022 00:12:33.381 # +sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 6:12:33 pm | 1:X 05 Apr 2022 00:12:33.914 # +sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 6:12:36 pm | 1:X 05 Apr 2022 00:12:36.709 # -sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 6:12:36 pm | 1:X 05 Apr 2022 00:12:36.931 # -sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 6:12:47 pm | 1:X 05 Apr 2022 00:12:47.992 # +tilt #tilt mode entered
Mon, Apr 4 2022 6:13:18 pm | 1:X 05 Apr 2022 00:13:18.012 # -tilt #tilt mode exited
Mon, Apr 4 2022 6:15:16 pm | 1:X 05 Apr 2022 00:15:16.327 # +sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 6:15:19 pm | 1:X 05 Apr 2022 00:15:19.653 # -sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 6:15:20 pm | 1:X 05 Apr 2022 00:15:20.319 # +sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 6:15:23 pm | 1:X 05 Apr 2022 00:15:23.662 # -sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 6:26:31 pm | 1:X 05 Apr 2022 00:26:31.059 # +sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 6:26:34 pm | 1:X 05 Apr 2022 00:26:34.609 # -sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 6:31:08 pm | 1:X 05 Apr 2022 00:31:08.298 # +tilt #tilt mode entered
Mon, Apr 4 2022 6:31:38 pm | 1:X 05 Apr 2022 00:31:38.376 # -tilt #tilt mode exited
Mon, Apr 4 2022 6:35:41 pm | 1:X 05 Apr 2022 00:35:41.686 # +tilt #tilt mode entered
Mon, Apr 4 2022 6:35:47 pm | 1:X 05 Apr 2022 00:35:47.182 # +tilt #tilt mode entered
Mon, Apr 4 2022 6:36:17 pm | 1:X 05 Apr 2022 00:36:17.193 # -tilt #tilt mode exited
Mon, Apr 4 2022 6:40:22 pm | 1:X 05 Apr 2022 00:40:22.826 # +tilt #tilt mode entered
Mon, Apr 4 2022 6:40:28 pm | 1:X 05 Apr 2022 00:40:28.539 # +tilt #tilt mode entered
Mon, Apr 4 2022 6:40:41 pm | 1:X 05 Apr 2022 00:40:41.700 # +tilt #tilt mode entered
Mon, Apr 4 2022 6:40:53 pm | 1:X 05 Apr 2022 00:40:53.560 # +tilt #tilt mode entered
Mon, Apr 4 2022 6:41:01 pm | 1:X 05 Apr 2022 00:41:01.778 # +tilt #tilt mode entered
Mon, Apr 4 2022 6:41:31 pm | 1:X 05 Apr 2022 00:41:31.841 # -tilt #tilt mode exited
Mon, Apr 4 2022 6:42:14 pm | 1:X 05 Apr 2022 00:42:14.368 # +sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 6:42:14 pm | 1:X 05 Apr 2022 00:42:14.498 # +sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 6:42:17 pm | 1:X 05 Apr 2022 00:42:17.347 # -sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 6:42:17 pm | 1:X 05 Apr 2022 00:42:17.856 # -sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 6:42:26 pm | 1:X 05 Apr 2022 00:42:26.578 # +tilt #tilt mode entered
Mon, Apr 4 2022 6:42:56 pm | 1:X 05 Apr 2022 00:42:56.588 # -tilt #tilt mode exited
Mon, Apr 4 2022 6:43:00 pm | 1:X 05 Apr 2022 00:43:00.092 # +sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 6:43:03 pm | 1:X 05 Apr 2022 00:43:03.483 # -sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 7:04:18 pm | 1:X 05 Apr 2022 01:04:18.538 # +tilt #tilt mode entered
Mon, Apr 4 2022 7:04:48 pm | 1:X 05 Apr 2022 01:04:48.576 # -tilt #tilt mode exited
Mon, Apr 4 2022 7:05:04 pm | 1:X 05 Apr 2022 01:05:04.876 # +sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 7:05:08 pm | 1:X 05 Apr 2022 01:05:08.921 # -sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 7:05:48 pm | 1:X 05 Apr 2022 01:05:48.718 # +sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 7:05:57 pm | 1:X 05 Apr 2022 01:05:57.477 # -sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 7:06:33 pm | 1:X 05 Apr 2022 01:06:33.731 # +sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 7:06:36 pm | 1:X 05 Apr 2022 01:06:36.460 # -sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 7:06:43 pm | 1:X 05 Apr 2022 01:06:43.575 # +tilt #tilt mode entered
Mon, Apr 4 2022 7:07:13 pm | 1:X 05 Apr 2022 01:07:13.625 # -tilt #tilt mode exited
Mon, Apr 4 2022 7:08:04 pm | 1:X 05 Apr 2022 01:08:04.225 # +tilt #tilt mode entered
Mon, Apr 4 2022 7:08:34 pm | 1:X 05 Apr 2022 01:08:34.266 # -tilt #tilt mode exited
Mon, Apr 4 2022 7:08:35 pm | 1:X 05 Apr 2022 01:08:35.095 # +sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 7:08:38 pm | 1:X 05 Apr 2022 01:08:38.395 # +sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 7:08:41 pm | 1:X 05 Apr 2022 01:08:41.623 # -sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 7:08:42 pm | 1:X 05 Apr 2022 01:08:42.940 # -sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 7:08:50 pm | 1:X 05 Apr 2022 01:08:50.476 # +sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 7:08:53 pm | 1:X 05 Apr 2022 01:08:53.914 # -sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 7:09:43 pm | 1:X 05 Apr 2022 01:09:43.447 # +sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 7:09:47 pm | 1:X 05 Apr 2022 01:09:47.018 # -sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 7:10:04 pm | 1:X 05 Apr 2022 01:10:04.790 # +sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 7:10:07 pm | 1:X 05 Apr 2022 01:10:07.356 # -sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 7:10:26 pm | 1:X 05 Apr 2022 01:10:26.712 # +sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 7:10:29 pm | 1:X 05 Apr 2022 01:10:29.417 # -sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 7:10:50 pm | 1:X 05 Apr 2022 01:10:50.378 # +sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 7:10:53 pm | 1:X 05 Apr 2022 01:10:53.472 # -sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 7:11:53 pm | 1:X 05 Apr 2022 01:11:53.579 # +sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 7:11:57 pm | 1:X 05 Apr 2022 01:11:57.069 # -sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 7:15:32 pm | 1:X 05 Apr 2022 01:15:32.047 # +tilt #tilt mode entered
Mon, Apr 4 2022 7:16:02 pm | 1:X 05 Apr 2022 01:16:02.062 # -tilt #tilt mode exited
Mon, Apr 4 2022 7:36:33 pm | 1:X 05 Apr 2022 01:36:33.739 # +sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 7:36:37 pm | 1:X 05 Apr 2022 01:36:37.526 # -sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 7:37:29 pm | 1:X 05 Apr 2022 01:37:29.027 # +tilt #tilt mode entered
Mon, Apr 4 2022 7:37:59 pm | 1:X 05 Apr 2022 01:37:59.049 # -tilt #tilt mode exited
Mon, Apr 4 2022 7:39:10 pm | 1:X 05 Apr 2022 01:39:10.739 # +sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 7:39:14 pm | 1:X 05 Apr 2022 01:39:14.236 # -sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 7:39:14 pm | 1:X 05 Apr 2022 01:39:14.934 # +sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 7:39:18 pm | 1:X 05 Apr 2022 01:39:18.204 # -sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 7:40:23 pm | 1:X 05 Apr 2022 01:40:23.961 # +sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 7:40:27 pm | 1:X 05 Apr 2022 01:40:27.390 # -sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 7:40:33 pm | 1:X 05 Apr 2022 01:40:33.185 # +tilt #tilt mode entered
Mon, Apr 4 2022 7:41:03 pm | 1:X 05 Apr 2022 01:41:03.250 # -tilt #tilt mode exited
Mon, Apr 4 2022 7:41:03 pm | 1:X 05 Apr 2022 01:41:03.250 # +sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 7:41:12 pm | 1:X 05 Apr 2022 01:41:12.354 # -sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 7:41:25 pm | 1:X 05 Apr 2022 01:41:25.881 # +sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 7:41:28 pm | 1:X 05 Apr 2022 01:41:28.917 # -sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 7:42:18 pm | 1:X 05 Apr 2022 01:42:18.768 # +tilt #tilt mode entered
Mon, Apr 4 2022 7:42:24 pm | 1:X 05 Apr 2022 01:42:24.740 # +tilt #tilt mode entered
Mon, Apr 4 2022 7:42:54 pm | 1:X 05 Apr 2022 01:42:54.762 # -tilt #tilt mode exited
Mon, Apr 4 2022 7:43:04 pm | 1:X 05 Apr 2022 01:43:04.522 # +sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 7:43:07 pm | 1:X 05 Apr 2022 01:43:07.787 # -sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 8:20:11 pm | 1:X 05 Apr 2022 02:20:11.954 # +sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 8:20:11 pm | 1:X 05 Apr 2022 02:20:11.954 # +sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 8:20:14 pm | 1:X 05 Apr 2022 02:20:14.629 # -sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 8:20:14 pm | 1:X 05 Apr 2022 02:20:14.629 # -sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 8:20:28 pm | 1:X 05 Apr 2022 02:20:28.214 # +sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 8:20:31 pm | 1:X 05 Apr 2022 02:20:31.501 # -sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 8:24:09 pm | 1:X 05 Apr 2022 02:24:09.933 # +sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 8:24:12 pm | 1:X 05 Apr 2022 02:24:12.907 # -sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 8:28:47 pm | 1:X 05 Apr 2022 02:28:47.699 # +tilt #tilt mode entered
Mon, Apr 4 2022 8:29:17 pm | 1:X 05 Apr 2022 02:29:17.735 # -tilt #tilt mode exited
Mon, Apr 4 2022 8:32:48 pm | 1:X 05 Apr 2022 02:32:48.816 # +sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 8:32:53 pm | 1:X 05 Apr 2022 02:32:53.076 # -sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 8:33:52 pm | 1:X 05 Apr 2022 02:33:52.541 # +sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 8:33:55 pm | 1:X 05 Apr 2022 02:33:55.602 # +sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 8:33:56 pm | 1:X 05 Apr 2022 02:33:55.998 # -sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 8:34:03 pm | 1:X 05 Apr 2022 02:34:03.954 # -sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 8:58:46 pm | 1:X 05 Apr 2022 02:58:46.179 # +sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 8:58:49 pm | 1:X 05 Apr 2022 02:58:49.037 # -sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 9:12:54 pm | 1:X 05 Apr 2022 03:12:54.818 # +sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 9:12:57 pm | 1:X 05 Apr 2022 03:12:57.783 # -sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 9:18:10 pm | 1:X 05 Apr 2022 03:18:10.185 # +sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 9:18:12 pm | 1:X 05 Apr 2022 03:18:12.841 # -sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 9:18:25 pm | 1:X 05 Apr 2022 03:18:25.447 # +tilt #tilt mode entered
Mon, Apr 4 2022 9:18:55 pm | 1:X 05 Apr 2022 03:18:55.469 # -tilt #tilt mode exited
Mon, Apr 4 2022 9:19:04 pm | 1:X 05 Apr 2022 03:19:04.369 # +tilt #tilt mode entered
Mon, Apr 4 2022 9:19:34 pm | 1:X 05 Apr 2022 03:19:34.398 # -tilt #tilt mode exited
Mon, Apr 4 2022 9:30:50 pm | 1:X 05 Apr 2022 03:30:50.867 # +tilt #tilt mode entered
Mon, Apr 4 2022 9:31:09 pm | 1:X 05 Apr 2022 03:31:09.762 # +tilt #tilt mode entered
Mon, Apr 4 2022 9:31:39 pm | 1:X 05 Apr 2022 03:31:39.839 # -tilt #tilt mode exited
Mon, Apr 4 2022 9:32:48 pm | 1:X 05 Apr 2022 03:32:48.719 # +sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 9:32:52 pm | 1:X 05 Apr 2022 03:32:52.115 # -sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 9:33:49 pm | 1:X 05 Apr 2022 03:33:49.500 # +sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 9:33:53 pm | 1:X 05 Apr 2022 03:33:53.015 # -sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 9:34:14 pm | 1:X 05 Apr 2022 03:34:14.849 # +sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 9:34:17 pm | 1:X 05 Apr 2022 03:34:17.819 # -sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 9:34:32 pm | 1:X 05 Apr 2022 03:34:32.451 # +sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 9:34:36 pm | 1:X 05 Apr 2022 03:34:36.613 # -sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 9:35:12 pm | 1:X 05 Apr 2022 03:35:12.113 # +sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 9:35:15 pm | 1:X 05 Apr 2022 03:35:15.396 # -sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 9:37:22 pm | 1:X 05 Apr 2022 03:37:22.709 # +tilt #tilt mode entered
Mon, Apr 4 2022 9:37:52 pm | 1:X 05 Apr 2022 03:37:52.796 # -tilt #tilt mode exited
Mon, Apr 4 2022 9:39:49 pm | 1:X 05 Apr 2022 03:39:49.641 # +tilt #tilt mode entered
Mon, Apr 4 2022 9:40:19 pm | 1:X 05 Apr 2022 03:40:19.698 # -tilt #tilt mode exited
Mon, Apr 4 2022 9:47:31 pm | 1:X 05 Apr 2022 03:47:31.739 # +tilt #tilt mode entered
Mon, Apr 4 2022 9:48:01 pm | 1:X 05 Apr 2022 03:48:01.793 # -tilt #tilt mode exited
Mon, Apr 4 2022 9:48:18 pm | 1:X 05 Apr 2022 03:48:18.242 # +sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 9:48:21 pm | 1:X 05 Apr 2022 03:48:21.916 # -sdown sentinel 034a2c242957d1854224b48c3583b3fb18173e51 redis1-node-2.redis1 30020 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 9:49:21 pm | 1:X 05 Apr 2022 03:49:21.832 # +sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 9:49:35 pm | 1:X 05 Apr 2022 03:49:35.659 # -sdown sentinel 33f1828dea40e299b1c83d4824a2c329cc4e3ac7 redis1-node-1.redis1 30018 @ mymaster redis1-node-1.redis1 30019
Mon, Apr 4 2022 9:49:47 pm | 1:X 05 Apr 2022 03:49:47.207 # +tilt #tilt mode entered
Mon, Apr 4 2022 9:50:03 pm | 1:X 05 Apr 2022 03:50:03.126 # +tilt #tilt mode entered
Mon, Apr 4 2022 9:50:14 pm | 1:X 05 Apr 2022 03:50:14.364 # +tilt #tilt mode entered
Mon, Apr 4 2022 9:50:26 pm | 1:X 05 Apr 2022 03:50:26.260 # +tilt #tilt mode entered
Mon, Apr 4 2022 9:50:36 pm | 1:X 05 Apr 2022 03:50:36.989 # +tilt #tilt mode entered
Mon, Apr 4 2022 9:50:37 pm | 1:signal-handler (1649130637) Received SIGTERM scheduling shutdown...
Mon, Apr 4 2022 9:50:38 pm | 1:X 05 Apr 2022 03:50:38.052 # User requested shutdown...
Mon, Apr 4 2022 9:50:38 pm | 1:X 05 Apr 2022 03:50:38.052 # Sentinel is now ready to exit, bye bye...

Redis container log.

20:47:48.10 INFO ==> about to run the command: timeout 220 redis-cli -h redis1.redis1.svc.cluster.local -p 30014 sentinel get-master-addr-by-name mymaster
--
Mon, Apr 4 2022 2:47:48 pm | 20:47:48.33 INFO ==> about to run the command: timeout 220 redis-cli -h redis1.redis1.svc.cluster.local -p 30014 sentinel get-master-addr-by-name mymaster
Mon, Apr 4 2022 2:47:48 pm | 20:47:48.39 INFO ==> Current master: REDIS_SENTINEL_INFO=(redis1-node-1.redis1,30019)
Mon, Apr 4 2022 2:47:48 pm | 20:47:48.39 INFO ==> Configuring the node as replica
Mon, Apr 4 2022 2:47:48 pm | 1:C 04 Apr 2022 20:47:48.465 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
Mon, Apr 4 2022 2:47:48 pm | 1:C 04 Apr 2022 20:47:48.465 # Redis version=6.2.6, bits=64, commit=00000000, modified=0, pid=1, just started
Mon, Apr 4 2022 2:47:48 pm | 1:C 04 Apr 2022 20:47:48.465 # Configuration loaded
Mon, Apr 4 2022 2:47:48 pm | 1:S 04 Apr 2022 20:47:48.466 * monotonic clock: POSIX clock_gettime
Mon, Apr 4 2022 2:47:48 pm | 1:S 04 Apr 2022 20:47:48.467 * Running mode=standalone, port=6379.
Mon, Apr 4 2022 2:47:48 pm | 1:S 04 Apr 2022 20:47:48.467 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
Mon, Apr 4 2022 2:47:48 pm | 1:S 04 Apr 2022 20:47:48.467 # Server initialized
Mon, Apr 4 2022 2:47:48 pm | 1:S 04 Apr 2022 20:47:48.468 * Ready to accept connections
Mon, Apr 4 2022 2:47:48 pm | 1:S 04 Apr 2022 20:47:48.468 * Connecting to MASTER redis1-node-1.redis1:30019
Mon, Apr 4 2022 2:47:48 pm | 1:S 04 Apr 2022 20:47:48.475 * MASTER <-> REPLICA sync started
Mon, Apr 4 2022 2:47:48 pm | 1:S 04 Apr 2022 20:47:48.476 * Non blocking connect for SYNC fired the event.
Mon, Apr 4 2022 2:47:48 pm | 1:S 04 Apr 2022 20:47:48.477 * Master replied to PING, replication can continue...
Mon, Apr 4 2022 2:47:48 pm | 1:S 04 Apr 2022 20:47:48.479 * Partial resynchronization not possible (no cached master)
Mon, Apr 4 2022 2:47:48 pm | 1:S 04 Apr 2022 20:47:48.480 * Full resync from master: a596e3271aa403b5cfb29c7c2f8126c412b5fe59:3429539
Mon, Apr 4 2022 2:47:48 pm | 1:S 04 Apr 2022 20:47:48.549 * MASTER <-> REPLICA sync: receiving 178 bytes from master to disk
Mon, Apr 4 2022 2:47:48 pm | 1:S 04 Apr 2022 20:47:48.549 * MASTER <-> REPLICA sync: Flushing old data
Mon, Apr 4 2022 2:47:48 pm | 1:S 04 Apr 2022 20:47:48.561 * MASTER <-> REPLICA sync: Loading DB in memory
Mon, Apr 4 2022 2:47:48 pm | 1:S 04 Apr 2022 20:47:48.601 * Loading RDB produced by version 6.2.6
Mon, Apr 4 2022 2:47:48 pm | 1:S 04 Apr 2022 20:47:48.601 * RDB age 0 seconds
Mon, Apr 4 2022 2:47:48 pm | 1:S 04 Apr 2022 20:47:48.601 * RDB memory usage when created 1.93 Mb
Mon, Apr 4 2022 2:47:48 pm | 1:S 04 Apr 2022 20:47:48.601 # Done loading RDB, keys loaded: 0, keys expired: 0.
Mon, Apr 4 2022 2:47:48 pm | 1:S 04 Apr 2022 20:47:48.601 * MASTER <-> REPLICA sync: Finished with success
Mon, Apr 4 2022 2:47:48 pm | 1:S 04 Apr 2022 20:47:48.602 * Background append only file rewriting started by pid 35
Mon, Apr 4 2022 2:47:48 pm | 1:S 04 Apr 2022 20:47:48.661 * AOF rewrite child asks to stop sending diffs.
Mon, Apr 4 2022 2:47:48 pm | 35:C 04 Apr 2022 20:47:48.661 * Parent agreed to stop sending diffs. Finalizing AOF...
Mon, Apr 4 2022 2:47:48 pm | 35:C 04 Apr 2022 20:47:48.661 * Concatenating 0.00 MB of AOF diff received from parent.
Mon, Apr 4 2022 2:47:48 pm | 35:C 04 Apr 2022 20:47:48.664 * SYNC append only file rewrite performed
Mon, Apr 4 2022 2:47:48 pm | 35:C 04 Apr 2022 20:47:48.665 * AOF rewrite: 0 MB of memory used by copy-on-write
Mon, Apr 4 2022 2:47:48 pm | 1:S 04 Apr 2022 20:47:48.703 * Background AOF rewrite terminated with success
Mon, Apr 4 2022 2:47:48 pm | 1:S 04 Apr 2022 20:47:48.704 * Residual parent diff successfully flushed to the rewritten AOF (0.00 MB)
Mon, Apr 4 2022 2:47:48 pm | 1:S 04 Apr 2022 20:47:48.704 * Background AOF rewrite finished successfully

Additional information

Seems like it may be the probes killing the container? Screenshot_20220405_103651 Screenshot_20220405_103718

This has also been happening for a while, we just have not noticed cause the redis container is not restarting, so all data is fine. Screenshot_20220405_103805

I think it might be host related somehow. The only reason I am thinking this is because we were testing in a multi-cluster environment. Part on-prem and the other part in a remote managed k8s. We see this behavior on-prem, and not in remote k8s.

I have been testing on a new cluster with Rocky Linux

Container Runtime Version | docker://20.10.13
Image | Rocky Linux 8.5 (Green Obsidian)
Kernel Version | 4.18.0-348.20.1.el8_5.x86_64
Kube Proxy Version | v1.21.9
Kubelet Version | v1.21.9

But this is also happening on the older deployment from the image above

Container Runtime Version | docker://20.10.8
Image | CentOS Linux 8
Kernel Version | 4.18.0-305.17.1.el8_4.x86_64
Kube Proxy Version | v1.20.6
Kubelet Version | v1.20.6

I am still testing, I am pairing down the values file more and more until I see the behavior stop. This is taking a while as it take a random long amount of time until sentinel enters tilt mode repeatedly and dies.

We have many things deployed onto these clusters, and have no other observed issues.

I have been poking around the hosts and trying to see if there was anything happening that would line up with the behavior explained here: https://redis.io/docs/manual/sentinel/#tilt-mode and I have not found anything yet. Metrics for the vms are all reasonable. No monitoring or alarming is going of for any of these clusters/workers.

I am unsure what is happening. If anyone reading this has any insight on what is happening, input would be greatly appreciated.

Thank you!

mgrzeszczak commented 1 year ago

@abhishekgupta2205 because in sentinel deployment you connect your application to sentinels (through a service) directly, and the redis client should be setinel-aware (java has those, jedis for example) then even if master changes the app can switch because sentinels will tell it to do so

gediminasmorkys commented 1 year ago

@abhishekgupta2205 because in sentinel deployment you connect your application to sentinels (through a service) directly, and the redis client should be setinel-aware (java has those, jedis for example) then even if master changes the app can switch because sentinels will tell it to do so

I apologize for not testing this yet with a Spring Boot app and asking this anyway, but the question for me is this: after applying useHostnames=false, can I still not know about specific IPs when connecting to sentinel from another application (and continue to use the headless service hostnames)?

mgrzeszczak commented 1 year ago

@abhishekgupta2205 you dont connect to sentinel via ip, there is a sentinel service created

abhishekGupta2205 commented 1 year ago

@gediminasmorkys i want that only that after applying hostaname=false , internat functioning may happen with pod ip but it should return some fixed svc name

mgrzeszczak commented 1 year ago

@abhishekgupta2205 this is the service you want, just use the sentinel port 26379

anyways this is offtopic

jotamartos commented 1 year ago

Just a quick note to let you know that the changes are merged. Version 17.7.0 of the Bitnami Redis Chart will include support to use IPs instead of domains when configuring the different nodes by using the useHostnames parameter.

mszumilak commented 1 year ago

Tilt is a time drift issue. Can you change the time drift tolerance and recompile redis? Aformentioned tolerance is not configurable and recompilation will be required

ChrisNoSim commented 1 year ago

Just a quick note to let you know that https://github.com/bitnami/charts/pull/14599. Version 17.7.0 of the Bitnami Redis Chart will include support to use IPs instead of domains when configuring the different nodes by using the useHostnames parameter.

Hey jotamartos :wave: thx for that hint. But the version 17.7.0 is not yet released (successfully), right? We get an error, when trying to deploy that chart version.

jotamartos commented 1 year ago

Hi @mszumilak,

We do not modify the source code of the components we build. If this is a problem in the Redis' source code, the developers should update that parameter or provide a way to modify that value using a compilation option.

@ChrisNoSim Regarding version 17.7.0, we noticed it was not published properly. We are working on it and expect to have an updated version soon. Will ping you when it's available.

jotamartos commented 1 year ago

Hi @ChrisNoSim,

Bitnami Redis chart is now updated with these changes. The latest available version is 17.7.1

$ helm search repo bitnami/redis
NAME                    CHART VERSION   APP VERSION     DESCRIPTION
bitnami/redis           17.7.1          7.0.8           Redis(R) is an open source, advanced key-value ...
PovilasKM commented 1 year ago

Thanks for fixing this issue. We've had the tilting issue for a long while and with 17.7.0 this was the first weekend without any restarts.

mgrzeszczak commented 1 year ago

@PovilasKM did you use the new useHostnames parameter and set it to false?

ChrisNoSim commented 1 year ago

Thanks for updating the REDIS chart. We set the useHostnames to false on Friday and since then we have not seen the tilt mode in this cluster. We will roll out that setting to our other REDIS clusters and lets see. :partying_face:

jotamartos commented 1 year ago

I'm really glad to hear that! 😄 Thanks for letting us know

PovilasKM commented 1 year ago

@mgrzeszczak we've used useHostnames and set it to false. Now there is no tilting in logs and no "random" restarts.

abhishekGupta2205 commented 1 year ago

@marcosbc , after disabling use hostname feature , connectivuty of pods from differnt k8s cluster would be impacted also . as now master address would be return in form of pod ip , which is only accessible inside same cluster. how can i access redis-master pod via sentinel from a diifferent cluster .

abhishekGupta2205 commented 1 year ago

@jotamartos , my issue is that after connecting with redis-sentinel , i need information of my redis-master in form of ip and port ( when using jedis or connecting from a java service ) . when the hostname usage is disabled , the ip which sentinel will return will be a pod ip . As you know pod ip is only connectible within same k8s cluster . but if my java service is outside the cluster where my redis server is running , my java application would not be able to connect to the pod ip . thats why we need something that is accessible irrespective of cluster .

my usage of redis --

java application --> sentinel-->(return ip of redis master to java appliction )----> redis master is connected to java application

is there any way we can address this issue also , i can get an ip and port , which are accessible from anywhere