Closed JashBook closed 4 months ago
After the second switch, all Sentinels believe that redis-imvtol-redis-0 is the master, but at this time, redis-imvtol-redis-0 has incorrect configuration (replicaof points to itself, so the info replication is displayed as a slave). Therefore, all Sentinels consider redis-imvtol-redis-0 to be in an unhealthy state. However, since there are no healthy slaves available, it is not possible to switch from redis-imvtol-redis-0 to another node.
The issue likely arises because during the second failover, the replicaof configuration should have been removed from redis-imvtol-redis-0. Due to some anomaly, redis-imvtol-redis-0 ended up with an incorrect configuration.
pod redis-imvtol-redis-0 kernel running log has two config rewrite
records:
logs sentinel failover success
kubectl logs redis-cluster-redis-sentinel-0
+ reset_redis_sentinel_conf
+ echo reset redis sentinel conf
+ sentinel_port=26379
+ [ -n 26379 ]
+ sentinel_port=26379
+ mkdir -p /data/sentinel
reset redis sentinel conf
+ [ -f /data/sentinel/redis-sentinel.conf ]
+ build_redis_sentinel_conf
build redis sentinel conf
+ echo build redis sentinel conf
+ kb_pod_fqdn=redis-cluster-redis-sentinel-0.redis-cluster-redis-sentinel-headless.default.svc
+ echo port 26379
+ echo sentinel announce-ip redis-cluster-redis-sentinel-0.redis-cluster-redis-sentinel-headless.default.svc
+ echo sentinel resolve-hostnames yes
+ echo sentinel announce-hostnames yes
+ [ -n JK9a184T2d ]
+ echo sentinel sentinel-user default
+ echo sentinel sentinel-pass JK9a184T2d
+ start_redis_sentinel_server
+ echo Starting redis sentinel server...
+ exec redis-server /data/sentinel/redis-sentinel.conf --sentinel
Starting redis sentinel server...
1:X 24 Jun 2024 07:50:01.457 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:X 24 Jun 2024 07:50:01.457 * Redis version=7.2.4, bits=64, commit=00000000, modified=0, pid=1, just started
1:X 24 Jun 2024 07:50:01.457 * Configuration loaded
1:X 24 Jun 2024 07:50:01.457 * monotonic clock: POSIX clock_gettime
1:X 24 Jun 2024 07:50:01.458 * Running mode=sentinel, port=26379.
1:X 24 Jun 2024 07:50:01.468 * Sentinel new configuration saved on disk
1:X 24 Jun 2024 07:50:01.468 * Sentinel ID is b2d0c0313ebd721ee18c41f23fb5b6ade4e7c6fd
1:X 24 Jun 2024 07:51:25.187 * Sentinel new configuration saved on disk
1:X 24 Jun 2024 07:51:25.187 # +monitor master redis-cluster-redis redis-cluster-redis-0.redis-cluster-redis-headless 6379 quorum 2
1:X 24 Jun 2024 07:51:25.194 # +set master redis-cluster-redis redis-cluster-redis-0.redis-cluster-redis-headless 6379 down-after-milliseconds 5000
1:X 24 Jun 2024 07:51:25.199 * Sentinel new configuration saved on disk
1:X 24 Jun 2024 07:51:25.206 # +set master redis-cluster-redis redis-cluster-redis-0.redis-cluster-redis-headless 6379 failover-timeout 60000
1:X 24 Jun 2024 07:51:25.211 * Sentinel new configuration saved on disk
1:X 24 Jun 2024 07:51:25.218 # +set master redis-cluster-redis redis-cluster-redis-0.redis-cluster-redis-headless 6379 parallel-syncs 1
1:X 24 Jun 2024 07:51:25.222 * Sentinel new configuration saved on disk
1:X 24 Jun 2024 07:51:25.229 # +set master redis-cluster-redis redis-cluster-redis-0.redis-cluster-redis-headless 6379 auth-user kbreplicator-sentinel
1:X 24 Jun 2024 07:51:25.233 * Sentinel new configuration saved on disk
1:X 24 Jun 2024 07:51:25.241 # +set master redis-cluster-redis redis-cluster-redis-0.redis-cluster-redis-headless 6379 auth-pass ******
1:X 24 Jun 2024 07:51:25.257 * Sentinel new configuration saved on disk
1:X 24 Jun 2024 07:51:27.439 * +sentinel sentinel 92b772e37b41eaf86e02941646601daab23b3ccb redis-cluster-redis-sentinel-1.redis-cluster-redis-sentinel-headless.default.svc 26379 @ redis-cluster-redis redis-cluster-redis-0.redis-cluster-redis-headless 6379
1:X 24 Jun 2024 07:51:27.443 * Sentinel new configuration saved on disk
1:X 24 Jun 2024 07:51:27.470 * +sentinel sentinel b45b3ca0341620d06a6cff7ae04102ecddb16194 redis-cluster-redis-sentinel-2.redis-cluster-redis-sentinel-headless.default.svc 26379 @ redis-cluster-redis redis-cluster-redis-0.redis-cluster-redis-headless 6379
1:X 24 Jun 2024 07:51:27.474 * Sentinel new configuration saved on disk
1:X 24 Jun 2024 07:51:46.388 * +slave slave redis-cluster-redis-1.redis-cluster-redis-headless.default.svc:6379 redis-cluster-redis-1.redis-cluster-redis-headless.default.svc 6379 @ redis-cluster-redis redis-cluster-redis-0.redis-cluster-redis-headless 6379
1:X 24 Jun 2024 07:51:46.392 * Sentinel new configuration saved on disk
1:X 24 Jun 2024 07:57:13.116 # +sdown master redis-cluster-redis redis-cluster-redis-0.redis-cluster-redis-headless 6379
1:X 24 Jun 2024 07:57:13.183 # +odown master redis-cluster-redis redis-cluster-redis-0.redis-cluster-redis-headless 6379 #quorum 2/2
1:X 24 Jun 2024 07:57:13.183 # +new-epoch 1
1:X 24 Jun 2024 07:57:13.183 # +try-failover master redis-cluster-redis redis-cluster-redis-0.redis-cluster-redis-headless 6379
1:X 24 Jun 2024 07:57:13.187 * Sentinel new configuration saved on disk
1:X 24 Jun 2024 07:57:13.187 # +vote-for-leader b2d0c0313ebd721ee18c41f23fb5b6ade4e7c6fd 1
1:X 24 Jun 2024 07:57:13.194 * b45b3ca0341620d06a6cff7ae04102ecddb16194 voted for b2d0c0313ebd721ee18c41f23fb5b6ade4e7c6fd 1
1:X 24 Jun 2024 07:57:13.197 * 92b772e37b41eaf86e02941646601daab23b3ccb voted for b2d0c0313ebd721ee18c41f23fb5b6ade4e7c6fd 1
1:X 24 Jun 2024 07:57:13.288 # +elected-leader master redis-cluster-redis redis-cluster-redis-0.redis-cluster-redis-headless 6379
1:X 24 Jun 2024 07:57:13.288 # +failover-state-select-slave master redis-cluster-redis redis-cluster-redis-0.redis-cluster-redis-headless 6379
1:X 24 Jun 2024 07:57:13.355 # +selected-slave slave redis-cluster-redis-1.redis-cluster-redis-headless.default.svc:6379 redis-cluster-redis-1.redis-cluster-redis-headless.default.svc 6379 @ redis-cluster-redis redis-cluster-redis-0.redis-cluster-redis-headless 6379
1:X 24 Jun 2024 07:57:13.355 * +failover-state-send-slaveof-noone slave redis-cluster-redis-1.redis-cluster-redis-headless.default.svc:6379 redis-cluster-redis-1.redis-cluster-redis-headless.default.svc 6379 @ redis-cluster-redis redis-cluster-redis-0.redis-cluster-redis-headless 6379
1:X 24 Jun 2024 07:57:13.411 * +failover-state-wait-promotion slave redis-cluster-redis-1.redis-cluster-redis-headless.default.svc:6379 redis-cluster-redis-1.redis-cluster-redis-headless.default.svc 6379 @ redis-cluster-redis redis-cluster-redis-0.redis-cluster-redis-headless 6379
1:X 24 Jun 2024 07:57:14.255 * Sentinel new configuration saved on disk
1:X 24 Jun 2024 07:57:14.255 # +promoted-slave slave redis-cluster-redis-1.redis-cluster-redis-headless.default.svc:6379 redis-cluster-redis-1.redis-cluster-redis-headless.default.svc 6379 @ redis-cluster-redis redis-cluster-redis-0.redis-cluster-redis-headless 6379
1:X 24 Jun 2024 07:57:14.255 # +failover-state-reconf-slaves master redis-cluster-redis redis-cluster-redis-0.redis-cluster-redis-headless 6379
1:X 24 Jun 2024 07:57:14.309 # +failover-end master redis-cluster-redis redis-cluster-redis-0.redis-cluster-redis-headless 6379
1:X 24 Jun 2024 07:57:14.309 # +switch-master redis-cluster-redis redis-cluster-redis-0.redis-cluster-redis-headless 6379 redis-cluster-redis-1.redis-cluster-redis-headless.default.svc 6379
1:X 24 Jun 2024 07:57:14.312 * +slave slave redis-cluster-redis-0.redis-cluster-redis-headless:6379 redis-cluster-redis-0.redis-cluster-redis-headless 6379 @ redis-cluster-redis redis-cluster-redis-1.redis-cluster-redis-headless.default.svc 6379
1:X 24 Jun 2024 07:57:14.316 * Sentinel new configuration saved on disk
1:X 24 Jun 2024 07:57:19.355 # +sdown slave redis-cluster-redis-0.redis-cluster-redis-headless:6379 redis-cluster-redis-0.redis-cluster-redis-headless 6379 @ redis-cluster-redis redis-cluster-redis-1.redis-cluster-redis-headless.default.svc 6379
1:X 24 Jun 2024 07:57:27.648 * +slave slave redis-cluster-redis-0.redis-cluster-redis-headless.default.svc:6379 redis-cluster-redis-0.redis-cluster-redis-headless.default.svc 6379 @ redis-cluster-redis redis-cluster-redis-1.redis-cluster-redis-headless.default.svc 6379
1:X 24 Jun 2024 07:57:27.653 * Sentinel new configuration saved on disk
1:X 24 Jun 2024 07:57:32.668 # +sdown slave redis-cluster-redis-0.redis-cluster-redis-headless.default.svc:6379 redis-cluster-redis-0.redis-cluster-redis-headless.default.svc 6379 @ redis-cluster-redis redis-cluster-redis-1.redis-cluster-redis-headless.default.svc 6379
1:X 24 Jun 2024 07:57:42.864 # -sdown slave redis-cluster-redis-0.redis-cluster-redis-headless.default.svc:6379 redis-cluster-redis-0.redis-cluster-redis-headless.default.svc 6379 @ redis-cluster-redis redis-cluster-redis-1.redis-cluster-redis-headless.default.svc 6379
1:X 24 Jun 2024 07:57:44.654 # -sdown slave redis-cluster-redis-0.redis-cluster-redis-headless:6379 redis-cluster-redis-0.redis-cluster-redis-headless 6379 @ redis-cluster-redis redis-cluster-redis-1.redis-cluster-redis-headless.default.svc 6379
➜ ~
➜ ~
➜ ~ kubectl logs redis-cluster-redis-sentinel-1
+ reset_redis_sentinel_conf
+ echo reset redis sentinel conf
+ sentinel_port=26379
+ [ -n 26379 ]
+ sentinel_port=26379
+ mkdir -p /data/sentinel
reset redis sentinel conf
+ [ -f /data/sentinel/redis-sentinel.conf ]
+ build_redis_sentinel_conf
+ echo build redis sentinel conf
build redis sentinel conf
+ kb_pod_fqdn=redis-cluster-redis-sentinel-1.redis-cluster-redis-sentinel-headless.default.svc
+ echo port 26379
+ echo sentinel announce-ip redis-cluster-redis-sentinel-1.redis-cluster-redis-sentinel-headless.default.svc
+ echo sentinel resolve-hostnames yes
+ echo sentinel announce-hostnames yes
+ [ -n JK9a184T2d ]
+ echo sentinel sentinel-user default
+ echo sentinel sentinel-pass JK9a184T2d
+ start_redis_sentinel_server
+ echo Starting redis sentinel server...
Starting redis sentinel server...
+ exec redis-server /data/sentinel/redis-sentinel.conf --sentinel
1:X 24 Jun 2024 07:50:00.870 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:X 24 Jun 2024 07:50:00.870 * Redis version=7.2.4, bits=64, commit=00000000, modified=0, pid=1, just started
1:X 24 Jun 2024 07:50:00.870 * Configuration loaded
1:X 24 Jun 2024 07:50:00.871 * monotonic clock: POSIX clock_gettime
1:X 24 Jun 2024 07:50:00.872 * Running mode=sentinel, port=26379.
1:X 24 Jun 2024 07:50:00.970 * Sentinel new configuration saved on disk
1:X 24 Jun 2024 07:50:00.970 * Sentinel ID is 92b772e37b41eaf86e02941646601daab23b3ccb
1:X 24 Jun 2024 07:51:25.373 * Sentinel new configuration saved on disk
1:X 24 Jun 2024 07:51:25.373 # +monitor master redis-cluster-redis redis-cluster-redis-0.redis-cluster-redis-headless 6379 quorum 2
1:X 24 Jun 2024 07:51:25.380 # +set master redis-cluster-redis redis-cluster-redis-0.redis-cluster-redis-headless 6379 down-after-milliseconds 5000
1:X 24 Jun 2024 07:51:25.382 * Sentinel new configuration saved on disk
1:X 24 Jun 2024 07:51:25.389 # +set master redis-cluster-redis redis-cluster-redis-0.redis-cluster-redis-headless 6379 failover-timeout 60000
1:X 24 Jun 2024 07:51:25.391 * Sentinel new configuration saved on disk
1:X 24 Jun 2024 07:51:25.398 # +set master redis-cluster-redis redis-cluster-redis-0.redis-cluster-redis-headless 6379 parallel-syncs 1
1:X 24 Jun 2024 07:51:25.400 * Sentinel new configuration saved on disk
1:X 24 Jun 2024 07:51:25.408 # +set master redis-cluster-redis redis-cluster-redis-0.redis-cluster-redis-headless 6379 auth-user kbreplicator-sentinel
1:X 24 Jun 2024 07:51:25.410 * Sentinel new configuration saved on disk
1:X 24 Jun 2024 07:51:25.416 # +set master redis-cluster-redis redis-cluster-redis-0.redis-cluster-redis-headless 6379 auth-pass ******
1:X 24 Jun 2024 07:51:25.418 * Sentinel new configuration saved on disk
1:X 24 Jun 2024 07:51:27.203 * +sentinel sentinel b2d0c0313ebd721ee18c41f23fb5b6ade4e7c6fd redis-cluster-redis-sentinel-0.redis-cluster-redis-sentinel-headless.default.svc 26379 @ redis-cluster-redis redis-cluster-redis-0.redis-cluster-redis-headless 6379
1:X 24 Jun 2024 07:51:27.206 * Sentinel new configuration saved on disk
1:X 24 Jun 2024 07:51:27.471 * +sentinel sentinel b45b3ca0341620d06a6cff7ae04102ecddb16194 redis-cluster-redis-sentinel-2.redis-cluster-redis-sentinel-headless.default.svc 26379 @ redis-cluster-redis redis-cluster-redis-0.redis-cluster-redis-headless 6379
1:X 24 Jun 2024 07:51:27.473 * Sentinel new configuration saved on disk
1:X 24 Jun 2024 07:51:45.565 * +slave slave redis-cluster-redis-1.redis-cluster-redis-headless.default.svc:6379 redis-cluster-redis-1.redis-cluster-redis-headless.default.svc 6379 @ redis-cluster-redis redis-cluster-redis-0.redis-cluster-redis-headless 6379
1:X 24 Jun 2024 07:51:45.567 * Sentinel new configuration saved on disk
1:X 24 Jun 2024 07:57:13.112 # +sdown master redis-cluster-redis redis-cluster-redis-0.redis-cluster-redis-headless 6379
1:X 24 Jun 2024 07:57:13.194 * Sentinel new configuration saved on disk
1:X 24 Jun 2024 07:57:13.194 # +new-epoch 1
1:X 24 Jun 2024 07:57:13.196 * Sentinel new configuration saved on disk
1:X 24 Jun 2024 07:57:13.196 # +vote-for-leader b2d0c0313ebd721ee18c41f23fb5b6ade4e7c6fd 1
1:X 24 Jun 2024 07:57:14.203 # +odown master redis-cluster-redis redis-cluster-redis-0.redis-cluster-redis-headless 6379 #quorum 3/2
1:X 24 Jun 2024 07:57:14.203 * Next failover delay: I will not start a failover before Mon Jun 24 07:59:13 2024
1:X 24 Jun 2024 07:57:14.320 # +config-update-from sentinel b2d0c0313ebd721ee18c41f23fb5b6ade4e7c6fd redis-cluster-redis-sentinel-0.redis-cluster-redis-sentinel-headless.default.svc 26379 @ redis-cluster-redis redis-cluster-redis-0.redis-cluster-redis-headless 6379
1:X 24 Jun 2024 07:57:14.320 # +switch-master redis-cluster-redis redis-cluster-redis-0.redis-cluster-redis-headless 6379 redis-cluster-redis-1.redis-cluster-redis-headless.default.svc 6379
1:X 24 Jun 2024 07:57:14.323 * +slave slave redis-cluster-redis-0.redis-cluster-redis-headless:6379 redis-cluster-redis-0.redis-cluster-redis-headless 6379 @ redis-cluster-redis redis-cluster-redis-1.redis-cluster-redis-headless.default.svc 6379
1:X 24 Jun 2024 07:57:14.326 * Sentinel new configuration saved on disk
1:X 24 Jun 2024 07:57:19.369 # +sdown slave redis-cluster-redis-0.redis-cluster-redis-headless:6379 redis-cluster-redis-0.redis-cluster-redis-headless 6379 @ redis-cluster-redis redis-cluster-redis-1.redis-cluster-redis-headless.default.svc 6379
1:X 24 Jun 2024 07:57:27.136 * +slave slave redis-cluster-redis-0.redis-cluster-redis-headless.default.svc:6379 redis-cluster-redis-0.redis-cluster-redis-headless.default.svc 6379 @ redis-cluster-redis redis-cluster-redis-1.redis-cluster-redis-headless.default.svc 6379
1:X 24 Jun 2024 07:57:27.138 * Sentinel new configuration saved on disk
1:X 24 Jun 2024 07:57:29.560 # -sdown slave redis-cluster-redis-0.redis-cluster-redis-headless:6379 redis-cluster-redis-0.redis-cluster-redis-headless 6379 @ redis-cluster-redis redis-cluster-redis-1.redis-cluster-redis-headless.default.svc 6379
➜ ~
➜ ~
➜ ~ kubectl logs redis-cluster-redis-sentinel-2
reset redis sentinel conf
+ reset_redis_sentinel_conf
+ echo reset redis sentinel conf
+ sentinel_port=26379
+ [ -n 26379 ]
+ sentinel_port=26379
+ mkdir -p /data/sentinel
+ [ -f /data/sentinel/redis-sentinel.conf ]
+ build_redis_sentinel_conf
build redis sentinel conf
+ echo build redis sentinel conf
+ kb_pod_fqdn=redis-cluster-redis-sentinel-2.redis-cluster-redis-sentinel-headless.default.svc
+ echo port 26379
+ echo sentinel announce-ip redis-cluster-redis-sentinel-2.redis-cluster-redis-sentinel-headless.default.svc
+ echo sentinel resolve-hostnames yes
+ echo sentinel announce-hostnames yes
+ [ -n JK9a184T2d ]
+ echo sentinel sentinel-user default
+ echo sentinel sentinel-pass JK9a184T2d
+ start_redis_sentinel_server
+ echo Starting redis sentinel server...
+ exec redis-server /data/sentinel/redis-sentinel.conf --sentinel
Starting redis sentinel server...
1:X 24 Jun 2024 07:50:06.711 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:X 24 Jun 2024 07:50:06.711 * Redis version=7.2.4, bits=64, commit=00000000, modified=0, pid=1, just started
1:X 24 Jun 2024 07:50:06.711 * Configuration loaded
1:X 24 Jun 2024 07:50:06.712 * monotonic clock: POSIX clock_gettime
1:X 24 Jun 2024 07:50:06.713 * Running mode=sentinel, port=26379.
1:X 24 Jun 2024 07:50:06.717 * Sentinel new configuration saved on disk
1:X 24 Jun 2024 07:50:06.717 * Sentinel ID is b45b3ca0341620d06a6cff7ae04102ecddb16194
1:X 24 Jun 2024 07:51:25.454 * Sentinel new configuration saved on disk
1:X 24 Jun 2024 07:51:25.454 # +monitor master redis-cluster-redis redis-cluster-redis-0.redis-cluster-redis-headless 6379 quorum 2
1:X 24 Jun 2024 07:51:25.461 # +set master redis-cluster-redis redis-cluster-redis-0.redis-cluster-redis-headless 6379 down-after-milliseconds 5000
1:X 24 Jun 2024 07:51:25.463 * Sentinel new configuration saved on disk
1:X 24 Jun 2024 07:51:25.470 # +set master redis-cluster-redis redis-cluster-redis-0.redis-cluster-redis-headless 6379 failover-timeout 60000
1:X 24 Jun 2024 07:51:25.472 * Sentinel new configuration saved on disk
1:X 24 Jun 2024 07:51:25.478 # +set master redis-cluster-redis redis-cluster-redis-0.redis-cluster-redis-headless 6379 parallel-syncs 1
1:X 24 Jun 2024 07:51:25.480 * Sentinel new configuration saved on disk
1:X 24 Jun 2024 07:51:25.487 # +set master redis-cluster-redis redis-cluster-redis-0.redis-cluster-redis-headless 6379 auth-user kbreplicator-sentinel
1:X 24 Jun 2024 07:51:25.489 * Sentinel new configuration saved on disk
1:X 24 Jun 2024 07:51:25.495 # +set master redis-cluster-redis redis-cluster-redis-0.redis-cluster-redis-headless 6379 auth-pass ******
1:X 24 Jun 2024 07:51:25.497 * Sentinel new configuration saved on disk
1:X 24 Jun 2024 07:51:27.202 * +sentinel sentinel b2d0c0313ebd721ee18c41f23fb5b6ade4e7c6fd redis-cluster-redis-sentinel-0.redis-cluster-redis-sentinel-headless.default.svc 26379 @ redis-cluster-redis redis-cluster-redis-0.redis-cluster-redis-headless 6379
1:X 24 Jun 2024 07:51:27.205 * Sentinel new configuration saved on disk
1:X 24 Jun 2024 07:51:27.439 * +sentinel sentinel 92b772e37b41eaf86e02941646601daab23b3ccb redis-cluster-redis-sentinel-1.redis-cluster-redis-sentinel-headless.default.svc 26379 @ redis-cluster-redis redis-cluster-redis-0.redis-cluster-redis-headless 6379
1:X 24 Jun 2024 07:51:27.441 * Sentinel new configuration saved on disk
1:X 24 Jun 2024 07:51:46.645 * +slave slave redis-cluster-redis-1.redis-cluster-redis-headless.default.svc:6379 redis-cluster-redis-1.redis-cluster-redis-headless.default.svc 6379 @ redis-cluster-redis redis-cluster-redis-0.redis-cluster-redis-headless 6379
1:X 24 Jun 2024 07:51:46.647 * Sentinel new configuration saved on disk
1:X 24 Jun 2024 07:53:02.381 # +tilt #tilt mode entered
1:X 24 Jun 2024 07:53:02.381 # waitpid() returned a pid (617) we can't find in our scripts execution queue!
1:X 24 Jun 2024 07:53:32.386 # -tilt #tilt mode exited
1:X 24 Jun 2024 07:57:13.187 # +sdown master redis-cluster-redis redis-cluster-redis-0.redis-cluster-redis-headless 6379
1:X 24 Jun 2024 07:57:13.192 * Sentinel new configuration saved on disk
1:X 24 Jun 2024 07:57:13.192 # +new-epoch 1
1:X 24 Jun 2024 07:57:13.194 * Sentinel new configuration saved on disk
1:X 24 Jun 2024 07:57:13.194 # +vote-for-leader b2d0c0313ebd721ee18c41f23fb5b6ade4e7c6fd 1
1:X 24 Jun 2024 07:57:13.277 # +odown master redis-cluster-redis redis-cluster-redis-0.redis-cluster-redis-headless 6379 #quorum 3/2
1:X 24 Jun 2024 07:57:13.277 * Next failover delay: I will not start a failover before Mon Jun 24 07:59:13 2024
1:X 24 Jun 2024 07:57:14.321 # +config-update-from sentinel b2d0c0313ebd721ee18c41f23fb5b6ade4e7c6fd redis-cluster-redis-sentinel-0.redis-cluster-redis-sentinel-headless.default.svc 26379 @ redis-cluster-redis redis-cluster-redis-0.redis-cluster-redis-headless 6379
1:X 24 Jun 2024 07:57:14.321 # +switch-master redis-cluster-redis redis-cluster-redis-0.redis-cluster-redis-headless 6379 redis-cluster-redis-1.redis-cluster-redis-headless.default.svc 6379
1:X 24 Jun 2024 07:57:14.324 * +slave slave redis-cluster-redis-0.redis-cluster-redis-headless:6379 redis-cluster-redis-0.redis-cluster-redis-headless 6379 @ redis-cluster-redis redis-cluster-redis-1.redis-cluster-redis-headless.default.svc 6379
1:X 24 Jun 2024 07:57:14.326 * Sentinel new configuration saved on disk
1:X 24 Jun 2024 07:57:19.330 # +sdown slave redis-cluster-redis-0.redis-cluster-redis-headless:6379 redis-cluster-redis-0.redis-cluster-redis-headless 6379 @ redis-cluster-redis redis-cluster-redis-1.redis-cluster-redis-headless.default.svc 6379
1:X 24 Jun 2024 07:57:28.121 * +slave slave redis-cluster-redis-0.redis-cluster-redis-headless.default.svc:6379 redis-cluster-redis-0.redis-cluster-redis-headless.default.svc 6379 @ redis-cluster-redis redis-cluster-redis-1.redis-cluster-redis-headless.default.svc 6379
1:X 24 Jun 2024 07:57:28.123 * Sentinel new configuration saved on disk
1:X 24 Jun 2024 07:57:33.212 # +sdown slave redis-cluster-redis-0.redis-cluster-redis-headless.default.svc:6379 redis-cluster-redis-0.redis-cluster-redis-headless.default.svc 6379 @ redis-cluster-redis redis-cluster-redis-1.redis-cluster-redis-headless.default.svc 6379
1:X 24 Jun 2024 07:57:43.347 # -sdown slave redis-cluster-redis-0.redis-cluster-redis-headless.default.svc:6379 redis-cluster-redis-0.redis-cluster-redis-headless.default.svc 6379 @ redis-cluster-redis redis-cluster-redis-1.redis-cluster-redis-headless.default.svc 6379
1:X 24 Jun 2024 07:57:44.631 # -sdown slave redis-cluster-redis-0.redis-cluster-redis-headless:6379 redis-cluster-redis-0.redis-cluster-redis-headless 6379 @ redis-cluster-redis redis-cluster-redis-1.redis-cluster-redis-headless.default.svc 6379
➜ ~
this is an occasional error, most likely caused by a network anomaly during the Sentinel switch. From the exceptional cases observed, it appears that the internal handling logic of Sentinel during network anomalies results in this error. Additionally, it cannot self-repair.
crash ops hscale in 3 -->2 and failover delete pod-1 logs pod-0
kubectl logs redis-szittw-redis-0 redis
+ declare -g primary
+ declare -g primary_port
+ declare -g default_initialize_pod_ordinal
+ declare -g headless_postfix=headless
+ parse_redis_advertised_svc_if_exist redis-szittw-redis-0
+ local pod_name=redis-szittw-redis-0
+ [[ -z '' ]]
+ echo 'Environment variable REDIS_ADVERTISED_PORT not found. Ignoring.'
+ return 0
+ build_redis_conf
+ load_redis_template_conf
+ echo 'include /etc/conf/redis.conf'
Environment variable REDIS_ADVERTISED_PORT not found. Ignoring.
+ build_announce_ip_and_port
+ '[' -n '' ']'
+ kb_pod_fqdn=redis-szittw-redis-0.redis-szittw-redis-headless.default.svc
+ echo 'redis use kb pod fqdn redis-szittw-redis-0.redis-szittw-redis-headless.default.svc to announce'
+ echo 'replica-announce-ip redis-szittw-redis-0.redis-szittw-redis-headless.default.svc'
redis use kb pod fqdn redis-szittw-redis-0.redis-szittw-redis-headless.default.svc to announce
+ build_redis_service_port
+ service_port=6379
+ '[' -n 6379 ']'
+ service_port=6379
+ echo 'port 6379'
+ build_replicaof_config
+ init_or_get_primary_node
+ init_or_get_primary_from_redis_sentinel
+ '[' -z redis-szittw-redis-sentinel ']'
+ '[' -z redis-szittw-redis-sentinel-0,redis-szittw-redis-sentinel-1,redis-szittw-redis-sentinel-2 ']'
+ '[' -z redis-szittw-redis-sentinel-headless ']'
+ old_ifs='
'
+ IFS=,
+ set -f
+ read -ra sentinel_pod_list
+ set +f
+ IFS='
'
+ declare -A master_count_map
+ local first_redis_primary_host=
+ local first_redis_primary_port=
+ for sentinel_pod in "${sentinel_pod_list[@]}"
+ sentinel_pod_fqdn=redis-szittw-redis-sentinel-0.redis-szittw-redis-sentinel-headless
+ retry_get_master_addr_by_name_from_sentinel redis-szittw-redis-sentinel-0.redis-szittw-redis-sentinel-headless
+ local sentinel_pod_fqdn=redis-szittw-redis-sentinel-0.redis-szittw-redis-sentinel-headless
+ local retry_count=0
+ local max_retry=3
+ local retry_delay=2
+ local timeout_value=5
+ '[' 0 -lt 3 ']'
+ echo 'execute command: timeout 5 redis-cli -h redis-szittw-redis-sentinel-0.redis-szittw-redis-sentinel-headless -p 26379 -a K44b706OAB sentinel get-master-addr-by-name redis-szittw-redis'
execute command: timeout 5 redis-cli -h redis-szittw-redis-sentinel-0.redis-szittw-redis-sentinel-headless -p 26379 -a K44b706OAB sentinel get-master-addr-by-name redis-szittw-redis
++ timeout 5 redis-cli -h redis-szittw-redis-sentinel-0.redis-szittw-redis-sentinel-headless -p 26379 -a K44b706OAB sentinel get-master-addr-by-name redis-szittw-redis
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
Could not connect to Redis at redis-szittw-redis-sentinel-0.redis-szittw-redis-sentinel-headless:26379: Name or service not known
+ output=
+ exit_code=1
+ '[' 1 -eq 0 ']'
+ '[' 1 -eq 124 ']'
+ echo 'Error occurred while retrieving primary info from sentinel: redis-szittw-redis-sentinel-0.redis-szittw-redis-sentinel-headless. Retrying...'
Error occurred while retrieving primary info from sentinel: redis-szittw-redis-sentinel-0.redis-szittw-redis-sentinel-headless. Retrying...
+ retry_count=1
+ sleep 2
+ '[' 1 -lt 3 ']'
+ echo 'execute command: timeout 5 redis-cli -h redis-szittw-redis-sentinel-0.redis-szittw-redis-sentinel-headless -p 26379 -a K44b706OAB sentinel get-master-addr-by-name redis-szittw-redis'
execute command: timeout 5 redis-cli -h redis-szittw-redis-sentinel-0.redis-szittw-redis-sentinel-headless -p 26379 -a K44b706OAB sentinel get-master-addr-by-name redis-szittw-redis
++ timeout 5 redis-cli -h redis-szittw-redis-sentinel-0.redis-szittw-redis-sentinel-headless -p 26379 -a K44b706OAB sentinel get-master-addr-by-name redis-szittw-redis
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
+ output='redis-szittw-redis-2.redis-szittw-redis-headless.default.svc
6379'
+ exit_code=0
+ '[' 0 -eq 0 ']'
+ old_ifs='
'
+ IFS='
'
+ set -f
+ read -r -d '' -a REDIS_SENTINEL_PRIMARY_INFO
+ set +f
+ IFS='
'
+ '[' 2 -eq 2 ']'
+ '[' -n redis-szittw-redis-2.redis-szittw-redis-headless.default.svc ']'
+ '[' -n 6379 ']'
Successfully retrieved primary info from sentinel: redis-szittw-redis-sentinel-0.redis-szittw-redis-sentinel-headless
sentinel:redis-szittw-redis-sentinel-0.redis-szittw-redis-sentinel-headless has master info: redis-szittw-redis-2.redis-szittw-redis-headless.default.svc 6379
+ echo 'Successfully retrieved primary info from sentinel: redis-szittw-redis-sentinel-0.redis-szittw-redis-sentinel-headless'
+ return 0
+ echo 'sentinel:redis-szittw-redis-sentinel-0.redis-szittw-redis-sentinel-headless has master info: redis-szittw-redis-2.redis-szittw-redis-headless.default.svc 6379'
+ '[' 2 -ne 2 ']'
+ '[' -z redis-szittw-redis-2.redis-szittw-redis-headless.default.svc ']'
+ '[' -z 6379 ']'
+ host_port_key=redis-szittw-redis-2.redis-szittw-redis-headless.default.svc:6379
+ master_count_map[$host_port_key]=1
+ [[ -z '' ]]
+ [[ -z '' ]]
+ first_redis_primary_host=redis-szittw-redis-2.redis-szittw-redis-headless.default.svc
+ first_redis_primary_port=6379
+ '[' redis-szittw-redis-2.redis-szittw-redis-headless.default.svc '!=' redis-szittw-redis-2.redis-szittw-redis-headless.default.svc ']'
+ '[' 6379 '!=' 6379 ']'
+ for sentinel_pod in "${sentinel_pod_list[@]}"
+ sentinel_pod_fqdn=redis-szittw-redis-sentinel-1.redis-szittw-redis-sentinel-headless
+ retry_get_master_addr_by_name_from_sentinel redis-szittw-redis-sentinel-1.redis-szittw-redis-sentinel-headless
+ local sentinel_pod_fqdn=redis-szittw-redis-sentinel-1.redis-szittw-redis-sentinel-headless
+ local retry_count=0
+ local max_retry=3
+ local retry_delay=2
+ local timeout_value=5
+ '[' 0 -lt 3 ']'
execute command: timeout 5 redis-cli -h redis-szittw-redis-sentinel-1.redis-szittw-redis-sentinel-headless -p 26379 -a K44b706OAB sentinel get-master-addr-by-name redis-szittw-redis
+ echo 'execute command: timeout 5 redis-cli -h redis-szittw-redis-sentinel-1.redis-szittw-redis-sentinel-headless -p 26379 -a K44b706OAB sentinel get-master-addr-by-name redis-szittw-redis'
++ timeout 5 redis-cli -h redis-szittw-redis-sentinel-1.redis-szittw-redis-sentinel-headless -p 26379 -a K44b706OAB sentinel get-master-addr-by-name redis-szittw-redis
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
+ output='redis-szittw-redis-2.redis-szittw-redis-headless.default.svc
6379'
+ exit_code=0
Successfully retrieved primary info from sentinel: redis-szittw-redis-sentinel-1.redis-szittw-redis-sentinel-headless
sentinel:redis-szittw-redis-sentinel-1.redis-szittw-redis-sentinel-headless has master info: redis-szittw-redis-2.redis-szittw-redis-headless.default.svc 6379
execute command: timeout 5 redis-cli -h redis-szittw-redis-sentinel-2.redis-szittw-redis-sentinel-headless -p 26379 -a K44b706OAB sentinel get-master-addr-by-name redis-szittw-redis
+ '[' 0 -eq 0 ']'
+ old_ifs='
'
+ IFS='
'
+ set -f
+ read -r -d '' -a REDIS_SENTINEL_PRIMARY_INFO
+ set +f
+ IFS='
'
+ '[' 2 -eq 2 ']'
+ '[' -n redis-szittw-redis-2.redis-szittw-redis-headless.default.svc ']'
+ '[' -n 6379 ']'
+ echo 'Successfully retrieved primary info from sentinel: redis-szittw-redis-sentinel-1.redis-szittw-redis-sentinel-headless'
+ return 0
+ echo 'sentinel:redis-szittw-redis-sentinel-1.redis-szittw-redis-sentinel-headless has master info: redis-szittw-redis-2.redis-szittw-redis-headless.default.svc 6379'
+ '[' 2 -ne 2 ']'
+ '[' -z redis-szittw-redis-2.redis-szittw-redis-headless.default.svc ']'
+ '[' -z 6379 ']'
+ host_port_key=redis-szittw-redis-2.redis-szittw-redis-headless.default.svc:6379
+ master_count_map[$host_port_key]=2
+ [[ -z redis-szittw-redis-2.redis-szittw-redis-headless.default.svc ]]
+ '[' redis-szittw-redis-2.redis-szittw-redis-headless.default.svc '!=' redis-szittw-redis-2.redis-szittw-redis-headless.default.svc ']'
+ '[' 6379 '!=' 6379 ']'
+ for sentinel_pod in "${sentinel_pod_list[@]}"
+ sentinel_pod_fqdn=redis-szittw-redis-sentinel-2.redis-szittw-redis-sentinel-headless
+ retry_get_master_addr_by_name_from_sentinel redis-szittw-redis-sentinel-2.redis-szittw-redis-sentinel-headless
+ local sentinel_pod_fqdn=redis-szittw-redis-sentinel-2.redis-szittw-redis-sentinel-headless
+ local retry_count=0
+ local max_retry=3
+ local retry_delay=2
+ local timeout_value=5
+ '[' 0 -lt 3 ']'
+ echo 'execute command: timeout 5 redis-cli -h redis-szittw-redis-sentinel-2.redis-szittw-redis-sentinel-headless -p 26379 -a K44b706OAB sentinel get-master-addr-by-name redis-szittw-redis'
++ timeout 5 redis-cli -h redis-szittw-redis-sentinel-2.redis-szittw-redis-sentinel-headless -p 26379 -a K44b706OAB sentinel get-master-addr-by-name redis-szittw-redis
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
+ output='redis-szittw-redis-2.redis-szittw-redis-headless.default.svc
6379'
+ exit_code=0
+ '[' 0 -eq 0 ']'
+ old_ifs='
'
+ IFS='
'
+ set -f
+ read -r -d '' -a REDIS_SENTINEL_PRIMARY_INFO
+ set +f
+ IFS='
'
+ '[' 2 -eq 2 ']'
+ '[' -n redis-szittw-redis-2.redis-szittw-redis-headless.default.svc ']'
+ '[' -n 6379 ']'
+ echo 'Successfully retrieved primary info from sentinel: redis-szittw-redis-sentinel-2.redis-szittw-redis-sentinel-headless'
+ return 0
Successfully retrieved primary info from sentinel: redis-szittw-redis-sentinel-2.redis-szittw-redis-sentinel-headless
+ echo 'sentinel:redis-szittw-redis-sentinel-2.redis-szittw-redis-sentinel-headless has master info: redis-szittw-redis-2.redis-szittw-redis-headless.default.svc 6379'
+ '[' 2 -ne 2 ']'
sentinel:redis-szittw-redis-sentinel-2.redis-szittw-redis-sentinel-headless has master info: redis-szittw-redis-2.redis-szittw-redis-headless.default.svc 6379
+ '[' -z redis-szittw-redis-2.redis-szittw-redis-headless.default.svc ']'
+ '[' -z 6379 ']'
+ host_port_key=redis-szittw-redis-2.redis-szittw-redis-headless.default.svc:6379
+ master_count_map[$host_port_key]=3
+ [[ -z redis-szittw-redis-2.redis-szittw-redis-headless.default.svc ]]
+ '[' redis-szittw-redis-2.redis-szittw-redis-headless.default.svc '!=' redis-szittw-redis-2.redis-szittw-redis-headless.default.svc ']'
+ '[' 6379 '!=' 6379 ']'
get all primary info from redis sentinel master_count_map: 3
+ echo 'get all primary info from redis sentinel master_count_map: 3'
+ '[' 1 -eq 0 ']'
+ max_count=0
+ for host_port in "${!master_count_map[@]}"
+ (( 3 > max_count ))
+ max_count=3
++ echo redis-szittw-redis-2.redis-szittw-redis-headless.default.svc:6379
++ cut -d: -f1
+ primary=redis-szittw-redis-2.redis-szittw-redis-headless.default.svc
++ echo redis-szittw-redis-2.redis-szittw-redis-headless.default.svc:6379
++ cut -d: -f2
+ primary_port=6379
+ check_current_pod_is_primary
+ current_pod=redis-szittw-redis-0.redis-szittw-redis
+ [[ redis-szittw-redis-2.redis-szittw-redis-headless.default.svc == *\r\e\d\i\s\-\s\z\i\t\t\w\-\r\e\d\i\s\-\0\.\r\e\d\i\s\-\s\z\i\t\t\w\-\r\e\d\i\s* ]]
+ '[' -n '' ']'
+ [[ redis-szittw-redis-2.redis-szittw-redis-headless.default.svc == \1\0\.\1\2\8\.\1\.\7\2 ]]
+ return 1
+ '[' -n wM1Tx1263W ']'
+ check_kernel_role_cmd='redis-cli -h redis-szittw-redis-2.redis-szittw-redis-headless.default.svc -p 6379 -a wM1Tx1263W info replication | grep '\''role:'\'' | awk -F: '\''{print $2}'\'''
+ retry_times=10
+ true
++ eval 'redis-cli -h redis-szittw-redis-2.redis-szittw-redis-headless.default.svc -p 6379 -a wM1Tx1263W info replication | grep '\''role:'\'' | awk -F: '\''{print $2}'\'''
+++ grep role:
+++ redis-cli -h redis-szittw-redis-2.redis-szittw-redis-headless.default.svc -p 6379 -a wM1Tx1263W info replication
+++ awk -F: '{print $2}'
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
+ check_role=$'master\r'
=~ master ]]
+ break
+ check_current_pod_is_primary
+ current_pod=redis-szittw-redis-0.redis-szittw-redis
+ [[ redis-szittw-redis-2.redis-szittw-redis-headless.default.svc == *\r\e\d\i\s\-\s\z\i\t\t\w\-\r\e\d\i\s\-\0\.\r\e\d\i\s\-\s\z\i\t\t\w\-\r\e\d\i\s* ]]
+ '[' -n '' ']'
+ [[ redis-szittw-redis-2.redis-szittw-redis-headless.default.svc == \1\0\.\1\2\8\.\1\.\7\2 ]]
+ return 1
+ echo 'replicaof redis-szittw-redis-2.redis-szittw-redis-headless.default.svc 6379'
+ rebuild_redis_acl_file
+ '[' -f /data/users.acl ']'
+ sed -i '/user default on/d' /data/users.acl
+ sed -i '/user kbreplicator on/d' /data/users.acl
+ sed -i '/user kbreplicator-sentinel on/d' /data/users.acl
+ build_redis_default_accounts
+ '[' -n wM1Tx1263W ']'
+ echo 'masteruser kbreplicator'
+ echo 'masterauth wM1Tx1263W'
+ echo 'user kbreplicator on +psync +replconf +ping >wM1Tx1263W'
+ '[' -n wM1Tx1263W ']'
+ echo 'user kbreplicator-sentinel on allchannels +multi +slaveof +ping +exec +subscribe +config|rewrite +role +publish +info +client|setname +client|kill +script|kill >wM1Tx1263W'
+ '[' -n wM1Tx1263W ']'
+ echo 'protected-mode yes'
+ echo 'user default on >wM1Tx1263W ~* &* +@all '
+ echo 'aclfile /data/users.acl'
+ start_redis_server
+ exec_cmd='exec redis-server /etc/redis/redis.conf'
+ '[' -f /opt/redis-stack/lib/redisearch.so ']'
+ exec_cmd='exec redis-server /etc/redis/redis.conf --loadmodule /opt/redis-stack/lib/redisearch.so '
+ '[' -f /opt/redis-stack/lib/redistimeseries.so ']'
+ exec_cmd='exec redis-server /etc/redis/redis.conf --loadmodule /opt/redis-stack/lib/redisearch.so --loadmodule /opt/redis-stack/lib/redistimeseries.so '
+ '[' -f /opt/redis-stack/lib/rejson.so ']'
+ exec_cmd='exec redis-server /etc/redis/redis.conf --loadmodule /opt/redis-stack/lib/redisearch.so --loadmodule /opt/redis-stack/lib/redistimeseries.so --loadmodule /opt/redis-stack/lib/rejson.so '
+ '[' -f /opt/redis-stack/lib/redisbloom.so ']'
+ exec_cmd='exec redis-server /etc/redis/redis.conf --loadmodule /opt/redis-stack/lib/redisearch.so --loadmodule /opt/redis-stack/lib/redistimeseries.so --loadmodule /opt/redis-stack/lib/rejson.so --loadmodule /opt/redis-stack/lib/redisbloom.so '
+ '[' -f /opt/redis-stack/lib/redisgraph.so ']'
+ '[' -f /opt/redis-stack/lib/rediscompat.so ']'
+ exec_cmd='exec redis-server /etc/redis/redis.conf --loadmodule /opt/redis-stack/lib/redisearch.so --loadmodule /opt/redis-stack/lib/redistimeseries.so --loadmodule /opt/redis-stack/lib/rejson.so --loadmodule /opt/redis-stack/lib/redisbloom.so --loadmodule /opt/redis-stack/lib/rediscompat.so'
+ '[' -f /opt/redis-stack/lib/redisgears.so ']'
+ exec_cmd='exec redis-server /etc/redis/redis.conf --loadmodule /opt/redis-stack/lib/redisearch.so --loadmodule /opt/redis-stack/lib/redistimeseries.so --loadmodule /opt/redis-stack/lib/rejson.so --loadmodule /opt/redis-stack/lib/redisbloom.so --loadmodule /opt/redis-stack/lib/rediscompat.so --loadmodule /opt/redis-stack/lib/redisgears.so v8-plugin-path /opt/redis-stack/lib/libredisgears_v8_plugin.so '
+ echo 'Starting redis server cmd: exec redis-server /etc/redis/redis.conf --loadmodule /opt/redis-stack/lib/redisearch.so --loadmodule /opt/redis-stack/lib/redistimeseries.so --loadmodule /opt/redis-stack/lib/rejson.so --loadmodule /opt/redis-stack/lib/redisbloom.so --loadmodule /opt/redis-stack/lib/rediscompat.so --loadmodule /opt/redis-stack/lib/redisgears.so v8-plugin-path /opt/redis-stack/lib/libredisgears_v8_plugin.so '
+ eval 'exec redis-server /etc/redis/redis.conf --loadmodule /opt/redis-stack/lib/redisearch.so --loadmodule /opt/redis-stack/lib/redistimeseries.so --loadmodule /opt/redis-stack/lib/rejson.so --loadmodule /opt/redis-stack/lib/redisbloom.so --loadmodule /opt/redis-stack/lib/rediscompat.so --loadmodule /opt/redis-stack/lib/redisgears.so v8-plugin-path /opt/redis-stack/lib/libredisgears_v8_plugin.so '
Starting redis server cmd: exec redis-server /etc/redis/redis.conf --loadmodule /opt/redis-stack/lib/redisearch.so --loadmodule /opt/redis-stack/lib/redistimeseries.so --loadmodule /opt/redis-stack/lib/rejson.so --loadmodule /opt/redis-stack/lib/redisbloom.so --loadmodule /opt/redis-stack/lib/rediscompat.so --loadmodule /opt/redis-stack/lib/redisgears.so v8-plugin-path /opt/redis-stack/lib/libredisgears_v8_plugin.so
++ exec redis-server /etc/redis/redis.conf --loadmodule /opt/redis-stack/lib/redisearch.so --loadmodule /opt/redis-stack/lib/redistimeseries.so --loadmodule /opt/redis-stack/lib/rejson.so --loadmodule /opt/redis-stack/lib/redisbloom.so --loadmodule /opt/redis-stack/lib/rediscompat.so --loadmodule /opt/redis-stack/lib/redisgears.so v8-plugin-path /opt/redis-stack/lib/libredisgears_v8_plugin.so
logs pod-1
kubectl logs redis-szittw-redis-1 redis --previous
+ declare -g primary
+ declare -g primary_port
+ declare -g default_initialize_pod_ordinal
+ declare -g headless_postfix=headless
+ parse_redis_advertised_svc_if_exist redis-szittw-redis-1
+ local pod_name=redis-szittw-redis-1
+ [[ -z '' ]]
+ echo 'Environment variable REDIS_ADVERTISED_PORT not found. Ignoring.'
+ return 0
Environment variable REDIS_ADVERTISED_PORT not found. Ignoring.
+ build_redis_conf
+ load_redis_template_conf
+ echo 'include /etc/conf/redis.conf'
+ build_announce_ip_and_port
+ '[' -n '' ']'
+ kb_pod_fqdn=redis-szittw-redis-1.redis-szittw-redis-headless.default.svc
+ echo 'redis use kb pod fqdn redis-szittw-redis-1.redis-szittw-redis-headless.default.svc to announce'
+ echo 'replica-announce-ip redis-szittw-redis-1.redis-szittw-redis-headless.default.svc'
redis use kb pod fqdn redis-szittw-redis-1.redis-szittw-redis-headless.default.svc to announce
+ build_redis_service_port
+ service_port=6379
+ '[' -n 6379 ']'
+ service_port=6379
+ echo 'port 6379'
+ build_replicaof_config
+ init_or_get_primary_node
+ init_or_get_primary_from_redis_sentinel
+ '[' -z redis-szittw-redis-sentinel ']'
+ '[' -z redis-szittw-redis-sentinel-0,redis-szittw-redis-sentinel-1,redis-szittw-redis-sentinel-2 ']'
+ '[' -z redis-szittw-redis-sentinel-headless ']'
+ old_ifs='
'
+ IFS=,
+ set -f
+ read -ra sentinel_pod_list
+ set +f
+ IFS='
'
+ declare -A master_count_map
+ local first_redis_primary_host=
+ local first_redis_primary_port=
+ for sentinel_pod in "${sentinel_pod_list[@]}"
+ sentinel_pod_fqdn=redis-szittw-redis-sentinel-0.redis-szittw-redis-sentinel-headless
+ retry_get_master_addr_by_name_from_sentinel redis-szittw-redis-sentinel-0.redis-szittw-redis-sentinel-headless
+ local sentinel_pod_fqdn=redis-szittw-redis-sentinel-0.redis-szittw-redis-sentinel-headless
+ local retry_count=0
+ local max_retry=3
+ local retry_delay=2
+ local timeout_value=5
+ '[' 0 -lt 3 ']'
+ echo 'execute command: timeout 5 redis-cli -h redis-szittw-redis-sentinel-0.redis-szittw-redis-sentinel-headless -p 26379 -a K44b706OAB sentinel get-master-addr-by-name redis-szittw-redis'
execute command: timeout 5 redis-cli -h redis-szittw-redis-sentinel-0.redis-szittw-redis-sentinel-headless -p 26379 -a K44b706OAB sentinel get-master-addr-by-name redis-szittw-redis
++ timeout 5 redis-cli -h redis-szittw-redis-sentinel-0.redis-szittw-redis-sentinel-headless -p 26379 -a K44b706OAB sentinel get-master-addr-by-name redis-szittw-redis
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
+ output='redis-szittw-redis-0.redis-szittw-redis-headless
6379'
+ exit_code=0
+ '[' 0 -eq 0 ']'
+ old_ifs='
'
+ IFS='
'
+ set -f
+ read -r -d '' -a REDIS_SENTINEL_PRIMARY_INFO
+ set +f
+ IFS='
'
+ '[' 2 -eq 2 ']'
+ '[' -n redis-szittw-redis-0.redis-szittw-redis-headless ']'
+ '[' -n 6379 ']'
+ echo 'Successfully retrieved primary info from sentinel: redis-szittw-redis-sentinel-0.redis-szittw-redis-sentinel-headless'
+ return 0
Successfully retrieved primary info from sentinel: redis-szittw-redis-sentinel-0.redis-szittw-redis-sentinel-headless
sentinel:redis-szittw-redis-sentinel-0.redis-szittw-redis-sentinel-headless has master info: redis-szittw-redis-0.redis-szittw-redis-headless 6379
+ echo 'sentinel:redis-szittw-redis-sentinel-0.redis-szittw-redis-sentinel-headless has master info: redis-szittw-redis-0.redis-szittw-redis-headless 6379'
+ '[' 2 -ne 2 ']'
+ '[' -z redis-szittw-redis-0.redis-szittw-redis-headless ']'
+ '[' -z 6379 ']'
+ host_port_key=redis-szittw-redis-0.redis-szittw-redis-headless:6379
+ master_count_map[$host_port_key]=1
+ [[ -z '' ]]
+ [[ -z '' ]]
+ first_redis_primary_host=redis-szittw-redis-0.redis-szittw-redis-headless
+ first_redis_primary_port=6379
+ '[' redis-szittw-redis-0.redis-szittw-redis-headless '!=' redis-szittw-redis-0.redis-szittw-redis-headless ']'
+ '[' 6379 '!=' 6379 ']'
+ for sentinel_pod in "${sentinel_pod_list[@]}"
+ sentinel_pod_fqdn=redis-szittw-redis-sentinel-1.redis-szittw-redis-sentinel-headless
+ retry_get_master_addr_by_name_from_sentinel redis-szittw-redis-sentinel-1.redis-szittw-redis-sentinel-headless
+ local sentinel_pod_fqdn=redis-szittw-redis-sentinel-1.redis-szittw-redis-sentinel-headless
+ local retry_count=0
+ local max_retry=3
+ local retry_delay=2
+ local timeout_value=5
+ '[' 0 -lt 3 ']'
+ echo 'execute command: timeout 5 redis-cli -h redis-szittw-redis-sentinel-1.redis-szittw-redis-sentinel-headless -p 26379 -a K44b706OAB sentinel get-master-addr-by-name redis-szittw-redis'
execute command: timeout 5 redis-cli -h redis-szittw-redis-sentinel-1.redis-szittw-redis-sentinel-headless -p 26379 -a K44b706OAB sentinel get-master-addr-by-name redis-szittw-redis
++ timeout 5 redis-cli -h redis-szittw-redis-sentinel-1.redis-szittw-redis-sentinel-headless -p 26379 -a K44b706OAB sentinel get-master-addr-by-name redis-szittw-redis
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
+ output='redis-szittw-redis-0.redis-szittw-redis-headless
6379'
+ exit_code=0
+ '[' 0 -eq 0 ']'
+ old_ifs='
'
+ IFS='
'
+ set -f
+ read -r -d '' -a REDIS_SENTINEL_PRIMARY_INFO
+ set +f
+ IFS='
'
+ '[' 2 -eq 2 ']'
+ '[' -n redis-szittw-redis-0.redis-szittw-redis-headless ']'
+ '[' -n 6379 ']'
+ echo 'Successfully retrieved primary info from sentinel: redis-szittw-redis-sentinel-1.redis-szittw-redis-sentinel-headless'
+ return 0
Successfully retrieved primary info from sentinel: redis-szittw-redis-sentinel-1.redis-szittw-redis-sentinel-headless
sentinel:redis-szittw-redis-sentinel-1.redis-szittw-redis-sentinel-headless has master info: redis-szittw-redis-0.redis-szittw-redis-headless 6379
+ echo 'sentinel:redis-szittw-redis-sentinel-1.redis-szittw-redis-sentinel-headless has master info: redis-szittw-redis-0.redis-szittw-redis-headless 6379'
+ '[' 2 -ne 2 ']'
+ '[' -z redis-szittw-redis-0.redis-szittw-redis-headless ']'
+ '[' -z 6379 ']'
+ host_port_key=redis-szittw-redis-0.redis-szittw-redis-headless:6379
+ master_count_map[$host_port_key]=2
+ [[ -z redis-szittw-redis-0.redis-szittw-redis-headless ]]
+ '[' redis-szittw-redis-0.redis-szittw-redis-headless '!=' redis-szittw-redis-0.redis-szittw-redis-headless ']'
+ '[' 6379 '!=' 6379 ']'
+ for sentinel_pod in "${sentinel_pod_list[@]}"
+ sentinel_pod_fqdn=redis-szittw-redis-sentinel-2.redis-szittw-redis-sentinel-headless
+ retry_get_master_addr_by_name_from_sentinel redis-szittw-redis-sentinel-2.redis-szittw-redis-sentinel-headless
+ local sentinel_pod_fqdn=redis-szittw-redis-sentinel-2.redis-szittw-redis-sentinel-headless
+ local retry_count=0
+ local max_retry=3
+ local retry_delay=2
+ local timeout_value=5
+ '[' 0 -lt 3 ']'
+ echo 'execute command: timeout 5 redis-cli -h redis-szittw-redis-sentinel-2.redis-szittw-redis-sentinel-headless -p 26379 -a K44b706OAB sentinel get-master-addr-by-name redis-szittw-redis'
execute command: timeout 5 redis-cli -h redis-szittw-redis-sentinel-2.redis-szittw-redis-sentinel-headless -p 26379 -a K44b706OAB sentinel get-master-addr-by-name redis-szittw-redis
++ timeout 5 redis-cli -h redis-szittw-redis-sentinel-2.redis-szittw-redis-sentinel-headless -p 26379 -a K44b706OAB sentinel get-master-addr-by-name redis-szittw-redis
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
+ output='redis-szittw-redis-0.redis-szittw-redis-headless
6379'
+ exit_code=0
+ '[' 0 -eq 0 ']'
+ old_ifs='
'
+ IFS='
'
+ set -f
+ read -r -d '' -a REDIS_SENTINEL_PRIMARY_INFO
+ set +f
+ IFS='
'
+ '[' 2 -eq 2 ']'
+ '[' -n redis-szittw-redis-0.redis-szittw-redis-headless ']'
+ '[' -n 6379 ']'
+ echo 'Successfully retrieved primary info from sentinel: redis-szittw-redis-sentinel-2.redis-szittw-redis-sentinel-headless'
Successfully retrieved primary info from sentinel: redis-szittw-redis-sentinel-2.redis-szittw-redis-sentinel-headless
+ return 0
sentinel:redis-szittw-redis-sentinel-2.redis-szittw-redis-sentinel-headless has master info: redis-szittw-redis-0.redis-szittw-redis-headless 6379
+ echo 'sentinel:redis-szittw-redis-sentinel-2.redis-szittw-redis-sentinel-headless has master info: redis-szittw-redis-0.redis-szittw-redis-headless 6379'
+ '[' 2 -ne 2 ']'
+ '[' -z redis-szittw-redis-0.redis-szittw-redis-headless ']'
+ '[' -z 6379 ']'
+ host_port_key=redis-szittw-redis-0.redis-szittw-redis-headless:6379
+ master_count_map[$host_port_key]=3
+ [[ -z redis-szittw-redis-0.redis-szittw-redis-headless ]]
+ '[' redis-szittw-redis-0.redis-szittw-redis-headless '!=' redis-szittw-redis-0.redis-szittw-redis-headless ']'
+ '[' 6379 '!=' 6379 ']'
+ echo 'get all primary info from redis sentinel master_count_map: 3'
get all primary info from redis sentinel master_count_map: 3
+ '[' 1 -eq 0 ']'
+ max_count=0
+ for host_port in "${!master_count_map[@]}"
+ (( 3 > max_count ))
+ max_count=3
++ echo redis-szittw-redis-0.redis-szittw-redis-headless:6379
++ cut -d: -f1
+ primary=redis-szittw-redis-0.redis-szittw-redis-headless
++ echo redis-szittw-redis-0.redis-szittw-redis-headless:6379
++ cut -d: -f2
+ primary_port=6379
+ check_current_pod_is_primary
+ current_pod=redis-szittw-redis-1.redis-szittw-redis
+ [[ redis-szittw-redis-0.redis-szittw-redis-headless == *\r\e\d\i\s\-\s\z\i\t\t\w\-\r\e\d\i\s\-\1\.\r\e\d\i\s\-\s\z\i\t\t\w\-\r\e\d\i\s* ]]
+ '[' -n '' ']'
+ [[ redis-szittw-redis-0.redis-szittw-redis-headless == \1\0\.\1\2\8\.\0\.\1\3\4 ]]
+ return 1
+ '[' -n wM1Tx1263W ']'
+ check_kernel_role_cmd='redis-cli -h redis-szittw-redis-0.redis-szittw-redis-headless -p 6379 -a wM1Tx1263W info replication | grep '\''role:'\'' | awk -F: '\''{print $2}'\'''
+ retry_times=10
+ true
++ eval 'redis-cli -h redis-szittw-redis-0.redis-szittw-redis-headless -p 6379 -a wM1Tx1263W info replication | grep '\''role:'\'' | awk -F: '\''{print $2}'\'''
+++ redis-cli -h redis-szittw-redis-0.redis-szittw-redis-headless -p 6379 -a wM1Tx1263W info replication
+++ grep role:
+++ awk -F: '{print $2}'
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
+ check_role=$'slave\r'
=~ master ]]
' echo 'the selected primary node is not the real master in kernel, existing primary node: redis-szittw-redis-0.redis-szittw-redis-headless, role: slave
+ sleep 3
the selected primary node is not the real master in kernel, existing primary node: redis-szittw-redis-0.redis-szittw-redis-headless, role: slave
+ retry_times=9
+ '[' 9 -eq 0 ']'
+ true
++ eval 'redis-cli -h redis-szittw-redis-0.redis-szittw-redis-headless -p 6379 -a wM1Tx1263W info replication | grep '\''role:'\'' | awk -F: '\''{print $2}'\'''
+++ redis-cli -h redis-szittw-redis-0.redis-szittw-redis-headless -p 6379 -a wM1Tx1263W info replication
+++ grep role:
+++ awk -F: '{print $2}'
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
+ check_role=$'slave\r'
=~ master ]]
' echo 'the selected primary node is not the real master in kernel, existing primary node: redis-szittw-redis-0.redis-szittw-redis-headless, role: slave
+ sleep 3
the selected primary node is not the real master in kernel, existing primary node: redis-szittw-redis-0.redis-szittw-redis-headless, role: slave
+ retry_times=8
+ '[' 8 -eq 0 ']'
+ true
++ eval 'redis-cli -h redis-szittw-redis-0.redis-szittw-redis-headless -p 6379 -a wM1Tx1263W info replication | grep '\''role:'\'' | awk -F: '\''{print $2}'\'''
+++ redis-cli -h redis-szittw-redis-0.redis-szittw-redis-headless -p 6379 -a wM1Tx1263W info replication
+++ grep role:
+++ awk -F: '{print $2}'
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
+ check_role=$'slave\r'
=~ master ]]
' echo 'the selected primary node is not the real master in kernel, existing primary node: redis-szittw-redis-0.redis-szittw-redis-headless, role: slave
+ sleep 3
the selected primary node is not the real master in kernel, existing primary node: redis-szittw-redis-0.redis-szittw-redis-headless, role: slave
+ retry_times=7
+ '[' 7 -eq 0 ']'
+ true
++ eval 'redis-cli -h redis-szittw-redis-0.redis-szittw-redis-headless -p 6379 -a wM1Tx1263W info replication | grep '\''role:'\'' | awk -F: '\''{print $2}'\'''
+++ redis-cli -h redis-szittw-redis-0.redis-szittw-redis-headless -p 6379 -a wM1Tx1263W info replication
+++ grep role:
+++ awk -F: '{print $2}'
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
+ check_role=$'slave\r'
=~ master ]]
' echo 'the selected primary node is not the real master in kernel, existing primary node: redis-szittw-redis-0.redis-szittw-redis-headless, role: slave
+ sleep 3
the selected primary node is not the real master in kernel, existing primary node: redis-szittw-redis-0.redis-szittw-redis-headless, role: slave
+ retry_times=6
+ '[' 6 -eq 0 ']'
+ true
++ eval 'redis-cli -h redis-szittw-redis-0.redis-szittw-redis-headless -p 6379 -a wM1Tx1263W info replication | grep '\''role:'\'' | awk -F: '\''{print $2}'\'''
+++ redis-cli -h redis-szittw-redis-0.redis-szittw-redis-headless -p 6379 -a wM1Tx1263W info replication
+++ grep role:
+++ awk -F: '{print $2}'
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
+ check_role=$'slave\r'
=~ master ]]
' echo 'the selected primary node is not the real master in kernel, existing primary node: redis-szittw-redis-0.redis-szittw-redis-headless, role: slave
+ sleep 3
the selected primary node is not the real master in kernel, existing primary node: redis-szittw-redis-0.redis-szittw-redis-headless, role: slave
+ retry_times=5
+ '[' 5 -eq 0 ']'
+ true
++ eval 'redis-cli -h redis-szittw-redis-0.redis-szittw-redis-headless -p 6379 -a wM1Tx1263W info replication | grep '\''role:'\'' | awk -F: '\''{print $2}'\'''
+++ redis-cli -h redis-szittw-redis-0.redis-szittw-redis-headless -p 6379 -a wM1Tx1263W info replication
+++ grep role:
+++ awk -F: '{print $2}'
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
+ check_role=$'slave\r'
=~ master ]]
' echo 'the selected primary node is not the real master in kernel, existing primary node: redis-szittw-redis-0.redis-szittw-redis-headless, role: slave
the selected primary node is not the real master in kernel, existing primary node: redis-szittw-redis-0.redis-szittw-redis-headless, role: slave
+ sleep 3
+ retry_times=4
+ '[' 4 -eq 0 ']'
+ true
++ eval 'redis-cli -h redis-szittw-redis-0.redis-szittw-redis-headless -p 6379 -a wM1Tx1263W info replication | grep '\''role:'\'' | awk -F: '\''{print $2}'\'''
+++ redis-cli -h redis-szittw-redis-0.redis-szittw-redis-headless -p 6379 -a wM1Tx1263W info replication
+++ grep role:
+++ awk -F: '{print $2}'
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
+ check_role=$'slave\r'
=~ master ]]
' echo 'the selected primary node is not the real master in kernel, existing primary node: redis-szittw-redis-0.redis-szittw-redis-headless, role: slave
+ sleep 3
the selected primary node is not the real master in kernel, existing primary node: redis-szittw-redis-0.redis-szittw-redis-headless, role: slave
+ retry_times=3
+ '[' 3 -eq 0 ']'
+ true
++ eval 'redis-cli -h redis-szittw-redis-0.redis-szittw-redis-headless -p 6379 -a wM1Tx1263W info replication | grep '\''role:'\'' | awk -F: '\''{print $2}'\'''
+++ redis-cli -h redis-szittw-redis-0.redis-szittw-redis-headless -p 6379 -a wM1Tx1263W info replication
+++ grep role:
+++ awk -F: '{print $2}'
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
+ check_role=$'slave\r'
=~ master ]]
' echo 'the selected primary node is not the real master in kernel, existing primary node: redis-szittw-redis-0.redis-szittw-redis-headless, role: slave
+ sleep 3
the selected primary node is not the real master in kernel, existing primary node: redis-szittw-redis-0.redis-szittw-redis-headless, role: slave
+ retry_times=2
+ '[' 2 -eq 0 ']'
+ true
++ eval 'redis-cli -h redis-szittw-redis-0.redis-szittw-redis-headless -p 6379 -a wM1Tx1263W info replication | grep '\''role:'\'' | awk -F: '\''{print $2}'\'''
+++ redis-cli -h redis-szittw-redis-0.redis-szittw-redis-headless -p 6379 -a wM1Tx1263W info replication
+++ grep role:
+++ awk -F: '{print $2}'
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
+ check_role=$'slave\r'
=~ master ]]
' echo 'the selected primary node is not the real master in kernel, existing primary node: redis-szittw-redis-0.redis-szittw-redis-headless, role: slave
+ sleep 3
the selected primary node is not the real master in kernel, existing primary node: redis-szittw-redis-0.redis-szittw-redis-headless, role: slave
+ retry_times=1
+ '[' 1 -eq 0 ']'
+ true
++ eval 'redis-cli -h redis-szittw-redis-0.redis-szittw-redis-headless -p 6379 -a wM1Tx1263W info replication | grep '\''role:'\'' | awk -F: '\''{print $2}'\'''
+++ redis-cli -h redis-szittw-redis-0.redis-szittw-redis-headless -p 6379 -a wM1Tx1263W info replication
+++ grep role:
+++ awk -F: '{print $2}'
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
+ check_role=$'slave\r'
=~ master ]]
' echo 'the selected primary node is not the real master in kernel, existing primary node: redis-szittw-redis-0.redis-szittw-redis-headless, role: slave
+ sleep 3
the selected primary node is not the real master in kernel, existing primary node: redis-szittw-redis-0.redis-szittw-redis-headless, role: slave
+ retry_times=0
+ '[' 0 -eq 0 ']'
' echo 'check primary node role failed after 20 times, existing primary node: redis-szittw-redis-0.redis-szittw-redis-headless, role: slave
+ exit 1
check primary node role failed after 20 times, existing primary node: redis-szittw-redis-0.redis-szittw-redis-headless, role: slave
pod-0 running.log pod-0-running-logs.txt
Sentinel logs pod-0
kubectl logs redis-szittw-redis-sentinel-0
+ reset_redis_sentinel_conf
+ echo reset redis sentinel conf
+ sentinel_port=26379
+ [ -n 26379 ]
reset redis sentinel conf
+ sentinel_port=26379
+ mkdir -p /data/sentinel
+ [ -f /data/sentinel/redis-sentinel.conf ]
+ sed -i /sentinel announce-ip/d /data/sentinel/redis-sentinel.conf
+ sed -i /sentinel resolve-hostnames/d /data/sentinel/redis-sentinel.conf
+ sed -i /sentinel announce-hostnames/d /data/sentinel/redis-sentinel.conf
+ [ -n K44b706OAB ]
+ sed -i /sentinel sentinel-user/d /data/sentinel/redis-sentinel.conf
+ sed -i /sentinel sentinel-pass/d /data/sentinel/redis-sentinel.conf
+ sed -i /port 26379/d /data/sentinel/redis-sentinel.conf
+ build_redis_sentinel_conf
+ echo build redis sentinel conf
+ kb_pod_fqdn=redis-szittw-redis-sentinel-0.redis-szittw-redis-sentinel-headless.default.svc
build redis sentinel conf
+ echo port 26379
+ echo sentinel announce-ip redis-szittw-redis-sentinel-0.redis-szittw-redis-sentinel-headless.default.svc
+ echo sentinel resolve-hostnames yes
+ echo sentinel announce-hostnames yes
+ [ -n K44b706OAB ]
+ echo sentinel sentinel-user default
+ echo sentinel sentinel-pass K44b706OAB
+ start_redis_sentinel_server
+ echo Starting redis sentinel server...
+ exec redis-server /data/sentinel/redis-sentinel.conf --sentinel
Starting redis sentinel server...
1:X 24 Jun 2024 09:05:36.467 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:X 24 Jun 2024 09:05:36.467 * Redis version=7.2.4, bits=64, commit=00000000, modified=0, pid=1, just started
1:X 24 Jun 2024 09:05:36.467 * Configuration loaded
1:X 24 Jun 2024 09:05:36.468 * monotonic clock: POSIX clock_gettime
1:X 24 Jun 2024 09:05:36.468 * Running mode=sentinel, port=26379.
1:X 24 Jun 2024 09:05:36.469 * Sentinel ID is 247ca38d1391606342d61e01ed372ad0ce812650
1:X 24 Jun 2024 09:05:36.469 # +monitor master redis-szittw-redis redis-szittw-redis-2.redis-szittw-redis-headless.default.svc 6379 quorum 2
1:X 24 Jun 2024 09:05:41.536 # +sdown slave redis-szittw-redis-0.redis-szittw-redis-headless:6379 redis-szittw-redis-0.redis-szittw-redis-headless 6379 @ redis-szittw-redis redis-szittw-redis-2.redis-szittw-redis-headless.default.svc 6379
1:X 24 Jun 2024 09:05:41.536 # +sdown slave redis-szittw-redis-0.redis-szittw-redis-headless.default.svc:6379 redis-szittw-redis-0.redis-szittw-redis-headless.default.svc 6379 @ redis-szittw-redis redis-szittw-redis-2.redis-szittw-redis-headless.default.svc 6379
1:X 24 Jun 2024 09:05:53.400 # +sdown master redis-szittw-redis redis-szittw-redis-2.redis-szittw-redis-headless.default.svc 6379
1:X 24 Jun 2024 09:05:53.515 * Sentinel new configuration saved on disk
1:X 24 Jun 2024 09:05:53.515 # +new-epoch 2
1:X 24 Jun 2024 09:05:53.519 * Sentinel new configuration saved on disk
1:X 24 Jun 2024 09:05:53.519 # +vote-for-leader a7318bcacddb15a7845caa4ad6f963d2ecc581ab 2
1:X 24 Jun 2024 09:05:54.472 # +odown master redis-szittw-redis redis-szittw-redis-2.redis-szittw-redis-headless.default.svc 6379 #quorum 3/2
1:X 24 Jun 2024 09:05:54.472 * Next failover delay: I will not start a failover before Mon Jun 24 09:07:53 2024
1:X 24 Jun 2024 09:05:54.593 # +config-update-from sentinel a7318bcacddb15a7845caa4ad6f963d2ecc581ab redis-szittw-redis-sentinel-2.redis-szittw-redis-sentinel-headless.default.svc 26379 @ redis-szittw-redis redis-szittw-redis-2.redis-szittw-redis-headless.default.svc 6379
1:X 24 Jun 2024 09:05:54.593 # +switch-master redis-szittw-redis redis-szittw-redis-2.redis-szittw-redis-headless.default.svc 6379 redis-szittw-redis-1.redis-szittw-redis-headless.default.svc 6379
1:X 24 Jun 2024 09:05:54.595 * +slave slave redis-szittw-redis-0.redis-szittw-redis-headless:6379 redis-szittw-redis-0.redis-szittw-redis-headless 6379 @ redis-szittw-redis redis-szittw-redis-1.redis-szittw-redis-headless.default.svc 6379
1:X 24 Jun 2024 09:05:54.597 * +slave slave redis-szittw-redis-0.redis-szittw-redis-headless.default.svc:6379 redis-szittw-redis-0.redis-szittw-redis-headless.default.svc 6379 @ redis-szittw-redis redis-szittw-redis-1.redis-szittw-redis-headless.default.svc 6379
1:X 24 Jun 2024 09:05:54.599 * +slave slave redis-szittw-redis-2.redis-szittw-redis-headless.default.svc:6379 redis-szittw-redis-2.redis-szittw-redis-headless.default.svc 6379 @ redis-szittw-redis redis-szittw-redis-1.redis-szittw-redis-headless.default.svc 6379
1:X 24 Jun 2024 09:05:54.603 * Sentinel new configuration saved on disk
1:X 24 Jun 2024 09:06:04.664 * +convert-to-slave slave redis-szittw-redis-2.redis-szittw-redis-headless.default.svc:6379 redis-szittw-redis-2.redis-szittw-redis-headless.default.svc 6379 @ redis-szittw-redis redis-szittw-redis-1.redis-szittw-redis-headless.default.svc 6379
1:X 24 Jun 2024 09:06:54.787 * +fix-slave-config slave redis-szittw-redis-0.redis-szittw-redis-headless:6379 redis-szittw-redis-0.redis-szittw-redis-headless 6379 @ redis-szittw-redis redis-szittw-redis-1.redis-szittw-redis-headless.default.svc 6379
1:X 24 Jun 2024 09:07:55.434 # +sdown slave redis-szittw-redis-2.redis-szittw-redis-headless.default.svc:6379 redis-szittw-redis-2.redis-szittw-redis-headless.default.svc 6379 @ redis-szittw-redis redis-szittw-redis-1.redis-szittw-redis-headless.default.svc 6379
1:X 24 Jun 2024 09:08:06.640 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
1:X 24 Jun 2024 09:08:21.787 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
1:X 24 Jun 2024 09:08:36.911 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
1:X 24 Jun 2024 09:08:52.048 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
1:X 24 Jun 2024 09:09:07.212 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
1:X 24 Jun 2024 09:09:13.613 * Sentinel new configuration saved on disk
1:X 24 Jun 2024 09:09:13.613 # +new-epoch 3
1:X 24 Jun 2024 09:09:13.617 * Sentinel new configuration saved on disk
1:X 24 Jun 2024 09:09:13.617 # +vote-for-leader a7318bcacddb15a7845caa4ad6f963d2ecc581ab 3
1:X 24 Jun 2024 09:09:13.624 # +sdown master redis-szittw-redis redis-szittw-redis-1.redis-szittw-redis-headless.default.svc 6379
1:X 24 Jun 2024 09:09:13.691 # +odown master redis-szittw-redis redis-szittw-redis-1.redis-szittw-redis-headless.default.svc 6379 #quorum 3/2
1:X 24 Jun 2024 09:09:13.691 * Next failover delay: I will not start a failover before Mon Jun 24 09:11:13 2024
1:X 24 Jun 2024 09:09:14.691 # +config-update-from sentinel a7318bcacddb15a7845caa4ad6f963d2ecc581ab redis-szittw-redis-sentinel-2.redis-szittw-redis-sentinel-headless.default.svc 26379 @ redis-szittw-redis redis-szittw-redis-1.redis-szittw-redis-headless.default.svc 6379
1:X 24 Jun 2024 09:09:14.691 # +switch-master redis-szittw-redis redis-szittw-redis-1.redis-szittw-redis-headless.default.svc 6379 redis-szittw-redis-0.redis-szittw-redis-headless 6379
1:X 24 Jun 2024 09:09:14.725 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
1:X 24 Jun 2024 09:09:14.725 * +slave slave redis-szittw-redis-2.redis-szittw-redis-headless.default.svc:6379 redis-szittw-redis-2.redis-szittw-redis-headless.default.svc 6379 @ redis-szittw-redis redis-szittw-redis-0.redis-szittw-redis-headless 6379
1:X 24 Jun 2024 09:09:14.728 * +slave slave redis-szittw-redis-1.redis-szittw-redis-headless.default.svc:6379 redis-szittw-redis-1.redis-szittw-redis-headless.default.svc 6379 @ redis-szittw-redis redis-szittw-redis-0.redis-szittw-redis-headless 6379
1:X 24 Jun 2024 09:09:14.731 * Sentinel new configuration saved on disk
1:X 24 Jun 2024 09:09:14.801 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
1:X 24 Jun 2024 09:09:15.814 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
1:X 24 Jun 2024 09:09:16.850 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
1:X 24 Jun 2024 09:09:17.918 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
1:X 24 Jun 2024 09:09:18.939 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
1:X 24 Jun 2024 09:09:19.735 # +sdown slave redis-szittw-redis-2.redis-szittw-redis-headless.default.svc:6379 redis-szittw-redis-2.redis-szittw-redis-headless.default.svc 6379 @ redis-szittw-redis redis-szittw-redis-0.redis-szittw-redis-headless 6379
1:X 24 Jun 2024 09:09:19.735 # +sdown slave redis-szittw-redis-1.redis-szittw-redis-headless.default.svc:6379 redis-szittw-redis-1.redis-szittw-redis-headless.default.svc 6379 @ redis-szittw-redis redis-szittw-redis-0.redis-szittw-redis-headless 6379
1:X 24 Jun 2024 09:09:19.960 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
1:X 24 Jun 2024 09:09:20.991 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
1:X 24 Jun 2024 09:09:22.066 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
1:X 24 Jun 2024 09:09:23.097 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
1:X 24 Jun 2024 09:09:24.129 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
1:X 24 Jun 2024 09:09:25.186 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
1:X 24 Jun 2024 09:09:26.247 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
1:X 24 Jun 2024 09:09:27.287 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
1:X 24 Jun 2024 09:09:28.297 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
1:X 24 Jun 2024 09:09:29.304 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
1:X 24 Jun 2024 09:09:30.341 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
1:X 24 Jun 2024 09:09:31.355 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
1:X 24 Jun 2024 09:09:32.385 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
1:X 24 Jun 2024 09:09:33.434 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
1:X 24 Jun 2024 09:09:34.525 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
1:X 24 Jun 2024 09:09:35.574 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
1:X 24 Jun 2024 09:09:36.581 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
1:X 24 Jun 2024 09:09:37.598 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
1:X 24 Jun 2024 09:09:38.666 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
1:X 24 Jun 2024 09:09:39.690 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
1:X 24 Jun 2024 09:09:40.774 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
1:X 24 Jun 2024 09:09:41.839 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
1:X 24 Jun 2024 09:09:42.831 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
1:X 24 Jun 2024 09:09:43.847 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
1:X 24 Jun 2024 09:09:44.882 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
1:X 24 Jun 2024 09:09:45.200 # -sdown slave redis-szittw-redis-1.redis-szittw-redis-headless.default.svc:6379 redis-szittw-redis-1.redis-szittw-redis-headless.default.svc 6379 @ redis-szittw-redis redis-szittw-redis-0.redis-szittw-redis-headless 6379
1:X 24 Jun 2024 09:09:45.886 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
1:X 24 Jun 2024 09:09:46.903 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
1:X 24 Jun 2024 09:09:47.959 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
1:X 24 Jun 2024 09:09:49.021 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
1:X 24 Jun 2024 09:09:50.065 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
1:X 24 Jun 2024 09:09:51.069 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
1:X 24 Jun 2024 09:09:52.148 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
1:X 24 Jun 2024 09:09:53.167 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
1:X 24 Jun 2024 09:09:54.200 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
1:X 24 Jun 2024 09:09:55.208 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
1:X 24 Jun 2024 09:09:56.211 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
1:X 24 Jun 2024 09:09:57.232 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
1:X 24 Jun 2024 09:09:58.245 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
1:X 24 Jun 2024 09:09:59.300 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
1:X 24 Jun 2024 09:10:00.322 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
1:X 24 Jun 2024 09:10:01.328 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
1:X 24 Jun 2024 09:10:02.337 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
1:X 24 Jun 2024 09:10:03.374 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
1:X 24 Jun 2024 09:10:04.411 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
1:X 24 Jun 2024 09:10:05.559 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
1:X 24 Jun 2024 09:10:06.510 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
1:X 24 Jun 2024 09:10:07.633 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
1:X 24 Jun 2024 09:10:08.608 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
1:X 24 Jun 2024 09:10:09.641 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
1:X 24 Jun 2024 09:10:10.735 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
1:X 24 Jun 2024 09:10:10.735 # +sdown slave redis-szittw-redis-1.redis-szittw-redis-headless.default.svc:6379 redis-szittw-redis-1.redis-szittw-redis-headless.default.svc 6379 @ redis-szittw-redis redis-szittw-redis-0.redis-szittw-redis-headless 6379
1:X 24 Jun 2024 09:10:11.741 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
1:X 24 Jun 2024 09:10:12.644 # +sdown master redis-szittw-redis redis-szittw-redis-0.redis-szittw-redis-headless 6379
1:X 24 Jun 2024 09:10:12.728 # +odown master redis-szittw-redis redis-szittw-redis-0.redis-szittw-redis-headless 6379 #quorum 2/2
1:X 24 Jun 2024 09:10:12.728 # +new-epoch 4
1:X 24 Jun 2024 09:10:12.728 # +try-failover master redis-szittw-redis redis-szittw-redis-0.redis-szittw-redis-headless 6379
1:X 24 Jun 2024 09:10:12.732 * Sentinel new configuration saved on disk
1:X 24 Jun 2024 09:10:12.732 # +vote-for-leader 247ca38d1391606342d61e01ed372ad0ce812650 4
1:X 24 Jun 2024 09:10:12.746 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
1:X 24 Jun 2024 09:10:12.751 * ce3ed1002fcbb0aba59cdb451f69b609b467f90f voted for 247ca38d1391606342d61e01ed372ad0ce812650 4
1:X 24 Jun 2024 09:10:12.818 # +elected-leader master redis-szittw-redis redis-szittw-redis-0.redis-szittw-redis-headless 6379
1:X 24 Jun 2024 09:10:12.818 # +failover-state-select-slave master redis-szittw-redis redis-szittw-redis-0.redis-szittw-redis-headless 6379
1:X 24 Jun 2024 09:10:12.871 # -failover-abort-no-good-slave master redis-szittw-redis redis-szittw-redis-0.redis-szittw-redis-headless 6379
1:X 24 Jun 2024 09:10:12.962 * Next failover delay: I will not start a failover before Mon Jun 24 09:12:13 2024
1:X 24 Jun 2024 09:10:13.814 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
1:X 24 Jun 2024 09:10:14.858 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
1:X 24 Jun 2024 09:10:15.870 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
1:X 24 Jun 2024 09:10:16.919 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
1:X 24 Jun 2024 09:10:17.985 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
1:X 24 Jun 2024 09:10:19.029 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
1:X 24 Jun 2024 09:10:20.053 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
1:X 24 Jun 2024 09:10:21.115 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
1:X 24 Jun 2024 09:10:22.172 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
1:X 24 Jun 2024 09:10:23.253 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
1:X 24 Jun 2024 09:10:24.286 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
1:X 24 Jun 2024 09:10:25.299 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
1:X 24 Jun 2024 09:10:26.338 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
1:X 24 Jun 2024 09:10:27.375 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
1:X 24 Jun 2024 09:10:28.376 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
1:X 24 Jun 2024 09:10:29.424 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
1:X 24 Jun 2024 09:10:30.469 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
1:X 24 Jun 2024 09:10:31.514 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
1:X 24 Jun 2024 09:10:32.521 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
1:X 24 Jun 2024 09:10:33.554 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
1:X 24 Jun 2024 09:10:34.656 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
1:X 24 Jun 2024 09:10:35.666 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
1:X 24 Jun 2024 09:10:36.664 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
1:X 24 Jun 2024 09:10:37.719 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
1:X 24 Jun 2024 09:10:38.695 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
1:X 24 Jun 2024 09:10:39.722 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
1:X 24 Jun 2024 09:10:40.143 * Sentinel new configuration saved on disk
1:X 24 Jun 2024 09:10:40.14
pod-1
kubectl logs redis-szittw-redis-sentinel-1
+ reset_redis_sentinel_conf
+ echo reset redis sentinel conf
+ sentinel_port=26379
+ [ -n 26379 ]
reset redis sentinel conf
+ sentinel_port=26379
+ mkdir -p /data/sentinel
+ [ -f /data/sentinel/redis-sentinel.conf ]
+ sed -i /sentinel announce-ip/d /data/sentinel/redis-sentinel.conf
+ sed -i /sentinel resolve-hostnames/d /data/sentinel/redis-sentinel.conf
+ sed -i /sentinel announce-hostnames/d /data/sentinel/redis-sentinel.conf
+ [ -n K44b706OAB ]
+ sed -i /sentinel sentinel-user/d /data/sentinel/redis-sentinel.conf
+ sed -i /sentinel sentinel-pass/d /data/sentinel/redis-sentinel.conf
+ sed -i /port 26379/d /data/sentinel/redis-sentinel.conf
+ build_redis_sentinel_conf
+ echo build redis sentinel conf
+ kb_pod_fqdn=redis-szittw-redis-sentinel-1.redis-szittw-redis-sentinel-headless.default.svc
+ echo port 26379
+ echo sentinel announce-ip redis-szittw-redis-sentinel-1.redis-szittw-redis-sentinel-headless.default.svc
+ echo sentinel resolve-hostnames yes
+ echo sentinel announce-hostnames yes
build redis sentinel conf
+ [ -n K44b706OAB ]
+ echo sentinel sentinel-user default
+ echo sentinel sentinel-pass K44b706OAB
+ start_redis_sentinel_server
+ echo Starting redis sentinel server...
+ exec redis-server /data/sentinel/redis-sentinel.conf --sentinel
Starting redis sentinel server...
1:X 24 Jun 2024 09:05:24.117 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:X 24 Jun 2024 09:05:24.117 * Redis version=7.2.4, bits=64, commit=00000000, modified=0, pid=1, just started
1:X 24 Jun 2024 09:05:24.117 * Configuration loaded
1:X 24 Jun 2024 09:05:24.118 * monotonic clock: POSIX clock_gettime
1:X 24 Jun 2024 09:05:24.119 * Running mode=sentinel, port=26379.
1:X 24 Jun 2024 09:05:24.119 * Sentinel ID is ce3ed1002fcbb0aba59cdb451f69b609b467f90f
1:X 24 Jun 2024 09:05:24.119 # +monitor master redis-szittw-redis redis-szittw-redis-2.redis-szittw-redis-headless.default.svc 6379 quorum 2
1:X 24 Jun 2024 09:05:29.085 # +sdown slave redis-szittw-redis-1.redis-szittw-redis-headless.default.svc:6379 redis-szittw-redis-1.redis-szittw-redis-headless.default.svc 6379 @ redis-szittw-redis redis-szittw-redis-2.redis-szittw-redis-headless.default.svc 6379
1:X 24 Jun 2024 09:05:29.168 # +sdown sentinel a7318bcacddb15a7845caa4ad6f963d2ecc581ab redis-szittw-redis-sentinel-2.redis-szittw-redis-sentinel-headless.default.svc 26379 @ redis-szittw-redis redis-szittw-redis-2.redis-szittw-redis-headless.default.svc 6379
1:X 24 Jun 2024 09:05:38.463 # +sdown sentinel 247ca38d1391606342d61e01ed372ad0ce812650 redis-szittw-redis-sentinel-0.redis-szittw-redis-sentinel-headless.default.svc 26379 @ redis-szittw-redis redis-szittw-redis-2.redis-szittw-redis-headless.default.svc 6379
1:X 24 Jun 2024 09:05:39.163 # +sdown slave redis-szittw-redis-0.redis-szittw-redis-headless.default.svc:6379 redis-szittw-redis-0.redis-szittw-redis-headless.default.svc 6379 @ redis-szittw-redis redis-szittw-redis-2.redis-szittw-redis-headless.default.svc 6379
1:X 24 Jun 2024 09:05:39.164 # +sdown slave redis-szittw-redis-0.redis-szittw-redis-headless:6379 redis-szittw-redis-0.redis-szittw-redis-headless 6379 @ redis-szittw-redis redis-szittw-redis-2.redis-szittw-redis-headless.default.svc 6379
1:X 24 Jun 2024 09:05:39.296 # -sdown sentinel a7318bcacddb15a7845caa4ad6f963d2ecc581ab redis-szittw-redis-sentinel-2.redis-szittw-redis-sentinel-headless.default.svc 26379 @ redis-szittw-redis redis-szittw-redis-2.redis-szittw-redis-headless.default.svc 6379
1:X 24 Jun 2024 09:05:40.369 # -sdown slave redis-szittw-redis-1.redis-szittw-redis-headless.default.svc:6379 redis-szittw-redis-1.redis-szittw-redis-headless.default.svc 6379 @ redis-szittw-redis redis-szittw-redis-2.redis-szittw-redis-headless.default.svc 6379
1:X 24 Jun 2024 09:05:49.644 # -sdown sentinel 247ca38d1391606342d61e01ed372ad0ce812650 redis-szittw-redis-sentinel-0.redis-szittw-redis-sentinel-headless.default.svc 26379 @ redis-szittw-redis redis-szittw-redis-2.redis-szittw-redis-headless.default.svc 6379
1:X 24 Jun 2024 09:05:53.477 # +sdown master redis-szittw-redis redis-szittw-redis-2.redis-szittw-redis-headless.default.svc 6379
1:X 24 Jun 2024 09:05:53.515 * Sentinel new configuration saved on disk
1:X 24 Jun 2024 09:05:53.515 # +new-epoch 2
1:X 24 Jun 2024 09:05:53.517 * Sentinel new configuration saved on disk
1:X 24 Jun 2024 09:05:53.517 # +vote-for-leader a7318bcacddb15a7845caa4ad6f963d2ecc581ab 2
1:X 24 Jun 2024 09:05:53.554 # +odown master redis-szittw-redis redis-szittw-redis-2.redis-szittw-redis-headless.default.svc 6379 #quorum 3/2
1:X 24 Jun 2024 09:05:53.554 * Next failover delay: I will not start a failover before Mon Jun 24 09:07:53 2024
1:X 24 Jun 2024 09:05:54.593 # +config-update-from sentinel a7318bcacddb15a7845caa4ad6f963d2ecc581ab redis-szittw-redis-sentinel-2.redis-szittw-redis-sentinel-headless.default.svc 26379 @ redis-szittw-redis redis-szittw-redis-2.redis-szittw-redis-headless.default.svc 6379
1:X 24 Jun 2024 09:05:54.593 # +switch-master redis-szittw-redis redis-szittw-redis-2.redis-szittw-redis-headless.default.svc 6379 redis-szittw-redis-1.redis-szittw-redis-headless.default.svc 6379
1:X 24 Jun 2024 09:05:54.597 * +slave slave redis-szittw-redis-0.redis-szittw-redis-headless.default.svc:6379 redis-szittw-redis-0.redis-szittw-redis-headless.default.svc 6379 @ redis-szittw-redis redis-szittw-redis-1.redis-szittw-redis-headless.default.svc 6379
1:X 24 Jun 2024 09:05:54.598 * +slave slave redis-szittw-redis-0.redis-szittw-redis-headless:6379 redis-szittw-redis-0.redis-szittw-redis-headless 6379 @ redis-szittw-redis redis-szittw-redis-1.redis-szittw-redis-headless.default.svc 6379
1:X 24 Jun 2024 09:05:54.599 * +slave slave redis-szittw-redis-2.redis-szittw-redis-headless.default.svc:6379 redis-szittw-redis-2.redis-szittw-redis-headless.default.svc 6379 @ redis-szittw-redis redis-szittw-redis-1.redis-szittw-redis-headless.default.svc 6379
1:X 24 Jun 2024 09:05:54.601 * Sentinel new configuration saved on disk
1:X 24 Jun 2024 09:05:59.644 # +sdown slave redis-szittw-redis-0.redis-szittw-redis-headless.default.svc:6379 redis-szittw-redis-0.redis-szittw-redis-headless.default.svc 6379 @ redis-szittw-redis redis-szittw-redis-1.redis-szittw-redis-headless.default.svc 6379
1:X 24 Jun 2024 09:05:59.644 # +sdown slave redis-szittw-redis-0.redis-szittw-redis-headless:6379 redis-szittw-redis-0.redis-szittw-redis-headless 6379 @ redis-szittw-redis redis-szittw-redis-1.redis-szittw-redis-headless.default.svc 6379
1:X 24 Jun 2024 09:05:59.645 # +sdown slave redis-szittw-redis-2.redis-szittw-redis-headless.default.svc:6379 redis-szittw-redis-2.redis-szittw-redis-headless.default.svc 6379 @ redis-szittw-redis redis-szittw-redis-1.redis-szittw-redis-headless.default.svc 6379
1:X 24 Jun 2024 09:06:09.813 # -sdown slave redis-szittw-redis-0.redis-szittw-redis-headless.default.svc:6379 redis-szittw-redis-0.redis-szittw-redis-headless.default.svc 6379 @ redis-szittw-redis redis-szittw-redis-1.redis-szittw-redis-headless.default.svc 6379
1:X 24 Jun 2024 09:06:09.813 # -sdown slave redis-szittw-redis-0.redis-szittw-redis-headless:6379 redis-szittw-redis-0.redis-szittw-redis-headless 6379 @ redis-szittw-redis redis-szittw-redis-1.redis-szittw-redis-headless.default.svc 6379
1:X 24 Jun 2024 09:06:09.813 # -sdown slave redis-szittw-redis-2.redis-szittw-redis-headless.default.svc:6379 redis-szittw-redis-2.redis-szittw-redis-headless.default.svc 6379 @ redis-szittw-redis redis-szittw-redis-1.redis-szittw-redis-headless.default.svc 6379
1:X 24 Jun 2024 09:07:55.255 # +sdown slave redis-szittw-redis-2.redis-szittw-redis-headless.default.svc:6379 redis-szittw-redis-2.redis-szittw-redis-headless.default.svc 6379 @ redis-szittw-redis redis-szittw-redis-1.redis-szittw-redis-headless.default.svc 6379
1:X 24 Jun 2024 09:08:06.640 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
1:X 24 Jun 2024 09:08:21.804 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
1:X 24 Jun 2024 09:08:36.992 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
1:X 24 Jun 2024 09:08:52.129 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
1:X 24 Jun 2024 09:09:07.413 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
1:X 24 Jun 2024 09:09:13.544 # +sdown master redis-szittw-redis redis-szittw-redis-1.redis-szittw-redis-headless.default.svc 6379
1:X 24 Jun 2024 09:09:13.611 * Sentinel new configuration saved on disk
1:X 24 Jun 2024 09:09:13.611 # +new-epoch 3
1:X 24 Jun 2024 09:09:13.613 * Sentinel new configuration saved on disk
1:X 24 Jun 2024 09:09:13.613 # +vote-for-leader a7318bcacddb15a7845caa4ad6f963d2ecc581ab 3
1:X 24 Jun 2024 09:09:14.619 # +odown master redis-szittw-redis redis-szittw-redis-1.redis-szittw-redis-headless.default.svc 6379 #quorum 3/2
1:X 24 Jun 2024 09:09:14.619 * Next failover delay: I will not start a failover before Mon Jun 24 09:11:14 2024
1:X 24 Jun 2024 09:09:14.690 # +config-update-from sentinel a7318bcacddb15a7845caa4ad6f963d2ecc581ab redis-szittw-redis-sentinel-2.redis-szittw-redis-sentinel-headless.default.svc 26379 @ redis-szittw-redis redis-szittw-redis-1.redis-szittw-redis-headless.default.svc 6379
1:X 24 Jun 2024 09:09:14.690 # +switch-master redis-szittw-redis redis-szittw-redis-1.redis-szittw-redis-headless.default.svc 6379 redis-szittw-redis-0.redis-szittw-redis-headless 6379
1:X 24 Jun 2024 09:09:14.721 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
1:X 24 Jun 2024 09:09:14.721 * +slave slave redis-szittw-redis-2.redis-szittw-redis-headless.default.svc:6379 redis-szittw-redis-2.redis-szittw-redis-headless.default.svc 6379 @ redis-szittw-redis redis-szittw-redis-0.redis-szittw-redis-headless 6379
1:X 24 Jun 2024 09:09:14.723 * +slave slave redis-szittw-redis-1.redis-szittw-redis-headless.default.svc:6379 redis-szittw-redis-1.redis-szittw-redis-headless.default.svc 6379 @ redis-szittw-redis redis-szittw-redis-0.redis-szittw-redis-headless 6379
1:X 24 Jun 2024 09:09:14.725 * Sentinel new configuration saved on disk
1:X 24 Jun 2024 09:09:14.746 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
1:X 24 Jun 2024 09:09:15.789 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
1:X 24 Jun 2024 09:09:16.829 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
1:X 24 Jun 2024 09:09:17.844 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
1:X 24 Jun 2024 09:09:18.857 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
1:X 24 Jun 2024 09:09:19.784 # +sdown slave redis-szittw-redis-1.redis-szittw-redis-headless.default.svc:6379 redis-szittw-redis-1.redis-szittw-redis-headless.default.svc 6379 @ redis-szittw-redis redis-szittw-redis-0.redis-szittw-redis-headless 6379
1:X 24 Jun 2024 09:09:19.784 # +sdown slave redis-szittw-redis-2.redis-szittw-redis-headless.default.svc:6379 redis-szittw-redis-2.redis-szittw-redis-headless.default.svc 6379 @ redis-szittw-redis redis-szittw-redis-0.redis-szittw-redis-headless 6379
1:X 24 Jun 2024 09:09:19.849 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
1:X 24 Jun 2024 09:09:20.897 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
1:X 24 Jun 2024 09:09:21.942 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
1:X 24 Jun 2024 09:09:22.970 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
1:X 24 Jun 2024 09:09:23.972 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
1:X 24 Jun 2024 09:09:25.034 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
1:X 24 Jun 2024 09:09:26.078 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
1:X 24 Jun 2024 09:09:27.143 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
1:X 24 Jun 2024 09:09:28.143 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
pod-2
kubectl logs redis-szittw-redis-sentinel-2
+ reset_redis_sentinel_conf
+ echo reset redis sentinel conf
+ sentinel_port=26379
+ [ -n 26379 ]
+ sentinel_port=26379
+ mkdir -p /data/sentinel
reset redis sentinel conf
+ [ -f /data/sentinel/redis-sentinel.conf ]
+ sed -i /sentinel announce-ip/d /data/sentinel/redis-sentinel.conf
+ sed -i /sentinel resolve-hostnames/d /data/sentinel/redis-sentinel.conf
+ sed -i /sentinel announce-hostnames/d /data/sentinel/redis-sentinel.conf
+ [ -n K44b706OAB ]
+ sed -i /sentinel sentinel-user/d /data/sentinel/redis-sentinel.conf
+ sed -i /sentinel sentinel-pass/d /data/sentinel/redis-sentinel.conf
+ sed -i /port 26379/d /data/sentinel/redis-sentinel.conf
+ build_redis_sentinel_conf
+ echo build redis sentinel conf
+ kb_pod_fqdn=redis-szittw-redis-sentinel-2.redis-szittw-redis-sentinel-headless.default.svc
+ echo port 26379
build redis sentinel conf
+ echo sentinel announce-ip redis-szittw-redis-sentinel-2.redis-szittw-redis-sentinel-headless.default.svc
+ echo sentinel resolve-hostnames yes
+ echo sentinel announce-hostnames yes
+ [ -n K44b706OAB ]
+ echo sentinel sentinel-user default
+ echo sentinel sentinel-pass K44b706OAB
+ start_redis_sentinel_server
+ echo Starting redis sentinel server...
+ exec redis-server /data/sentinel/redis-sentinel.conf --sentinel
Starting redis sentinel server...
1:X 24 Jun 2024 09:05:06.881 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:X 24 Jun 2024 09:05:06.881 * Redis version=7.2.4, bits=64, commit=00000000, modified=0, pid=1, just started
1:X 24 Jun 2024 09:05:06.881 * Configuration loaded
1:X 24 Jun 2024 09:05:06.882 * monotonic clock: POSIX clock_gettime
1:X 24 Jun 2024 09:05:06.883 * Running mode=sentinel, port=26379.
1:X 24 Jun 2024 09:05:06.883 * Sentinel ID is a7318bcacddb15a7845caa4ad6f963d2ecc581ab
1:X 24 Jun 2024 09:05:06.883 # +monitor master redis-szittw-redis redis-szittw-redis-2.redis-szittw-redis-headless.default.svc 6379 quorum 2
1:X 24 Jun 2024 09:05:11.942 # +sdown slave redis-szittw-redis-1.redis-szittw-redis-headless.default.svc:6379 redis-szittw-redis-1.redis-szittw-redis-headless.default.svc 6379 @ redis-szittw-redis redis-szittw-redis-2.redis-szittw-redis-headless.default.svc 6379
1:X 24 Jun 2024 09:05:26.246 # +sdown sentinel ce3ed1002fcbb0aba59cdb451f69b609b467f90f redis-szittw-redis-sentinel-1.redis-szittw-redis-sentinel-headless.default.svc 26379 @ redis-szittw-redis redis-szittw-redis-2.redis-szittw-redis-headless.default.svc 6379
1:X 24 Jun 2024 09:05:37.515 # -sdown sentinel ce3ed1002fcbb0aba59cdb451f69b609b467f90f redis-szittw-redis-sentinel-1.redis-szittw-redis-sentinel-headless.default.svc 26379 @ redis-szittw-redis redis-szittw-redis-2.redis-szittw-redis-headless.default.svc 6379
1:X 24 Jun 2024 09:05:38.256 # -sdown slave redis-szittw-redis-1.redis-szittw-redis-headless.default.svc:6379 redis-szittw-redis-1.redis-szittw-redis-headless.default.svc 6379 @ redis-szittw-redis redis-szittw-redis-2.redis-szittw-redis-headless.default.svc 6379
1:X 24 Jun 2024 09:05:38.476 # +sdown sentinel 247ca38d1391606342d61e01ed372ad0ce812650 redis-szittw-redis-sentinel-0.redis-szittw-redis-sentinel-headless.default.svc 26379 @ redis-szittw-redis redis-szittw-redis-2.redis-szittw-redis-headless.default.svc 6379
1:X 24 Jun 2024 09:05:39.135 # +sdown slave redis-szittw-redis-0.redis-szittw-redis-headless:6379 redis-szittw-redis-0.redis-szittw-redis-headless 6379 @ redis-szittw-redis redis-szittw-redis-2.redis-szittw-redis-headless.default.svc 6379
1:X 24 Jun 2024 09:05:39.135 # +sdown slave redis-szittw-redis-0.redis-szittw-redis-headless.default.svc:6379 redis-szittw-redis-0.redis-szittw-redis-headless.default.svc 6379 @ redis-szittw-redis redis-szittw-redis-2.redis-szittw-redis-headless.default.svc 6379
1:X 24 Jun 2024 09:05:49.637 # -sdown sentinel 247ca38d1391606342d61e01ed372ad0ce812650 redis-szittw-redis-sentinel-0.redis-szittw-redis-sentinel-headless.default.svc 26379 @ redis-szittw-redis redis-szittw-redis-2.redis-szittw-redis-headless.default.svc 6379
1:X 24 Jun 2024 09:05:53.451 # +sdown master redis-szittw-redis redis-szittw-redis-2.redis-szittw-redis-headless.default.svc 6379
1:X 24 Jun 2024 09:05:53.506 # +odown master redis-szittw-redis redis-szittw-redis-2.redis-szittw-redis-headless.default.svc 6379 #quorum 2/2
1:X 24 Jun 2024 09:05:53.506 # +new-epoch 2
1:X 24 Jun 2024 09:05:53.506 # +try-failover master redis-szittw-redis redis-szittw-redis-2.redis-szittw-redis-headless.default.svc 6379
1:X 24 Jun 2024 09:05:53.510 * Sentinel new configuration saved on disk
1:X 24 Jun 2024 09:05:53.510 # +vote-for-leader a7318bcacddb15a7845caa4ad6f963d2ecc581ab 2
1:X 24 Jun 2024 09:05:53.517 * ce3ed1002fcbb0aba59cdb451f69b609b467f90f voted for a7318bcacddb15a7845caa4ad6f963d2ecc581ab 2
1:X 24 Jun 2024 09:05:53.519 * 247ca38d1391606342d61e01ed372ad0ce812650 voted for a7318bcacddb15a7845caa4ad6f963d2ecc581ab 2
1:X 24 Jun 2024 09:05:53.600 # +elected-leader master redis-szittw-redis redis-szittw-redis-2.redis-szittw-redis-headless.default.svc 6379
1:X 24 Jun 2024 09:05:53.600 # +failover-state-select-slave master redis-szittw-redis redis-szittw-redis-2.redis-szittw-redis-headless.default.svc 6379
1:X 24 Jun 2024 09:05:53.655 # +selected-slave slave redis-szittw-redis-1.redis-szittw-redis-headless.default.svc:6379 redis-szittw-redis-1.redis-szittw-redis-headless.default.svc 6379 @ redis-szittw-redis redis-szittw-redis-2.redis-szittw-redis-headless.default.svc 6379
1:X 24 Jun 2024 09:05:53.655 * +failover-state-send-slaveof-noone slave redis-szittw-redis-1.redis-szittw-redis-headless.default.svc:6379 redis-szittw-redis-1.redis-szittw-redis-headless.default.svc 6379 @ redis-szittw-redis redis-szittw-redis-2.redis-szittw-redis-headless.default.svc 6379
1:X 24 Jun 2024 09:05:53.718 * +failover-state-wait-promotion slave redis-szittw-redis-1.redis-szittw-redis-headless.default.svc:6379 redis-szittw-redis-1.redis-szittw-redis-headless.default.svc 6379 @ redis-szittw-redis redis-szittw-redis-2.redis-szittw-redis-headless.default.svc 6379
1:X 24 Jun 2024 09:05:54.528 * Sentinel new configuration saved on disk
1:X 24 Jun 2024 09:05:54.528 # +promoted-slave slave redis-szittw-redis-1.redis-szittw-redis-headless.default.svc:6379 redis-szittw-redis-1.redis-szittw-redis-headless.default.svc 6379 @ redis-szittw-redis redis-szittw-redis-2.redis-szittw-redis-headless.default.svc 6379
1:X 24 Jun 2024 09:05:54.528 # +failover-state-reconf-slaves master redis-szittw-redis redis-szittw-redis-2.redis-szittw-redis-headless.default.svc 6379
1:X 24 Jun 2024 09:05:54.578 * +slave-reconf-sent slave redis-szittw-redis-0.redis-szittw-redis-headless:6379 redis-szittw-redis-0.redis-szittw-redis-headless 6379 @ redis-szittw-redis redis-szittw-redis-2.redis-szittw-redis-headless.default.svc 6379
1:X 24 Jun 2024 09:05:54.578 # +failover-end master redis-szittw-redis redis-szittw-redis-2.redis-szittw-redis-headless.default.svc 6379
1:X 24 Jun 2024 09:05:54.579 # +switch-master redis-szittw-redis redis-szittw-redis-2.redis-szittw-redis-headless.default.svc 6379 redis-szittw-redis-1.redis-szittw-redis-headless.default.svc 6379
1:X 24 Jun 2024 09:05:54.582 * +slave slave redis-szittw-redis-0.redis-szittw-redis-headless:6379 redis-szittw-redis-0.redis-szittw-redis-headless 6379 @ redis-szittw-redis redis-szittw-redis-1.redis-szittw-redis-headless.default.svc 6379
1:X 24 Jun 2024 09:05:54.585 * +slave slave redis-szittw-redis-0.redis-szittw-redis-headless.default.svc:6379 redis-szittw-redis-0.redis-szittw-redis-headless.default.svc 6379 @ redis-szittw-redis redis-szittw-redis-1.redis-szittw-redis-headless.default.svc 6379
1:X 24 Jun 2024 09:05:54.587 * +slave slave redis-szittw-redis-2.redis-szittw-redis-headless.default.svc:6379 redis-szittw-redis-2.redis-szittw-redis-headless.default.svc 6379 @ redis-szittw-redis redis-szittw-redis-1.redis-szittw-redis-headless.default.svc 6379
1:X 24 Jun 2024 09:05:54.589 * Sentinel new configuration saved on disk
1:X 24 Jun 2024 09:05:59.643 # +sdown slave redis-szittw-redis-0.redis-szittw-redis-headless.default.svc:6379 redis-szittw-redis-0.redis-szittw-redis-headless.default.svc 6379 @ redis-szittw-redis redis-szittw-redis-1.redis-szittw-redis-headless.default.svc 6379
1:X 24 Jun 2024 09:06:09.888 # -sdown slave redis-szittw-redis-0.redis-szittw-redis-headless.default.svc:6379 redis-szittw-redis-0.redis-szittw-redis-headless.default.svc 6379 @ redis-szittw-redis redis-szittw-redis-1.redis-szittw-redis-headless.default.svc 6379
1:X 24 Jun 2024 09:07:55.442 # +sdown slave redis-szittw-redis-2.redis-szittw-redis-headless.default.svc:6379 redis-szittw-redis-2.redis-szittw-redis-headless.default.svc 6379 @ redis-szittw-redis redis-szittw-redis-1.redis-szittw-redis-headless.default.svc 6379
1:X 24 Jun 2024 09:08:06.640 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
1:X 24 Jun 2024 09:08:21.864 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
1:X 24 Jun 2024 09:08:36.992 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
1:X 24 Jun 2024 09:08:52.167 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
1:X 24 Jun 2024 09:09:07.312 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
1:X 24 Jun 2024 09:09:13.549 # +sdown master redis-szittw-redis redis-szittw-redis-1.redis-szittw-redis-headless.default.svc 6379
1:X 24 Jun 2024 09:09:13.602 # +odown master redis-szittw-redis redis-szittw-redis-1.redis-szittw-redis-headless.default.svc 6379 #quorum 2/2
1:X 24 Jun 2024 09:09:13.602 # +new-epoch 3
1:X 24 Jun 2024 09:09:13.602 # +try-failover master redis-szittw-redis redis-szittw-redis-1.redis-szittw-redis-headless.default.svc 6379
1:X 24 Jun 2024 09:09:13.605 * Sentinel new configuration saved on disk
1:X 24 Jun 2024 09:09:13.605 # +vote-for-leader a7318bcacddb15a7845caa4ad6f963d2ecc581ab 3
1:X 24 Jun 2024 09:09:13.613 * ce3ed1002fcbb0aba59cdb451f69b609b467f90f voted for a7318bcacddb15a7845caa4ad6f963d2ecc581ab 3
1:X 24 Jun 2024 09:09:13.617 * 247ca38d1391606342d61e01ed372ad0ce812650 voted for a7318bcacddb15a7845caa4ad6f963d2ecc581ab 3
1:X 24 Jun 2024 09:09:13.672 # +elected-leader master redis-szittw-redis redis-szittw-redis-1.redis-szittw-redis-headless.default.svc 6379
1:X 24 Jun 2024 09:09:13.672 # +failover-state-select-slave master redis-szittw-redis redis-szittw-redis-1.redis-szittw-redis-headless.default.svc 6379
1:X 24 Jun 2024 09:09:13.731 # +selected-slave slave redis-szittw-redis-0.redis-szittw-redis-headless:6379 redis-szittw-redis-0.redis-szittw-redis-headless 6379 @ redis-szittw-redis redis-szittw-redis-1.redis-szittw-redis-headless.default.svc 6379
1:X 24 Jun 2024 09:09:13.731 * +failover-state-send-slaveof-noone slave redis-szittw-redis-0.redis-szittw-redis-headless:6379 redis-szittw-redis-0.redis-szittw-redis-headless 6379 @ redis-szittw-redis redis-szittw-redis-1.redis-szittw-redis-headless.default.svc 6379
1:X 24 Jun 2024 09:09:13.815 * +failover-state-wait-promotion slave redis-szittw-redis-0.redis-szittw-redis-headless:6379 redis-szittw-redis-0.redis-szittw-redis-headless 6379 @ redis-szittw-redis redis-szittw-redis-1.redis-szittw-redis-headless.default.svc 6379
1:X 24 Jun 2024 09:09:14.611 * Sentinel new configuration saved on disk
1:X 24 Jun 2024 09:09:14.611 # +promoted-slave slave redis-szittw-redis-0.redis-szittw-redis-headless:6379 redis-szittw-redis-0.redis-szittw-redis-headless 6379 @ redis-szittw-redis redis-szittw-redis-1.redis-szittw-redis-headless.default.svc 6379
1:X 24 Jun 2024 09:09:14.611 # +failover-state-reconf-slaves master redis-szittw-redis redis-szittw-redis-1.redis-szittw-redis-headless.default.svc 6379
1:X 24 Jun 2024 09:09:14.685 * +slave-reconf-sent slave redis-szittw-redis-2.redis-szittw-redis-headless.default.svc:6379 redis-szittw-redis-2.redis-szittw-redis-headless.default.svc 6379 @ redis-szittw-redis redis-szittw-redis-1.redis-szittw-redis-headless.default.svc 6379
1:X 24 Jun 2024 09:09:15.856 # -odown master redis-szittw-redis redis-szittw-redis-1.redis-szittw-redis-headless.default.svc 6379
1:X 24 Jun 2024 09:09:22.442 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
1:X 24 Jun 2024 09:09:37.633 # Failed to resolve hostname 'redis-szittw-redis-2.redis-szittw-redis-headless.default.svc'
1:X 24 Jun 2024 09:09:40.791 * +reboot master redis-szittw-redis redis-szittw-redis-1.redis-szittw-redis-headless.default.svc 6379
1:X 24 Jun 2024 09:09:40.882 # -sdown master redis-szittw-redis redis-szittw-redis-1.redis-szittw-redis-headless.default.svc 6379
1:X 24 Jun 2024 09:09:41.864 * +convert-to-slave slave redis-szittw-redis-0.redis-szittw-redis-headless.default.svc:6379 redis-szittw-redis-0.redis-szittw-redis-headless.default.svc 6379 @ redis-szittw-redis redis-szittw-redis-1.redis-szittw-redis-headless.default.svc 6379
Describe the bug
To Reproduce Steps to reproduce the behavior:
kubectl get ops NAME TYPE CLUSTER STATUS PROGRESS AGE redis-cluster-reconfiguring-tzvd7 Reconfiguring redis-cluster Succeed 2/2 7m13s
kbcli cluster list-instances redis-cluster NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME us-central1-f 100m / 100m 512Mi / 512Mi data:1Gi gke-infracreate-gke-kbdata-e2-standar-25c8fd47-9yic/10.10.0.70 Jun 24,2024 15:10 UTC+0800 us-central1-f 100m / 100m 512Mi / 512Mi data:1Gi gke-infracreate-gke-kbdata-4c8g-a94cd103-pp75/10.10.0.83 Jun 24,2024 15:10 UTC+0800 us-central1-f 100m / 100m 512Mi / 512Mi data:1Gi gke-infracreate-gke-kbdata-e2-standar-25c8fd47-9yic/10.10.0.70 Jun 24,2024 15:08 UTC+0800 us-central1-f 100m / 100m 512Mi / 512Mi data:1Gi gke-infracreate-gke-kbdata-4c8g-a94cd103-pp75/10.10.0.83 Jun 24,2024 15:08 UTC+0800 us-central1-f 100m / 100m 512Mi / 512Mi data:1Gi gke-infracreate-gke-kbdata-e2-standar-25c8fd47-9yic/10.10.0.70 Jun 24,2024 15:08 UTC+0800
redis-cluster-redis-0 default redis-cluster redis Running primary
redis-cluster-redis-1 default redis-cluster redis Running secondary
redis-cluster-redis-sentinel-0 default redis-cluster redis-sentinel Running
redis-cluster-redis-sentinel-1 default redis-cluster redis-sentinel Running
redis-cluster-redis-sentinel-2 default redis-cluster redis-sentinel Running
kubectl delete pod redis-cluster-redis-0
kbcli cluster list-instances redis-cluster NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME us-central1-f 100m / 100m 512Mi / 512Mi data:1Gi gke-infracreate-gke-kbdata-e2-standar-25c8fd47-9yic/10.10.0.70 Jun 24,2024 15:14 UTC+0800 us-central1-f 100m / 100m 512Mi / 512Mi data:1Gi gke-infracreate-gke-kbdata-4c8g-a94cd103-pp75/10.10.0.83 Jun 24,2024 15:14 UTC+0800 us-central1-f 100m / 100m 512Mi / 512Mi data:1Gi gke-infracreate-gke-kbdata-e2-standar-25c8fd47-9yic/10.10.0.70 Jun 24,2024 15:08 UTC+0800 us-central1-f 100m / 100m 512Mi / 512Mi data:1Gi gke-infracreate-gke-kbdata-4c8g-a94cd103-pp75/10.10.0.83 Jun 24,2024 15:08 UTC+0800 us-central1-f 100m / 100m 512Mi / 512Mi data:1Gi gke-infracreate-gke-kbdata-e2-standar-25c8fd47-9yic/10.10.0.70 Jun 24,2024 15:08 UTC+0800
redis-cluster-redis-0 default redis-cluster redis Running secondary
redis-cluster-redis-1 default redis-cluster redis Running primary
redis-cluster-redis-sentinel-0 default redis-cluster redis-sentinel Running
redis-cluster-redis-sentinel-1 default redis-cluster redis-sentinel Running
redis-cluster-redis-sentinel-2 default redis-cluster redis-sentinel Running
kubectl delete pod redis-cluster-redis-1
kbcli cluster list-instances redis-cluster NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME us-central1-f 100m / 100m 512Mi / 512Mi data:1Gi gke-infracreate-gke-kbdata-e2-standar-25c8fd47-9yic/10.10.0.70 Jun 24,2024 15:14 UTC+0800 us-central1-f 100m / 100m 512Mi / 512Mi data:1Gi gke-infracreate-gke-kbdata-4c8g-a94cd103-pp75/10.10.0.83 Jun 24,2024 15:15 UTC+0800 us-central1-f 100m / 100m 512Mi / 512Mi data:1Gi gke-infracreate-gke-kbdata-e2-standar-25c8fd47-9yic/10.10.0.70 Jun 24,2024 15:08 UTC+0800 us-central1-f 100m / 100m 512Mi / 512Mi data:1Gi gke-infracreate-gke-kbdata-4c8g-a94cd103-pp75/10.10.0.83 Jun 24,2024 15:08 UTC+0800 us-central1-f 100m / 100m 512Mi / 512Mi data:1Gi gke-infracreate-gke-kbdata-e2-standar-25c8fd47-9yic/10.10.0.70 Jun 24,2024 15:08 UTC+0800
redis-cluster-redis-0 default redis-cluster redis Running secondary
redis-cluster-redis-1 default redis-cluster redis Running
redis-cluster-redis-sentinel-0 default redis-cluster redis-sentinel Running
redis-cluster-redis-sentinel-1 default redis-cluster redis-sentinel Running
redis-cluster-redis-sentinel-2 default redis-cluster redis-sentinel Running
kubectl get pod
NAME READY STATUS RESTARTS AGE redis-cluster-redis-0 3/3 Running 0 8m51s redis-cluster-redis-1 2/3 CrashLoopBackOff 5 (81s ago) 7m26s redis-cluster-redis-sentinel-0 1/1 Running 0 14m redis-cluster-redis-sentinel-1 1/1 Running 0 14m redis-cluster-redis-sentinel-2 1/1 Running 0 14m
kubectl logs redis-cluster-redis-0 redis
'
'
'
'
'
'
'
'
kubectl logs redis-cluster-redis-1 redis --previous
'
'
'
'
'
'
'
'
Expected behavior A clear and concise description of what you expected to happen.
Screenshots If applicable, add screenshots to help explain your problem.
Desktop (please complete the following information):
Additional context Add any other context about the problem here.