[BUG]redis pod crash after restart for Duplicate hostname and port for replica

[ahjing99]

➜ ~ kbcli version Kubernetes: v1.27.11-gke.1062004 KubeBlocks: 0.9.0-beta.35 kbcli: 0.9.0-beta.27

1. create cluster

   
   apiVersion: apps.kubeblocks.io/v1alpha1
   kind: Cluster
   metadata:
   name: redis-jzenfb
   namespace: default
   spec:
   terminationPolicy: Halt
   componentSpecs:
   - name: redis
     componentDef: redis-7
     replicas: 2
     resources:
       requests:
         cpu: 100m
         memory: 0.5Gi
       limits:
         cpu: 100m
         memory: 0.5Gi
     switchPolicy:
       type: Noop
     volumeClaimTemplates:
       - name: data
         spec:
           storageClassName:
           accessModes:
             - ReadWriteOnce
           resources:
             requests:
               storage: 1Gi
   - name: redis-sentinel
     componentDef: redis-sentinel-7
     replicas: 3
     resources:
       requests:
         cpu: 100m
         memory: 0.5Gi
       limits:
         cpu: 100m
         memory: 0.5Gi
     volumeClaimTemplates:
       - name: data
         spec:
           storageClassName:
           accessModes:
             - ReadWriteOnce
           resources:
             requests:
               storage: 1Gi
   - name: redis-twemproxy
     componentDef: redis-twemproxy-0.5
     replicas: 3
     resources:
       requests:
         cpu: 100m
         memory: 0.5Gi
       limits:
         cpu: 100m
         memory: 0.5Gi
     volumeClaimTemplates:
       - name: data
         spec:
           storageClassName:
           accessModes:
             - ReadWriteOnce
           resources:
             requests:
               storage: 1Gi
   
     `kubectl apply -f test_create_redis-jzenfb.yaml`

cluster.apps.kubeblocks.io/redis-jzenfb created

2.  Do some ops

kbcli cluster expose redis-jzenfb --auto-approve --force=true --type internet --enable true --components redis --namespace default

OpsRequest redis-jzenfb-expose-ltxpk created successfully, you can view the progress: kbcli cluster describe-ops redis-jzenfb-expose-ltxpk -n default

kbcli cluster configure redis-jzenfb --auto-approve --force=true --set maxclients=10001 --components redis --config-spec redis-replication-config --config-file redis.conf --namespace default

Will updated configure file meta: ConfigSpec: redis-replication-config ConfigFile: redis.conf ComponentName: redis ClusterName: redis-jzenfb OpsRequest redis-jzenfb-reconfiguring-tqxcr created successfully, you can view the progress: kbcli cluster describe-ops redis-jzenfb-reconfiguring-tqxcr -n default

kubectl apply -f benchtest-redis-jzenfb.yaml

pod/benchtest-redis-jzenfb created apply benchtest-redis-jzenfb.yaml Success

kbcli cluster volume-expand redis-jzenfb --auto-approve --force=true --components redis --volume-claim-templates data --storage 3Gi --namespace default

OpsRequest redis-jzenfb-volumeexpansion-mjgx5 created successfully, you can view the progress: kbcli cluster describe-ops redis-jzenfb-volumeexpansion-mjgx5 -n default

3. Stop  then start

kbcli cluster stop redis-jzenfb --auto-approve --force=true --namespace default

OpsRequest redis-jzenfb-stop-2r4pj created successfully, you can view the progress: kbcli cluster describe-ops redis-jzenfb-stop-2r4pj -n default

  `kbcli cluster start redis-jzenfb --force=true --namespace default `

OpsRequest redis-jzenfb-start-wnb7h created successfully, you can view the progress: kbcli cluster describe-ops redis-jzenfb-start-wnb7h -n default

4. Pod crash

➜ ~ k get pod | grep redis redis-jzenfb-redis-0 3/3 Running 0 12m redis-jzenfb-redis-1 2/3 CrashLoopBackOff 6 (106s ago) 12m redis-jzenfb-redis-sentinel-0 0/1 CrashLoopBackOff 7 (38s ago) 11m redis-jzenfb-redis-sentinel-1 1/1 Running 0 12m redis-jzenfb-redis-sentinel-2 1/1 Running 0 12m redis-jzenfb-redis-twemproxy-0 1/1 Running 0 10m redis-jzenfb-redis-twemproxy-1 1/1 Running 0 11m redis-jzenfb-redis-twemproxy-2 1/1 Running 0 12m

➜ ~ k logs redis-jzenfb-redis-sentinel-0

• reset_redis_sentinel_conf
• echo reset redis sentinel conf
• sentinel_port=26379
• [ -n 26379 ] reset redis sentinel conf
• sentinel_port=26379
• mkdir -p /data/sentinel
• [ -f /data/sentinel/redis-sentinel.conf ]
• sed -i /sentinel announce-ip/d /data/sentinel/redis-sentinel.conf
• sed -i /sentinel resolve-hostnames/d /data/sentinel/redis-sentinel.conf
• sed -i /sentinel announce-hostnames/d /data/sentinel/redis-sentinel.conf
• [ -n M15t2GB89c ]
• sed -i /sentinel sentinel-user/d /data/sentinel/redis-sentinel.conf
• sed -i /sentinel sentinel-pass/d /data/sentinel/redis-sentinel.conf
• sed -i /port 26379/d /data/sentinel/redis-sentinel.conf
• build_redis_sentinel_conf
• echo build redis sentinel conf
• kb_pod_fqdn=redis-jzenfb-redis-sentinel-0.redis-jzenfb-redis-sentinel-headless.default.svc
• echo port 26379
• echo sentinel announce-ip redis-jzenfb-redis-sentinel-0.redis-jzenfb-redis-sentinel-headless.default.svc
• echo sentinel resolve-hostnames yes build redis sentinel conf
• echo sentinel announce-hostnames yes
• [ -n M15t2GB89c ]
• echo sentinel sentinel-user default
• echo sentinel sentinel-pass M15t2GB89c Starting redis sentinel server...
• start_redis_sentinel_server
• echo Starting redis sentinel server...
• exec redis-server /data/sentinel/redis-sentinel.conf --sentinel

FATAL CONFIG FILE ERROR (Redis 7.2.4) Reading the configuration file, at line 28

'sentinel known-replica redis-jzenfb-redis redis-jzenfb-redis-1.redis-jzenfb-redis-headless.default.svc 6379' Duplicate hostname and port for replica.

➜ ~ k logs redis-jzenfb-redis-1 Defaulted container "redis" out of: redis, metrics, lorry, init-lorry (init)

• declare -g primary
• declare -g primary_port
• declare -g default_initialize_pod_ordinal
• declare -g headless_postfix=headless
• parse_redis_advertised_svc_if_exist redis-jzenfb-redis-1
• local pod_name=redis-jzenfb-redis-1
• [[ -z '' ]]
• echo 'Environment variable REDIS_ADVERTISED_PORT not found. Ignoring.'
• return 0
• build_redis_conf
• load_redis_template_conf
• echo 'include /etc/conf/redis.conf'
• build_announce_ip_and_port Environment variable REDIS_ADVERTISED_PORT not found. Ignoring.
• '[' -n '' ']'
• kb_pod_fqdn=redis-jzenfb-redis-1.redis-jzenfb-redis-headless.default.svc
• echo 'redis use kb pod fqdn redis-jzenfb-redis-1.redis-jzenfb-redis-headless.default.svc to announce'
• echo 'replica-announce-ip redis-jzenfb-redis-1.redis-jzenfb-redis-headless.default.svc'
• build_redis_service_port
• service_port=6379
• '[' -n 6379 ']'
• service_port=6379
• echo 'port 6379'
• build_replicaof_config
• init_or_get_primary_node redis use kb pod fqdn redis-jzenfb-redis-1.redis-jzenfb-redis-headless.default.svc to announce
• init_or_get_primary_from_redis_sentinel
• '[' -z redis-jzenfb-redis-sentinel ']'
• '[' -z redis-jzenfb-redis-sentinel-0,redis-jzenfb-redis-sentinel-1,redis-jzenfb-redis-sentinel-2 ']'
• '[' -z redis-jzenfb-redis-sentinel-headless ']'
• old_ifs=' '
• IFS=,
• set -f
• read -ra sentinel_pod_list
• set +f
• IFS=' '
• declare -A master_count_map
• local first_redis_primary_host=
• local first_redis_primary_port=
• for sentinel_pod in "${sentinel_pod_list[@]}"
• sentinel_pod_fqdn=redis-jzenfb-redis-sentinel-0.redis-jzenfb-redis-sentinel-headless
• retry_get_master_addr_by_name_from_sentinel redis-jzenfb-redis-sentinel-0.redis-jzenfb-redis-sentinel-headless
• local sentinel_pod_fqdn=redis-jzenfb-redis-sentinel-0.redis-jzenfb-redis-sentinel-headless
• local retry_count=0
• local max_retry=3
• local retry_delay=2
• local timeout_value=5
• '[' 0 -lt 3 ']'
• echo 'execute command: timeout 5 redis-cli -h redis-jzenfb-redis-sentinel-0.redis-jzenfb-redis-sentinel-headless -p 26379 -a M15t2GB89c sentinel get-master-addr-by-name redis-jzenfb-redis' ++ timeout 5 redis-cli -h redis-jzenfb-redis-sentinel-0.redis-jzenfb-redis-sentinel-headless -p 26379 -a M15t2GB89c sentinel get-master-addr-by-name redis-jzenfb-redis execute command: timeout 5 redis-cli -h redis-jzenfb-redis-sentinel-0.redis-jzenfb-redis-sentinel-headless -p 26379 -a M15t2GB89c sentinel get-master-addr-by-name redis-jzenfb-redis Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. Could not connect to Redis at redis-jzenfb-redis-sentinel-0.redis-jzenfb-redis-sentinel-headless:26379: Connection refused
• output=
• exit_code=1
• '[' 1 -eq 0 ']'
• '[' 1 -eq 124 ']'
• echo 'Error occurred while retrieving primary info from sentinel: redis-jzenfb-redis-sentinel-0.redis-jzenfb-redis-sentinel-headless. Retrying...' Error occurred while retrieving primary info from sentinel: redis-jzenfb-redis-sentinel-0.redis-jzenfb-redis-sentinel-headless. Retrying...
• retry_count=1
• sleep 2 execute command: timeout 5 redis-cli -h redis-jzenfb-redis-sentinel-0.redis-jzenfb-redis-sentinel-headless -p 26379 -a M15t2GB89c sentinel get-master-addr-by-name redis-jzenfb-redis
• '[' 1 -lt 3 ']'
• echo 'execute command: timeout 5 redis-cli -h redis-jzenfb-redis-sentinel-0.redis-jzenfb-redis-sentinel-headless -p 26379 -a M15t2GB89c sentinel get-master-addr-by-name redis-jzenfb-redis' ++ timeout 5 redis-cli -h redis-jzenfb-redis-sentinel-0.redis-jzenfb-redis-sentinel-headless -p 26379 -a M15t2GB89c sentinel get-master-addr-by-name redis-jzenfb-redis Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. Could not connect to Redis at redis-jzenfb-redis-sentinel-0.redis-jzenfb-redis-sentinel-headless:26379: Connection refused
• output=
• exit_code=1
• '[' 1 -eq 0 ']'
• '[' 1 -eq 124 ']' Error occurred while retrieving primary info from sentinel: redis-jzenfb-redis-sentinel-0.redis-jzenfb-redis-sentinel-headless. Retrying...
• echo 'Error occurred while retrieving primary info from sentinel: redis-jzenfb-redis-sentinel-0.redis-jzenfb-redis-sentinel-headless. Retrying...'
• retry_count=2
• sleep 2
• '[' 2 -lt 3 ']'
• echo 'execute command: timeout 5 redis-cli -h redis-jzenfb-redis-sentinel-0.redis-jzenfb-redis-sentinel-headless -p 26379 -a M15t2GB89c sentinel get-master-addr-by-name redis-jzenfb-redis' execute command: timeout 5 redis-cli -h redis-jzenfb-redis-sentinel-0.redis-jzenfb-redis-sentinel-headless -p 26379 -a M15t2GB89c sentinel get-master-addr-by-name redis-jzenfb-redis ++ timeout 5 redis-cli -h redis-jzenfb-redis-sentinel-0.redis-jzenfb-redis-sentinel-headless -p 26379 -a M15t2GB89c sentinel get-master-addr-by-name redis-jzenfb-redis Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. Could not connect to Redis at redis-jzenfb-redis-sentinel-0.redis-jzenfb-redis-sentinel-headless:26379: Connection refused
• output=
• exit_code=1
• '[' 1 -eq 0 ']'
• '[' 1 -eq 124 ']'
• echo 'Error occurred while retrieving primary info from sentinel: redis-jzenfb-redis-sentinel-0.redis-jzenfb-redis-sentinel-headless. Retrying...'
• retry_count=3 Error occurred while retrieving primary info from sentinel: redis-jzenfb-redis-sentinel-0.redis-jzenfb-redis-sentinel-headless. Retrying...
• sleep 2 Failed to retrieve primary info from sentinel: redis-jzenfb-redis-sentinel-0.redis-jzenfb-redis-sentinel-headless after 3 retries. Failed to retrieve primary info from sentinel: redis-jzenfb-redis-sentinel-0.redis-jzenfb-redis-sentinel-headless. Skipping this sentinel.
• '[' 3 -lt 3 ']'
• echo 'Failed to retrieve primary info from sentinel: redis-jzenfb-redis-sentinel-0.redis-jzenfb-redis-sentinel-headless after 3 retries.'
• return 1
• echo 'Failed to retrieve primary info from sentinel: redis-jzenfb-redis-sentinel-0.redis-jzenfb-redis-sentinel-headless. Skipping this sentinel.'
• for sentinel_pod in "${sentinel_pod_list[@]}"
• sentinel_pod_fqdn=redis-jzenfb-redis-sentinel-1.redis-jzenfb-redis-sentinel-headless
• retry_get_master_addr_by_name_from_sentinel redis-jzenfb-redis-sentinel-1.redis-jzenfb-redis-sentinel-headless
• local sentinel_pod_fqdn=redis-jzenfb-redis-sentinel-1.redis-jzenfb-redis-sentinel-headless
• local retry_count=0 execute command: timeout 5 redis-cli -h redis-jzenfb-redis-sentinel-1.redis-jzenfb-redis-sentinel-headless -p 26379 -a M15t2GB89c sentinel get-master-addr-by-name redis-jzenfb-redis
• local max_retry=3
• local retry_delay=2
• local timeout_value=5
• '[' 0 -lt 3 ']'
• echo 'execute command: timeout 5 redis-cli -h redis-jzenfb-redis-sentinel-1.redis-jzenfb-redis-sentinel-headless -p 26379 -a M15t2GB89c sentinel get-master-addr-by-name redis-jzenfb-redis' ++ timeout 5 redis-cli -h redis-jzenfb-redis-sentinel-1.redis-jzenfb-redis-sentinel-headless -p 26379 -a M15t2GB89c sentinel get-master-addr-by-name redis-jzenfb-redis Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
• output='redis-jzenfb-redis-0.redis-jzenfb-redis-headless.default.svc 6379'
• exit_code=0
• '[' 0 -eq 0 ']'
• old_ifs=' '
• IFS=' '
• set -f
• read -r -d '' -a REDIS_SENTINEL_PRIMARY_INFO
• set +f
• IFS=' '
• '[' 2 -eq 2 ']'
• '[' -n redis-jzenfb-redis-0.redis-jzenfb-redis-headless.default.svc ']'
• '[' -n 6379 ']' Successfully retrieved primary info from sentinel: redis-jzenfb-redis-sentinel-1.redis-jzenfb-redis-sentinel-headless sentinel:redis-jzenfb-redis-sentinel-1.redis-jzenfb-redis-sentinel-headless has master info: redis-jzenfb-redis-0.redis-jzenfb-redis-headless.default.svc 6379
• echo 'Successfully retrieved primary info from sentinel: redis-jzenfb-redis-sentinel-1.redis-jzenfb-redis-sentinel-headless'
• return 0
• echo 'sentinel:redis-jzenfb-redis-sentinel-1.redis-jzenfb-redis-sentinel-headless has master info: redis-jzenfb-redis-0.redis-jzenfb-redis-headless.default.svc 6379'
• '[' 2 -ne 2 ']'
• '[' -z redis-jzenfb-redis-0.redis-jzenfb-redis-headless.default.svc ']'
• '[' -z 6379 ']'
• host_port_key=redis-jzenfb-redis-0.redis-jzenfb-redis-headless.default.svc:6379
• master_count_map[$host_port_key]=1
• [[ -z '' ]]
• [[ -z '' ]]
• first_redis_primary_host=redis-jzenfb-redis-0.redis-jzenfb-redis-headless.default.svc
• first_redis_primary_port=6379
• '[' redis-jzenfb-redis-0.redis-jzenfb-redis-headless.default.svc '!=' redis-jzenfb-redis-0.redis-jzenfb-redis-headless.default.svc ']'
• '[' 6379 '!=' 6379 ']'
• for sentinel_pod in "${sentinel_pod_list[@]}"
• sentinel_pod_fqdn=redis-jzenfb-redis-sentinel-2.redis-jzenfb-redis-sentinel-headless
• retry_get_master_addr_by_name_from_sentinel redis-jzenfb-redis-sentinel-2.redis-jzenfb-redis-sentinel-headless
• local sentinel_pod_fqdn=redis-jzenfb-redis-sentinel-2.redis-jzenfb-redis-sentinel-headless
• local retry_count=0
• local max_retry=3
• local retry_delay=2
• local timeout_value=5
• '[' 0 -lt 3 ']'
• echo 'execute command: timeout 5 redis-cli -h redis-jzenfb-redis-sentinel-2.redis-jzenfb-redis-sentinel-headless -p 26379 -a M15t2GB89c sentinel get-master-addr-by-name redis-jzenfb-redis' execute command: timeout 5 redis-cli -h redis-jzenfb-redis-sentinel-2.redis-jzenfb-redis-sentinel-headless -p 26379 -a M15t2GB89c sentinel get-master-addr-by-name redis-jzenfb-redis ++ timeout 5 redis-cli -h redis-jzenfb-redis-sentinel-2.redis-jzenfb-redis-sentinel-headless -p 26379 -a M15t2GB89c sentinel get-master-addr-by-name redis-jzenfb-redis Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
• output='redis-jzenfb-redis-0.redis-jzenfb-redis-headless.default.svc 6379'
• exit_code=0
• '[' 0 -eq 0 ']'
• old_ifs=' '
• IFS=' '
• set -f
• read -r -d '' -a REDIS_SENTINEL_PRIMARY_INFO
• set +f
• IFS=' '
• '[' 2 -eq 2 ']'
• '[' -n redis-jzenfb-redis-0.redis-jzenfb-redis-headless.default.svc ']'
• '[' -n 6379 ']'
• echo 'Successfully retrieved primary info from sentinel: redis-jzenfb-redis-sentinel-2.redis-jzenfb-redis-sentinel-headless'
• return 0 Successfully retrieved primary info from sentinel: redis-jzenfb-redis-sentinel-2.redis-jzenfb-redis-sentinel-headless sentinel:redis-jzenfb-redis-sentinel-2.redis-jzenfb-redis-sentinel-headless has master info: redis-jzenfb-redis-0.redis-jzenfb-redis-headless.default.svc 6379
• echo 'sentinel:redis-jzenfb-redis-sentinel-2.redis-jzenfb-redis-sentinel-headless has master info: redis-jzenfb-redis-0.redis-jzenfb-redis-headless.default.svc 6379'
• '[' 2 -ne 2 ']'
• '[' -z redis-jzenfb-redis-0.redis-jzenfb-redis-headless.default.svc ']'
• '[' -z 6379 ']'
• host_port_key=redis-jzenfb-redis-0.redis-jzenfb-redis-headless.default.svc:6379
• master_count_map[$host_port_key]=2
• [[ -z redis-jzenfb-redis-0.redis-jzenfb-redis-headless.default.svc ]]
• '[' redis-jzenfb-redis-0.redis-jzenfb-redis-headless.default.svc '!=' redis-jzenfb-redis-0.redis-jzenfb-redis-headless.default.svc ']'
• '[' 6379 '!=' 6379 ']'
• echo 'get all primary info from redis sentinel master_count_map: 2'
• '[' 1 -eq 0 ']'
• max_count=0
• for host_port in "${!master_count_map[@]}"
• (( 2 > max_count ))
• max_count=2 get all primary info from redis sentinel master_count_map: 2 ++ echo redis-jzenfb-redis-0.redis-jzenfb-redis-headless.default.svc:6379 ++ cut -d: -f1
• primary=redis-jzenfb-redis-0.redis-jzenfb-redis-headless.default.svc ++ echo redis-jzenfb-redis-0.redis-jzenfb-redis-headless.default.svc:6379 ++ cut -d: -f2
• primary_port=6379
• check_current_pod_is_primary
• current_pod=redis-jzenfb-redis-1.redis-jzenfb-redis
• [[ redis-jzenfb-redis-0.redis-jzenfb-redis-headless.default.svc == \r\e\d\i\s-\j\z\e\n\f\b-\r\e\d\i\s-\1.\r\e\d\i\s-\j\z\e\n\f\b-\r\e\d\i\s ]]
• '[' -n '' ']'
• [[ redis-jzenfb-redis-0.redis-jzenfb-redis-headless.default.svc == \1\0.\2\8.\2.\1\8 ]]
• return 1
• '[' -n 1862p3JoRu ']'
• check_kernel_role_cmd='redis-cli -h redis-jzenfb-redis-0.redis-jzenfb-redis-headless.default.svc -p 6379 -a 1862p3JoRu info replication | grep '\''role:'\'' | awk -F: '\''{print $2}'\'''
• retry_times=10
• true ++ eval 'redis-cli -h redis-jzenfb-redis-0.redis-jzenfb-redis-headless.default.svc -p 6379 -a 1862p3JoRu info replication | grep '\''role:'\'' | awk -F: '\''{print $2}'\''' +++ redis-cli -h redis-jzenfb-redis-0.redis-jzenfb-redis-headless.default.svc -p 6379 -a 1862p3JoRu info replication +++ grep role: +++ awk -F: '{print $2}' Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
• check_role=$'slave\r' =~ master ]]
• echo 'the selected primary node is not the real master in kernel, existing primary node: redis-jzenfb-redis-0.redis-jzenfb-redis-headless.default.svc, role: sla'e
• sleep 3 the selected primary node is not the real master in kernel, existing primary node: redis-jzenfb-redis-0.redis-jzenfb-redis-headless.default.svc, role: slave
• retry_times=9
• '[' 9 -eq 0 ']'
• true ++ eval 'redis-cli -h redis-jzenfb-redis-0.redis-jzenfb-redis-headless.default.svc -p 6379 -a 1862p3JoRu info replication | grep '\''role:'\'' | awk -F: '\''{print $2}'\''' +++ redis-cli -h redis-jzenfb-redis-0.redis-jzenfb-redis-headless.default.svc -p 6379 -a 1862p3JoRu info replication +++ grep role: +++ awk -F: '{print $2}' Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
• check_role=$'slave\r' =~ master ]]
• echo 'the selected primary node is not the real master in kernel, existing primary node: redis-jzenfb-redis-0.redis-jzenfb-redis-headless.default.svc, role: sla'e
• sleep 3 the selected primary node is not the real master in kernel, existing primary node: redis-jzenfb-redis-0.redis-jzenfb-redis-headless.default.svc, role: slave
• retry_times=8
• '[' 8 -eq 0 ']'
• true ++ eval 'redis-cli -h redis-jzenfb-redis-0.redis-jzenfb-redis-headless.default.svc -p 6379 -a 1862p3JoRu info replication | grep '\''role:'\'' | awk -F: '\''{print $2}'\''' +++ redis-cli -h redis-jzenfb-redis-0.redis-jzenfb-redis-headless.default.svc -p 6379 -a 1862p3JoRu info replication +++ grep role: +++ awk -F: '{print $2}' Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
• check_role=$'slave\r' =~ master ]] the selected primary node is not the real master in kernel, existing primary node: redis-jzenfb-redis-0.redis-jzenfb-redis-headless.default.svc, role: slave
• echo 'the selected primary node is not the real master in kernel, existing primary node: redis-jzenfb-redis-0.redis-jzenfb-redis-headless.default.svc, role: sla'e
• sleep 3
• retry_times=7
• '[' 7 -eq 0 ']'
• true ++ eval 'redis-cli -h redis-jzenfb-redis-0.redis-jzenfb-redis-headless.default.svc -p 6379 -a 1862p3JoRu info replication | grep '\''role:'\'' | awk -F: '\''{print $2}'\''' +++ redis-cli -h redis-jzenfb-redis-0.redis-jzenfb-redis-headless.default.svc -p 6379 -a 1862p3JoRu info replication +++ grep role: +++ awk -F: '{print $2}' Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
• check_role=$'slave\r' =~ master ]]
• echo 'the selected primary node is not the real master in kernel, existing primary node: redis-jzenfb-redis-0.redis-jzenfb-redis-headless.default.svc, role: sla'e
• sleep 3 the selected primary node is not the real master in kernel, existing primary node: redis-jzenfb-redis-0.redis-jzenfb-redis-headless.default.svc, role: slave
• retry_times=6
• '[' 6 -eq 0 ']'
• true ++ eval 'redis-cli -h redis-jzenfb-redis-0.redis-jzenfb-redis-headless.default.svc -p 6379 -a 1862p3JoRu info replication | grep '\''role:'\'' | awk -F: '\''{print $2}'\''' +++ redis-cli -h redis-jzenfb-redis-0.redis-jzenfb-redis-headless.default.svc -p 6379 -a 1862p3JoRu info replication +++ grep role: +++ awk -F: '{print $2}' Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
• check_role=$'slave\r' =~ master ]] the selected primary node is not the real master in kernel, existing primary node: redis-jzenfb-redis-0.redis-jzenfb-redis-headless.default.svc, role: slave
• echo 'the selected primary node is not the real master in kernel, existing primary node: redis-jzenfb-redis-0.redis-jzenfb-redis-headless.default.svc, role: sla'e
• sleep 3
• retry_times=5
• '[' 5 -eq 0 ']'
• true ++ eval 'redis-cli -h redis-jzenfb-redis-0.redis-jzenfb-redis-headless.default.svc -p 6379 -a 1862p3JoRu info replication | grep '\''role:'\'' | awk -F: '\''{print $2}'\''' +++ grep role: +++ redis-cli -h redis-jzenfb-redis-0.redis-jzenfb-redis-headless.default.svc -p 6379 -a 1862p3JoRu info replication +++ awk -F: '{print $2}' Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
• check_role=$'slave\r' =~ master ]] the selected primary node is not the real master in kernel, existing primary node: redis-jzenfb-redis-0.redis-jzenfb-redis-headless.default.svc, role: slave
• echo 'the selected primary node is not the real master in kernel, existing primary node: redis-jzenfb-redis-0.redis-jzenfb-redis-headless.default.svc, role: sla'e
• sleep 3
• retry_times=4
• '[' 4 -eq 0 ']'
• true ++ eval 'redis-cli -h redis-jzenfb-redis-0.redis-jzenfb-redis-headless.default.svc -p 6379 -a 1862p3JoRu info replication | grep '\''role:'\'' | awk -F: '\''{print $2}'\''' +++ redis-cli -h redis-jzenfb-redis-0.redis-jzenfb-redis-headless.default.svc -p 6379 -a 1862p3JoRu info replication +++ grep role: +++ awk -F: '{print $2}' Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. the selected primary node is not the real master in kernel, existing primary node: redis-jzenfb-redis-0.redis-jzenfb-redis-headless.default.svc, role: slave
• check_role=$'slave\r' =~ master ]]
• echo 'the selected primary node is not the real master in kernel, existing primary node: redis-jzenfb-redis-0.redis-jzenfb-redis-headless.default.svc, role: sla'e
• sleep 3
• retry_times=3
• '[' 3 -eq 0 ']'
• true ++ eval 'redis-cli -h redis-jzenfb-redis-0.redis-jzenfb-redis-headless.default.svc -p 6379 -a 1862p3JoRu info replication | grep '\''role:'\'' | awk -F: '\''{print $2}'\''' +++ redis-cli -h redis-jzenfb-redis-0.redis-jzenfb-redis-headless.default.svc -p 6379 -a 1862p3JoRu info replication +++ grep role: +++ awk -F: '{print $2}' Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
• check_role=$'slave\r' =~ master ]]
• echo 'the selected primary node is not the real master in kernel, existing primary node: redis-jzenfb-redis-0.redis-jzenfb-redis-headless.default.svc, role: sla'e the selected primary node is not the real master in kernel, existing primary node: redis-jzenfb-redis-0.redis-jzenfb-redis-headless.default.svc, role: slave
• sleep 3
• retry_times=2
• '[' 2 -eq 0 ']'
• true ++ eval 'redis-cli -h redis-jzenfb-redis-0.redis-jzenfb-redis-headless.default.svc -p 6379 -a 1862p3JoRu info replication | grep '\''role:'\'' | awk -F: '\''{print $2}'\''' +++ redis-cli -h redis-jzenfb-redis-0.redis-jzenfb-redis-headless.default.svc -p 6379 -a 1862p3JoRu info replication +++ grep role: +++ awk -F: '{print $2}' Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
• check_role=$'slave\r' =~ master ]]
• echo 'the selected primary node is not the real master in kernel, existing primary node: redis-jzenfb-redis-0.redis-jzenfb-redis-headless.default.svc, role: sla'e the selected primary node is not the real master in kernel, existing primary node: redis-jzenfb-redis-0.redis-jzenfb-redis-headless.default.svc, role: slave
• sleep 3
• retry_times=1
• '[' 1 -eq 0 ']'
• true ++ eval 'redis-cli -h redis-jzenfb-redis-0.redis-jzenfb-redis-headless.default.svc -p 6379 -a 1862p3JoRu info replication | grep '\''role:'\'' | awk -F: '\''{print $2}'\''' +++ redis-cli -h redis-jzenfb-redis-0.redis-jzenfb-redis-headless.default.svc -p 6379 -a 1862p3JoRu info replication +++ grep role: +++ awk -F: '{print $2}' Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
• check_role=$'slave\r' =~ master ]] the selected primary node is not the real master in kernel, existing primary node: redis-jzenfb-redis-0.redis-jzenfb-redis-headless.default.svc, role: slave
• echo 'the selected primary node is not the real master in kernel, existing primary node: redis-jzenfb-redis-0.redis-jzenfb-redis-headless.default.svc, role: sla'e
• sleep 3 check primary node role failed after 20 times, existing primary node: redis-jzenfb-redis-0.redis-jzenfb-redis-headless.default.svc, role: slave
• retry_times=0
• '[' 0 -eq 0 ']' ' echo 'check primary node role failed after 20 times, existing primary node: redis-jzenfb-redis-0.redis-jzenfb-redis-headless.default.svc, role: slave
• exit 1

[ahjing99]

lower the severity since cannot reproduce

[Y-Rookie]

this issue might be caused by the simultaneous restart of both the Redis and Redis Sentinel components, leading to some kind of anomaly in the Sentinel. It is recommended to restart each component individually. In future versions, the restart operation will be made sequential.