apecloud / kubeblocks

KubeBlocks is an open-source control plane software that runs and manages databases, message queues and other stateful applications on K8s.
https://kubeblocks.io
GNU Affero General Public License v3.0
2.08k stars 170 forks source link

[BUG]redis is always updating after failover #7655

Closed ahjing99 closed 3 months ago

ahjing99 commented 3 months ago

➜ ~ kbcli version Kubernetes: v1.29.4-gke.1043002 KubeBlocks: 0.9.0-beta.39 kbcli: 0.9.0-beta.27

apiVersion: apps.kubeblocks.io/v1alpha1
kind: Cluster
metadata:
  name: redis-lviuoe
  namespace: default
spec:
  terminationPolicy: DoNotTerminate
  componentSpecs:
    - name: redis
      componentDef: redis-7
      replicas: 2
      resources:
        requests:
          cpu: 100m
          memory: 0.5Gi
        limits:
          cpu: 100m
          memory: 0.5Gi
      switchPolicy:
        type: Noop
      volumeClaimTemplates:
        - name: data
          spec:
            storageClassName:
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 1Gi
    - name: redis-sentinel
      componentDef: redis-sentinel-7
      replicas: 3
      resources:
        requests:
          cpu: 100m
          memory: 0.5Gi
        limits:
          cpu: 100m
          memory: 0.5Gi
      volumeClaimTemplates:
        - name: data
          spec:
            storageClassName:
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 1Gi
    - name: redis-twemproxy
      componentDef: redis-twemproxy-0.5
      replicas: 3
      resources:
        requests:
          cpu: 100m
          memory: 0.5Gi
        limits:
          cpu: 100m
          memory: 0.5Gi
      volumeClaimTemplates:
        - name: data
          spec:
            storageClassName:
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 1Gi

      `kubectl apply -f test_create_redis-lviuoe.yaml`

      `kbcli cluster hscale redis-lviuoe --auto-approve --force=true --components redis --replicas 4 --namespace default `

OpsRequest redis-lviuoe-horizontalscaling-nxjvb created successfully, you can view the progress:
    kbcli cluster describe-ops redis-lviuoe-horizontalscaling-nxjvb -n default

      `kbcli cluster list-instances redis-lviuoe --namespace default `

NAME                             NAMESPACE   CLUSTER        COMPONENT         STATUS    ROLE        ACCESSMODE   AZ              CPU(REQUEST/LIMIT)   MEMORY(REQUEST/LIMIT)   STORAGE    NODE                                                CREATED-TIME
redis-lviuoe-redis-0             default     redis-lviuoe   redis             Running   primary     <none>       us-central1-c   100m / 100m          512Mi / 512Mi           data:1Gi   gke-yjtest-default-pool-2619f239-vflk/10.128.0.31   Jun 27,2024 15:54 UTC+0800
redis-lviuoe-redis-1             default     redis-lviuoe   redis             Running   secondary   <none>       us-central1-c   100m / 100m          512Mi / 512Mi           data:1Gi   gke-yjtest-default-pool-2619f239-4tr1/10.128.0.33   Jun 27,2024 15:53 UTC+0800
redis-lviuoe-redis-2             default     redis-lviuoe   redis             Running   secondary   <none>       us-central1-c   100m / 100m          512Mi / 512Mi           data:1Gi   gke-yjtest-default-pool-2619f239-vflk/10.128.0.31   Jun 27,2024 15:56 UTC+0800
redis-lviuoe-redis-3             default     redis-lviuoe   redis             Running   secondary   <none>       us-central1-c   100m / 100m          512Mi / 512Mi           data:1Gi   gke-yjtest-default-pool-2619f239-mmwv/10.128.0.34   Jun 27,2024 15:56 UTC+0800
redis-lviuoe-redis-sentinel-0    default     redis-lviuoe   redis-sentinel    Running   <none>      <none>       us-central1-c   100m / 100m          512Mi / 512Mi           data:1Gi   gke-yjtest-default-pool-2619f239-mmwv/10.128.0.34   Jun 27,2024 15:53 UTC+0800
redis-lviuoe-redis-sentinel-1    default     redis-lviuoe   redis-sentinel    Running   <none>      <none>       us-central1-c   100m / 100m          512Mi / 512Mi           data:1Gi   gke-yjtest-default-pool-2619f239-vflk/10.128.0.31   Jun 27,2024 15:53 UTC+0800
redis-lviuoe-redis-sentinel-2    default     redis-lviuoe   redis-sentinel    Running   <none>      <none>       us-central1-c   100m / 100m          512Mi / 512Mi           data:1Gi   gke-yjtest-default-pool-2619f239-4tr1/10.128.0.33   Jun 27,2024 15:53 UTC+0800
redis-lviuoe-redis-twemproxy-0   default     redis-lviuoe   redis-twemproxy   Running   <none>      <none>       us-central1-c   100m / 100m          512Mi / 512Mi           data:1Gi   gke-yjtest-default-pool-2619f239-4tr1/10.128.0.33   Jun 27,2024 15:53 UTC+0800
redis-lviuoe-redis-twemproxy-1   default     redis-lviuoe   redis-twemproxy   Running   <none>      <none>       us-central1-c   100m / 100m          512Mi / 512Mi           data:1Gi   gke-yjtest-default-pool-2619f239-vflk/10.128.0.31   Jun 27,2024 15:53 UTC+0800
redis-lviuoe-redis-twemproxy-2   default     redis-lviuoe   redis-twemproxy   Running   <none>      <none>       us-central1-c   100m / 100m          512Mi / 512Mi           data:1Gi   gke-yjtest-default-pool-2619f239-mmwv/10.128.0.34   Jun 27,2024 15:53 UTC+0800

 `kubectl delete pod redis-lviuoe-redis-0  --namespace default `

pod "redis-lviuoe-redis-0" deleted

➜  ~ k logs kb-post-provision-job-redis-lviuoe-redis-l8smz
+ declare -g default_initialize_pod_ordinal
+ declare -g redis_advertised_svc_host_value
+ declare -g redis_advertised_svc_port_value
+ declare -g headless_postfix=headless
+ declare -g redis_default_service_port=6379
redis sentinel component replicas found, register to sentinel.
+ echo 'redis sentinel component replicas found, register to sentinel.'
+ register_to_sentinel_wrapper
+ '[' -z redis-lviuoe-redis-sentinel-0,redis-lviuoe-redis-sentinel-1,redis-lviuoe-redis-sentinel-2 ']'
+ '[' -z redis-lviuoe-redis-sentinel-headless ']'
+ get_minimum_initialize_pod_ordinal
+ '[' -z redis-lviuoe-redis-0,redis-lviuoe-redis-1,redis-lviuoe-redis-2,redis-lviuoe-redis-3 ']'
+ IFS=,
+ read -ra pod_list
+ for pod in "${pod_list[@]}"
+ '[' -z '' ']'
++ extract_ordinal_from_object_name redis-lviuoe-redis-0
++ local object_name=redis-lviuoe-redis-0
++ local ordinal=0
++ echo 0
+ default_initialize_pod_ordinal=0
+ continue
+ for pod in "${pod_list[@]}"
+ '[' -z 0 ']'
++ extract_ordinal_from_object_name redis-lviuoe-redis-1
++ local object_name=redis-lviuoe-redis-1
++ local ordinal=1
++ echo 1
+ pod_ordinal=1
+ '[' 1 -lt 0 ']'
+ for pod in "${pod_list[@]}"
+ '[' -z 0 ']'
++ extract_ordinal_from_object_name redis-lviuoe-redis-2
++ local object_name=redis-lviuoe-redis-2
++ local ordinal=2
++ echo 2
+ pod_ordinal=2
+ '[' 2 -lt 0 ']'
+ for pod in "${pod_list[@]}"
+ '[' -z 0 ']'
++ extract_ordinal_from_object_name redis-lviuoe-redis-3
++ local object_name=redis-lviuoe-redis-3
++ local ordinal=3
++ echo 3
+ pod_ordinal=3
+ '[' 3 -lt 0 ']'
+ default_redis_primary_pod_name=redis-lviuoe-redis-0
+ redis_default_primary_pod_headless_fqdn=redis-lviuoe-redis-0.redis-lviuoe-redis-headless.default.svc
Environment variable REDIS_ADVERTISED_PORT not found. Ignoring.
register to sentinel:redis-lviuoe-redis-sentinel-0.redis-lviuoe-redis-sentinel-headless with ClusterIP service: redis_default_primary_pod_fqdn=redis-lviuoe-redis-0.redis-lviuoe-redis-headless.default.svc, redis_default_service_port=6379
+ init_redis_service_port
+ '[' -n 6379 ']'
+ redis_default_service_port=6379
+ parse_redis_advertised_svc_if_exist redis-lviuoe-redis-0
+ local pod_name=redis-lviuoe-redis-0
+ [[ -z '' ]]
+ echo 'Environment variable REDIS_ADVERTISED_PORT not found. Ignoring.'
+ return 0
+ old_ifs='
'
+ IFS=,
+ set -f
+ read -ra sentinel_pod_list
+ set +f
+ IFS='
'
+ for sentinel_pod in "${sentinel_pod_list[@]}"
+ sentinel_pod_fqdn=redis-lviuoe-redis-sentinel-0.redis-lviuoe-redis-sentinel-headless
+ '[' -n '' ']'
+ echo 'register to sentinel:redis-lviuoe-redis-sentinel-0.redis-lviuoe-redis-sentinel-headless with ClusterIP service: redis_default_primary_pod_fqdn=redis-lviuoe-redis-0.redis-lviuoe-redis-headless.default.svc, redis_default_service_port=6379'
+ register_to_sentinel redis-lviuoe-redis-sentinel-0.redis-lviuoe-redis-sentinel-headless redis-lviuoe-redis redis-lviuoe-redis-0.redis-lviuoe-redis-headless.default.svc 6379
+ local sentinel_host=redis-lviuoe-redis-sentinel-0.redis-lviuoe-redis-sentinel-headless
+ local master_name=redis-lviuoe-redis
+ local sentinel_port=26379
+ local redis_primary_host=redis-lviuoe-redis-0.redis-lviuoe-redis-headless.default.svc
+ local redis_primary_port=6379
+ local timeout=600
++ date +%s
+ local start_time=1719475030
+ local current_time
+ wait_for_connectivity redis-lviuoe-redis-sentinel-0.redis-lviuoe-redis-sentinel-headless 26379 y86F6Bs8W3
+ local host=redis-lviuoe-redis-sentinel-0.redis-lviuoe-redis-sentinel-headless
+ local port=26379
+ local password=y86F6Bs8W3
+ echo 'Checking connectivity to redis-lviuoe-redis-sentinel-0.redis-lviuoe-redis-sentinel-headless on port 26379 using redis-cli...'
+ true
Checking connectivity to redis-lviuoe-redis-sentinel-0.redis-lviuoe-redis-sentinel-headless on port 26379 using redis-cli...
++ date +%s
+ current_time=1719475030
+ '[' 0 -gt 600 ']'
+ redis-cli -h redis-lviuoe-redis-sentinel-0.redis-lviuoe-redis-sentinel-headless -p 26379 -a y86F6Bs8W3 PING
+ grep -q PONG
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
redis-lviuoe-redis-sentinel-0.redis-lviuoe-redis-sentinel-headless is reachable on port 26379.
+ echo 'redis-lviuoe-redis-sentinel-0.redis-lviuoe-redis-sentinel-headless is reachable on port 26379.'
+ break
+ wait_for_connectivity redis-lviuoe-redis-0.redis-lviuoe-redis-headless.default.svc 6379 Eo0y130PK5
+ local host=redis-lviuoe-redis-0.redis-lviuoe-redis-headless.default.svc
+ local port=6379
+ local password=Eo0y130PK5
Checking connectivity to redis-lviuoe-redis-0.redis-lviuoe-redis-headless.default.svc on port 6379 using redis-cli...
+ echo 'Checking connectivity to redis-lviuoe-redis-0.redis-lviuoe-redis-headless.default.svc on port 6379 using redis-cli...'
+ true
++ date +%s
+ current_time=1719475030
+ '[' 0 -gt 600 ']'
+ redis-cli -h redis-lviuoe-redis-0.redis-lviuoe-redis-headless.default.svc -p 6379 -a Eo0y130PK5 PING
+ grep -q PONG
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
+ echo 'redis-lviuoe-redis-0.redis-lviuoe-redis-headless.default.svc is reachable on port 6379.'
redis-lviuoe-redis-0.redis-lviuoe-redis-headless.default.svc is reachable on port 6379.
+ break
+ execute_redis_cli SENTINEL monitor redis-lviuoe-redis redis-lviuoe-redis-0.redis-lviuoe-redis-headless.default.svc 6379 2
+ echo 'Executing: redis-cli -h redis-lviuoe-redis-sentinel-0.redis-lviuoe-redis-sentinel-headless -p 26379 -a y86F6Bs8W3 SENTINEL monitor redis-lviuoe-redis redis-lviuoe-redis-0.redis-lviuoe-redis-headless.default.svc 6379 2'
+ local output
Executing: redis-cli -h redis-lviuoe-redis-sentinel-0.redis-lviuoe-redis-sentinel-headless -p 26379 -a y86F6Bs8W3 SENTINEL monitor redis-lviuoe-redis redis-lviuoe-redis-0.redis-lviuoe-redis-headless.default.svc 6379 2
++ redis-cli -h redis-lviuoe-redis-sentinel-0.redis-lviuoe-redis-sentinel-headless -p 26379 -a y86F6Bs8W3 SENTINEL monitor redis-lviuoe-redis redis-lviuoe-redis-0.redis-lviuoe-redis-headless.default.svc 6379 2
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
ERR Duplicate master name.
Command failed with status 0 or output not OK.
+ output='ERR Duplicate master name.'
+ local status=0
+ echo 'ERR Duplicate master name.'
+ '[' 0 -ne 0 ']'
+ '[' 'ERR Duplicate master name.' '!=' OK ']'
+ echo 'Command failed with status 0 or output not OK.'
+ exit 1