[BUG]redis pod crash after upgrade kb from 0.7.2 to 0.8

Create cluster

Upgrade kb


➜  ~ kbcli kubeblocks upgrade --version 0.8.0-beta.8
Current KubeBlocks version 0.7.2-beta.28.
Kubernetes version 1.27.3
Kubernetes provider GKE
kbcli version 0.8.0-beta.8
Upgrade KubeBlocks from 0.7.2-beta.28 to 0.8.0-beta.8
Please type 'Yes/yes' to confirm your operation: yes
Add and update repo kubeblocks                     OK
Stop KubeBlocks 0.7.2-beta.28                      OK
Stop DataProtection                                OK
Upgrading KubeBlocks to 0.8.0-beta.8               OK

KubeBlocks has been upgraded to 0.8.0-beta.8 SUCCESSFULLY!

-> Basic commands for cluster: kbcli cluster create -h # help information about creating a database cluster kbcli cluster list # list all database clusters kbcli cluster describe # get cluster information

-> Uninstall KubeBlocks: kbcli kubeblocks uninstall

3. redis pod crash

➜ ~ k logs redis-eolbzo-redis-0 Defaulted container "redis" out of: redis, metrics, kb-checkrole

echo include /etc/conf/redis.conf
echo replica-announce-ip redis-eolbzo-redis-0.redis-eolbzo-redis-headless.default.svc
[ -f /data/users.acl ]
sed -i /user default on/d /data/users.acl
sed -i /user kbreplicator on/d /data/users.acl
sed -i /user kbreplicator-sentinel on/d /data/users.acl
[ ! -z km4bznzr ]
echo masteruser kbreplicator
echo masterauth km4bznzr
echo user kbreplicator on +psync +replconf +ping >km4bznzr
[ ! -z km4bznzr ]
echo user kbreplicator-sentinel on allchannels +multi +slaveof +ping +exec +subscribe +config|rewrite +role +publish +info +client|setname +client|kill +script|kill >km4bznzr
[ ! -z km4bznzr ]
echo protected-mode yes
echo user default on >km4bznzr ~ & +@all
echo aclfile /data/users.acl
start_redis_server
exec redis-server /etc/redis/redis.conf --loadmodule /opt/redis-stack/lib/redisearch.so --loadmodule /opt/redis-stack/lib/redisgraph.so --loadmodule /opt/redis-stack/lib/redistimeseries.so --loadmodule /opt/redis-stack/lib/rejson.so --loadmodule /opt/redis-stack/lib/redisbloom.so
create_replication
[ ! -z km4bznzr ]
retry redis-cli -h 127.0.0.1 -p 6379 -a km4bznzr ping
local max_attempts=20
local attempt=1
redis-cli -h 127.0.0.1 -p 6379 -a km4bznzr ping Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. Could not connect to Redis at 127.0.0.1:6379: Connection refused Command 'redis-cli -h 127.0.0.1 -p 6379 -a km4bznzr ping' failed. Attempt 1 of 20. Retrying in 5 seconds...
[ 1 -eq 20 ]
echo Command 'redis-cli -h 127.0.0.1 -p 6379 -a km4bznzr ping' failed. Attempt 1 of 20. Retrying in 5 seconds...
attempt=2
sleep 3

➜ ~ kbcli cluster describe redis-eolbzo Name: redis-eolbzo Created Time: Dec 18,2023 16:25 UTC+0800 NAMESPACE CLUSTER-DEFINITION VERSION STATUS TERMINATION-POLICY default redis redis-7.0.6 ConditionsError WipeOut

Endpoints: COMPONENT MODE INTERNAL EXTERNAL redis ReadWrite redis-eolbzo-redis.default.svc.cluster.local:6379 redis-sentinel ReadWrite redis-eolbzo-redis-sentinel.default.svc.cluster.local:26379

Topology: COMPONENT INSTANCE ROLE STATUS AZ NODE CREATED-TIME redis redis-eolbzo-redis-0 Running us-central1-c gke-yjtest-default-pool-9095b644-x2l2/10.128.15.192 Dec 18,2023 17:08 UTC+0800 redis redis-eolbzo-redis-1 primary Running us-central1-c gke-yjtest-default-pool-9095b644-mtwm/10.128.15.193 Dec 18,2023 16:27 UTC+0800 redis-sentinel redis-eolbzo-redis-sentinel-0 Running us-central1-c gke-yjtest-default-pool-9095b644-mtwm/10.128.15.193 Dec 18,2023 16:25 UTC+0800 redis-sentinel redis-eolbzo-redis-sentinel-1 Running us-central1-c gke-yjtest-default-pool-9095b644-jv44/10.128.15.196 Dec 18,2023 16:25 UTC+0800 redis-sentinel redis-eolbzo-redis-sentinel-2 Running us-central1-c gke-yjtest-default-pool-9095b644-x2l2/10.128.15.192 Dec 18,2023 16:25 UTC+0800

Resources Allocation: COMPONENT DEDICATED CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE-SIZE STORAGE-CLASS redis false 100m / 100m 512Mi / 512Mi data:1Gi kb-default-sc redis-sentinel false 100m / 100m 512Mi / 512Mi data:1Gi kb-default-sc

Images: COMPONENT TYPE IMAGE redis redis infracreate-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/redis-stack-server:7.0.6-RC8 redis-sentinel redis-sentinel infracreate-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/redis-stack-server:7.0.6-RC8

Data Protection: BACKUP-REPO AUTO-BACKUP BACKUP-SCHEDULE BACKUP-METHOD BACKUP-RETENTION

Show cluster events: kbcli cluster list-events -n default redis-eolbzo

➜ ~ k get cluster redis-eolbzo -o yaml apiVersion: apps.kubeblocks.io/v1alpha1 kind: Cluster metadata: annotations: kubeblocks.io/reconcile: "2023-12-18T09:18:25.127110327Z" creationTimestamp: "2023-12-18T08:25:22Z" finalizers:

cluster.kubeblocks.io/finalizer generation: 1 labels: app.kubernetes.io/instance: redis-eolbzo clusterdefinition.kubeblocks.io/name: redis clusterversion.kubeblocks.io/name: redis-7.0.6 name: redis-eolbzo namespace: default resourceVersion: "318637" uid: 780fefb1-efc6-4953-9304-82b9e863843d spec: affinity: podAntiAffinity: Preferred tenancy: SharedNode clusterDefinitionRef: redis clusterVersionRef: redis-7.0.6 componentSpecs:
componentDefRef: redis enabledLogs:
- running monitor: false name: redis noCreatePDB: false replicas: 2 resources: limits: cpu: 100m memory: 512Mi requests: cpu: 100m memory: 512Mi serviceAccountName: kb-redis-eolbzo switchPolicy: type: Noop volumeClaimTemplates:
- name: data spec: accessModes:
  - ReadWriteOnce resources: requests: storage: 1Gi
componentDefRef: redis-sentinel monitor: false name: redis-sentinel noCreatePDB: false replicas: 3 resources: limits: cpu: 100m memory: 512Mi requests: cpu: 100m memory: 512Mi serviceAccountName: kb-redis-eolbzo volumeClaimTemplates:
- name: data spec: accessModes:
  - ReadWriteOnce resources: requests: storage: 1Gi monitor: {} resources: cpu: "0" memory: "0" storage: size: "0" terminationPolicy: WipeOut status: clusterDefGeneration: 2 components: redis: membersStatus:
  - podName: redis-eolbzo-redis-1 role: accessMode: ReadWrite canVote: true isLeader: true name: primary
  - podName: redis-eolbzo-redis-0 role: accessMode: Readonly canVote: true isLeader: false name: secondary phase: Running podsReady: true podsReadyTime: "2023-12-18T08:30:01Z" replicationSetStatus: primary: pod: redis-eolbzo-redis-1 secondaries:
  - pod: redis-eolbzo-redis-0 redis-sentinel: phase: Running podsReady: true podsReadyTime: "2023-12-18T08:26:49Z" conditions:
lastTransitionTime: "2023-12-18T08:25:24Z" message: 'The operator has started the provisioning of Cluster: redis-eolbzo' observedGeneration: 1 reason: PreCheckSucceed status: "True" type: ProvisioningStarted
lastTransitionTime: "2023-12-18T09:07:55Z" message: Component.apps.kubeblocks.io "redis-eolbzo-redis" not found reason: ApplyResourcesFailed status: "False" type: ApplyResources
lastTransitionTime: "2023-12-18T08:30:01Z" message: all pods of components are ready, waiting for the probe detection successful reason: AllReplicasReady status: "True" type: ReplicasReady

lastTransitionTime: "2023-12-18T08:30:01Z" message: 'Cluster: redis-eolbzo is ready, current phase is Running' reason: ClusterReady status: "True" type: Ready observedGeneration: 1 phase: Running


➜  ~ kbcli report cluster redis-eolbzo --with-logs --all-containers
reporting cluster information to report-cluster-redis-eolbzo-2023-12-18-17-21-59.zip
processing manifests                               OK
processing events                                  OK
process pod logs                                   OK
➜  ~ kbcli report kubeblocks --with-logs --all-containers --output yaml

reporting KubeBlocks information to report-kubeblocks-2023-12-18-17-22-28.zip processing manifests OK processing events OK process pod logs OK report-kubeblocks-2023-12-18-17-22-28.zip report-cluster-redis-eolbzo-2023-12-18-17-21-59.zip

apecloud / kubeblocks

[BUG]redis pod crash after upgrade kb from 0.7.2 to 0.8 #6131