Closed ahjing99 closed 4 months ago
After we successfully created the redis cluster, we executed an sql to connect to the database with the same problem and simply disconnected the connection. We found that the cluster status is Updating and the pod status CrashLoopBackOff
---
# Source: redis-cluster/templates/rbac.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: kb-redis-cluster
namespace: default
labels:
helm.sh/chart: redis-cluster-0.7.0-alpha.0
app.kubernetes.io/version: "7.0.6"
app.kubernetes.io/instance: redis-cluster
---
# Source: redis-cluster/templates/rbac.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: kb-redis-cluster
labels:
helm.sh/chart: redis-cluster-0.7.0-alpha.0
app.kubernetes.io/version: "7.0.6"
app.kubernetes.io/instance: redis-cluster
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: kubeblocks-volume-protection-pod-role
subjects:
- kind: ServiceAccount
name: kb-redis-cluster
namespace: default
---
# Source: redis-cluster/templates/rbac.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: kb-redis-cluster
labels:
helm.sh/chart: redis-cluster-0.7.0-alpha.0
app.kubernetes.io/version: "7.0.6"
app.kubernetes.io/instance: redis-cluster
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: kubeblocks-cluster-pod-role
subjects:
- kind: ServiceAccount
name: kb-redis-cluster
namespace: default
---
# Source: redis-cluster/templates/cluster.yaml
apiVersion: apps.kubeblocks.io/v1alpha1
kind: Cluster
metadata:
name: redis-cluster
namespace: default
labels:
helm.sh/chart: redis-cluster-0.7.0-alpha.0
app.kubernetes.io/version: "7.0.6"
app.kubernetes.io/instance: redis-cluster
spec:
clusterVersionRef: redis-7.0.6
terminationPolicy: Delete
affinity:
podAntiAffinity: Preferred
topologyKeys:
- kubernetes.io/hostname
tenancy: SharedNode
clusterDefinitionRef: redis # ref clusterDefinition.name
componentSpecs:
- name: redis
componentDefRef: redis # ref clusterDefinition componentDefs.name
monitor: false
replicas: 1
enabledLogs:
- running
serviceAccountName: kb-redis-cluster
switchPolicy:
type: Noop
resources:
limits:
cpu: "0.5"
memory: "0.5Gi"
requests:
cpu: "0.5"
memory: "0.5Gi"
volumeClaimTemplates:
- name: data # ref clusterDefinition components.containers.volumeMounts.name
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 20Gi
services:
kbcli cluster connect redis-cluster
Connect to instance redis-cluster-redis-0
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
[127.0.0.1:6379](http://127.0.0.1:6379/)> config get rdbcompression
1) "rdbcompression"
2) "yes"
[127.0.0.1:6379](http://127.0.0.1:6379/)> command terminated with exit code 137
➜ dbass git:(main) ✗ k get cluster
NAME CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS AGE
mysql-cluster apecloud-mysql ac-mysql-8.0.30 Delete Running 59m
redis-cluster redis redis-7.0.6 Delete Updating 2m3s
➜ dbass git:(main) ✗ k get pod
NAME READY STATUS RESTARTS AGE
mysql-cluster-mysql-0 8/8 Running 0 59m
redis-cluster-redis-0 5/5 Running 1 (13s ago) 2m9s
➜ dbass git:(main) ✗ k get cluster
NAME CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS AGE
mysql-cluster apecloud-mysql ac-mysql-8.0.30 Delete Running 61m
redis-cluster redis redis-7.0.6 Delete Updating 3m41s
➜ dbass git:(main) ✗ k get pod
NAME READY STATUS RESTARTS AGE
mysql-cluster-mysql-0 8/8 Running 0 61m
redis-cluster-redis-0 4/5 CrashLoopBackOff 1 (9s ago) 3m45s
k describe pod redis-cluster-redis-0
Name: redis-cluster-redis-0
Namespace: default
Priority: 0
Service Account: kb-redis-cluster
Node: ip-10-0-3-15.us-west-2.compute.internal/10.0.3.15
Start Time: Mon, 25 Sep 2023 15:49:35 +0800
Labels: app.kubernetes.io/component=redis
app.kubernetes.io/instance=redis-cluster
app.kubernetes.io/managed-by=kubeblocks
app.kubernetes.io/name=redis
app.kubernetes.io/version=redis-7.0.6
apps.kubeblocks.io/component-name=redis
apps.kubeblocks.io/workload-type=Replication
controller-revision-hash=redis-cluster-redis-765b949b
kubeblocks.io/role=primary
rsm.workloads.kubeblocks.io/access-mode=ReadWrite
statefulset.kubernetes.io/pod-name=redis-cluster-redis-0
Annotations: apps.kubeblocks.io/component-replicas: 1
apps.kubeblocks.io/last-role-snapshot-version: 2023-09-25T07:55:42.899242Z
Status: Running
IP: 10.0.3.36
IPs:
IP: 10.0.3.36
Controlled By: StatefulSet/redis-cluster-redis
Init Containers:
role-agent-installer:
Container ID: containerd://b90d16bf26fc9dfe171c45daba61b2b440c4e3000e9033496828da50763c8303
Image: msoap/shell2http:1.16.0
Image ID: docker.io/msoap/shell2http@sha256:a20bdde2f679de2cba6bf3d9f470489c7836d4d0d28232a2b295450809cd43ef
Port: <none>
Host Port: <none>
Command:
cp
/app/shell2http
/role-probe/agent
State: Terminated
Reason: Completed
Exit Code: 0
Started: Mon, 25 Sep 2023 15:49:43 +0800
Finished: Mon, 25 Sep 2023 15:49:43 +0800
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/role-probe from role-agent (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-29l7t (ro)
Containers:
redis:
Container ID: containerd://765d51960d9dc58aca819019a0257895864c850fd76bed995c7399799fe3ff47
Image: registry.cn-hangzhou.aliyuncs.com/apecloud/redis-stack-server:7.0.6-RC8
Image ID: registry.cn-hangzhou.aliyuncs.com/apecloud/redis-stack-server@sha256:511808b267ab8d800283604ef5c01f4fe94792bfb746bb6dba236cc29ff5495b
Port: 6379/TCP
Host Port: 0/TCP
Command:
/scripts/redis-start.sh
State: Running
Started: Mon, 25 Sep 2023 15:55:41 +0800
Last State: Terminated
Reason: Completed
Exit Code: 0
Started: Mon, 25 Sep 2023 15:53:25 +0800
Finished: Mon, 25 Sep 2023 15:55:08 +0800
Ready: True
Restart Count: 3
Limits:
cpu: 500m
memory: 512Mi
Requests:
cpu: 500m
memory: 512Mi
Readiness: exec [sh -c /scripts/redis-ping.sh 1] delay=10s timeout=1s period=5s #success=1 #failure=5
Environment Variables from:
redis-cluster-redis-env ConfigMap Optional: false
redis-cluster-redis-rsm-env ConfigMap Optional: false
Environment:
KB_POD_NAME: redis-cluster-redis-0 (v1:metadata.name)
KB_POD_UID: (v1:metadata.uid)
KB_NAMESPACE: default (v1:metadata.namespace)
KB_SA_NAME: (v1:spec.serviceAccountName)
KB_NODENAME: (v1:spec.nodeName)
KB_HOST_IP: (v1:status.hostIP)
KB_POD_IP: (v1:status.podIP)
KB_POD_IPS: (v1:status.podIPs)
KB_HOSTIP: (v1:status.hostIP)
KB_PODIP: (v1:status.podIP)
KB_PODIPS: (v1:status.podIPs)
KB_CLUSTER_NAME: redis-cluster
KB_COMP_NAME: redis
KB_CLUSTER_COMP_NAME: redis-cluster-redis
KB_CLUSTER_UID_POSTFIX_8: 632eba18
KB_POD_FQDN: $(KB_POD_NAME).$(KB_CLUSTER_COMP_NAME)-headless.$(KB_NAMESPACE).svc
REDIS_REPL_USER: kbreplicator
REDIS_REPL_PASSWORD: <set to the key 'password' in secret 'redis-cluster-conn-credential'> Optional: false
REDIS_DEFAULT_PASSWORD: <set to the key 'password' in secret 'redis-cluster-conn-credential'> Optional: false
REDIS_SENTINEL_USER: $(REDIS_REPL_USER)-sentinel
REDIS_SENTINEL_PASSWORD: <set to the key 'password' in secret 'redis-cluster-conn-credential'> Optional: false
REDIS_ARGS: --requirepass $(REDIS_PASSWORD)
Mounts:
/data from data (rw)
/etc/conf from redis-config (rw)
/etc/redis from redis-conf (rw)
/kb-podinfo from pod-info (rw)
/scripts from scripts (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-29l7t (ro)
metrics:
Container ID: containerd://a9cf81c8ba177a6f3171fa4e20c691982ba72077a0603e654932e6df09d3ad79
Image: registry.cn-hangzhou.aliyuncs.com/apecloud/agamotto:0.1.2-beta.1
Image ID: registry.cn-hangzhou.aliyuncs.com/apecloud/agamotto@sha256:cbab349b90490807a8d5039bf01bc7e37334f20c98c7dd75bc7fc4cf9e5b10ee
Port: 9121/TCP
Host Port: 0/TCP
Command:
/bin/agamotto
--config=/opt/conf/metrics-config.yaml
State: Running
Started: Mon, 25 Sep 2023 15:49:44 +0800
Ready: True
Restart Count: 0
Limits:
cpu: 0
memory: 0
Requests:
cpu: 0
memory: 0
Environment Variables from:
redis-cluster-redis-env ConfigMap Optional: false
redis-cluster-redis-rsm-env ConfigMap Optional: false
Environment:
KB_POD_NAME: redis-cluster-redis-0 (v1:metadata.name)
KB_POD_UID: (v1:metadata.uid)
KB_NAMESPACE: default (v1:metadata.namespace)
KB_SA_NAME: (v1:spec.serviceAccountName)
KB_NODENAME: (v1:spec.nodeName)
KB_HOST_IP: (v1:status.hostIP)
KB_POD_IP: (v1:status.podIP)
KB_POD_IPS: (v1:status.podIPs)
KB_HOSTIP: (v1:status.hostIP)
KB_PODIP: (v1:status.podIP)
KB_PODIPS: (v1:status.podIPs)
KB_CLUSTER_NAME: redis-cluster
KB_COMP_NAME: redis
KB_CLUSTER_COMP_NAME: redis-cluster-redis
KB_CLUSTER_UID_POSTFIX_8: 632eba18
KB_POD_FQDN: $(KB_POD_NAME).$(KB_CLUSTER_COMP_NAME)-headless.$(KB_NAMESPACE).svc
ENDPOINT: localhost:6379
REDIS_USER: <set to the key 'username' in secret 'redis-cluster-conn-credential'> Optional: false
REDIS_PASSWORD: <set to the key 'password' in secret 'redis-cluster-conn-credential'> Optional: false
Mounts:
/opt/conf from redis-metrics-config (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-29l7t (ro)
kb-we-syncer:
Container ID: containerd://ec89909de439a018bbf973f46619b078f1de32768b771e20bba92351587d7a43
Image: registry.cn-hangzhou.aliyuncs.com/apecloud/kubeblocks-tools:0.7.0-alpha.16
Image ID: registry.cn-hangzhou.aliyuncs.com/apecloud/kubeblocks-tools@sha256:19ba1eb3d920a0c6e90aee6b5a54c544663c27889e5d29a79bc80e6656030297
Ports: 3501/TCP, 50001/TCP
Host Ports: 0/TCP, 0/TCP
Command:
lorry
--port
3501
State: Running
Started: Mon, 25 Sep 2023 15:49:44 +0800
Ready: True
Restart Count: 0
Limits:
cpu: 0
memory: 0
Requests:
cpu: 0
memory: 0
Startup: tcp-socket :3501 delay=0s timeout=1s period=10s #success=1 #failure=3
Environment Variables from:
redis-cluster-redis-env ConfigMap Optional: false
redis-cluster-redis-rsm-env ConfigMap Optional: false
Environment:
KB_POD_NAME: redis-cluster-redis-0 (v1:metadata.name)
KB_POD_UID: (v1:metadata.uid)
KB_NAMESPACE: default (v1:metadata.namespace)
KB_SA_NAME: (v1:spec.serviceAccountName)
KB_NODENAME: (v1:spec.nodeName)
KB_HOST_IP: (v1:status.hostIP)
KB_POD_IP: (v1:status.podIP)
KB_POD_IPS: (v1:status.podIPs)
KB_HOSTIP: (v1:status.hostIP)
KB_PODIP: (v1:status.podIP)
KB_PODIPS: (v1:status.podIPs)
KB_CLUSTER_NAME: redis-cluster
KB_COMP_NAME: redis
KB_CLUSTER_COMP_NAME: redis-cluster-redis
KB_CLUSTER_UID_POSTFIX_8: 632eba18
KB_POD_FQDN: $(KB_POD_NAME).$(KB_CLUSTER_COMP_NAME)-headless.$(KB_NAMESPACE).svc
KB_SERVICE_USER: <set to the key 'username' in secret 'redis-cluster-conn-credential'> Optional: false
KB_SERVICE_PASSWORD: <set to the key 'password' in secret 'redis-cluster-conn-credential'> Optional: false
KB_SERVICE_PORT: 6379
KB_DATA_PATH: /data
KB_SERVICE_CHARACTER_TYPE: redis
KB_WORKLOAD_TYPE: Replication
Mounts:
/data from data (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-29l7t (ro)
action-0:
Container ID: containerd://ea8e8d886de6697a50f3ebb26819c17220d8b98fe364cf6e2f2a5f31c55ad70a
Image: registry.cn-hangzhou.aliyuncs.com/apecloud/redis-stack-server:7.0.6-RC8
Image ID: registry.cn-hangzhou.aliyuncs.com/apecloud/redis-stack-server@sha256:511808b267ab8d800283604ef5c01f4fe94792bfb746bb6dba236cc29ff5495b
Port: <none>
Host Port: <none>
Command:
/role-probe/agent
-port
36501
-export-all-vars
-form
/role
Role=$(redis-cli --user $KB_RSM_USERNAME --pass $KB_RSM_PASSWORD --no-auth-warning info | grep role | awk -F ':' '{print $2}' | tr '[:upper:]' '[:lower:]' | tr -d '\r' | tr -d '
') && if [ "master" = "$Role" ]; then echo -n "primary"; else echo -n "secondary"; fi
State: Running
Started: Mon, 25 Sep 2023 15:49:44 +0800
Ready: True
Restart Count: 0
Environment:
KB_RSM_USERNAME: <set to the key 'username' in secret 'redis-cluster-conn-credential'> Optional: false
KB_RSM_PASSWORD: <set to the key 'password' in secret 'redis-cluster-conn-credential'> Optional: false
Mounts:
/role-probe from role-agent (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-29l7t (ro)
kb-role-probe:
Container ID: containerd://7900dac506d8124a33a9e1b6f892288279663bf5f179a5dc6b8fa133a3d6df5c
Image: apecloud/kubeblocks-tools:latest
Image ID: docker.io/apecloud/kubeblocks-tools@sha256:ca0b5179d21fc74bf598966c9f62b22b070e95f6798bb2de7d533ba3fe2fca9d
Port: 7373/TCP
Host Port: 0/TCP
Command:
lorry
--port
7373
State: Running
Started: Mon, 25 Sep 2023 15:49:44 +0800
Ready: True
Restart Count: 0
Readiness: http-get http://:7373/v1.0/bindings/custom%3Foperation=checkRole delay=0s timeout=1s period=2s #success=1 #failure=2
Environment:
KB_RSM_USERNAME: <set to the key 'username' in secret 'redis-cluster-conn-credential'> Optional: false
KB_RSM_PASSWORD: <set to the key 'password' in secret 'redis-cluster-conn-credential'> Optional: false
KB_RSM_ACTION_SVC_LIST: [36501]
KB_SERVICE_USER: <set to the key 'username' in secret 'redis-cluster-conn-credential'> Optional: false
KB_SERVICE_PASSWORD: <set to the key 'password' in secret 'redis-cluster-conn-credential'> Optional: false
KB_RSM_SERVICE_PORT: 6379
KB_SERVICE_PORT: 6379
KB_RSM_ROLE_UPDATE_MECHANISM: DirectAPIServerEventUpdate
KB_POD_NAME: redis-cluster-redis-0 (v1:metadata.name)
KB_NAMESPACE: default (v1:metadata.namespace)
KB_POD_UID: (v1:metadata.uid)
KB_NODENAME: (v1:spec.nodeName)
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-29l7t (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
data:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: data-redis-cluster-redis-0
ReadOnly: false
pod-info:
Type: DownwardAPI (a volume populated by information about the pod)
Items:
metadata.labels['kubeblocks.io/role'] -> pod-role
metadata.annotations['rs.apps.kubeblocks.io/primary'] -> primary-pod
metadata.annotations['apps.kubeblocks.io/component-replicas'] -> component-replicas
redis-metrics-config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: redis-cluster-redis-redis-metrics-config
Optional: false
redis-config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: redis-cluster-redis-redis-replication-config
Optional: false
scripts:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: redis-cluster-redis-redis-scripts
Optional: false
redis-conf:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
role-agent:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
kube-api-access-29l7t:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: kb-data=true:NoSchedule
node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Topology Spread Constraints: kubernetes.io/hostname:ScheduleAnyway when max skew 1 is exceeded for selector app.kubernetes.io/instance=redis-cluster,apps.kubeblocks.io/component-name=redis
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 6m50s default-scheduler Successfully assigned default/redis-cluster-redis-0 to ip-10-0-3-15.us-west-2.compute.internal
Normal SuccessfulAttachVolume 6m49s attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-49e88858-1284-49c8-8cc6-3817acef3a06"
Normal Pulled 6m42s kubelet Container image "msoap/shell2http:1.16.0" already present on machine
Normal Created 6m42s kubelet Created container role-agent-installer
Normal Started 6m42s kubelet Started container role-agent-installer
Normal Created 6m41s kubelet Created container kb-role-probe
Normal Started 6m41s kubelet Started container action-0
Normal Started 6m41s kubelet Started container kb-role-probe
Normal Pulled 6m41s kubelet Container image "registry.cn-hangzhou.aliyuncs.com/apecloud/agamotto:0.1.2-beta.1" already present on machine
Normal Created 6m41s kubelet Created container metrics
Normal Started 6m41s kubelet Started container metrics
Normal Pulled 6m41s kubelet Container image "registry.cn-hangzhou.aliyuncs.com/apecloud/kubeblocks-tools:0.7.0-alpha.16" already present on machine
Normal Created 6m41s kubelet Created container kb-we-syncer
Normal Started 6m41s kubelet Started container kb-we-syncer
Normal Pulled 6m41s kubelet Container image "registry.cn-hangzhou.aliyuncs.com/apecloud/redis-stack-server:7.0.6-RC8" already present on machine
Normal Created 6m41s kubelet Created container action-0
Normal Pulled 6m41s kubelet Container image "apecloud/kubeblocks-tools:latest" already present on machine
Normal checkRole 6m39s sqlchannel {"event":"Success","operation":"checkRole","originalRole":"","role":"primary"}
Normal Created 4m58s (x2 over 6m41s) kubelet Created container redis
Normal Pulled 4m58s (x2 over 6m41s) kubelet Container image "registry.cn-hangzhou.aliyuncs.com/apecloud/redis-stack-server:7.0.6-RC8" already present on machine
Normal Started 4m58s (x2 over 6m41s) kubelet Started container redis
Warning Unhealthy 3m18s kubelet Readiness probe errored: rpc error: code = Unknown desc = failed to exec in container: container is in CONTAINER_EXITED state
Normal checkRole 3m16s sqlchannel {"event":"Success","operation":"checkRole","originalRole":"primary","role":"secondary"}
Normal checkRole 2m58s sqlchannel {"event":"Success","operation":"checkRole","originalRole":"secondary","role":"primary"}
Warning BackOff 77s (x4 over 3m17s) kubelet Back-off restarting failed container
Normal checkRole 76s sqlchannel {"event":"Success","operation":"checkRole","originalRole":"primary","role":"secondary"}
Normal checkRole 42s sqlchannel {"event":"Success","operation":"checkRole","originalRole":"secondary","role":"primary"}
➜ ~ git:(main) ✗ k get pod
NAME READY STATUS RESTARTS AGE
mysql-cluster-mysql-0 8/8 Running 0 65m
redis-cluster-redis-0 5/5 Running 3 (104s ago) 7m21s
➜ ~ git:(main) ✗ k logs redis-cluster-redis-0
Defaulted container "redis" out of: redis, metrics, kb-we-syncer, action-0, kb-role-probe, role-agent-installer (init)
+ echo include /etc/conf/redis.conf
+ echo replica-announce-ip redis-cluster-redis-0.redis-cluster-redis-headless.default.svc
+ [ -f /data/users.acl ]
+ sed -i /user default on/d /data/users.acl
+ sed -i /user kbreplicator on/d /data/users.acl
+ sed -i /user kbreplicator-sentinel on/d /data/users.acl
+ [ ! -z dqr6jqvn ]
+ echo masteruser kbreplicator
+ echo masterauth dqr6jqvn
+ echo user kbreplicator on +psync +replconf +ping >dqr6jqvn
+ [ ! -z dqr6jqvn ]
+ echo user kbreplicator-sentinel on allchannels +multi +slaveof +ping +exec +subscribe +config|rewrite +role +publish +info +client|setname +client|kill +script|kill >dqr6jqvn
+ [ ! -z dqr6jqvn ]
+ echo protected-mode yes
+ echo user default on allcommands allkeys >dqr6jqvn
+ echo aclfile /data/users.acl
+ start_redis_server
+ exec redis-server /etc/redis/redis.conf --loadmodule /opt/redis-stack/lib/redisearch.so --loadmodule /opt/redis-stack/lib/redisgraph.so --loadmodule /opt/redis-stack/lib/redistimeseries.so+ --loadmodule /opt/redis-stack/lib/rejson.so --loadmodule /opt/redis-stack/lib/redisbloom.so
create_replication
+ [ ! -z dqr6jqvn ]
+ retry redis-cli -h 127.0.0.1 -p 6379 -a dqr6jqvn ping
+ local max_attempts=20
+ local attempt=1
+ redis-cli -h 127.0.0.1 -p 6379 -a dqr6jqvn ping
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
Could not connect to Redis at 127.0.0.1:6379: Connection refused
+ [ 1 -eq 20 ]
+ echo Command 'redis-cli -h 127.0.0.1 -p 6379 -a dqr6jqvn ping' failed. Attempt 1 of 20. Retrying in 5 seconds...
+ attempt=2
+ sleep 3
Command 'redis-cli -h 127.0.0.1 -p 6379 -a dqr6jqvn ping' failed. Attempt 1 of 20. Retrying in 5 seconds...
+ redis-cli -h 127.0.0.1 -p 6379 -a dqr6jqvn ping
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
PONG
+ [ 2 -eq 20 ]
+ attempt=1
+ max_attempts=20
+ [ 1 -le 20 ]
+ cat /kb-podinfo/primary-pod
+ [ -z ]
+ echo Waiting for primary pod information from the DownwardAPI annotation to be available, attempt 1 of 20...
Waiting for primary pod information from the DownwardAPI annotation to be available, attempt 1 of 20...
+ sleep 5
+ attempt=2
+ [ 2 -le 20 ]
+ cat /kb-podinfo/primary-pod
+ [ -z ]
+ echo Waiting for primary pod information from the DownwardAPI annotation to be available, attempt 2 of 20...
+ sleep 5
Waiting for primary pod information from the DownwardAPI annotation to be available, attempt 2 of 20...
+ attempt=3
+ [ 3 -le 20 ]
+ cat /kb-podinfo/primary-pod
+ [ -z ]
+ echo Waiting for primary pod information from the DownwardAPI annotation to be available, attempt 3 of 20...
+ sleep 5
Waiting for primary pod information from the DownwardAPI annotation to be available, attempt 3 of 20...
+ attempt=4
+ [ 4 -le 20 ]
+ cat /kb-podinfo/primary-pod
Waiting for primary pod information from the DownwardAPI annotation to be available, attempt 4 of 20...
+ [ -z ]
+ echo Waiting for primary pod information from the DownwardAPI annotation to be available, attempt 4 of 20...
+ sleep 5
+ attempt=5
+ [ 5 -le 20 ]
+ cat /kb-podinfo/primary-pod
Waiting for primary pod information from the DownwardAPI annotation to be available, attempt 5 of 20...
+ [ -z ]
+ echo Waiting for primary pod information from the DownwardAPI annotation to be available, attempt 5 of 20...
+ sleep 5
+ attempt=6
+ [ 6 -le 20 ]
+ cat /kb-podinfo/primary-pod
+ [ -z ]
+ echo Waiting for primary pod information from the DownwardAPI annotation to be available, attempt 6 of 20...
+ sleep 5
Waiting for primary pod information from the DownwardAPI annotation to be available, attempt 6 of 20...
+ attempt=7
+ [ 7 -le 20 ]
+ cat /kb-podinfo/primary-pod
+ [ -z ]
+ echo Waiting for primary pod information from the DownwardAPI annotation to be available, attempt 7 of 20...
+ sleep 5
Waiting for primary pod information from the DownwardAPI annotation to be available, attempt 7 of 20...
+ attempt=8
+ [ 8 -le 20 ]
+ cat /kb-podinfo/primary-pod
+ [ -z ]
+ echo Waiting for primary pod information from the DownwardAPI annotation to be available, attempt 8 of 20...
+ sleep 5
Waiting for primary pod information from the DownwardAPI annotation to be available, attempt 8 of 20...
+ attempt=9
+ [ 9 -le 20 ]
+ cat /kb-podinfo/primary-pod
Waiting for primary pod information from the DownwardAPI annotation to be available, attempt 9 of 20...
+ [ -z ]
+ echo Waiting for primary pod information from the DownwardAPI annotation to be available, attempt 9 of 20...
+ sleep 5
+ attempt=10
+ [ 10 -le 20 ]
+ cat /kb-podinfo/primary-pod
Waiting for primary pod information from the DownwardAPI annotation to be available, attempt 10 of 20...
+ [ -z ]
+ echo Waiting for primary pod information from the DownwardAPI annotation to be available, attempt 10 of 20...
+ sleep 5
+ attempt=11
+ [ 11 -le 20 ]
+ cat /kb-podinfo/primary-pod
+ [ -z ]
+ echo Waiting for primary pod information from the DownwardAPI annotation to be available, attempt 11 of 20...
+ sleep 5
Waiting for primary pod information from the DownwardAPI annotation to be available, attempt 11 of 20...
+ attempt=12
+ [ 12 -le 20 ]
+ cat /kb-podinfo/primary-pod
+ [ -z ]
+ echo Waiting for primary pod information from the DownwardAPI annotation to be available, attempt 12 of 20...
+ sleep 5
Waiting for primary pod information from the DownwardAPI annotation to be available, attempt 12 of 20...
+ attempt=13
+ [ 13 -le 20 ]
+ cat /kb-podinfo/primary-pod
Waiting for primary pod information from the DownwardAPI annotation to be available, attempt 13 of 20...
+ [ -z ]
+ echo Waiting for primary pod information from the DownwardAPI annotation to be available, attempt 13 of 20...
+ sleep 5
+ attempt=14
+ [ 14 -le 20 ]
+ cat /kb-podinfo/primary-pod
+ [ -z ]
+ echo Waiting for primary pod information from the DownwardAPI annotation to be available, attempt 14 of 20...
+ sleep 5
Waiting for primary pod information from the DownwardAPI annotation to be available, attempt 14 of 20...
+ attempt=15
+ [ 15 -le 20 ]
+ cat /kb-podinfo/primary-pod
+ [ -z ]
+ echo Waiting for primary pod information from the DownwardAPI annotation to be available, attempt 15 of 20...
+ sleep 5
Waiting for primary pod information from the DownwardAPI annotation to be available, attempt 15 of 20...
+ attempt=16
+ [ 16 -le 20 ]
+ cat /kb-podinfo/primary-pod
+ [ -z ]
+ echo Waiting for primary pod information from the DownwardAPI annotation to be available, attempt 16 of 20...
+ sleep 5
Waiting for primary pod information from the DownwardAPI annotation to be available, attempt 16 of 20...
➜ ~ k describe pod redis-2q8t6-redis-0 Name: redis-2q8t6-redis-0 Namespace: default Priority: 0 Node: gke-cluster-1-default-pool-56d1d4e0-fhtn/10.128.0.51 Start Time: Mon, 04 Sep 2023 16:56:44 +0800 Labels: app.kubernetes.io/component=redis app.kubernetes.io/instance=redis-2q8t6 app.kubernetes.io/managed-by=kubeblocks app.kubernetes.io/name=redis app.kubernetes.io/version=redis-7.0.6 apps.kubeblocks.io/component-name=redis apps.kubeblocks.io/workload-type=Replication controller-revision-hash=redis-2q8t6-redis-5659cb9975 kubeblocks.io/role=primary statefulset.kubernetes.io/pod-name=redis-2q8t6-redis-0 Annotations: apps.kubeblocks.io/component-replicas: 1 apps.kubeblocks.io/last-role-changed-event-timestamp: 2023-09-04T08:58:39Z Status: Running IP: 10.4.2.201 IPs: IP: 10.4.2.201 Controlled By: StatefulSet/redis-2q8t6-redis Containers: redis: Container ID: containerd://05b607d8f80cecb3a9ab041e8498e1be3caa03c523938d578a13aa1bc89d777e Image: registry.cn-hangzhou.aliyuncs.com/apecloud/redis-stack-server:7.0.6-RC8 Image ID: docker.io/redis/redis-stack-server@sha256:511808b267ab8d800283604ef5c01f4fe94792bfb746bb6dba236cc29ff5495b Port: 6379/TCP Host Port: 0/TCP Command: /scripts/redis-start.sh State: Waiting Reason: CrashLoopBackOff Last State: Terminated Reason: Completed Exit Code: 0 Started: Mon, 04 Sep 2023 17:09:10 +0800 Finished: Mon, 04 Sep 2023 17:10:53 +0800 Ready: False Restart Count: 5 Limits: cpu: 1 memory: 1Gi Requests: cpu: 100m memory: 102Mi Readiness: exec [sh -c /scripts/redis-ping.sh 1] delay=10s timeout=1s period=5s #success=1 #failure=5 Environment Variables from: redis-2q8t6-redis-env ConfigMap Optional: false Environment: KB_POD_NAME: redis-2q8t6-redis-0 (v1:metadata.name) KB_POD_UID: (v1:metadata.uid) KB_NAMESPACE: default (v1:metadata.namespace) KB_SA_NAME: (v1:spec.serviceAccountName) KB_NODENAME: (v1:spec.nodeName) KB_HOST_IP: (v1:status.hostIP) KB_POD_IP: (v1:status.podIP) KB_POD_IPS: (v1:status.podIPs) KB_HOSTIP: (v1:status.hostIP) KB_PODIP: (v1:status.podIP) KB_PODIPS: (v1:status.podIPs) KB_CLUSTER_NAME: redis-2q8t6 KB_COMP_NAME: redis KB_CLUSTER_COMP_NAME: redis-2q8t6-redis KB_CLUSTER_UID_POSTFIX_8: ce6e6d04 KB_POD_FQDN: $(KB_POD_NAME).$(KB_CLUSTER_COMP_NAME)-headless.$(KB_NAMESPACE).svc REDIS_REPL_USER: kbreplicator REDIS_REPL_PASSWORD: <set to the key 'password' in secret 'redis-2q8t6-conn-credential'> Optional: false REDIS_DEFAULT_PASSWORD: <set to the key 'password' in secret 'redis-2q8t6-conn-credential'> Optional: false REDIS_SENTINEL_USER: $(REDIS_REPL_USER)-sentinel REDIS_SENTINEL_PASSWORD: <set to the key 'password' in secret 'redis-2q8t6-conn-credential'> Optional: false REDIS_ARGS: --requirepass $(REDIS_PASSWORD) Mounts: /data from data (rw) /etc/conf from redis-config (rw) /etc/redis from redis-conf (rw) /kb-podinfo from pod-info (rw) /scripts from scripts (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-5wjmb (ro) metrics: Container ID: containerd://90def8238d455240d149adb04e75a01aaa3434c93d6bf1454b28ddb48487e633 Image: registry.cn-hangzhou.aliyuncs.com/apecloud/agamotto:0.1.2-beta.1 Image ID: registry.cn-hangzhou.aliyuncs.com/apecloud/agamotto@sha256:cbab349b90490807a8d5039bf01bc7e37334f20c98c7dd75bc7fc4cf9e5b10ee Port: 9121/TCP Host Port: 0/TCP Command: /bin/agamotto --config=/opt/conf/metrics-config.yaml State: Running Started: Mon, 04 Sep 2023 16:57:36 +0800 Ready: True Restart Count: 0 Limits: cpu: 0 memory: 0 Requests: cpu: 0 memory: 0 Environment Variables from: redis-2q8t6-redis-env ConfigMap Optional: false Environment: KB_POD_NAME: redis-2q8t6-redis-0 (v1:metadata.name) KB_POD_UID: (v1:metadata.uid) KB_NAMESPACE: default (v1:metadata.namespace) KB_SA_NAME: (v1:spec.serviceAccountName) KB_NODENAME: (v1:spec.nodeName) KB_HOST_IP: (v1:status.hostIP) KB_POD_IP: (v1:status.podIP) KB_POD_IPS: (v1:status.podIPs) KB_HOSTIP: (v1:status.hostIP) KB_PODIP: (v1:status.podIP) KB_PODIPS: (v1:status.podIPs) KB_CLUSTER_NAME: redis-2q8t6 KB_COMP_NAME: redis KB_CLUSTER_COMP_NAME: redis-2q8t6-redis KB_CLUSTER_UID_POSTFIX_8: ce6e6d04 KB_POD_FQDN: $(KB_POD_NAME).$(KB_CLUSTER_COMP_NAME)-headless.$(KB_NAMESPACE).svc ENDPOINT: localhost:6379 REDIS_USER: <set to the key 'username' in secret 'redis-2q8t6-conn-credential'> Optional: false REDIS_PASSWORD: <set to the key 'password' in secret 'redis-2q8t6-conn-credential'> Optional: false Mounts: /opt/conf from redis-metrics-config (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-5wjmb (ro) kb-checkrole: Container ID: containerd://f2a847808bcc02a974074e34944f8857b098c546558b6c3306f91a4bad05a817 Image: registry.cn-hangzhou.aliyuncs.com/apecloud/kubeblocks-tools:0.6.2-beta.1 Image ID: registry.cn-hangzhou.aliyuncs.com/apecloud/kubeblocks-tools@sha256:6760e31e4094714bbe126134d43d19c7d4bd82c1619695908c1c6c696474cb9f Ports: 3501/TCP, 50001/TCP Host Ports: 0/TCP, 0/TCP Command: probe --app-id batch-sdk --dapr-http-port 3501 --dapr-grpc-port 50001 --log-level info --config /config/probe/config.yaml --components-path /config/probe/components State: Running Started: Mon, 04 Sep 2023 16:57:38 +0800 Ready: True Restart Count: 0 Limits: cpu: 0 memory: 0 Requests: cpu: 0 memory: 0 Readiness: http-get http://:3501/v1.0/bindings/redis%3Foperation=checkRole&workloadType=Replication delay=0s timeout=1s period=2s #success=1 #failure=2 Startup: tcp-socket :3501 delay=0s timeout=1s period=10s #success=1 #failure=3 Environment Variables from: redis-2q8t6-redis-env ConfigMap Optional: false Environment: KB_POD_NAME: redis-2q8t6-redis-0 (v1:metadata.name) KB_POD_UID: (v1:metadata.uid) KB_NAMESPACE: default (v1:metadata.namespace) KB_SA_NAME: (v1:spec.serviceAccountName) KB_NODENAME: (v1:spec.nodeName) KB_HOST_IP: (v1:status.hostIP) KB_POD_IP: (v1:status.podIP) KB_POD_IPS: (v1:status.podIPs) KB_HOSTIP: (v1:status.hostIP) KB_PODIP: (v1:status.podIP) KB_PODIPS: (v1:status.podIPs) KB_CLUSTER_NAME: redis-2q8t6 KB_COMP_NAME: redis KB_CLUSTER_COMP_NAME: redis-2q8t6-redis KB_CLUSTER_UID_POSTFIX_8: ce6e6d04 KB_POD_FQDN: $(KB_POD_NAME).$(KB_CLUSTER_COMP_NAME)-headless.$(KB_NAMESPACE).svc KB_SERVICE_USER: <set to the key 'username' in secret 'redis-2q8t6-conn-credential'> Optional: false KB_SERVICE_PASSWORD: <set to the key 'password' in secret 'redis-2q8t6-conn-credential'> Optional: false KB_SERVICE_PORT: 6379 KB_SERVICE_ROLES: {} KB_SERVICE_CHARACTER_TYPE: redis KB_WORKLOAD_TYPE: Replication Mounts: /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-5wjmb (ro) Conditions: Type Status Initialized True Ready False ContainersReady False PodScheduled True Volumes: data: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: data-redis-2q8t6-redis-0 ReadOnly: false pod-info: Type: DownwardAPI (a volume populated by information about the pod) Items: metadata.labels['kubeblocks.io/role'] -> pod-role metadata.annotations['rs.apps.kubeblocks.io/primary'] -> primary-pod metadata.annotations['apps.kubeblocks.io/component-replicas'] -> component-replicas redis-metrics-config: Type: ConfigMap (a volume populated by a ConfigMap) Name: redis-2q8t6-redis-redis-metrics-config Optional: false redis-config: Type: ConfigMap (a volume populated by a ConfigMap) Name: redis-2q8t6-redis-redis-replication-config Optional: false scripts: Type: ConfigMap (a volume populated by a ConfigMap) Name: redis-2q8t6-redis-redis-scripts Optional: false redis-conf: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: SizeLimit:
kube-api-access-5wjmb:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional:
DownwardAPI: true
QoS Class: Burstable
Node-Selectors:
Tolerations: kb-data=true:NoSchedule
node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
Warning FailedAttachVolume 14m attachdetach-controller Multi-Attach error for volume "pvc-af1ac4d1-0247-4e62-9e70-41c3a37aebf2" Volume is already exclusively attached to one node and can't be attached to another Normal Scheduled 14m default-scheduler Successfully assigned default/redis-2q8t6-redis-0 to gke-cluster-1-default-pool-56d1d4e0-fhtn Normal SuccessfulAttachVolume 14m attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-af1ac4d1-0247-4e62-9e70-41c3a37aebf2" Normal Pulled 13m kubelet Container image "registry.cn-hangzhou.aliyuncs.com/apecloud/agamotto:0.1.2-beta.1" already present on machine Normal Created 13m kubelet Created container metrics Normal Created 13m kubelet Created container kb-checkrole Normal Pulled 13m kubelet Container image "registry.cn-hangzhou.aliyuncs.com/apecloud/kubeblocks-tools:0.6.2-beta.1" already present on machine Normal Started 13m kubelet Started container metrics Normal Started 13m kubelet Started container kb-checkrole Normal checkRole 13m sqlchannel {"event":"Failed","message":"role check delay","operation":"checkRole","originalRole":""} Normal checkRole 12m sqlchannel {"event":"Success","operation":"checkRole","originalRole":"","role":"primary"} Normal checkRole 12m sqlchannel {"event":"Failed","message":"dial tcp 127.0.0.1:6379: connect: connection refused","operation":"checkRole","originalRole":"primary"} Normal checkRole 10m sqlchannel {"event":"Failed","message":"dial tcp 127.0.0.1:6379: connect: connection refused","operation":"checkRole","originalRole":"primary"} Warning Unhealthy 8m30s kubelet Readiness probe errored: rpc error: code = NotFound desc = failed to exec in container: failed to load task: no running task found: task 26e57b3fdbdd486d0e7adc2de42fb8708d45fbbab9423c9e048a207045c762dc not found: not found Normal checkRole 8m28s sqlchannel {"event":"Failed","message":"dial tcp 127.0.0.1:6379: connect: connection refused","operation":"checkRole","originalRole":"primary"} Normal Pulled 7m58s (x4 over 14m) kubelet Container image "registry.cn-hangzhou.aliyuncs.com/apecloud/redis-stack-server:7.0.6-RC8" already present on machine Normal Created 7m58s (x4 over 14m) kubelet Created container redis Normal Started 7m56s (x4 over 13m) kubelet Started container redis Normal checkRole 6m12s sqlchannel {"event":"Failed","message":"dial tcp 127.0.0.1:6379: connect: connection refused","operation":"checkRole","originalRole":"primary"} Normal checkRole 3m46s sqlchannel {"event":"Failed","message":"dial tcp 127.0.0.1:6379: connect: connection refused","operation":"checkRole","originalRole":"primary"} Warning BackOff 3m45s (x10 over 10m) kubelet Back-off restarting failed container redis in pod redis-2q8t6-redis-0_default(800d96ed-aa47-4205-80d1-f3f202ecb606) Normal checkRole 36s sqlchannel {"event":"Failed","message":"dial tcp 127.0.0.1:6379: connect: connection refused","operation":"checkRole","originalRole":"primary"} ➜ ~ k logs redis-2q8t6-redis-0 Defaulted container "redis" out of: redis, metrics, kb-checkrole
➜ ~ k logs kubeblocks-86c89fcc67-zqwg4 -n kb-system >kblog.txt Defaulted container "manager" out of: manager, tools (init), datascript (init) kblog.txt