search

apecloud / kubeblocks

KubeBlocks is an open-source control plane software that runs and manages databases, message queues and other stateful applications on K8s.
https://kubeblocks.io
GNU Affero General Public License v3.0
2.15k stars 176 forks source link

[BUG] `kbcli cluster create` cluster `STATUS` is `Failed` #935

Closed JashBook closed 1 year ago

JashBook commented 1 year ago

Describe the bug kbcli cluster create pod status is Running ,but cluster STATUS is Failed 

Message:               pods are unavailable in Components: [replicasets], refer to related component message in Cluster.status.components
    Reason:                ComponentsNotReady
    Status:                False
    Type:                  Ready

To Reproduce Steps to reproduce the behavior:

  1. install kbcli 
    curl -fsSL http://161.189.136.182:8000/apecloud/kubeblocks/install_cli.sh |bash -s v0.3.0-alpha.6
  2. install kubeblocks 
    kbcli kubeblocks install
  3. create cluster 
    echo "- name: replicasets
type: replicasets
replicas: 1
volumeClaimTemplates:
- name: data
  spec:
    accessModes:
      - ReadWriteOnce
    resources:
      requests:
        storage: 1Gi" >  create_1replica.yaml
    kbcli cluster create wesql-cluster \
--cluster-definition=apecloud-wesql \
--cluster-version=wesql-8.0.30 \
--termination-policy=Halt \
--components=create_1replica.yaml \
--monitor=false \
--enable-all-logs=false
  4. see error (This mistake is not guaranteed to happen 100% of the time, but the probability is high.) 
    
kubectl get cluster,pod    
NAME                                        CLUSTER-DEFINITION   VERSION        TERMINATION-POLICY   STATUS   AGE
cluster.dbaas.kubeblocks.io/wesql-cluster   apecloud-wesql       wesql-8.0.30   Halt                 Failed   2m30s

NAME READY STATUS RESTARTS AGE pod/kubeblocks-5464f47c54-bx4dk 1/1 Running 0 3h57m pod/kubeblocks-grafana-94c6975ff-gsm4d 3/3 Running 0 3h57m pod/kubeblocks-prometheus-alertmanager-84bcc88b44-f65fk 2/2 Running 0 3h57m pod/kubeblocks-prometheus-server-b5db88865-d7kqb 2/2 Running 0 3h57m pod/wesql-cluster-replicasets-0 2/2 Running 0 2m30s


logs pod container mysql

kubectl logs wesql-cluster-replicasets-0 -c mysql

wesql-cluster-replicasets-0

no need to call add 0 wesql-cluster-replicasets-0.wesql-cluster-replicasets-headless:13306@1

docker-entrypoint.sh mysqld --defaults-file=/opt/mysql/my.cnf --cluster-start-index=1 --cluster-info="wesql-cluster-replicasets-0.wesql-cluster-replicasets-headless:13306@1" --cluster-id=1 2023-01-09 07:06:20+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server started. 2023-01-09 07:06:21+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql' 2023-01-09 07:06:21+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server started. '/var/lib/mysql/mysql.sock' -> '/var/run/mysqld/mysqld.sock' 2023-01-09T07:06:21.720016Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.30) starting as process 15 2023-01-09T07:06:21.723047Z 0 [Warning] [MY-000054] [Server] World-writable config file '/data/mysql/data/auto.cnf' is ignored. 2023-01-09T07:06:21.723138Z 0 [Warning] [MY-010107] [Server] World-writable config file '/data/mysql/data/auto.cnf' has been removed. 2023-01-09T07:06:21.723478Z 0 [Warning] [MY-010075] [Server] No existing UUID has been found, so we assume that this is the first time that this server has been started. Generating a new UUID: 1f38f53a-8fec-11ed-9b51-0242ac11000f. 2023-01-09T07:06:21.728982Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started. 2023-01-09T07:06:21.999874Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended. 2023-01-09T07:06:22.104472Z 0 [System] [MY-010229] [Server] Starting XA crash recovery... 2023-01-09T07:06:22.127280Z 0 [System] [MY-010232] [Server] XA crash recovery finished. 2023-01-09T07:06:22.129305Z 0 [Warning] [MY-000000] [Server] Recover consensus index is 1 2023-01-09T07:06:22.141725Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed. 2023-01-09T07:06:22.141768Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel. 2023-01-09T07:06:22.159578Z 0 [Warning] [MY-000000] [Server] last log term is 2, last log term index is 1. [2023-01-09 07:06:31.881200] [Default] Server 1 : Enter startElectionCallback [2023-01-09 07:06:31.881200] [Default] Server 1 : Paxos state change from FOLL to CAND !! [2023-01-09 07:06:31.881200] [Default] Server 1 : Epoch task currentEpoch(0) during requestVote [2023-01-09 07:06:31.881200] [Default] Server 1 : Start new requestVote: new term(3) [2023-01-09 07:06:31.881200] [Default] Server 1 : Paxos state change from CAND to LEDR !! [2023-01-09 07:06:31.881200] [Default] Server 1 : become Leader (currentTerm 3, lli:2, llt:3)!! 2023-01-09T07:06:31.892007Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Bind-address: '::' port: 33060, socket: /var/run/mysqld/mysqlx.sock 2023-01-09T07:06:31.892219Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.30' socket: '/var/run/mysqld/mysqld.sock' port: 3306 WeSQL Server - GPL, Release 4, Revision ba56235. 2023-01-09T07:06:31.900039Z 5 [System] [MY-025002] [Server] Consensus apply thread start, recover status = 1, consensus start apply index = 0, rli consensus index = 0. 2023-01-09T07:06:31.900157Z 5 [System] [MY-025003] [Server] Consensus apply thread group relay log file name = './master-bin.000001', pos = 364, rli apply index = 1. 2023-01-09T07:06:31.903732Z 5 [System] [MY-025004] [Server] Consensus apply thread stop, opt_consensus_leader_stop_apply: false, seconds_behind_master: 1673247991, opt_consensus_leader_stop_apply_time: 0. 2023-01-09T07:06:31.903784Z 5 [System] [MY-025005] [Server] Consensus apply thread catchup commit index, consensus index: 1, current term: 2, stop term: 3.

logs pod container kb-rolechangedcheck

kubectl logs wesql-cluster-replicasets-0 kb-rolechangedcheck time="2023-01-09T07:06:20.981326219Z" level=info msg="starting Dapr Runtime -- version edge -- commit " app_id=batch-sdk instance=wesql-cluster-replicasets-0 scope=dapr.runtime type=log ver=unknown time="2023-01-09T07:06:20.981394094Z" level=info msg="log level set to: info" app_id=batch-sdk instance=wesql-cluster-replicasets-0 scope=dapr.runtime type=log ver=unknown time="2023-01-09T07:06:20.981649552Z" level=info msg="metrics server started on :9090/" app_id=batch-sdk instance=wesql-cluster-replicasets-0 scope=dapr.metrics type=log ver=unknown time="2023-01-09T07:06:20.982932635Z" level=info msg="standalone mode configured" app_id=batch-sdk instance=wesql-cluster-replicasets-0 scope=dapr.runtime type=log ver=unknown time="2023-01-09T07:06:20.982958719Z" level=info msg="app id: batch-sdk" app_id=batch-sdk instance=wesql-cluster-replicasets-0 scope=dapr.runtime type=log ver=unknown time="2023-01-09T07:06:20.983175844Z" level=info msg="mTLS is disabled. Skipping certificate request and tls validation" app_id=batch-sdk instance=wesql-cluster-replicasets-0 scope=dapr.runtime type=log ver=unknown time="2023-01-09T07:06:20.983276135Z" level=info msg="Dapr trace sampler initialized: DaprTraceSampler(P=0.000100)" app_id=batch-sdk instance=wesql-cluster-replicasets-0 scope=dapr.runtime type=log ver=unknown time="2023-01-09T07:06:20.984319094Z" level=info msg="local service entry announced: batch-sdk -> 172.17.0.15:39253" app_id=batch-sdk instance=wesql-cluster-replicasets-0 scope=dapr.contrib type=log ver=unknown time="2023-01-09T07:06:20.98437076Z" level=info msg="Initialized name resolution to mdns" app_id=batch-sdk instance=wesql-cluster-replicasets-0 scope=dapr.runtime type=log ver=unknown time="2023-01-09T07:06:20.984548177Z" level=info msg="loading components" app_id=batch-sdk instance=wesql-cluster-replicasets-0 scope=dapr.runtime type=log ver=unknown time="2023-01-09T07:06:20.986328927Z" level=info msg="successful init for output binding etcd (bindings.etcd/v1)" app_id=batch-sdk instance=wesql-cluster-replicasets-0 scope=dapr.runtime type=log ver=unknown time="2023-01-09T07:06:20.986362344Z" level=info msg="component loaded. name: etcd, type: bindings.etcd/v1" app_id=batch-sdk instance=wesql-cluster-replicasets-0 scope=dapr.runtime type=log ver=unknown time="2023-01-09T07:06:20.986511635Z" level=info msg="successful init for output binding mongodb (bindings.mongodb/v1)" app_id=batch-sdk instance=wesql-cluster-replicasets-0 scope=dapr.runtime type=log ver=unknown time="2023-01-09T07:06:20.986588719Z" level=info msg="component loaded. name: mongodb, type: bindings.mongodb/v1" app_id=batch-sdk instance=wesql-cluster-replicasets-0 scope=dapr.runtime type=log ver=unknown time="2023-01-09T07:06:20.98665876Z" level=info msg="successful init for output binding mysql (bindings.mysql/v1)" app_id=batch-sdk instance=wesql-cluster-replicasets-0 scope=dapr.runtime type=log ver=unknown time="2023-01-09T07:06:20.986678844Z" level=info msg="component loaded. name: mysql, type: bindings.mysql/v1" app_id=batch-sdk instance=wesql-cluster-replicasets-0 scope=dapr.runtime type=log ver=unknown time="2023-01-09T07:06:20.986692635Z" level=info msg="waiting for all outstanding components to be processed" app_id=batch-sdk instance=wesql-cluster-replicasets-0 scope=dapr.runtime type=log ver=unknown time="2023-01-09T07:06:20.986703427Z" level=info msg="component loaded. name: probe, type: middleware.http.probe/v1" app_id=batch-sdk instance=wesql-cluster-replicasets-0 scope=dapr.runtime type=log ver=unknown time="2023-01-09T07:06:20.986801802Z" level=info msg="all outstanding components processed" app_id=batch-sdk instance=wesql-cluster-replicasets-0 scope=dapr.runtime type=log ver=unknown time="2023-01-09T07:06:20.98682126Z" level=info msg="enabled middleware.http.probe/http middleware" app_id=batch-sdk instance=wesql-cluster-replicasets-0 scope=dapr.runtime type=log ver=unknown time="2023-01-09T07:06:20.986836052Z" level=info msg="gRPC proxy enabled" app_id=batch-sdk instance=wesql-cluster-replicasets-0 scope=dapr.runtime type=log ver=unknown time="2023-01-09T07:06:20.986900177Z" level=info msg="enabled gRPC tracing middleware" app_id=batch-sdk instance=wesql-cluster-replicasets-0 scope=dapr.runtime.grpc.api type=log ver=unknown time="2023-01-09T07:06:20.986919469Z" level=info msg="enabled gRPC metrics middleware" app_id=batch-sdk instance=wesql-cluster-replicasets-0 scope=dapr.runtime.grpc.api type=log ver=unknown time="2023-01-09T07:06:20.987009844Z" level=info msg="API gRPC server is running on port 50001" app_id=batch-sdk instance=wesql-cluster-replicasets-0 scope=dapr.runtime type=log ver=unknown time="2023-01-09T07:06:20.98776351Z" level=info msg="enabled metrics http middleware" app_id=batch-sdk instance=wesql-cluster-replicasets-0 scope=dapr.runtime.http type=log ver=unknown time="2023-01-09T07:06:20.987786802Z" level=info msg="enabled tracing http middleware" app_id=batch-sdk instance=wesql-cluster-replicasets-0 scope=dapr.runtime.http type=log ver=unknown time="2023-01-09T07:06:20.987829677Z" level=info msg="http server is running on port 3501" app_id=batch-sdk instance=wesql-cluster-replicasets-0 scope=dapr.runtime type=log ver=unknown time="2023-01-09T07:06:20.98784401Z" level=info msg="The request body size parameter is: 4" app_id=batch-sdk instance=wesql-cluster-replicasets-0 scope=dapr.runtime type=log ver=unknown time="2023-01-09T07:06:20.987869302Z" level=info msg="enabled gRPC tracing middleware" app_id=batch-sdk instance=wesql-cluster-replicasets-0 scope=dapr.runtime.grpc.internal type=log ver=unknown time="2023-01-09T07:06:20.987883052Z" level=info msg="enabled gRPC metrics middleware" app_id=batch-sdk instance=wesql-cluster-replicasets-0 scope=dapr.runtime.grpc.internal type=log ver=unknown time="2023-01-09T07:06:20.987907427Z" level=info msg="internal gRPC server is running on port 39253" app_id=batch-sdk instance=wesql-cluster-replicasets-0 scope=dapr.runtime type=log ver=unknown time="2023-01-09T07:06:20.987926635Z" level=warning msg="App channel is not initialized. Did you configure an app-port?" app_id=batch-sdk instance=wesql-cluster-replicasets-0 scope=dapr.runtime type=log ver=unknown time="2023-01-09T07:06:20.988007802Z" level=warning msg="failed to read from bindings: app channel not initialized " app_id=batch-sdk instance=wesql-cluster-replicasets-0 scope=dapr.runtime type=log ver=unknown time="2023-01-09T07:06:20.988092177Z" level=info msg="dapr initialized. Status: Running. Init Elapsed 5ms" app_id=batch-sdk instance=wesql-cluster-replicasets-0 scope=dapr.runtime type=log ver=unknown time="2023-01-09T07:06:31.319706626Z" level=info msg="request: &{POST /v1.0/bindings/mysql HTTP/1.1 1 1 map[Accept:[/] Content-Length:[49] Content-Type:[application/json] Host:[localhost:3501] User-Agent:[curl/7.85.0]] {0x4000738ae0} 49 [] false localhost:3501 map[] map[] map[] 127.0.0.1:55620 /v1.0/bindings/mysql 0x400064cc00}" app_id=batch-sdk instance=wesql-cluster-replicasets-0 scope=dapr.runtime type=log ver=unknown time="2023-01-09T07:06:31.468408959Z" level=info msg="request: &{POST /v1.0/bindings/mysql HTTP/1.1 1 1 map[Accept:[/] Content-Length:[49] Content-Type:[application/json] Host:[localhost:3501] User-Agent:[curl/7.85.0]] {0x4000739350} 49 [] false localhost:3501 map[] map[] map[] 127.0.0.1:55626 /v1.0/bindings/mysql 0x40007e2c00}" app_id=batch-sdk instance=wesql-cluster-replicasets-0 scope=dapr.runtime type=log ver=unknown time="2023-01-09T07:06:31.897103085Z" level=info msg="unable to ping the DB" app_id=batch-sdk instance=wesql-cluster-replicasets-0 scope=dapr.contrib type=log ver=unknown time="2023-01-09T07:06:31.901018001Z" level=info msg="unable to ping the DB" app_id=batch-sdk instance=wesql-cluster-replicasets-0 scope=dapr.contrib type=log ver=unknown ... time="2023-01-09T07:17:06.459594087Z" level=info msg="request: &{POST /v1.0/bindings/mysql HTTP/1.1 1 1 map[Accept:[/] Content-Length:[49] Content-Type:[application/json] Host:[localhost:3501] User-Agent:[curl/7.85.0]] {0x40005502d0} 49 [] false localhost:3501 map[] map[] map[] 127.0.0.1:38320 /v1.0/bindings/mysql 0x40003c4600}" app_id=batch-sdk instance=wesql-cluster-replicasets-0 scope=dapr.runtime type=log ver=unknown time="2023-01-09T07:17:06.464601003Z" level=info msg="unable to ping the DB" app_id=batch-sdk instance=wesql-cluster-replicasets-0 scope=dapr.contrib type=log ver=unknown

describe cluster

kubectl describe cluster wesql-cluster Name: wesql-cluster Namespace: default Labels: clusterdefinition.kubeblocks.io/name=apecloud-wesql clusterversion.kubeblocks.io/name=wesql-8.0.30 Annotations: kubeblocks.io/storage-class: csi-hostpath-sc API Version: dbaas.kubeblocks.io/v1alpha1 Kind: Cluster Metadata: Creation Timestamp: 2023-01-09T07:06:11Z Finalizers: cluster.kubeblocks.io/finalizer Generation: 1 Managed Fields: API Version: dbaas.kubeblocks.io/v1alpha1 Fields Type: FieldsV1 fieldsV1: f:spec: .: f:affinity: .: f:nodeLabels: f:podAntiAffinity: f:topologyKeys: f:clusterDefinitionRef: f:clusterVersionRef: f:components: .: k:{"name":"replicasets"}: .: f:monitor: f:name: f:replicas: f:serviceType: f:type: f:volumeClaimTemplates: f:terminationPolicy: f:tolerations: Manager: kbcli Operation: Update Time: 2023-01-09T07:06:11Z API Version: dbaas.kubeblocks.io/v1alpha1 Fields Type: FieldsV1 fieldsV1: f:metadata: f:annotations: .: f:kubeblocks.io/storage-class: f:finalizers: .: v:"cluster.kubeblocks.io/finalizer": f:labels: .: f:clusterdefinition.kubeblocks.io/name: f:clusterversion.kubeblocks.io/name: Manager: manager Operation: Update Time: 2023-01-09T07:06:11Z API Version: dbaas.kubeblocks.io/v1alpha1 Fields Type: FieldsV1 fieldsV1: f:status: .: f:clusterDefGeneration: f:components: .: f:replicasets: .: f:consensusSetStatus: .: f:leader: .: f:accessMode: f:name: f:pod: f:message: .: f:Pod/wesql-cluster-replicasets-0: f:phase: f:podsReady: f:podsReadyTime: f:type: f:conditions: f:observedGeneration: f:operations: .: f:horizontalScalable: f:restartable: f:verticalScalable: f:volumeExpandable: f:phase: Manager: manager Operation: Update Subresource: status Time: 2023-01-09T07:08:11Z Resource Version: 63479 UID: 05f62a17-4c3e-4f89-b19f-4f13de69c91a Spec: Affinity: Pod Anti Affinity: Preferred Cluster Definition Ref: apecloud-wesql Cluster Version Ref: wesql-8.0.30 Components: Monitor: false Name: replicasets Replicas: 1 Resources: Service Type: ClusterIP Type: replicasets Volume Claim Templates: Name: data Spec: Access Modes: ReadWriteOnce Resources: Requests: Storage: 1Gi Termination Policy: Halt Status: Cluster Def Generation: 2 Components: Replicasets: Consensus Set Status: Leader: Access Mode: None Name:
Pod: Unknown Message: Pod/wesql-cluster-replicasets-0: Role probe timeout, check whether the application is available Phase: Failed Pods Ready: true Pods Ready Time: 2023-01-09T07:08:12Z Type: replicasets Conditions: Last Transition Time: 2023-01-09T07:06:11Z Message: The operator has started the provisioning of Cluster: wesql-cluster Reason: PreCheckSucceed Status: True Type: ProvisioningStarted Last Transition Time: 2023-01-09T07:06:11Z Message: Successfully applied for resources Reason: ApplyResourcesSucceed Status: True Type: ApplyResources Last Transition Time: 2023-01-09T07:06:31Z Message: all pods of components are ready, waiting for the probe detection successful Reason: AllReplicasReady Status: True Type: ReplicasReady Last Transition Time: 2023-01-09T07:08:12Z Message: pods are unavailable in Components: [replicasets], refer to related component message in Cluster.status.components Reason: ComponentsNotReady Status: False Type: Ready Observed Generation: 1 Operations: Horizontal Scalable: Name: replicasets Restartable: replicasets Vertical Scalable: replicasets Volume Expandable: Name: replicasets Volume Claim Template Names: data Phase: Failed Events: Type Reason Age From Message

Normal Creating 4m27s cluster-controller Start Creating in Cluster: wesql-cluster Normal PreCheckSucceed 4m27s cluster-controller The operator has started the provisioning of Cluster: wesql-cluster Normal ApplyResourcesSucceed 4m27s cluster-controller Successfully applied for resources Normal AllReplicasReady 2m26s (x6 over 4m7s) cluster-controller all pods of components are ready, waiting for the probe detection successful

describe pod

kubectl describe pod wesql-cluster-replicasets-0 Name: wesql-cluster-replicasets-0 Namespace: default Priority: 0 Service Account: default Node: minikube/192.168.76.2 Start Time: Mon, 09 Jan 2023 15:06:11 +0800 Labels: app.kubernetes.io/component-name=replicasets app.kubernetes.io/instance=wesql-cluster app.kubernetes.io/managed-by=kubeblocks app.kubernetes.io/name=state.mysql-apecloud-wesql controller-revision-hash=wesql-cluster-replicasets-7f9cfb764b statefulset.kubernetes.io/pod-name=wesql-cluster-replicasets-0 Annotations: Status: Running IP: 172.17.0.15 IPs: IP: 172.17.0.15 Controlled By: StatefulSet/wesql-cluster-replicasets Containers: mysql: Container ID: docker://ad7987b6575900812427b49d8ffb9fb758f5c4bf5c1e49663c1c1d7d45f5383b Image: docker.io/apecloud/wesql-server:8.0.30-4.alpha4.20221117.gba56235 Image ID: docker-pullable://apecloud/wesql-server@sha256:50263dfa6d69c049d0bd4db55768c95aaec2f87e105b89ef9657d185b5d9be5c Ports: 3306/TCP, 13306/TCP Host Ports: 0/TCP, 0/TCP Command: /scripts/setup.sh State: Running Started: Mon, 09 Jan 2023 15:06:20 +0800 Ready: True Restart Count: 0 Environment Variables from: wesql-cluster-replicasets-env ConfigMap Optional: false Environment: KB_POD_NAME: wesql-cluster-replicasets-0 (v1:metadata.name) KB_NAMESPACE: default (v1:metadata.namespace) KB_SA_NAME: (v1:spec.serviceAccountName) KB_NODENAME: (v1:spec.nodeName) KB_HOSTIP: (v1:status.hostIP) KB_PODIP: (v1:status.podIP) KB_PODIPS: (v1:status.podIPs) KB_CLUSTER_NAME: wesql-cluster KB_COMP_NAME: replicasets KB_CLUSTER_COMP_NAME: wesql-cluster-replicasets MYSQL_ROOT_HOST: % MYSQL_ROOT_USER: <set to the key 'username' in secret 'wesql-cluster-conn-credential'> Optional: false MYSQL_ROOT_PASSWORD: <set to the key 'password' in secret 'wesql-cluster-conn-credential'> Optional: false MYSQL_DATABASE: mydb MYSQL_USER: u1 MYSQL_PASSWORD: u1 CLUSTER_ID: 1 CLUSTER_START_INDEX: 1 REPLICATIONUSER: replicator REPLICATION_PASSWORD:
MYSQL_TEMPLATE_CONFIG:
MYSQL_CUSTOM_CONFIG:
MYSQL_DYNAMIC_CONFIG:
Mounts: /data/mysql from data (rw) /opt/mysql from mysql-config (rw) /scripts from scripts (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-bpjj9 (ro) kb-rolechangedcheck: Container ID: docker://a9f4649fea6cc380be52448876aa87740c959bc431676950dcf786254a780666 Image: docker.io/apecloud/kubeblocks:0.3.0-alpha.6 Image ID: docker-pullable://apecloud/kubeblocks@sha256:245b8cfd8ff6ee3e781c44a7d33ea47682ecd854d38d4eea04b7fbd04c813aba Port: 3501/TCP Host Port: 0/TCP Command: probe --app-id batch-sdk --dapr-http-port 3501 --dapr-grpc-port 50001 --app-protocol http --log-level info --config /config/probe/config.yaml --components-path /config/probe/components State: Running Started: Mon, 09 Jan 2023 15:06:20 +0800 Ready: True Restart Count: 0 Readiness: exec [curl -X POST --max-time 1 --fail-with-body --silent -H Content-Type: application/json http://localhost:3501/v1.0/bindings/mysql -d {"operation": "roleCheck", "metadata":{"sql":""}}] delay=0s timeout=1s period=1s #success=1 #failure=2 Startup: tcp-socket :3501 delay=0s timeout=1s period=10s #success=1 #failure=3 Environment Variables from: wesql-cluster-replicasets-env ConfigMap Optional: false Environment: KB_POD_NAME: wesql-cluster-replicasets-0 (v1:metadata.name) KB_NAMESPACE: default (v1:metadata.namespace) KB_SA_NAME: (v1:spec.serviceAccountName) KB_NODENAME: (v1:spec.nodeName) KB_HOSTIP: (v1:status.hostIP) KB_PODIP: (v1:status.podIP) KB_PODIPS: (v1:status.podIPs) KB_CLUSTER_NAME: wesql-cluster KB_COMP_NAME: replicasets KB_CLUSTER_COMP_NAME: wesql-cluster-replicasets KB_SERVICE_USER: <set to the key 'username' in secret 'wesql-cluster-conn-credential'> Optional: false KB_SERVICE_PASSWORD: <set to the key 'password' in secret 'wesql-cluster-conn-credential'> Optional: false KB_SERVICE_PORT: 3306 KB_SERVICE_ROLES: {"follower":"Readonly","leader":"ReadWrite"} KB_SERVICE_CHARACTER_TYPE: mysql Mounts: /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-bpjj9 (ro) Conditions: Type Status Initialized True Ready True ContainersReady True PodScheduled True Volumes: data: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: data-wesql-cluster-replicasets-0 ReadOnly: false mysql-config: Type: ConfigMap (a volume populated by a ConfigMap) Name: wesql-cluster-replicasets-mysql-config Optional: false scripts: Type: ConfigMap (a volume populated by a ConfigMap) Name: wesql-cluster-replicasets-scripts Optional: false kube-api-access-bpjj9: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true QoS Class: BestEffort Node-Selectors: Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message

Normal Scheduled 12m default-scheduler Successfully assigned default/wesql-cluster-replicasets-0 to minikube Normal SuccessfulAttachVolume 12m attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-48300e50-8b7a-4c60-b1ef-dfea949fa876" Warning VolumeConditionAbnormal 12m (x6 over 12m) csi-pv-monitor-agent-hostpath.csi.k8s.io The volume isn't mounted Normal Pulled 12m kubelet Container image "docker.io/apecloud/wesql-server:8.0.30-4.alpha4.20221117.gba56235" already present on machine Normal Created 12m kubelet Created container mysql Normal Started 12m kubelet Started container mysql Normal Pulled 12m kubelet Container image "docker.io/apecloud/kubeblocks:0.3.0-alpha.6" already present on machine Normal Created 12m kubelet Created container kb-rolechangedcheck Normal Started 12m kubelet Started container kb-rolechangedcheck Normal VolumeConditionNormal 2m26s (x95 over 11m) csi-pv-monitor-agent-hostpath.csi.k8s.io The Volume returns to the healthy state



**Expected behavior**
`kbcil cluster create` success.

**Desktop (please complete the following information):**
 - OS: [e.g. iOS] KubeBlocks
 - Browser [e.g. chrome, safari]
 - Version [e.g. 22] [v0.3.0-alpha.6](https://github.com/apecloud/kubeblocks/releases/tag/v0.3.0-alpha.6)
xuriwuyun commented 1 year ago

time="2023-01-09T07:17:06.464601003Z" level=info msg="unable to ping the DB" app_id=batch-sdk instance=wesql-cluster-replicasets-0 scope=dapr.contrib type=log ver=unknown mysql is not ready, please check mysql container status in the pod yaml.

JashBook commented 1 year ago

description pod already shows the status of mysql container is Running.

Containers:
  mysql:
    Container ID:  docker://ad7987b6575900812427b49d8ffb9fb758f5c4bf5c1e49663c1c1d7d45f5383b
    Image:         docker.io/apecloud/wesql-server:8.0.30-4.alpha4.20221117.gba56235
    Image ID:      docker-pullable://apecloud/wesql-server@sha256:50263dfa6d69c049d0bd4db55768c95aaec2f87e105b89ef9657d185b5d9be5c
    Ports:         3306/TCP, 13306/TCP
    Host Ports:    0/TCP, 0/TCP
    Command:
      /scripts/setup.sh
    State:          Running
      Started:      Mon, 09 Jan 2023 15:06:20 +0800
    Ready:          True
    Restart Count:  0
    Environment Variables from:
      wesql-cluster-replicasets-env  ConfigMap  Optional: false
    Environment:
      KB_POD_NAME:            wesql-cluster-replicasets-0 (v1:metadata.name)
      KB_NAMESPACE:           default (v1:metadata.namespace)
      KB_SA_NAME:              (v1:spec.serviceAccountName)
      KB_NODENAME:             (v1:spec.nodeName)
      KB_HOSTIP:               (v1:status.hostIP)
      KB_PODIP:                (v1:status.podIP)
      KB_PODIPS:               (v1:status.podIPs)
      KB_CLUSTER_NAME:        wesql-cluster
      KB_COMP_NAME:           replicasets
      KB_CLUSTER_COMP_NAME:   wesql-cluster-replicasets
      MYSQL_ROOT_HOST:        %
      MYSQL_ROOT_USER:        <set to the key 'username' in secret 'wesql-cluster-conn-credential'>  Optional: false
      MYSQL_ROOT_PASSWORD:    <set to the key 'password' in secret 'wesql-cluster-conn-credential'>  Optional: false
      MYSQL_DATABASE:         mydb
      MYSQL_USER:             u1
      MYSQL_PASSWORD:         u1
      CLUSTER_ID:             1
      CLUSTER_START_INDEX:    1
      REPLICATIONUSER:        replicator
      REPLICATION_PASSWORD:   
      MYSQL_TEMPLATE_CONFIG:  
      MYSQL_CUSTOM_CONFIG:    
      MYSQL_DYNAMIC_CONFIG:   
    Mounts:
      /data/mysql from data (rw)
      /opt/mysql from mysql-config (rw)
      /scripts from scripts (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-bpjj9 (ro)
JashBook commented 1 year ago

Has been fixed.