search

apecloud / kubeblocks

KubeBlocks is an open-source control plane software that runs and manages databases, message queues and other stateful applications on K8s.
https://kubeblocks.io
GNU Affero General Public License v3.0
2.14k stars 176 forks source link

[BUG]Upgraded Mysql-scale cannot write with The consensus follower is not allowed to to do current operation returned #5754

Closed ahjing99 closed 1 year ago

ahjing99 commented 1 year ago
  1. Install kb 0.6.4
  2. Create mysql scale cluster
  3. Upgrade kb to 0.7
  4. Write to mysqlscale Seems the leader kbcli connect is not consistent with kbcli cluster describe 
    
➜  ~ kbcli cluster connect scale-uyvljv
Connect to instance scale-uyvljv-mysql-1: out of scale-uyvljv-mysql-1(leader), scale-uyvljv-mysql-0(follower), scale-uyvljv-mysql-2(follower)
Defaulted container "mysql" out of: mysql, metrics, vttablet, kb-checkrole, config-manager
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 125
Server version: 8.0.30 WeSQL Server - GPL, Release 5, Revision 4ca1eb8

Copyright (c) 2000, 2022, Oracle and/or its affiliates.

Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> use mydb; Database changed mysql> DROP TABLE IF EXISTS tmp_table; ERROR 7504 (HY000): The consensus follower is not allowed to to do current operation. mysql>

➜ ~ kbcli cluster describe scale-uyvljv Name: scale-uyvljv Created Time: Nov 02,2023 20:59 UTC+0800 NAMESPACE CLUSTER-DEFINITION VERSION STATUS TERMINATION-POLICY default apecloud-mysql ac-mysql-8.0.30 ConditionsError DoNotTerminate

Endpoints: COMPONENT MODE INTERNAL EXTERNAL mysql ReadWrite scale-uyvljv-mysql.default.svc.cluster.local:3306 vtgate ReadWrite scale-uyvljv-vtgate.default.svc.cluster.local:15001 scale-uyvljv-vtgate.default.svc.cluster.local:15991 scale-uyvljv-vtgate.default.svc.cluster.local:15306 scale-uyvljv-vtgate.default.svc.cluster.local:40000

Topology: COMPONENT INSTANCE ROLE STATUS AZ NODE CREATED-TIME mysql scale-uyvljv-mysql-0 leader Running us-central1-c gke-yijing-default-pool-3e14ea35-hqtr/10.128.0.30 Nov 03,2023 09:21 UTC+0800 mysql scale-uyvljv-mysql-1 follower Running us-central1-c gke-yijing-default-pool-3e14ea35-hxpl/10.128.0.28 Nov 03,2023 09:23 UTC+0800 mysql scale-uyvljv-mysql-2 follower Running us-central1-c gke-yijing-default-pool-3e14ea35-wg54/10.128.0.35 Nov 03,2023 09:22 UTC+0800 vtcontroller scale-uyvljv-vtcontroller-0 Running us-central1-c gke-yijing-default-pool-3e14ea35-hxpl/10.128.0.28 Nov 03,2023 09:21 UTC+0800 vtgate scale-uyvljv-vtgate-d9b7fb447-gxhnb Running us-central1-c gke-yijing-default-pool-3e14ea35-wg54/10.128.0.35 Nov 02,2023 20:59 UTC+0800

Resources Allocation: COMPONENT DEDICATED CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE-SIZE STORAGE-CLASS mysql false 500m / 500m 1Gi / 1Gi data:20Gi kb-default-sc vtcontroller false 500m / 500m 128Mi / 128Mi data:20Gi kb-default-sc vtgate false 500m / 500m

Images: COMPONENT TYPE IMAGE mysql mysql infracreate-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/apecloud-mysql-server:8.0.30-5.beta2.20230830.g4ca1eb8.13 vtcontroller vtcontroller infracreate-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/apecloud-mysql-scale:0.1.1 vtgate vtgate registry.cn-hangzhou.aliyuncs.com/apecloud/apecloud-mysql-scale:0.1.1

Data Protection: BACKUP-REPO AUTO-BACKUP BACKUP-SCHEDULE BACKUP-METHOD BACKUP-RETENTION

Show cluster events: kbcli cluster list-events -n default scale-uyvljv

ahjing99 commented 1 year ago 
➜  ~ echo 'show vitess_tablets;'|kbcli cluster connect  scale-uyvljv  --component  vtgate
Connect to instance scale-uyvljv-vtgate-d9b7fb447-gxhnb
Defaulted container "vtgate" out of: vtgate, wait-vtctld-ready (init)
Unable to use a TTY - input is not a terminal or the right kind of file
Cell    Keyspace    Shard   TabletType  State   Alias   Hostname    PrimaryTermStartTime
zone1   mysql   0   PRIMARY SERVING zone1-0000000000    scale-uyvljv-mysql-0.scale-uyvljv-mysql-headless    2023-11-03T01:22:54Z
zone1   mysql   0   REPLICA SERVING zone1-0000000001    scale-uyvljv-mysql-1.scale-uyvljv-mysql-headless
zone1   mysql   0   REPLICA SERVING zone1-0000000002    scale-uyvljv-mysql-2.scale-uyvljv-mysql-headless
ahjing99 commented 1 year ago 
➜  ~ k exec -it scale-uyvljv-mysql-0 -c mysql -- bash
[root@scale-uyvljv-mysql-0 /]# mysql -p$MYSQL_ROOT_PASSWORD
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 1392
Server version: 8.0.30 WeSQL Server - GPL, Release 5, Revision 4ca1eb8

Copyright (c) 2000, 2022, Oracle and/or its affiliates.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> use mydb;
Database changed
mysql> DROP TABLE IF EXISTS tmp_table;
Query OK, 0 rows affected, 1 warning (0.00 sec)

mysql>
ahjing99 commented 1 year ago

img_v3_004r_fe433a44-c790-4c5e-8620-331c2c17c9ag

JashBook commented 1 year ago

node action can't be nil component is Updating

kubectl describe cluster  scale-wqrrjr
Name:         scale-wqrrjr
Namespace:    default
Labels:       app.kubernetes.io/instance=scale-wqrrjr
              app.kubernetes.io/version=8.0.30
              clusterdefinition.kubeblocks.io/name=apecloud-mysql
              clusterversion.kubeblocks.io/name=ac-mysql-8.0.30
              helm.sh/chart=apecloud-mysql-cluster-0.6.0-alpha.0
Annotations:  kubeblocks.io/extra-env: {"KB_PROXY_ENABLED":"on"}
API Version:  apps.kubeblocks.io/v1alpha1
Kind:         Cluster
Metadata:
  Creation Timestamp:  2023-11-03T02:16:48Z
  Finalizers:
    cluster.kubeblocks.io/finalizer
  Generation:        1
  Resource Version:  23979487
  UID:               e57d0f06-30cb-4e76-b499-adae0edc4f90
Spec:
  Affinity:
    Pod Anti Affinity:     Required
    Tenancy:               SharedNode
  Cluster Definition Ref:  apecloud-mysql
  Cluster Version Ref:     ac-mysql-8.0.30
  Component Specs:
    Component Def Ref:  mysql
    Enabled Logs:
      slow
      error
    Monitor:        false
    Name:           mysql
    No Create PDB:  false
    Replicas:       3
    Resources:
      Limits:
        Cpu:     0.5
        Memory:  1Gi
      Requests:
        Cpu:     0.5
        Memory:  1Gi
    Volume Claim Templates:
      Name:  data
      Spec:
        Access Modes:
          ReadWriteOnce
        Resources:
          Requests:
            Storage:    20Gi
    Component Def Ref:  vtcontroller
    Enabled Logs:
      error
      warning
      info
    Monitor:        false
    Name:           vtcontroller
    No Create PDB:  false
    Replicas:       1
    Resources:
      Limits:
        Cpu:     500m
        Memory:  128Mi
    Volume Claim Templates:
      Name:  data
      Spec:
        Access Modes:
          ReadWriteOnce
        Resources:
          Requests:
            Storage:    20Gi
    Component Def Ref:  vtgate
    Enabled Logs:
      error
      warning
      info
      queryLog
    Monitor:        false
    Name:           vtgate
    No Create PDB:  false
    Replicas:       1
    Resources:
      Limits:
        Cpu:  0.5
      Requests:
        Cpu:           0.5
  Termination Policy:  Delete
Status:
  Cluster Def Generation:  2
  Components:
    Mysql:
      Consensus Set Status:
        Followers:
          Access Mode:  Readonly
          Name:         follower
          Pod:          scale-wqrrjr-mysql-1
          Access Mode:  Readonly
          Name:         follower
          Pod:          scale-wqrrjr-mysql-2
        Leader:
          Access Mode:  ReadWrite
          Name:         leader
          Pod:          scale-wqrrjr-mysql-0
      Phase:            Running
      Pods Ready:       true
      Pods Ready Time:  2023-11-03T02:21:04Z
    Vtcontroller:
      Phase:            Running
      Pods Ready:       true
      Pods Ready Time:  2023-11-03T02:18:41Z
    Vtgate:
      Phase:            Running
      Pods Ready:       true
      Pods Ready Time:  2023-11-03T02:20:51Z
  Conditions:
    Last Transition Time:  2023-11-03T02:16:48Z
    Message:               The operator has started the provisioning of Cluster: scale-wqrrjr
    Observed Generation:   1
    Reason:                PreCheckSucceed
    Status:                True
    Type:                  ProvisioningStarted
    Last Transition Time:  2023-11-03T02:37:35Z
    Message:               node action can't be nil
    Reason:                ApplyResourcesFailed
    Status:                False
    Type:                  ApplyResources
    Last Transition Time:  2023-11-03T02:21:04Z
    Message:               all pods of components are ready, waiting for the probe detection successful
    Reason:                AllReplicasReady
    Status:                True
    Type:                  ReplicasReady
    Last Transition Time:  2023-11-03T02:21:04Z
    Message:               Cluster: scale-wqrrjr is ready, current phase is Running
    Reason:                ClusterReady
    Status:                True
    Type:                  Ready
  Observed Generation:     1
  Phase:                   Running
Events:
  Type     Reason                    Age                   From                       Message
  ----     ------                    ----                  ----                       -------
  Normal   ComponentPhaseTransition  31m (x3 over 31m)     cluster-controller         Create a new component
  Normal   PreCheckSucceed           31m                   cluster-controller         The operator has started the provisioning of Cluster: scale-wqrrjr
  Normal   ApplyResourcesSucceed     31m                   cluster-controller         Successfully applied for resources
  Warning  ReplicasNotReady          29m                   cluster-controller         pods are not ready in Components: [mysql vtgate], refer to related component message in Cluster.status.components
  Warning  ComponentsNotReady        29m                   cluster-controller         pods are unavailable in Components: [mysql vtgate], refer to related component message in Cluster.status.components
  Warning  ComponentsNotReady        27m                   cluster-controller         pods are unavailable in Components: [mysql], refer to related component message in Cluster.status.components
  Warning  ReplicasNotReady          27m                   cluster-controller         pods are not ready in Components: [mysql], refer to related component message in Cluster.status.components
  Normal   AllReplicasReady          26m                   cluster-controller         all pods of components are ready, waiting for the probe detection successful
  Normal   ComponentPhaseTransition  26m (x3 over 29m)     cluster-controller         Running: true, PodsReady: true, PodsTimedout: false
  Normal   ClusterReady              26m                   cluster-controller         Cluster: scale-wqrrjr is ready, current phase is Running
  Normal   Running                   26m                   cluster-controller         Cluster: scale-wqrrjr is ready, current phase is Running
  Normal   SysAcctCreate             26m                   system-account-controller  Created accounts for cluster: scale-wqrrjr, component: mysql, accounts: kbdataprotection
  Normal   SysAcctCreate             26m                   system-account-controller  Created accounts for cluster: scale-wqrrjr, component: mysql, accounts: kbmonitoring
  Normal   SysAcctCreate             26m                   system-account-controller  Created accounts for cluster: scale-wqrrjr, component: mysql, accounts: kbprobe
  Normal   SysAcctCreate             26m                   system-account-controller  Created accounts for cluster: scale-wqrrjr, component: mysql, accounts: kbadmin
  Normal   SysAcctCreate             26m                   system-account-controller  Created accounts for cluster: scale-wqrrjr, component: mysql, accounts: kbreplicator
  Warning  ApplyResourcesFailed      10m (x2 over 10m)     cluster-controller         BackupPolicy.dataprotection.kubeblocks.io "scale-wqrrjr-mysql-backup-policy" is invalid: spec.backupMethods: Required value
  Warning  ApplyResourcesFailed      9m6s (x6 over 9m44s)  cluster-controller         node action can't be nil
  Normal   ComponentPhaseTransition  8m58s (x17 over 10m)  cluster-controller         component is Updating
ahjing99 commented 1 year ago

Did more test

  1. before upgrade kb, all three cluster's lead is pod2 
    
➜  ~ kbcli cluster describe scale-kveuzq
Name: scale-kveuzq   Created Time: Nov 03,2023 17:10 UTC+0800
NAMESPACE   CLUSTER-DEFINITION   VERSION           STATUS    TERMINATION-POLICY
default     apecloud-mysql       ac-mysql-8.0.30   Running   Delete

Endpoints: COMPONENT MODE INTERNAL EXTERNAL mysql ReadWrite scale-kveuzq-mysql.default.svc.cluster.local:3306 vtcontroller ReadWrite scale-kveuzq-vtcontroller.default.svc.cluster.local:2379 scale-kveuzq-vtcontroller.default.svc.cluster.local:15000 scale-kveuzq-vtcontroller.default.svc.cluster.local:15999 vtgate ReadWrite scale-kveuzq-vtgate.default.svc.cluster.local:15001 scale-kveuzq-vtgate.default.svc.cluster.local:15991 scale-kveuzq-vtgate.default.svc.cluster.local:15306 scale-kveuzq-vtgate.default.svc.cluster.local:40000

Topology: COMPONENT INSTANCE ROLE STATUS AZ NODE CREATED-TIME mysql scale-kveuzq-mysql-0 follower Running us-central1-c gke-yijing-default-pool-3e14ea35-klwc/10.128.0.26 Nov 03,2023 17:10 UTC+0800 mysql scale-kveuzq-mysql-1 follower Running us-central1-c gke-yijing-default-pool-3e14ea35-84st/10.128.0.31 Nov 03,2023 17:10 UTC+0800 mysql scale-kveuzq-mysql-2 leader Running us-central1-c gke-yijing-default-pool-3e14ea35-wg54/10.128.0.35 Nov 03,2023 17:10 UTC+0800 vtcontroller scale-kveuzq-vtcontroller-0 Running us-central1-c gke-yijing-default-pool-3e14ea35-84st/10.128.0.31 Nov 03,2023 17:10 UTC+0800 vtgate scale-kveuzq-vtgate-665d5b798f-x5kgn Running us-central1-c gke-yijing-default-pool-3e14ea35-klwc/10.128.0.26 Nov 03,2023 17:10 UTC+0800

Resources Allocation: COMPONENT DEDICATED CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE-SIZE STORAGE-CLASS mysql false 500m / 500m 1Gi / 1Gi data:20Gi kb-default-sc vtcontroller false 500m / 500m 128Mi / 128Mi data:20Gi kb-default-sc vtgate false 500m / 500m

Images: COMPONENT TYPE IMAGE mysql mysql registry.cn-hangzhou.aliyuncs.com/apecloud/apecloud-mysql-server:8.0.30-5.beta1.20230802.g5b589f1.12 vtcontroller vtcontroller registry.cn-hangzhou.aliyuncs.com/apecloud/apecloud-mysql-scale:0.1.1 vtgate vtgate registry.cn-hangzhou.aliyuncs.com/apecloud/apecloud-mysql-scale:0.1.1

Data Protection: AUTO-BACKUP BACKUP-SCHEDULE TYPE BACKUP-TTL LAST-SCHEDULE RECOVERABLE-TIME Disabled 7d

Show cluster events: kbcli cluster list-events -n default scale-kveuzq ➜ ~ kbcli cluster describe scale-ozhnvf Name: scale-ozhnvf Created Time: Nov 03,2023 17:10 UTC+0800 NAMESPACE CLUSTER-DEFINITION VERSION STATUS TERMINATION-POLICY default apecloud-mysql ac-mysql-8.0.30 Running DoNotTerminate

Endpoints: COMPONENT MODE INTERNAL EXTERNAL mysql ReadWrite scale-ozhnvf-mysql.default.svc.cluster.local:3306 vtcontroller ReadWrite scale-ozhnvf-vtcontroller.default.svc.cluster.local:2379 scale-ozhnvf-vtcontroller.default.svc.cluster.local:15000 scale-ozhnvf-vtcontroller.default.svc.cluster.local:15999 vtgate ReadWrite scale-ozhnvf-vtgate.default.svc.cluster.local:15001 scale-ozhnvf-vtgate.default.svc.cluster.local:15991 scale-ozhnvf-vtgate.default.svc.cluster.local:15306 scale-ozhnvf-vtgate.default.svc.cluster.local:40000

Topology: COMPONENT INSTANCE ROLE STATUS AZ NODE CREATED-TIME mysql scale-ozhnvf-mysql-0 follower Running us-central1-c gke-yijing-default-pool-3e14ea35-klwc/10.128.0.26 Nov 03,2023 17:10 UTC+0800 mysql scale-ozhnvf-mysql-1 follower Running us-central1-c gke-yijing-default-pool-3e14ea35-84st/10.128.0.31 Nov 03,2023 17:10 UTC+0800 mysql scale-ozhnvf-mysql-2 leader Running us-central1-c gke-yijing-default-pool-3e14ea35-wg54/10.128.0.35 Nov 03,2023 17:10 UTC+0800 vtcontroller scale-ozhnvf-vtcontroller-0 Running us-central1-c gke-yijing-default-pool-3e14ea35-wg54/10.128.0.35 Nov 03,2023 17:10 UTC+0800 vtgate scale-ozhnvf-vtgate-74955cf48b-cb26w Running us-central1-c gke-yijing-default-pool-3e14ea35-klwc/10.128.0.26 Nov 03,2023 17:10 UTC+0800

Resources Allocation: COMPONENT DEDICATED CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE-SIZE STORAGE-CLASS mysql false 500m / 500m 1Gi / 1Gi data:20Gi kb-default-sc vtcontroller false 500m / 500m 128Mi / 128Mi data:20Gi kb-default-sc vtgate false 500m / 500m

Images: COMPONENT TYPE IMAGE mysql mysql registry.cn-hangzhou.aliyuncs.com/apecloud/apecloud-mysql-server:8.0.30-5.beta1.20230802.g5b589f1.12 vtcontroller vtcontroller registry.cn-hangzhou.aliyuncs.com/apecloud/apecloud-mysql-scale:0.1.1 vtgate vtgate registry.cn-hangzhou.aliyuncs.com/apecloud/apecloud-mysql-scale:0.1.1

Data Protection: AUTO-BACKUP BACKUP-SCHEDULE TYPE BACKUP-TTL LAST-SCHEDULE RECOVERABLE-TIME Disabled 7d

Show cluster events: kbcli cluster list-events -n default scale-ozhnvf ➜ ~ kbcli cluster describe scale-xjvgmc Name: scale-xjvgmc Created Time: Nov 03,2023 17:10 UTC+0800 NAMESPACE CLUSTER-DEFINITION VERSION STATUS TERMINATION-POLICY default apecloud-mysql ac-mysql-8.0.30 Running DoNotTerminate

Endpoints: COMPONENT MODE INTERNAL EXTERNAL mysql ReadWrite scale-xjvgmc-mysql.default.svc.cluster.local:3306 vtcontroller ReadWrite scale-xjvgmc-vtcontroller.default.svc.cluster.local:2379 scale-xjvgmc-vtcontroller.default.svc.cluster.local:15000 scale-xjvgmc-vtcontroller.default.svc.cluster.local:15999 vtgate ReadWrite scale-xjvgmc-vtgate.default.svc.cluster.local:15001 scale-xjvgmc-vtgate.default.svc.cluster.local:15991 scale-xjvgmc-vtgate.default.svc.cluster.local:15306 scale-xjvgmc-vtgate.default.svc.cluster.local:40000

Topology: COMPONENT INSTANCE ROLE STATUS AZ NODE CREATED-TIME mysql scale-xjvgmc-mysql-0 follower Running us-central1-c gke-yijing-default-pool-3e14ea35-klwc/10.128.0.26 Nov 03,2023 17:10 UTC+0800 mysql scale-xjvgmc-mysql-1 follower Running us-central1-c gke-yijing-default-pool-3e14ea35-84st/10.128.0.31 Nov 03,2023 17:10 UTC+0800 mysql scale-xjvgmc-mysql-2 leader Running us-central1-c gke-yijing-default-pool-3e14ea35-wg54/10.128.0.35 Nov 03,2023 17:10 UTC+0800 vtcontroller scale-xjvgmc-vtcontroller-0 Running us-central1-c gke-yijing-default-pool-3e14ea35-wg54/10.128.0.35 Nov 03,2023 17:10 UTC+0800 vtgate scale-xjvgmc-vtgate-75d9867df6-7j9zz Running us-central1-c gke-yijing-default-pool-3e14ea35-84st/10.128.0.31 Nov 03,2023 17:10 UTC+0800

Resources Allocation: COMPONENT DEDICATED CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE-SIZE STORAGE-CLASS mysql false 500m / 500m 1Gi / 1Gi data:20Gi kb-default-sc vtcontroller false 500m / 500m 128Mi / 128Mi data:20Gi kb-default-sc vtgate false 500m / 500m

Images: COMPONENT TYPE IMAGE mysql mysql registry.cn-hangzhou.aliyuncs.com/apecloud/apecloud-mysql-server:8.0.30-5.beta1.20230802.g5b589f1.12 vtcontroller vtcontroller registry.cn-hangzhou.aliyuncs.com/apecloud/apecloud-mysql-scale:0.1.1 vtgate vtgate registry.cn-hangzhou.aliyuncs.com/apecloud/apecloud-mysql-scale:0.1.1

Data Protection: AUTO-BACKUP BACKUP-SCHEDULE TYPE BACKUP-TTL LAST-SCHEDULE RECOVERABLE-TIME Disabled 7d

Show cluster events: kbcli cluster list-events -n default scale-xjvgmc

2. after upgrade kb to 0.7.0 , seems vtgate pod does not restart

➜ ~ k get pod NAME READY STATUS RESTARTS AGE scale-kveuzq-mysql-0 5/5 Running 0 6m18s scale-kveuzq-mysql-1 5/5 Running 0 5m9s scale-kveuzq-mysql-2 5/5 Running 0 4m14s scale-kveuzq-vtcontroller-0 3/3 Running 0 5m45s scale-kveuzq-vtgate-665d5b798f-x5kgn 1/1 Running 0 26m scale-ozhnvf-mysql-0 5/5 Running 0 5m18s scale-ozhnvf-mysql-1 5/5 Running 0 6m4s scale-ozhnvf-mysql-2 5/5 Running 0 4m34s scale-ozhnvf-vtcontroller-0 3/3 Running 0 5m54s scale-ozhnvf-vtgate-74955cf48b-cb26w 1/1 Running 0 26m scale-xjvgmc-mysql-0 5/5 Running 0 3m27s scale-xjvgmc-mysql-1 5/5 Running 0 5m18s scale-xjvgmc-mysql-2 5/5 Running 0 4m25s scale-xjvgmc-vtcontroller-0 3/3 Running 0 5m38s scale-xjvgmc-vtgate-75d9867df6-7j9zz 1/1 Running 0 26m

3. Two clusters have changed leader to pod1, and all status are ConditionsError when describe cluster

➜ ~ k get cluster NAME CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS AGE scale-kveuzq apecloud-mysql ac-mysql-8.0.30 Delete Running 28m scale-ozhnvf apecloud-mysql ac-mysql-8.0.30 DoNotTerminate Running 28m scale-xjvgmc apecloud-mysql ac-mysql-8.0.30 DoNotTerminate Running 28m

➜ ~ kbcli cluster describe scale-kveuzq Name: scale-kveuzq Created Time: Nov 03,2023 17:10 UTC+0800 NAMESPACE CLUSTER-DEFINITION VERSION STATUS TERMINATION-POLICY default apecloud-mysql ac-mysql-8.0.30 ConditionsError Delete

Endpoints: COMPONENT MODE INTERNAL EXTERNAL mysql ReadWrite scale-kveuzq-mysql.default.svc.cluster.local:3306 vtgate ReadWrite scale-kveuzq-vtgate.default.svc.cluster.local:15001 scale-kveuzq-vtgate.default.svc.cluster.local:15991 scale-kveuzq-vtgate.default.svc.cluster.local:15306 scale-kveuzq-vtgate.default.svc.cluster.local:40000

Topology: COMPONENT INSTANCE ROLE STATUS AZ NODE CREATED-TIME mysql scale-kveuzq-mysql-0 leader Running us-central1-c gke-yijing-default-pool-3e14ea35-klwc/10.128.0.26 Nov 03,2023 17:30 UTC+0800 mysql scale-kveuzq-mysql-1 follower Running us-central1-c gke-yijing-default-pool-3e14ea35-84st/10.128.0.31 Nov 03,2023 17:31 UTC+0800 mysql scale-kveuzq-mysql-2 follower Running us-central1-c gke-yijing-default-pool-3e14ea35-wg54/10.128.0.35 Nov 03,2023 17:32 UTC+0800 vtcontroller scale-kveuzq-vtcontroller-0 Running us-central1-c gke-yijing-default-pool-3e14ea35-84st/10.128.0.31 Nov 03,2023 17:31 UTC+0800 vtgate scale-kveuzq-vtgate-665d5b798f-x5kgn Running us-central1-c gke-yijing-default-pool-3e14ea35-klwc/10.128.0.26 Nov 03,2023 17:10 UTC+0800

Resources Allocation: COMPONENT DEDICATED CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE-SIZE STORAGE-CLASS mysql false 500m / 500m 1Gi / 1Gi data:20Gi kb-default-sc vtcontroller false 500m / 500m 128Mi / 128Mi data:20Gi kb-default-sc vtgate false 500m / 500m

Images: COMPONENT TYPE IMAGE mysql mysql infracreate-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/apecloud-mysql-server:8.0.30-5.beta2.20230830.g4ca1eb8.13 vtcontroller vtcontroller infracreate-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/apecloud-mysql-scale:0.1.1 vtgate vtgate registry.cn-hangzhou.aliyuncs.com/apecloud/apecloud-mysql-scale:0.1.1

Data Protection: BACKUP-REPO AUTO-BACKUP BACKUP-SCHEDULE BACKUP-METHOD BACKUP-RETENTION

Show cluster events: kbcli cluster list-events -n default scale-kveuzq ➜ ~ kbcli cluster describe scale-ozhnvf Name: scale-ozhnvf Created Time: Nov 03,2023 17:10 UTC+0800 NAMESPACE CLUSTER-DEFINITION VERSION STATUS TERMINATION-POLICY default apecloud-mysql ac-mysql-8.0.30 ConditionsError DoNotTerminate

Endpoints: COMPONENT MODE INTERNAL EXTERNAL mysql ReadWrite scale-ozhnvf-mysql.default.svc.cluster.local:3306 vtgate ReadWrite scale-ozhnvf-vtgate.default.svc.cluster.local:15001 scale-ozhnvf-vtgate.default.svc.cluster.local:15991 scale-ozhnvf-vtgate.default.svc.cluster.local:15306 scale-ozhnvf-vtgate.default.svc.cluster.local:40000

Topology: COMPONENT INSTANCE ROLE STATUS AZ NODE CREATED-TIME mysql scale-ozhnvf-mysql-0 leader Running us-central1-c gke-yijing-default-pool-3e14ea35-klwc/10.128.0.26 Nov 03,2023 17:31 UTC+0800 mysql scale-ozhnvf-mysql-1 follower Running us-central1-c gke-yijing-default-pool-3e14ea35-84st/10.128.0.31 Nov 03,2023 17:30 UTC+0800 mysql scale-ozhnvf-mysql-2 follower Running us-central1-c gke-yijing-default-pool-3e14ea35-wg54/10.128.0.35 Nov 03,2023 17:32 UTC+0800 vtcontroller scale-ozhnvf-vtcontroller-0 Running us-central1-c gke-yijing-default-pool-3e14ea35-wg54/10.128.0.35 Nov 03,2023 17:31 UTC+0800 vtgate scale-ozhnvf-vtgate-74955cf48b-cb26w Running us-central1-c gke-yijing-default-pool-3e14ea35-klwc/10.128.0.26 Nov 03,2023 17:10 UTC+0800

Resources Allocation: COMPONENT DEDICATED CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE-SIZE STORAGE-CLASS mysql false 500m / 500m 1Gi / 1Gi data:20Gi kb-default-sc vtcontroller false 500m / 500m 128Mi / 128Mi data:20Gi kb-default-sc vtgate false 500m / 500m

Images: COMPONENT TYPE IMAGE mysql mysql infracreate-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/apecloud-mysql-server:8.0.30-5.beta2.20230830.g4ca1eb8.13 vtcontroller vtcontroller infracreate-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/apecloud-mysql-scale:0.1.1 vtgate vtgate registry.cn-hangzhou.aliyuncs.com/apecloud/apecloud-mysql-scale:0.1.1

Data Protection: BACKUP-REPO AUTO-BACKUP BACKUP-SCHEDULE BACKUP-METHOD BACKUP-RETENTION

Show cluster events: kbcli cluster list-events -n default scale-ozhnvf ➜ ~ kbcli cluster describe scale-xjvgmc Name: scale-xjvgmc Created Time: Nov 03,2023 17:10 UTC+0800 NAMESPACE CLUSTER-DEFINITION VERSION STATUS TERMINATION-POLICY default apecloud-mysql ac-mysql-8.0.30 ConditionsError DoNotTerminate

Endpoints: COMPONENT MODE INTERNAL EXTERNAL mysql ReadWrite scale-xjvgmc-mysql.default.svc.cluster.local:3306 vtgate ReadWrite scale-xjvgmc-vtgate.default.svc.cluster.local:15001 scale-xjvgmc-vtgate.default.svc.cluster.local:15991 scale-xjvgmc-vtgate.default.svc.cluster.local:15306 scale-xjvgmc-vtgate.default.svc.cluster.local:40000

Topology: COMPONENT INSTANCE ROLE STATUS AZ NODE CREATED-TIME mysql scale-xjvgmc-mysql-0 follower Running us-central1-c gke-yijing-default-pool-3e14ea35-klwc/10.128.0.26 Nov 03,2023 17:33 UTC+0800 mysql scale-xjvgmc-mysql-1 follower Running us-central1-c gke-yijing-default-pool-3e14ea35-84st/10.128.0.31 Nov 03,2023 17:31 UTC+0800 mysql scale-xjvgmc-mysql-2 leader Running us-central1-c gke-yijing-default-pool-3e14ea35-wg54/10.128.0.35 Nov 03,2023 17:32 UTC+0800 vtcontroller scale-xjvgmc-vtcontroller-0 Running us-central1-c gke-yijing-default-pool-3e14ea35-wg54/10.128.0.35 Nov 03,2023 17:31 UTC+0800 vtgate scale-xjvgmc-vtgate-75d9867df6-7j9zz Running us-central1-c gke-yijing-default-pool-3e14ea35-84st/10.128.0.31 Nov 03,2023 17:10 UTC+0800

Resources Allocation: COMPONENT DEDICATED CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE-SIZE STORAGE-CLASS mysql false 500m / 500m 1Gi / 1Gi data:20Gi kb-default-sc vtcontroller false 500m / 500m 128Mi / 128Mi data:20Gi kb-default-sc vtgate false 500m / 500m

Images: COMPONENT TYPE IMAGE mysql mysql infracreate-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/apecloud-mysql-server:8.0.30-5.beta2.20230830.g4ca1eb8.13 vtcontroller vtcontroller infracreate-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/apecloud-mysql-scale:0.1.1 vtgate vtgate registry.cn-hangzhou.aliyuncs.com/apecloud/apecloud-mysql-scale:0.1.1

Data Protection: BACKUP-REPO AUTO-BACKUP BACKUP-SCHEDULE BACKUP-METHOD BACKUP-RETENTION

Show cluster events: kbcli cluster list-events -n default scale-xjvgmc

4. The two clusters which has changed leader cannot write with kbcli cluster connect for the status role is not correct

➜ ~ k describe cluster scale-kveuzq Name: scale-kveuzq Namespace: default Labels: app.kubernetes.io/instance=scale-kveuzq app.kubernetes.io/version=8.0.30 clusterdefinition.kubeblocks.io/name=apecloud-mysql clusterversion.kubeblocks.io/name=ac-mysql-8.0.30 helm.sh/chart=apecloud-mysql-cluster-0.6.0-alpha.0 Annotations: kubeblocks.io/extra-env: {"KB_PROXY_ENABLED":"on"} API Version: apps.kubeblocks.io/v1alpha1 Kind: Cluster Metadata: Creation Timestamp: 2023-11-03T09:10:17Z Finalizers: cluster.kubeblocks.io/finalizer Generation: 1 Managed Fields: API Version: apps.kubeblocks.io/v1alpha1 Fields Type: FieldsV1 fieldsV1: f:metadata: f:annotations: .: f:kubeblocks.io/extra-env: f:labels: .: f:app.kubernetes.io/instance: f:app.kubernetes.io/version: f:helm.sh/chart: f:spec: .: f:affinity: .: f:podAntiAffinity: f:tenancy: f:clusterDefinitionRef: f:clusterVersionRef: f:componentSpecs: .: k:{"name":"mysql"}: .: f:componentDefRef: f:enabledLogs: .: v:"error": v:"slow": f:monitor: f:name: f:noCreatePDB: f:replicas: f:resources: .: f:limits: .: f:cpu: f:memory: f:requests: .: f:cpu: f:memory: f:volumeClaimTemplates: k:{"name":"vtcontroller"}: .: f:componentDefRef: f:enabledLogs: .: v:"error": v:"info": v:"warning": f:monitor: f:name: f:noCreatePDB: f:replicas: f:resources: .: f:limits: .: f:cpu: f:memory: f:volumeClaimTemplates: k:{"name":"vtgate"}: .: f:componentDefRef: f:enabledLogs: .: v:"error": v:"info": v:"queryLog": v:"warning": f:monitor: f:name: f:noCreatePDB: f:replicas: f:resources: .: f:limits: .: f:cpu: f:requests: .: f:cpu: f:terminationPolicy: Manager: kbcli Operation: Update Time: 2023-11-03T09:10:17Z API Version: apps.kubeblocks.io/v1alpha1 Fields Type: FieldsV1 fieldsV1: f:metadata: f:finalizers: .: v:"cluster.kubeblocks.io/finalizer": f:labels: f:clusterdefinition.kubeblocks.io/name: f:clusterversion.kubeblocks.io/name: Manager: manager Operation: Update Time: 2023-11-03T09:10:20Z API Version: apps.kubeblocks.io/v1alpha1 Fields Type: FieldsV1 fieldsV1: f:status: .: f:clusterDefGeneration: f:components: .: f:mysql: .: f:consensusSetStatus: .: f:followers: f:leader: .: f:accessMode: f:name: f:pod: f:phase: f:podsReady: f:podsReadyTime: f:vtcontroller: .: f:phase: f:podsReady: f:podsReadyTime: f:vtgate: .: f:phase: f:podsReady: f:podsReadyTime: f:conditions: f:observedGeneration: f:phase: Manager: manager Operation: Update Subresource: status Time: 2023-11-03T09:29:39Z Resource Version: 3940863 UID: 44ed82fa-c20e-46d6-b891-249f8d93593f Spec: Affinity: Pod Anti Affinity: Required Tenancy: SharedNode Cluster Definition Ref: apecloud-mysql Cluster Version Ref: ac-mysql-8.0.30 Component Specs: Component Def Ref: mysql Enabled Logs: slow error Monitor: false Name: mysql No Create PDB: false Replicas: 3 Resources: Limits: Cpu: 0.5 Memory: 1Gi Requests: Cpu: 0.5 Memory: 1Gi Volume Claim Templates: Name: data Spec: Access Modes: ReadWriteOnce Resources: Requests: Storage: 20Gi Component Def Ref: vtcontroller Enabled Logs: error warning info Monitor: false Name: vtcontroller No Create PDB: false Replicas: 1 Resources: Limits: Cpu: 500m Memory: 128Mi Volume Claim Templates: Name: data Spec: Access Modes: ReadWriteOnce Resources: Requests: Storage: 20Gi Component Def Ref: vtgate Enabled Logs: error warning info queryLog Monitor: false Name: vtgate No Create PDB: false Replicas: 1 Resources: Limits: Cpu: 0.5 Requests: Cpu: 0.5 Termination Policy: Delete Status: Cluster Def Generation: 2 Components: Mysql: Consensus Set Status: Followers: Access Mode: Readonly Name: follower Pod: scale-kveuzq-mysql-0 Access Mode: Readonly Name: follower Pod: scale-kveuzq-mysql-1 Leader: Access Mode: ReadWrite Name: leader Pod: scale-kveuzq-mysql-2 Phase: Running Pods Ready: true Pods Ready Time: 2023-11-03T09:11:06Z Vtcontroller: Phase: Running Pods Ready: true Pods Ready Time: 2023-11-03T09:10:42Z Vtgate: Phase: Running Pods Ready: true Pods Ready Time: 2023-11-03T09:11:06Z Conditions: Last Transition Time: 2023-11-03T09:10:18Z Message: The operator has started the provisioning of Cluster: scale-kveuzq Observed Generation: 1 Reason: PreCheckSucceed Status: True Type: ProvisioningStarted Last Transition Time: 2023-11-03T09:29:39Z Message: node action can't be nil Reason: ApplyResourcesFailed Status: False Type: ApplyResources Last Transition Time: 2023-11-03T09:11:06Z Message: all pods of components are ready, waiting for the probe detection successful Reason: AllReplicasReady Status: True Type: ReplicasReady Last Transition Time: 2023-11-03T09:11:20Z Message: Cluster: scale-kveuzq is ready, current phase is Running Reason: ClusterReady Status: True Type: Ready Observed Generation: 1 Phase: Running Events: Type Reason Age From Message

Normal ComponentPhaseTransition 31m (x3 over 31m) cluster-controller Create a new component Normal PreCheckSucceed 31m cluster-controller The operator has started the provisioning of Cluster: scale-kveuzq Normal ApplyResourcesSucceed 31m cluster-controller Successfully applied for resources Warning ReplicasNotReady 31m cluster-controller pods are not ready in Components: [mysql vtgate], refer to related component message in Cluster.status.components Warning ComponentsNotReady 31m cluster-controller pods are unavailable in Components: [mysql vtgate], refer to related component message in Cluster.status.components Normal AllReplicasReady 31m cluster-controller all pods of components are ready, waiting for the probe detection successful Normal WaitingForProbeSuccess 30m (x3 over 31m) cluster-controller Waiting for probe success Normal ComponentPhaseTransition 30m (x3 over 31m) cluster-controller Running: true, PodsReady: true, PodsTimedout: false Normal Running 30m cluster-controller Cluster: scale-kveuzq is ready, current phase is Running Normal ClusterReady 30m cluster-controller Cluster: scale-kveuzq is ready, current phase is Running Normal SysAcctCreate 30m system-account-controller Created accounts for cluster: scale-kveuzq, component: mysql, accounts: kbprobe Normal SysAcctCreate 30m system-account-controller Created accounts for cluster: scale-kveuzq, component: mysql, accounts: kbadmin Normal SysAcctCreate 30m system-account-controller Created accounts for cluster: scale-kveuzq, component: mysql, accounts: kbdataprotection Normal SysAcctCreate 30m system-account-controller Created accounts for cluster: scale-kveuzq, component: mysql, accounts: kbreplicator Normal SysAcctCreate 30m system-account-controller Created accounts for cluster: scale-kveuzq, component: mysql, accounts: kbmonitoring Normal ComponentPhaseTransition 11m (x17 over 12m) cluster-controller component is Updating Warning ApplyResourcesFailed 5m54s (x44 over 12m) cluster-controller node action can't be nil

➜ ~ kbcli cluster connect scale-kveuzq Connect to instance scale-kveuzq-mysql-2: out of scale-kveuzq-mysql-2(leader), scale-kveuzq-mysql-0(follower), scale-kveuzq-mysql-1(follower) Defaulted container "mysql" out of: mysql, metrics, vttablet, kb-checkrole, config-manager mysql: [Warning] Using a password on the command line interface can be insecure. Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 77 Server version: 8.0.30 WeSQL Server - GPL, Release 5, Revision 4ca1eb8

Copyright (c) 2000, 2022, Oracle and/or its affiliates.

Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> use mydb; Database changed mysql> create table t1(c1 int); ERROR 7504 (HY000): The consensus follower is not allowed to to do current operation. mysql>

➜ ~ k describe cluster scale-ozhnvf Name: scale-ozhnvf Namespace: default Labels: app.kubernetes.io/instance=scale-ozhnvf app.kubernetes.io/version=8.0.30 clusterdefinition.kubeblocks.io/name=apecloud-mysql clusterversion.kubeblocks.io/name=ac-mysql-8.0.30 helm.sh/chart=apecloud-mysql-cluster-0.6.0-alpha.0 Annotations: kubeblocks.io/extra-env: {"KB_PROXY_ENABLED":"on"} kubeblocks.io/reconcile: 2023-11-03T09:31:37.250275519Z API Version: apps.kubeblocks.io/v1alpha1 Kind: Cluster Metadata: Creation Timestamp: 2023-11-03T09:10:17Z Finalizers: cluster.kubeblocks.io/finalizer Generation: 1 Managed Fields: API Version: apps.kubeblocks.io/v1alpha1 Fields Type: FieldsV1 fieldsV1: f:metadata: f:annotations: .: f:kubeblocks.io/extra-env: f:labels: .: f:app.kubernetes.io/instance: f:app.kubernetes.io/version: f:helm.sh/chart: f:spec: .: f:affinity: .: f:podAntiAffinity: f:tenancy: f:clusterDefinitionRef: f:clusterVersionRef: f:componentSpecs: .: k:{"name":"mysql"}: .: f:componentDefRef: f:enabledLogs: .: v:"error": v:"slow": f:monitor: f:name: f:noCreatePDB: f:replicas: f:resources: .: f:limits: .: f:cpu: f:memory: f:requests: .: f:cpu: f:memory: f:volumeClaimTemplates: k:{"name":"vtcontroller"}: .: f:componentDefRef: f:enabledLogs: .: v:"error": v:"info": v:"warning": f:monitor: f:name: f:noCreatePDB: f:replicas: f:resources: .: f:limits: .: f:cpu: f:memory: f:volumeClaimTemplates: k:{"name":"vtgate"}: .: f:componentDefRef: f:enabledLogs: .: v:"error": v:"info": v:"queryLog": v:"warning": f:monitor: f:name: f:noCreatePDB: f:replicas: f:resources: .: f:limits: .: f:cpu: f:requests: .: f:cpu: f:terminationPolicy: Manager: kbcli Operation: Update Time: 2023-11-03T09:10:17Z API Version: apps.kubeblocks.io/v1alpha1 Fields Type: FieldsV1 fieldsV1: f:metadata: f:annotations: f:kubeblocks.io/reconcile: f:finalizers: .: v:"cluster.kubeblocks.io/finalizer": f:labels: f:clusterdefinition.kubeblocks.io/name: f:clusterversion.kubeblocks.io/name: Manager: manager Operation: Update Time: 2023-11-03T09:31:37Z API Version: apps.kubeblocks.io/v1alpha1 Fields Type: FieldsV1 fieldsV1: f:status: .: f:clusterDefGeneration: f:components: .: f:mysql: .: f:consensusSetStatus: .: f:followers: f:leader: .: f:accessMode: f:name: f:pod: f:phase: f:podsReady: f:podsReadyTime: f:vtcontroller: .: f:phase: f:podsReady: f:podsReadyTime: f:vtgate: .: f:phase: f:podsReady: f:podsReadyTime: f:conditions: f:observedGeneration: f:phase: Manager: manager Operation: Update Subresource: status Time: 2023-11-03T09:32:22Z Resource Version: 3943325 UID: 5fbe7f70-7fd0-4626-bc80-94634e2bc9e1 Spec: Affinity: Pod Anti Affinity: Required Tenancy: SharedNode Cluster Definition Ref: apecloud-mysql Cluster Version Ref: ac-mysql-8.0.30 Component Specs: Component Def Ref: mysql Enabled Logs: slow error Monitor: false Name: mysql No Create PDB: false Replicas: 3 Resources: Limits: Cpu: 0.5 Memory: 1Gi Requests: Cpu: 0.5 Memory: 1Gi Volume Claim Templates: Name: data Spec: Access Modes: ReadWriteOnce Resources: Requests: Storage: 20Gi Component Def Ref: vtcontroller Enabled Logs: error warning info Monitor: false Name: vtcontroller No Create PDB: false Replicas: 1 Resources: Limits: Cpu: 500m Memory: 128Mi Volume Claim Templates: Name: data Spec: Access Modes: ReadWriteOnce Resources: Requests: Storage: 20Gi Component Def Ref: vtgate Enabled Logs: error warning info queryLog Monitor: false Name: vtgate No Create PDB: false Replicas: 1 Resources: Limits: Cpu: 0.5 Requests: Cpu: 0.5 Termination Policy: DoNotTerminate Status: Cluster Def Generation: 2 Components: Mysql: Consensus Set Status: Followers: Access Mode: Readonly Name: follower Pod: scale-ozhnvf-mysql-0 Access Mode: Readonly Name: follower Pod: scale-ozhnvf-mysql-1 Leader: Access Mode: ReadWrite Name: leader Pod: scale-ozhnvf-mysql-2 Phase: Running Pods Ready: true Pods Ready Time: 2023-11-03T09:11:04Z Vtcontroller: Phase: Running Pods Ready: true Pods Ready Time: 2023-11-03T09:10:37Z Vtgate: Phase: Running Pods Ready: true Pods Ready Time: 2023-11-03T09:11:04Z Conditions: Last Transition Time: 2023-11-03T09:10:17Z Message: The operator has started the provisioning of Cluster: scale-ozhnvf Observed Generation: 1 Reason: PreCheckSucceed Status: True Type: ProvisioningStarted Last Transition Time: 2023-11-03T09:29:33Z Message: node action can't be nil Reason: ApplyResourcesFailed Status: False Type: ApplyResources Last Transition Time: 2023-11-03T09:11:04Z Message: all pods of components are ready, waiting for the probe detection successful Reason: AllReplicasReady Status: True Type: ReplicasReady Last Transition Time: 2023-11-03T09:11:22Z Message: Cluster: scale-ozhnvf is ready, current phase is Running Reason: ClusterReady Status: True Type: Ready Observed Generation: 1 Phase: Running Events: Type Reason Age From Message

Normal ComponentPhaseTransition 33m (x3 over 33m) cluster-controller Create a new component Normal PreCheckSucceed 33m cluster-controller The operator has started the provisioning of Cluster: scale-ozhnvf Warning ReplicasNotReady 33m cluster-controller pods are not ready in Components: [mysql vtgate], refer to related component message in Cluster.status.components Warning ComponentsNotReady 33m cluster-controller pods are unavailable in Components: [mysql vtgate], refer to related component message in Cluster.status.components Normal AllReplicasReady 32m cluster-controller all pods of components are ready, waiting for the probe detection successful Normal WaitingForProbeSuccess 32m (x3 over 32m) cluster-controller Waiting for probe success Warning ApplyResourcesFailed 32m cluster-controller Operation cannot be fulfilled on pods "scale-ozhnvf-mysql-0": the object has been modified; please apply your changes to the latest version and try again Normal ComponentPhaseTransition 32m (x4 over 33m) cluster-controller Running: true, PodsReady: true, PodsTimedout: false Normal ApplyResourcesSucceed 32m (x2 over 33m) cluster-controller Successfully applied for resources Normal ClusterReady 32m cluster-controller Cluster: scale-ozhnvf is ready, current phase is Running Normal Running 32m cluster-controller Cluster: scale-ozhnvf is ready, current phase is Running Normal SysAcctCreate 32m system-account-controller Created accounts for cluster: scale-ozhnvf, component: mysql, accounts: kbdataprotection Normal SysAcctCreate 32m system-account-controller Created accounts for cluster: scale-ozhnvf, component: mysql, accounts: kbreplicator Normal SysAcctCreate 32m system-account-controller Created accounts for cluster: scale-ozhnvf, component: mysql, accounts: kbadmin Normal SysAcctCreate 32m system-account-controller Created accounts for cluster: scale-ozhnvf, component: mysql, accounts: kbmonitoring Normal SysAcctCreate 32m system-account-controller Created accounts for cluster: scale-ozhnvf, component: mysql, accounts: kbprobe Warning ApplyResourcesFailed 14m (x2 over 14m) cluster-controller BackupPolicy.dataprotection.kubeblocks.io "scale-ozhnvf-mysql-backup-policy" is invalid: spec.backupMethods: Required value Warning ApplyResourcesFailed 13m (x6 over 14m) cluster-controller node action can't be nil Normal ComponentPhaseTransition 13m (x17 over 14m) cluster-controller component is Updating Warning RecreatingFailedPod 12m (x3 over 12m) event-controller StatefulSet default/scale-ozhnvf-mysql is recreating failed Pod scale-ozhnvf-mysql-0 ➜ ~ kbcli cluster connect scale-ozhnvf Connect to instance scale-ozhnvf-mysql-2: out of scale-ozhnvf-mysql-2(leader), scale-ozhnvf-mysql-0(follower), scale-ozhnvf-mysql-1(follower) Defaulted container "mysql" out of: mysql, metrics, vttablet, kb-checkrole, config-manager mysql: [Warning] Using a password on the command line interface can be insecure. Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 83 Server version: 8.0.30 WeSQL Server - GPL, Release 5, Revision 4ca1eb8

Copyright (c) 2000, 2022, Oracle and/or its affiliates.

Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> use mydb;\ Database changed mysql> create table t1(c1 int); ERROR 7504 (HY000): The consensus follower is not allowed to to do current operation. mysql>

5. The 3rd cluster which didn't change leader can write normally

➜ ~ kbcli cluster connect scale-ozhnvf Connect to instance scale-ozhnvf-mysql-2: out of scale-ozhnvf-mysql-2(leader), scale-ozhnvf-mysql-0(follower), scale-ozhnvf-mysql-1(follower) Defaulted container "mysql" out of: mysql, metrics, vttablet, kb-checkrole, config-manager mysql: [Warning] Using a password on the command line interface can be insecure. Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 83 Server version: 8.0.30 WeSQL Server - GPL, Release 5, Revision 4ca1eb8

Copyright (c) 2000, 2022, Oracle and/or its affiliates.

Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> use mydb;\ Database changed mysql> create table t1(c1 int); ERROR 7504 (HY000): The consensus follower is not allowed to to do current operation. mysql> exit Bye ➜ ~ k describe cluster scale-xjvgmc Name: scale-xjvgmc Namespace: default Labels: app.kubernetes.io/instance=scale-xjvgmc app.kubernetes.io/version=8.0.30 clusterdefinition.kubeblocks.io/name=apecloud-mysql clusterversion.kubeblocks.io/name=ac-mysql-8.0.30 helm.sh/chart=apecloud-mysql-cluster-0.6.0-alpha.0 Annotations: kubeblocks.io/extra-env: {"KB_PROXY_ENABLED":"on"} kubeblocks.io/reconcile: 2023-11-03T09:31:16.619031433Z API Version: apps.kubeblocks.io/v1alpha1 Kind: Cluster Metadata: Creation Timestamp: 2023-11-03T09:10:08Z Finalizers: cluster.kubeblocks.io/finalizer Generation: 1 Managed Fields: API Version: apps.kubeblocks.io/v1alpha1 Fields Type: FieldsV1 fieldsV1: f:metadata: f:annotations: .: f:kubeblocks.io/extra-env: f:labels: .: f:app.kubernetes.io/instance: f:app.kubernetes.io/version: f:helm.sh/chart: f:spec: .: f:affinity: .: f:podAntiAffinity: f:tenancy: f:clusterDefinitionRef: f:clusterVersionRef: f:componentSpecs: .: k:{"name":"mysql"}: .: f:componentDefRef: f:enabledLogs: .: v:"error": v:"slow": f:monitor: f:name: f:noCreatePDB: f:replicas: f:resources: .: f:limits: .: f:cpu: f:memory: f:requests: .: f:cpu: f:memory: f:volumeClaimTemplates: k:{"name":"vtcontroller"}: .: f:componentDefRef: f:enabledLogs: .: v:"error": v:"info": v:"warning": f:monitor: f:name: f:noCreatePDB: f:replicas: f:resources: .: f:limits: .: f:cpu: f:memory: f:volumeClaimTemplates: k:{"name":"vtgate"}: .: f:componentDefRef: f:enabledLogs: .: v:"error": v:"info": v:"queryLog": v:"warning": f:monitor: f:name: f:noCreatePDB: f:replicas: f:resources: .: f:limits: .: f:cpu: f:requests: .: f:cpu: f:terminationPolicy: Manager: kbcli Operation: Update Time: 2023-11-03T09:10:08Z API Version: apps.kubeblocks.io/v1alpha1 Fields Type: FieldsV1 fieldsV1: f:metadata: f:annotations: f:kubeblocks.io/reconcile: f:finalizers: .: v:"cluster.kubeblocks.io/finalizer": f:labels: f:clusterdefinition.kubeblocks.io/name: f:clusterversion.kubeblocks.io/name: Manager: manager Operation: Update Time: 2023-11-03T09:31:16Z API Version: apps.kubeblocks.io/v1alpha1 Fields Type: FieldsV1 fieldsV1: f:status: .: f:clusterDefGeneration: f:components: .: f:mysql: .: f:consensusSetStatus: .: f:followers: f:leader: .: f:accessMode: f:name: f:pod: f:phase: f:podsReady: f:podsReadyTime: f:vtcontroller: .: f:phase: f:podsReady: f:podsReadyTime: f:vtgate: .: f:phase: f:podsReady: f:podsReadyTime: f:conditions: f:observedGeneration: f:phase: Manager: manager Operation: Update Subresource: status Time: 2023-11-03T09:33:28Z Resource Version: 3944039 UID: 3497da49-446b-4e25-8fef-0c7046334a39 Spec: Affinity: Pod Anti Affinity: Required Tenancy: SharedNode Cluster Definition Ref: apecloud-mysql Cluster Version Ref: ac-mysql-8.0.30 Component Specs: Component Def Ref: mysql Enabled Logs: slow error Monitor: false Name: mysql No Create PDB: false Replicas: 3 Resources: Limits: Cpu: 0.5 Memory: 1Gi Requests: Cpu: 0.5 Memory: 1Gi Volume Claim Templates: Name: data Spec: Access Modes: ReadWriteOnce Resources: Requests: Storage: 20Gi Component Def Ref: vtcontroller Enabled Logs: error warning info Monitor: false Name: vtcontroller No Create PDB: false Replicas: 1 Resources: Limits: Cpu: 500m Memory: 128Mi Volume Claim Templates: Name: data Spec: Access Modes: ReadWriteOnce Resources: Requests: Storage: 20Gi Component Def Ref: vtgate Enabled Logs: error warning info queryLog Monitor: false Name: vtgate No Create PDB: false Replicas: 1 Resources: Limits: Cpu: 0.5 Requests: Cpu: 0.5 Termination Policy: DoNotTerminate Status: Cluster Def Generation: 2 Components: Mysql: Consensus Set Status: Followers: Access Mode: Readonly Name: follower Pod: scale-xjvgmc-mysql-0 Access Mode: Readonly Name: follower Pod: scale-xjvgmc-mysql-1 Leader: Access Mode: ReadWrite Name: leader Pod: scale-xjvgmc-mysql-2 Phase: Running Pods Ready: true Pods Ready Time: 2023-11-03T09:11:05Z Vtcontroller: Phase: Running Pods Ready: true Pods Ready Time: 2023-11-03T09:10:31Z Vtgate: Phase: Running Pods Ready: true Pods Ready Time: 2023-11-03T09:10:44Z Conditions: Last Transition Time: 2023-11-03T09:10:08Z Message: The operator has started the provisioning of Cluster: scale-xjvgmc Observed Generation: 1 Reason: PreCheckSucceed Status: True Type: ProvisioningStarted Last Transition Time: 2023-11-03T09:29:35Z Message: node action can't be nil Reason: ApplyResourcesFailed Status: False Type: ApplyResources Last Transition Time: 2023-11-03T09:11:05Z Message: all pods of components are ready, waiting for the probe detection successful Reason: AllReplicasReady Status: True Type: ReplicasReady Last Transition Time: 2023-11-03T09:11:21Z Message: Cluster: scale-xjvgmc is ready, current phase is Running Reason: ClusterReady Status: True Type: Ready Observed Generation: 1 Phase: Running Events: Type Reason Age From Message

Normal ComponentPhaseTransition 34m (x3 over 34m) cluster-controller Create a new component Normal PreCheckSucceed 34m cluster-controller The operator has started the provisioning of Cluster: scale-xjvgmc Warning ApplyResourcesFailed 34m cluster-controller Operation cannot be fulfilled on pods "scale-xjvgmc-mysql-0": the object has been modified; please apply your changes to the latest version and try again Normal ApplyResourcesSucceed 34m (x2 over 34m) cluster-controller Successfully applied for resources Warning ComponentsNotReady 34m cluster-controller pods are unavailable in Components: [mysql vtgate], refer to related component message in Cluster.status.components Warning ReplicasNotReady 34m cluster-controller pods are not ready in Components: [mysql vtgate], refer to related component message in Cluster.status.components Warning ReplicasNotReady 34m cluster-controller pods are not ready in Components: [mysql], refer to related component message in Cluster.status.components Warning ComponentsNotReady 34m cluster-controller pods are unavailable in Components: [mysql], refer to related component message in Cluster.status.components Normal AllReplicasReady 33m cluster-controller all pods of components are ready, waiting for the probe detection successful Normal WaitingForProbeSuccess 33m (x5 over 33m) cluster-controller Waiting for probe success Normal ComponentPhaseTransition 33m (x3 over 34m) cluster-controller Running: true, PodsReady: true, PodsTimedout: false Normal Running 33m cluster-controller Cluster: scale-xjvgmc is ready, current phase is Running Normal ClusterReady 33m cluster-controller Cluster: scale-xjvgmc is ready, current phase is Running Normal SysAcctCreate 33m system-account-controller Created accounts for cluster: scale-xjvgmc, component: mysql, accounts: kbmonitoring Normal SysAcctCreate 33m system-account-controller Created accounts for cluster: scale-xjvgmc, component: mysql, accounts: kbreplicator Normal SysAcctCreate 33m system-account-controller Created accounts for cluster: scale-xjvgmc, component: mysql, accounts: kbdataprotection Normal SysAcctCreate 33m system-account-controller Created accounts for cluster: scale-xjvgmc, component: mysql, accounts: kbadmin Normal SysAcctCreate 33m system-account-controller Created accounts for cluster: scale-xjvgmc, component: mysql, accounts: kbprobe Warning ApplyResourcesFailed 15m (x2 over 15m) cluster-controller BackupPolicy.dataprotection.kubeblocks.io "scale-xjvgmc-mysql-backup-policy-hscale" is invalid: spec.backupMethods: Required value Normal ComponentPhaseTransition 14m (x17 over 15m) cluster-controller component is Updating Warning RecreatingFailedPod 13m (x5 over 13m) event-controller StatefulSet default/scale-xjvgmc-vtcontroller is recreating failed Pod scale-xjvgmc-vtcontroller-0 Warning ApplyResourcesFailed 3m7s (x53 over 15m) cluster-controller node action can't be nil ➜ ~ kbcli cluster connect scale-xjvgmc Connect to instance scale-xjvgmc-mysql-2: out of scale-xjvgmc-mysql-2(leader), scale-xjvgmc-mysql-0(follower), scale-xjvgmc-mysql-1(follower) Defaulted container "mysql" out of: mysql, metrics, vttablet, kb-checkrole, config-manager mysql: [Warning] Using a password on the command line interface can be insecure. Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 609 Server version: 8.0.30 WeSQL Server - GPL, Release 5, Revision 4ca1eb8

Copyright (c) 2000, 2022, Oracle and/or its affiliates.

Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> use mydb; Database changed mysql> create table t1(c1 int); Query OK, 0 rows affected (0.03 sec)

mysql>

ahjing99 commented 1 year ago

Then I restart the first cluster scale-kveuzq to restart vtgate pod

➜  ~ kbcli cluster restart scale-kveuzq
Please type the name again(separate with white space when more than one): scale-kveuzq
OpsRequest scale-kveuzq-restart-gnzzp created successfully, you can view the progress:
    kbcli cluster describe-ops scale-kveuzq-restart-gnzzp -n default

➜  ~ k get pod | grep scale-kveuzq
scale-kveuzq-mysql-0                   5/5     Running       0          17m
scale-kveuzq-mysql-1                   5/5     Running       0          32s
scale-kveuzq-mysql-2                   5/5     Terminating   0          15m
scale-kveuzq-vtcontroller-0            3/3     Running       0          32s
scale-kveuzq-vtgate-0                  1/1     Running       0          61s
scale-kveuzq-vtgate-576ddb7954-bqwwr   1/1     Running       0          63s
scale-kveuzq-vtgate-665d5b798f-x5kgn   1/1     Terminating   0          38m

The cluster status keep Updating, but now can write since the role is correct 

➜  ~ kbcli cluster describe scale-kveuzq
Name: scale-kveuzq   Created Time: Nov 03,2023 17:10 UTC+0800
NAMESPACE   CLUSTER-DEFINITION   VERSION           STATUS     TERMINATION-POLICY
default     apecloud-mysql       ac-mysql-8.0.30   Updating   Delete

Endpoints:
COMPONENT   MODE        INTERNAL                                              EXTERNAL
mysql       ReadWrite   scale-kveuzq-mysql.default.svc.cluster.local:3306     <none>
vtgate      ReadWrite   scale-kveuzq-vtgate.default.svc.cluster.local:15306   <none>

Topology:
COMPONENT      INSTANCE                               ROLE       STATUS    AZ              NODE                                                CREATED-TIME
mysql          scale-kveuzq-mysql-0                   follower   Running   us-central1-c   gke-yijing-default-pool-3e14ea35-klwc/10.128.0.26   Nov 03,2023 17:49 UTC+0800
mysql          scale-kveuzq-mysql-1                   follower   Running   us-central1-c   gke-yijing-default-pool-3e14ea35-84st/10.128.0.31   Nov 03,2023 17:48 UTC+0800
mysql          scale-kveuzq-mysql-2                   leader     Running   us-central1-c   gke-yijing-default-pool-3e14ea35-wg54/10.128.0.35   Nov 03,2023 17:48 UTC+0800
vtcontroller   scale-kveuzq-vtcontroller-0            <none>     Running   us-central1-c   gke-yijing-default-pool-3e14ea35-84st/10.128.0.31   Nov 03,2023 17:48 UTC+0800
vtgate         scale-kveuzq-vtgate-0                  <none>     Running   us-central1-c   gke-yijing-default-pool-3e14ea35-84st/10.128.0.31   Nov 03,2023 17:47 UTC+0800
vtgate         scale-kveuzq-vtgate-576ddb7954-bqwwr   <none>     Running   us-central1-c   gke-yijing-default-pool-3e14ea35-klwc/10.128.0.26   Nov 03,2023 17:47 UTC+0800

Resources Allocation:
COMPONENT      DEDICATED   CPU(REQUEST/LIMIT)   MEMORY(REQUEST/LIMIT)   STORAGE-SIZE   STORAGE-CLASS
mysql          false       500m / 500m          1Gi / 1Gi               data:20Gi      kb-default-sc
vtcontroller   false       500m / 500m          128Mi / 128Mi           data:20Gi      kb-default-sc
vtgate         false       500m / 500m          <none>                  <none>         <none>

Images:
COMPONENT      TYPE           IMAGE
mysql          mysql          infracreate-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/apecloud-mysql-server:8.0.30-5.beta2.20230830.g4ca1eb8.13
vtcontroller   vtcontroller   infracreate-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/apecloud-mysql-scale:0.1.1
vtgate         vtgate         infracreate-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/apecloud-mysql-scale:0.1.1

Data Protection:
BACKUP-REPO   AUTO-BACKUP   BACKUP-SCHEDULE   BACKUP-METHOD   BACKUP-RETENTION

Show cluster events: kbcli cluster list-events -n default scale-kveuzq

➜  ~ kbcli cluster connect scale-kveuzq
Connect to instance scale-kveuzq-mysql-2: out of scale-kveuzq-mysql-2(leader), scale-kveuzq-mysql-1(follower), scale-kveuzq-mysql-0(follower)
Defaulted container "mysql" out of: mysql, metrics, vttablet, kb-checkrole, config-manager
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 101
Server version: 8.0.30 WeSQL Server - GPL, Release 5, Revision 4ca1eb8

Copyright (c) 2000, 2022, Oracle and/or its affiliates.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> use mydb;
Database changed
mysql> create table t1(c1 int);
Query OK, 0 rows affected (0.02 sec)

mysql>