search

apecloud / kubeblocks

KubeBlocks is an open-source control plane software that runs and manages databases, message queues and other stateful applications on K8s.
https://kubeblocks.io
GNU Affero General Public License v3.0
2.05k stars 167 forks source link

[BUG]OceanBase restart failed for ERROR 5156 (HY000) at line 1: tenant already exist #5146

Closed ahjing99 closed 12 months ago

ahjing99 commented 1 year ago

➜ ~ kbcli version Kubernetes: v1.27.3-gke.100 KubeBlocks: 0.7.0-alpha.11 kbcli: 0.7.0-alpha.11

➜  ~ helm install oceanbase kubeblocks/oceanbase --version "0.7.0-alpha.10"
NAME: oceanbase
LAST DEPLOYED: Fri Sep 15 12:57:10 2023
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
/**
 * Copyright (c) 2021 OceanBase
 * OceanBase CE is licensed under Mulan PubL v2.
 * You can use this software according to the terms and conditions of the Mulan PubL v2.
 * You may obtain a copy of Mulan PubL v2 at:
 *          http://license.coscl.org.cn/MulanPubL-2.0
 * THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND,
 * EITHER EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT,
 * MERCHANTABILITY OR FIT FOR A PARTICULAR PURPOSE.
 * See the Mulan PubL v2 for more details.
 */

Unlimited scalable distributed database for data-intensive transactional and real-time operational analytics workloads, with ultra-fast performance that has once achieved world records in the TPC-C benchmark test.

OceanBase has served over 400 customers across the globe and has been supporting all mission critical systems in Alipay.
➜  ~ helm install ocbase-cluster kubeblocks/oceanbase-cluster --version "0.7.0-alpha.10"
NAME: ocbase-cluster
LAST DEPLOYED: Fri Sep 15 12:57:25 2023
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
/**
 * Copyright (c) 2021 OceanBase
 * OceanBase CE is licensed under Mulan PubL v2.
 * You can use this software according to the terms and conditions of the Mulan PubL v2.
 * You may obtain a copy of Mulan PubL v2 at:
 *          http://license.coscl.org.cn/MulanPubL-2.0
 * THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND,
 * EITHER EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT,
 * MERCHANTABILITY OR FIT FOR A PARTICULAR PURPOSE.
 * See the Mulan PubL v2 for more details.
 */

Unlimited scalable distributed database for data-intensive transactional and real-time operational analytics workloads, with ultra-fast performance that has once achieved world records in the TPC-C benchmark test.

OceanBase has served over 400 customers across the globe and has been supporting all mission critical systems in Alipay.

➜  ~ k get cluster
NAME                               CLUSTER-DEFINITION   VERSION                                TERMINATION-POLICY   STATUS    AGE
ocbase-cluster-oceanbase-cluster   oceanbase            oceanbase-4.2.0.0-100010032023083021   Delete               Running   7m15s
➜  ~ kbcli cluster restart ocbase-cluster-oceanbase-cluster
Please type the name again(separate with white space when more than one): ocbase-cluster-oceanbase-cluster
OpsRequest ocbase-cluster-oceanbase-cluster-restart-zw9nt created successfully, you can view the progress:
    kbcli cluster describe-ops ocbase-cluster-oceanbase-cluster-restart-zw9nt -n default

➜  ~ kbcli cluster describe-ops ocbase-cluster-oceanbase-cluster-restart-zw9nt -n default
Spec:
  Name: ocbase-cluster-oceanbase-cluster-restart-zw9nt  NameSpace: default  Cluster: ocbase-cluster-oceanbase-cluster   Type: Restart

Command:
  kbcli cluster restart ocbase-cluster-oceanbase-cluster --components=ob-bundle --namespace=default

Status:
  Start Time:         Sep 15,2023 13:05 UTC+0800
  Completion Time:    Sep 15,2023 13:07 UTC+0800
  Duration:           2m16s
  Status:             Succeed
  Progress:           3/3
                      OBJECT-KEY                                         STATUS    DURATION    MESSAGE
                      Pod/ocbase-cluster-oceanbase-cluster-ob-bundle-0   Succeed   <Unknown>   Successfully restart: Pod/ocbase-cluster-oceanbase-cluster-ob-bundle-0 in Component: ob-bundle
                      Pod/ocbase-cluster-oceanbase-cluster-ob-bundle-1   Succeed   <Unknown>   Successfully restart: Pod/ocbase-cluster-oceanbase-cluster-ob-bundle-1 in Component: ob-bundle
                      Pod/ocbase-cluster-oceanbase-cluster-ob-bundle-2   Succeed   2m16s       Successfully restart: Pod/ocbase-cluster-oceanbase-cluster-ob-bundle-2 in Component: ob-bundle

Conditions:
LAST-TRANSITION-TIME         TYPE          REASON                            STATUS   MESSAGE
Sep 15,2023 13:05 UTC+0800   Progressing   OpsRequestProgressingStarted      True     Start to process the OpsRequest: ocbase-cluster-oceanbase-cluster-restart-zw9nt in Cluster: ocbase-cluster-oceanbase-cluster
Sep 15,2023 13:05 UTC+0800   Validated     ValidateOpsRequestPassed          True     OpsRequest: ocbase-cluster-oceanbase-cluster-restart-zw9nt is validated
Sep 15,2023 13:05 UTC+0800   Restarting    RestartStarted                    True     Start to restart database in Cluster: ocbase-cluster-oceanbase-cluster
Sep 15,2023 13:07 UTC+0800   Succeed       OpsRequestProcessedSuccessfully   True     Successfully processed the OpsRequest: ocbase-cluster-oceanbase-cluster-restart-zw9nt in Cluster: ocbase-cluster-oceanbase-cluster

Warning Events: <none>

➜  ~ k get pod
NAME                                           READY   STATUS             RESTARTS       AGE
ocbase-cluster-oceanbase-cluster-ob-bundle-0   1/1     Running            0              9m21s
ocbase-cluster-oceanbase-cluster-ob-bundle-1   1/1     Running            0              9m54s
ocbase-cluster-oceanbase-cluster-ob-bundle-2   0/1     CrashLoopBackOff   6 (107s ago)   10m

➜  ~ k describe pod ocbase-cluster-oceanbase-cluster-ob-bundle-2
Name:         ocbase-cluster-oceanbase-cluster-ob-bundle-2
Namespace:    default
Priority:     0
Node:         gke-yjtest-default-pool-cf88b660-psg9/10.128.15.227
Start Time:   Fri, 15 Sep 2023 13:05:31 +0800
Labels:       app.kubernetes.io/component=ob-bundle
              app.kubernetes.io/instance=ocbase-cluster-oceanbase-cluster
              app.kubernetes.io/managed-by=kubeblocks
              app.kubernetes.io/name=oceanbase
              app.kubernetes.io/version=oceanbase-4.2.0.0-100010032023083021
              apps.kubeblocks.io/component-name=ob-bundle
              apps.kubeblocks.io/workload-type=Stateful
              controller-revision-hash=ocbase-cluster-oceanbase-cluster-ob-bundle-578679bbf9
              statefulset.kubernetes.io/pod-name=ocbase-cluster-oceanbase-cluster-ob-bundle-2
Annotations:  apps.kubeblocks.io/component-replicas: 3
              kubeblocks.io/restart: 2023-09-15T05:05:00Z
Status:       Running
IP:           10.104.0.24
IPs:
  IP:           10.104.0.24
Controlled By:  StatefulSet/ocbase-cluster-oceanbase-cluster-ob-bundle
Containers:
  observer-container:
    Container ID:  containerd://bfe4f37f5c5463b0fab994f7be2a71e33279a683002d245055314562fbd1188d
    Image:         oceanbasedev/oceanbase-chart:4.2.0.0-100010032023083021
    Image ID:      docker.io/oceanbasedev/oceanbase-chart@sha256:27a23a89b2cd7c5259ad46913b53192dcd214e9125383eb1fc6ada441cdcd9f7
    Ports:         2881/TCP, 2882/TCP
    Host Ports:    0/TCP, 0/TCP
    Command:
      bash
      -c
      ./scripts/entrypoint.sh
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Fri, 15 Sep 2023 13:13:55 +0800
      Finished:     Fri, 15 Sep 2023 13:14:11 +0800
    Ready:          False
    Restart Count:  6
    Limits:
      cpu:     2
      memory:  10Gi
    Requests:
      cpu:     2
      memory:  10Gi
    Environment Variables from:
      ocbase-cluster-oceanbase-cluster-ob-bundle-env      ConfigMap  Optional: false
      ocbase-cluster-oceanbase-cluster-ob-bundle-rsm-env  ConfigMap  Optional: false
    Environment:
      KB_POD_NAME:               ocbase-cluster-oceanbase-cluster-ob-bundle-2 (v1:metadata.name)
      KB_POD_UID:                 (v1:metadata.uid)
      KB_NAMESPACE:              default (v1:metadata.namespace)
      KB_SA_NAME:                 (v1:spec.serviceAccountName)
      KB_NODENAME:                (v1:spec.nodeName)
      KB_HOST_IP:                 (v1:status.hostIP)
      KB_POD_IP:                  (v1:status.podIP)
      KB_POD_IPS:                 (v1:status.podIPs)
      KB_HOSTIP:                  (v1:status.hostIP)
      KB_PODIP:                   (v1:status.podIP)
      KB_PODIPS:                  (v1:status.podIPs)
      KB_CLUSTER_NAME:           ocbase-cluster-oceanbase-cluster
      KB_COMP_NAME:              ob-bundle
      KB_CLUSTER_COMP_NAME:      ocbase-cluster-oceanbase-cluster-ob-bundle
      KB_CLUSTER_UID_POSTFIX_8:  7af7900b
      KB_POD_FQDN:               $(KB_POD_NAME).$(KB_CLUSTER_COMP_NAME)-headless.$(KB_NAMESPACE).svc
      LD_LIBRARY_PATH:           /home/admin/oceanbase/lib
      ZONE_COUNT:                3
      CLUSTER_NAME:              $(KB_CLUSTER_COMP_NAME)
      DB_ROOT_PASSWORD:          <set to the key 'password' in secret 'ocbase-cluster-oceanbase-cluster-conn-credential'>  Optional: false
    Mounts:
      /home/admin/data-file from data-file (rw)
      /home/admin/data-log from data-log (rw)
      /home/admin/log from log (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-9c69q (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  data-log:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  data-log-ocbase-cluster-oceanbase-cluster-ob-bundle-2
    ReadOnly:   false
  log:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  log-ocbase-cluster-oceanbase-cluster-ob-bundle-2
    ReadOnly:   false
  data-file:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  data-file-ocbase-cluster-oceanbase-cluster-ob-bundle-2
    ReadOnly:   false
  kube-api-access-9c69q:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Guaranteed
Node-Selectors:              <none>
Tolerations:                 kb-data=true:NoSchedule
                             node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                  From               Message
  ----     ------     ----                 ----               -------
  Normal   Scheduled  11m                  default-scheduler  Successfully assigned default/ocbase-cluster-oceanbase-cluster-ob-bundle-2 to gke-yjtest-default-pool-cf88b660-psg9
  Normal   Pulled     7m29s (x5 over 11m)  kubelet            Container image "oceanbasedev/oceanbase-chart:4.2.0.0-100010032023083021" already present on machine
  Normal   Created    7m29s (x5 over 11m)  kubelet            Created container observer-container
  Normal   Started    7m29s (x5 over 11m)  kubelet            Started container observer-container
  Warning  BackOff    60s (x34 over 10m)   kubelet            Back-off restarting failed container observer-container in pod ocbase-cluster-oceanbase-cluster-ob-bundle-2_default(0604d8eb-767a-4dea-b679-5a6d361a5110)

➜  ~ k logs ocbase-cluster-oceanbase-cluster-ob-bundle-2
Getting dynamic replica ips
REPLICA_NUM: 3
nslookup ocbase-cluster-oceanbase-cluster-ob-bundle-0.ocbase-cluster-oceanbase-cluster-ob-bundle-headless
10.104.3.4
nslookup ocbase-cluster-oceanbase-cluster-ob-bundle-1.ocbase-cluster-oceanbase-cluster-ob-bundle-headless
10.104.2.22
nslookup ocbase-cluster-oceanbase-cluster-ob-bundle-2.ocbase-cluster-oceanbase-cluster-ob-bundle-headless
ocbase-cluster-oceanbase-cluster-ob-bundle-2.ocbase-cluster-oceanbase-cluster-ob-bundle-headless is not ready yet
nslookup ocbase-cluster-oceanbase-cluster-ob-bundle-2.ocbase-cluster-oceanbase-cluster-ob-bundle-headless
10.104.0.24
Recovering: True
Others running: False
Prepare config folders
Run as RootService
ZONE_NAME: zone2 RS_LIST: 10.104.3.4:2882:2881;10.104.2.22:2882:2881;10.104.0.24:2882:2881
/home/admin/oceanbase/bin/observer --appname obcluster -r 10.104.3.4:2882:2881;10.104.2.22:2882:2881;10.104.0.24:2882:2881 --cluster_id 1 --zone zone2 --devname eth0 -p 2881 -P 2882 -d /home/admin/oceanbase/store/ -l info -o config_additional_dir=/home/admin/oceanbase/store/etc,cpu_count=16,memory_limit=8G,system_memory=1G,__min_full_resource_pool_memory=1073741824,datafile_size=40G,log_disk_size=40G,net_thread_count=2,stack_size=512K,cache_wash_threshold=1G,schema_history_expire_time=1d,enable_separate_sys_clog=false,enable_merge_by_turn=false,enable_syslog_recycle=true,enable_syslog_wf=false,max_syslog_file_count=4
appname: obcluster
rs list: 10.104.3.4:2882:2881;10.104.2.22:2882:2881;10.104.0.24:2882:2881
cluster id: 1
zone: zone2
devname: eth0
mysql port: 2881
rpc port: 2882
data_dir: /home/admin/oceanbase/store/
log level: info
optstr: config_additional_dir=/home/admin/oceanbase/store/etc,cpu_count=16,memory_limit=8G,system_memory=1G,__min_full_resource_pool_memory=1073741824,datafile_size=40G,log_disk_size=40G,net_thread_count=2,stack_size=512K,cache_wash_threshold=1G,schema_history_expire_time=1d,enable_separate_sys_clog=false,enable_merge_by_turn=false,enable_syslog_recycle=true,enable_syslog_wf=false,max_syslog_file_count=4
Choose the last RS to bootstrap cluster
hostname.svc: ocbase-cluster-oceanbase-cluster-ob-bundle-0.ocbase-cluster-oceanbase-cluster-ob-bundle-headless ip: 10.104.3.4
hostname.svc: ocbase-cluster-oceanbase-cluster-ob-bundle-1.ocbase-cluster-oceanbase-cluster-ob-bundle-headless ip: 10.104.2.22
hostname.svc: ocbase-cluster-oceanbase-cluster-ob-bundle-2.ocbase-cluster-oceanbase-cluster-ob-bundle-headless ip: 10.104.0.24
Replica ocbase-cluster-oceanbase-cluster-ob-bundle-2.ocbase-cluster-oceanbase-cluster-ob-bundle-headless is not up yet
SET SESSION ob_query_timeout=1000000000;
ALTER SYSTEM BOOTSTRAP ZONE 'zone0' SERVER '10.104.3.4:2882',ZONE 'zone1' SERVER '10.104.2.22:2882',ZONE 'zone2' SERVER '10.104.0.24:2882';
ERROR 5156 (HY000) at line 1: tenant already exist
Bootstrap failed, please check the store

➜  ~ k describe cluster ocbase-cluster-oceanbase-cluster
Name:         ocbase-cluster-oceanbase-cluster
Namespace:    default
Labels:       app.kubernetes.io/instance=ocbase-cluster
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=oceanbase-cluster
              app.kubernetes.io/version=4.2.0.0-100010032023083021
              clusterdefinition.kubeblocks.io/name=oceanbase
              clusterversion.kubeblocks.io/name=oceanbase-4.2.0.0-100010032023083021
              helm.sh/chart=oceanbase-cluster-0.7.0-alpha.10
Annotations:  kubeblocks.io/reconcile: 2023-09-15T05:20:35.406862131Z
              meta.helm.sh/release-name: ocbase-cluster
              meta.helm.sh/release-namespace: default
API Version:  apps.kubeblocks.io/v1alpha1
Kind:         Cluster
Metadata:
  Creation Timestamp:  2023-09-15T04:57:34Z
  Finalizers:
    cluster.kubeblocks.io/finalizer
  Generation:  4
  Managed Fields:
    API Version:  apps.kubeblocks.io/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:meta.helm.sh/release-name:
          f:meta.helm.sh/release-namespace:
        f:labels:
          .:
          f:app.kubernetes.io/instance:
          f:app.kubernetes.io/managed-by:
          f:app.kubernetes.io/name:
          f:app.kubernetes.io/version:
          f:helm.sh/chart:
      f:spec:
        .:
        f:clusterDefinitionRef:
        f:clusterVersionRef:
        f:componentSpecs:
          .:
          k:{"name":"ob-bundle"}:
            .:
            f:componentDefRef:
            f:monitor:
            f:name:
            f:noCreatePDB:
            f:serviceAccountName:
            f:volumeClaimTemplates:
        f:terminationPolicy:
    Manager:      helm
    Operation:    Update
    Time:         2023-09-15T04:57:34Z
    API Version:  apps.kubeblocks.io/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        .:
        f:clusterDefGeneration:
        f:components:
          .:
          f:ob-bundle:
            .:
            f:message:
              .:
              f:Pod/ocbase-cluster-oceanbase-cluster-ob-bundle-2:
            f:phase:
            f:podsReady:
            f:podsReadyTime:
        f:conditions:
        f:observedGeneration:
        f:phase:
    Manager:      manager
    Operation:    Update
    Subresource:  status
    Time:         2023-09-15T05:19:42Z
    API Version:  apps.kubeblocks.io/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          f:kubeblocks.io/reconcile:
        f:finalizers:
          .:
          v:"cluster.kubeblocks.io/finalizer":
        f:labels:
          f:clusterdefinition.kubeblocks.io/name:
          f:clusterversion.kubeblocks.io/name:
      f:spec:
        f:componentSpecs:
          k:{"name":"ob-bundle"}:
            f:replicas:
            f:resources:
        f:monitor:
        f:resources:
          .:
          f:cpu:
          f:memory:
        f:storage:
          .:
          f:size:
    Manager:         manager
    Operation:       Update
    Time:            2023-09-15T05:20:35Z
  Resource Version:  98596
  UID:               13e486b6-050b-4596-b8ab-d3957af7900b
Spec:
  Cluster Definition Ref:  oceanbase
  Cluster Version Ref:     oceanbase-4.2.0.0-100010032023083021
  Component Specs:
    Component Def Ref:  ob-bundle
    Monitor:            false
    Name:               ob-bundle
    No Create PDB:      false
    Replicas:           3
    Resources:
    Service Account Name:  obcluster-observer-sa
    Volume Claim Templates:
      Name:  data-file
      Spec:
        Access Modes:
          ReadWriteOnce
        Resources:
          Requests:
            Storage:  50Gi
      Name:           data-log
      Spec:
        Access Modes:
          ReadWriteOnce
        Resources:
          Requests:
            Storage:  50Gi
      Name:           log
      Spec:
        Access Modes:
          ReadWriteOnce
        Resources:
          Requests:
            Storage:  20Gi
  Monitor:
  Resources:
    Cpu:     0
    Memory:  0
  Storage:
    Size:              0
  Termination Policy:  Delete
Status:
  Cluster Def Generation:  1
  Components:
    Ob - Bundle:
      Message:
        Pod/ocbase-cluster-oceanbase-cluster-ob-bundle-2:  back-off 5m0s restarting failed container=observer-container pod=ocbase-cluster-oceanbase-cluster-ob-bundle-2_default(0604d8eb-767a-4dea-b679-5a6d361a5110)
      Phase:                                               Failed
      Pods Ready:                                          false
      Pods Ready Time:                                     2023-09-15T05:19:15Z
  Conditions:
    Last Transition Time:  2023-09-15T04:57:34Z
    Message:               The operator has started the provisioning of Cluster: ocbase-cluster-oceanbase-cluster
    Observed Generation:   4
    Reason:                PreCheckSucceed
    Status:                True
    Type:                  ProvisioningStarted
    Last Transition Time:  2023-09-15T05:05:31Z
    Message:               Successfully applied for resources
    Observed Generation:   4
    Reason:                ApplyResourcesSucceed
    Status:                True
    Type:                  ApplyResources
    Last Transition Time:  2023-09-15T05:19:32Z
    Message:               pods are not ready in Components: [ob-bundle], refer to related component message in Cluster.status.components
    Reason:                ReplicasNotReady
    Status:                False
    Type:                  ReplicasReady
    Last Transition Time:  2023-09-15T05:19:32Z
    Message:               pods are unavailable in Components: [ob-bundle], refer to related component message in Cluster.status.components
    Reason:                ComponentsNotReady
    Status:                False
    Type:                  Ready
  Observed Generation:     4
  Phase:                   Failed
Events:
  Type     Reason                    Age                  From                Message
  ----     ------                    ----                 ----                -------
  Normal   ComponentPhaseTransition  23m                  cluster-controller  Create a new component
  Normal   HorizontalScale           19m                  cluster-controller  start horizontal scale component ob-bundle of cluster ocbase-cluster-oceanbase-cluster from 3 to 0
  Normal   ComponentPhaseTransition  19m                  cluster-controller  Component is Stopping
  Normal   ComponentPhaseTransition  18m                  cluster-controller  Component is Stopped
  Normal   Stopped                   18m                  cluster-controller  Cluster: ocbase-cluster-oceanbase-cluster stopped successfully.
  Warning  ComponentsNotReady        17m                  cluster-controller  pods are unavailable in Components: [ob-bundle], refer to related component message in Cluster.status.components
  Warning  ReplicasNotReady          17m                  cluster-controller  pods are not ready in Components: [ob-bundle], refer to related component message in Cluster.status.components
  Normal   HorizontalScale           17m                  cluster-controller  start horizontal scale component ob-bundle of cluster ocbase-cluster-oceanbase-cluster from 0 to 3
  Normal   ApplyResourcesSucceed     17m (x4 over 23m)    cluster-controller  Successfully applied for resources
  Normal   PreCheckSucceed           17m (x4 over 23m)    cluster-controller  The operator has started the provisioning of Cluster: ocbase-cluster-oceanbase-cluster
  Normal   Running                   16m (x2 over 21m)    cluster-controller  Cluster: ocbase-cluster-oceanbase-cluster is ready, current phase is Running
  Normal   ClusterReady              16m (x2 over 21m)    cluster-controller  Cluster: ocbase-cluster-oceanbase-cluster is ready, current phase is Running
  Normal   AllReplicasReady          16m (x2 over 21m)    cluster-controller  all pods of components are ready, waiting for the probe detection successful
  Normal   ComponentPhaseTransition  16m (x2 over 21m)    cluster-controller  Component is Running
  Normal   ComponentPhaseTransition  13m                  cluster-controller  Component is Failed
  Normal   ComponentPhaseTransition  2m9s (x12 over 17m)  cluster-controller  Component is Updating
  Warning  BackOff                   49s (x12 over 13m)   event-controller    Pod ocbase-cluster-oceanbase-cluster-ob-bundle-2: Back-off restarting failed container observer-container in pod ocbase-cluster-oceanbase-cluster-ob-bundle-2_default(0604d8eb-767a-4dea-b679-5a6d361a5110)
shanshanying commented 1 year ago

Restart

AFAIK, to restart an OBServer, we should: 1.ALTER SYSTEM STOP SERVER, to switch all LEADER partitions on this server to others.

  1. restart OBServer
  2. ALTER SYSTEM START SERVER to add this server back.

    Scale-In

    And for HSacle, we should join/leave an observer, we should: on leave, 

    ALTER SYSTEM DELETE SERVER 'ip:port' [,'ip:port'...] [ZONE [=] 'zone']

    Scale-out

    on join:

    ALTER SYSTEM ADD SERVER '$IP:$PORT' ZONE '$ZONE';
powerfooI commented 12 months ago

:) OceanBase cluster managed by StatefulSet workload does not support restart by now. I'll work on it recently.