search

apecloud / kubeblocks

KubeBlocks is an open-source control plane software that runs and manages databases, message queues and other stateful applications on K8s.
https://kubeblocks.io
GNU Affero General Public License v3.0
2.11k stars 174 forks source link

[BUG]OceanBase Hscale 4->3 in not work #5148

Closed ahjing99 closed 1 year ago

ahjing99 commented 1 year ago

➜ ~ kbcli version Kubernetes: v1.27.3-gke.100 KubeBlocks: 0.7.0-alpha.11 kbcli: 0.7.0-alpha.11

  1. Create cluster
➜  ~ helm install oceanbase kubeblocks/oceanbase --version "0.7.0-alpha.10"
NAME: oceanbase
LAST DEPLOYED: Fri Sep 15 12:57:10 2023
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
/**
 * Copyright (c) 2021 OceanBase
 * OceanBase CE is licensed under Mulan PubL v2.
 * You can use this software according to the terms and conditions of the Mulan PubL v2.
 * You may obtain a copy of Mulan PubL v2 at:
 *          http://license.coscl.org.cn/MulanPubL-2.0
 * THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND,
 * EITHER EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT,
 * MERCHANTABILITY OR FIT FOR A PARTICULAR PURPOSE.
 * See the Mulan PubL v2 for more details.
 */

Unlimited scalable distributed database for data-intensive transactional and real-time operational analytics workloads, with ultra-fast performance that has once achieved world records in the TPC-C benchmark test.

OceanBase has served over 400 customers across the globe and has been supporting all mission critical systems in Alipay.
➜  ~ helm install ob kubeblocks/oceanbase-cluster --version "0.7.0-alpha.10"
NAME: ocbase-cluster
LAST DEPLOYED: Fri Sep 15 12:57:25 2023
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
/**
 * Copyright (c) 2021 OceanBase
 * OceanBase CE is licensed under Mulan PubL v2.
 * You can use this software according to the terms and conditions of the Mulan PubL v2.
 * You may obtain a copy of Mulan PubL v2 at:
 *          http://license.coscl.org.cn/MulanPubL-2.0
 * THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND,
 * EITHER EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT,
 * MERCHANTABILITY OR FIT FOR A PARTICULAR PURPOSE.
 * See the Mulan PubL v2 for more details.
 */

Unlimited scalable distributed database for data-intensive transactional and real-time operational analytics workloads, with ultra-fast performance that has once achieved world records in the TPC-C benchmark test.

OceanBase has served over 400 customers across the globe and has been supporting all mission critical systems in Alipay.
  1. Hscale out 3-4 success 
    
➜  ~ kbcli cluster hscale ob-oceanbase-cluster  --components ob-bundle  --replicas 4
Please type the name again(separate with white space when more than one): ob-oceanbase-cluster
OpsRequest ob-oceanbase-cluster-horizontalscaling-jj57g created successfully, you can view the progress:
kbcli cluster describe-ops ob-oceanbase-cluster-horizontalscaling-jj57g -n default

➜ ~ k get pod NAME READY STATUS RESTARTS AGE ob-oceanbase-cluster-ob-bundle-0 1/1 Running 0 6m12s ob-oceanbase-cluster-ob-bundle-1 1/1 Running 0 5m55s ob-oceanbase-cluster-ob-bundle-2 1/1 Running 0 5m36s ob-oceanbase-cluster-ob-bundle-3 1/1 Running 0 45s ➜ ~ k get cluster NAME CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS AGE ob-oceanbase-cluster oceanbase oceanbase-4.2.0.0-100010032023083021 Delete Running 6m18s

3. Hscale in 4-3 not work

➜ ~ kbcli cluster hscale ob-oceanbase-cluster --components ob-bundle --replicas 3 Please type the name again(separate with white space when more than one): ob-oceanbase-cluster OpsRequest ob-oceanbase-cluster-horizontalscaling-mnr26 created successfully, you can view the progress: kbcli cluster describe-ops ob-oceanbase-cluster-horizontalscaling-mnr26 -n default

➜ ~ k get ops NAME TYPE CLUSTER STATUS PROGRESS AGE ob-oceanbase-cluster-horizontalscaling-jj57g HorizontalScaling ob-oceanbase-cluster Succeed 1/1 10m ob-oceanbase-cluster-horizontalscaling-mnr26 HorizontalScaling ob-oceanbase-cluster Running -/- 8m46s

➜ ~ k describe ops ob-oceanbase-cluster-horizontalscaling-mnr26 Name: ob-oceanbase-cluster-horizontalscaling-mnr26 Namespace: default Labels: app.kubernetes.io/instance=ob-oceanbase-cluster ops.kubeblocks.io/ops-type=HorizontalScaling Annotations: API Version: apps.kubeblocks.io/v1alpha1 Kind: OpsRequest Metadata: Creation Timestamp: 2023-09-15T06:07:18Z Finalizers: opsrequest.kubeblocks.io/finalizer Generate Name: ob-oceanbase-cluster-horizontalscaling- Generation: 2 Managed Fields: API Version: apps.kubeblocks.io/v1alpha1 Fields Type: FieldsV1 fieldsV1: f:metadata: f:generateName: f:spec: .: f:clusterRef: f:horizontalScaling: .: k:{"componentName":"ob-bundle"}: .: f:componentName: f:replicas: f:ttlSecondsBeforeAbort: f:type: Manager: kbcli Operation: Update Time: 2023-09-15T06:07:18Z API Version: apps.kubeblocks.io/v1alpha1 Fields Type: FieldsV1 fieldsV1: f:metadata: f:finalizers: .: v:"opsrequest.kubeblocks.io/finalizer": f:labels: .: f:app.kubernetes.io/instance: f:ops.kubeblocks.io/ops-type: f:ownerReferences: .: k:{"uid":"0ae2b18c-8e1c-4833-835a-457a7ce0f8e7"}: Manager: manager Operation: Update Time: 2023-09-15T06:07:18Z API Version: apps.kubeblocks.io/v1alpha1 Fields Type: FieldsV1 fieldsV1: f:status: .: f:clusterGeneration: f:conditions: .: k:{"type":"HorizontalScaling"}: .: f:lastTransitionTime: f:message: f:reason: f:status: f:type: k:{"type":"Progressing"}: .: f:lastTransitionTime: f:message: f:reason: f:status: f:type: k:{"type":"Validated"}: .: f:lastTransitionTime: f:message: f:reason: f:status: f:type: f:lastConfiguration: .: f:components: .: f:ob-bundle: .: f:replicas: f:targetResources: .: f:pods: f:phase: f:progress: f:startTimestamp: Manager: manager Operation: Update Subresource: status Time: 2023-09-15T06:07:18Z Owner References: API Version: apps.kubeblocks.io/v1alpha1 Kind: Cluster Name: ob-oceanbase-cluster UID: 0ae2b18c-8e1c-4833-835a-457a7ce0f8e7 Resource Version: 127985 UID: 4f7b3e74-0cdb-4740-aad1-926225daddff Spec: Cluster Ref: ob-oceanbase-cluster Horizontal Scaling: Component Name: ob-bundle Replicas: 3 Ttl Seconds Before Abort: 0 Type: HorizontalScaling Status: Cluster Generation: 4 Conditions: Last Transition Time: 2023-09-15T06:07:18Z Message: Start to process the OpsRequest: ob-oceanbase-cluster-horizontalscaling-mnr26 in Cluster: ob-oceanbase-cluster Reason: OpsRequestProgressingStarted Status: True Type: Progressing Last Transition Time: 2023-09-15T06:07:18Z Message: OpsRequest: ob-oceanbase-cluster-horizontalscaling-mnr26 is validated Reason: ValidateOpsRequestPassed Status: True Type: Validated Last Transition Time: 2023-09-15T06:07:18Z Message: Start to horizontal scale replicas in Cluster: ob-oceanbase-cluster Reason: HorizontalScalingStarted Status: True Type: HorizontalScaling Last Configuration: Components: Ob - Bundle: Replicas: 4 Target Resources: Pods: ob-oceanbase-cluster-ob-bundle-2 ob-oceanbase-cluster-ob-bundle-3 ob-oceanbase-cluster-ob-bundle-0 ob-oceanbase-cluster-ob-bundle-1 Phase: Running Progress: -/- Start Timestamp: 2023-09-15T06:07:18Z Events: Type Reason Age From Message

Normal OpsRequestProgressingStarted 10m ops-request-controller Start to process the OpsRequest: ob-oceanbase-cluster-horizontalscaling-mnr26 in Cluster: ob-oceanbase-cluster Normal ValidateOpsRequestPassed 10m ops-request-controller OpsRequest: ob-oceanbase-cluster-horizontalscaling-mnr26 is validated Normal HorizontalScalingStarted 10m ops-request-controller Start to horizontal scale replicas in Cluster: ob-oceanbase-cluster

➜ ~ k describe cluster ob-oceanbase-cluster Name: ob-oceanbase-cluster Namespace: default Labels: app.kubernetes.io/instance=ob app.kubernetes.io/managed-by=Helm app.kubernetes.io/name=oceanbase-cluster app.kubernetes.io/version=4.2.0.0-100010032023083021 clusterdefinition.kubeblocks.io/name=oceanbase clusterversion.kubeblocks.io/name=oceanbase-4.2.0.0-100010032023083021 helm.sh/chart=oceanbase-cluster-0.7.0-alpha.10 Annotations: kubeblocks.io/ops-request: [{"name":"ob-oceanbase-cluster-horizontalscaling-mnr26","type":"HorizontalScaling"}] meta.helm.sh/release-name: ob meta.helm.sh/release-namespace: default API Version: apps.kubeblocks.io/v1alpha1 Kind: Cluster Metadata: Creation Timestamp: 2023-09-15T05:59:50Z Finalizers: cluster.kubeblocks.io/finalizer Generation: 4 Managed Fields: API Version: apps.kubeblocks.io/v1alpha1 Fields Type: FieldsV1 fieldsV1: f:metadata: f:annotations: .: f:meta.helm.sh/release-name: f:meta.helm.sh/release-namespace: f:labels: .: f:app.kubernetes.io/instance: f:app.kubernetes.io/managed-by: f:app.kubernetes.io/name: f:app.kubernetes.io/version: f:helm.sh/chart: f:spec: .: f:clusterDefinitionRef: f:clusterVersionRef: f:componentSpecs: .: k:{"name":"ob-bundle"}: .: f:componentDefRef: f:monitor: f:name: f:noCreatePDB: f:serviceAccountName: f:volumeClaimTemplates: f:terminationPolicy: Manager: helm Operation: Update Time: 2023-09-15T05:59:50Z API Version: apps.kubeblocks.io/v1alpha1 Fields Type: FieldsV1 fieldsV1: f:metadata: f:annotations: f:kubeblocks.io/ops-request: f:finalizers: .: v:"cluster.kubeblocks.io/finalizer": f:labels: f:clusterdefinition.kubeblocks.io/name: f:clusterversion.kubeblocks.io/name: f:spec: f:componentSpecs: k:{"name":"ob-bundle"}: f:replicas: f:resources: f:monitor: f:resources: .: f:cpu: f:memory: f:storage: .: f:size: Manager: manager Operation: Update Time: 2023-09-15T06:07:18Z API Version: apps.kubeblocks.io/v1alpha1 Fields Type: FieldsV1 fieldsV1: f:status: .: f:clusterDefGeneration: f:components: .: f:ob-bundle: .: f:phase: f:podsReady: f:podsReadyTime: f:conditions: f:observedGeneration: f:phase: Manager: manager Operation: Update Subresource: status Time: 2023-09-15T06:07:18Z Resource Version: 127987 UID: 0ae2b18c-8e1c-4833-835a-457a7ce0f8e7 Spec: Cluster Definition Ref: oceanbase Cluster Version Ref: oceanbase-4.2.0.0-100010032023083021 Component Specs: Component Def Ref: ob-bundle Monitor: false Name: ob-bundle No Create PDB: false Replicas: 3 Resources: Service Account Name: obcluster-observer-sa Volume Claim Templates: Name: data-file Spec: Access Modes: ReadWriteOnce Resources: Requests: Storage: 50Gi Name: data-log Spec: Access Modes: ReadWriteOnce Resources: Requests: Storage: 50Gi Name: log Spec: Access Modes: ReadWriteOnce Resources: Requests: Storage: 20Gi Monitor: Resources: Cpu: 0 Memory: 0 Storage: Size: 0 Termination Policy: Delete Status: Cluster Def Generation: 1 Components: Ob - Bundle: Phase: Running Pods Ready: true Pods Ready Time: 2023-09-15T06:06:59Z Conditions: Last Transition Time: 2023-09-15T05:59:50Z Message: The operator has started the provisioning of Cluster: ob-oceanbase-cluster Observed Generation: 4 Reason: PreCheckSucceed Status: True Type: ProvisioningStarted Last Transition Time: 2023-09-15T06:07:18Z Message: port probe-http-port not found Reason: ApplyResourcesFailed Status: False Type: ApplyResources Last Transition Time: 2023-09-15T06:06:59Z Message: all pods of components are ready, waiting for the probe detection successful Reason: AllReplicasReady Status: True Type: ReplicasReady Last Transition Time: 2023-09-15T06:06:59Z Message: Cluster: ob-oceanbase-cluster is ready, current phase is Running Reason: ClusterReady Status: True Type: Ready Observed Generation: 3 Phase: Running Events: Type Reason Age From Message

Normal ComponentPhaseTransition 18m cluster-controller Create a new component Normal PreCheckSucceed 13m (x3 over 18m) cluster-controller The operator has started the provisioning of Cluster: ob-oceanbase-cluster Normal ApplyResourcesSucceed 13m (x3 over 18m) cluster-controller Successfully applied for resources Normal HorizontalScale 13m (x2 over 13m) cluster-controller start horizontal scale component ob-bundle of cluster ob-oceanbase-cluster from 3 to 4 Normal Running 12m (x2 over 17m) cluster-controller Cluster: ob-oceanbase-cluster is ready, current phase is Running Warning ReplicasNotReady 12m (x2 over 13m) cluster-controller pods are not ready in Components: [ob-bundle], refer to related component message in Cluster.status.components Warning ComponentsNotReady 12m (x2 over 13m) cluster-controller pods are unavailable in Components: [ob-bundle], refer to related component message in Cluster.status.components Normal ComponentPhaseTransition 12m (x2 over 13m) cluster-controller Component is Updating Normal ClusterReady 11m (x3 over 17m) cluster-controller Cluster: ob-oceanbase-cluster is ready, current phase is Running Normal ComponentPhaseTransition 11m (x3 over 17m) cluster-controller Component is Running Normal AllReplicasReady 11m (x3 over 17m) cluster-controller all pods of components are ready, waiting for the probe detection successful Warning ApplyResourcesFailed 8s (x19 over 11m) cluster-controller port probe-http-port not found ➜ ~ ➜ ~ k get pod NAME READY STATUS RESTARTS AGE ob-oceanbase-cluster-ob-bundle-0 1/1 Running 0 12m ob-oceanbase-cluster-ob-bundle-1 1/1 Running 0 19m ob-oceanbase-cluster-ob-bundle-2 1/1 Running 0 18m ob-oceanbase-cluster-ob-bundle-3 1/1 Running 0 13m

powerfooI commented 1 year ago

Sorry, I can not reproduce the results.

# powerfooI @ dev in ~/kubeblocks on git:main o [14:51:22] 
$ helm install ob deploy/oceanbase-cluster --version "0.7.0-alpha.10"    
NAME: ob
LAST DEPLOYED: Tue Sep 19 14:53:53 2023
NAMESPACE: oceanbase
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
/**
 * Copyright (c) 2021 OceanBase
 * OceanBase CE is licensed under Mulan PubL v2.
 * You can use this software according to the terms and conditions of the Mulan PubL v2.
 * You may obtain a copy of Mulan PubL v2 at:
 *          http://license.coscl.org.cn/MulanPubL-2.0
 * THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND,
 * EITHER EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT,
 * MERCHANTABILITY OR FIT FOR A PARTICULAR PURPOSE.
 * See the Mulan PubL v2 for more details.
 */

Unlimited scalable distributed database for data-intensive transactional and real-time operational analytics workloads, with ultra-fast performance that has once achieved world records in the TPC-C benchmark test. 

OceanBase has served over 400 customers across the globe and has been supporting all mission critical systems in Alipay.

# powerfooI @ dev in ~/kubeblocks on git:main o [14:53:53] 
$ kbcli cluster hscale ob-oceanbase-cluster  --components ob-bundle  --replicas 4
Please type the name again(separate with white space when more than one): ob-oceanbase-cluster
OpsRequest ob-oceanbase-cluster-horizontalscaling-4lvd4 created successfully, you can view the progress:
        kbcli cluster describe-ops ob-oceanbase-cluster-horizontalscaling-4lvd4 -n oceanbase

# powerfooI @ dev in ~/kubeblocks on git:main o [14:56:36] 
$ kubectl get pods 
NAME                               READY   STATUS    RESTARTS   AGE
ob-oceanbase-cluster-ob-bundle-2   1/1     Running   0          2m45s
ob-oceanbase-cluster-ob-bundle-0   1/1     Running   0          2m55s
ob-oceanbase-cluster-ob-bundle-1   1/1     Running   0          2m50s
ob-oceanbase-cluster-ob-bundle-3   1/1     Running   0          12s

# powerfooI @ dev in ~/kubeblocks on git:main o [14:57:39] 
$ kbcli cluster hscale ob-oceanbase-cluster  --components ob-bundle  --replicas 3
Please type the name again(separate with white space when more than one): ob-oceanbase-cluster█
OpsRequest ob-oceanbase-cluster-horizontalscaling-kc5ng created successfully, you can view the progress:
        kbcli cluster describe-ops ob-oceanbase-cluster-horizontalscaling-kc5ng -n oceanbase

# powerfooI @ dev in ~/kubeblocks on git:main o [14:57:53] 
$ kubectl get pods                                                               
NAME                               READY   STATUS        RESTARTS   AGE
ob-oceanbase-cluster-ob-bundle-0   1/1     Running       0          4m11s
ob-oceanbase-cluster-ob-bundle-1   1/1     Running       0          4m6s
ob-oceanbase-cluster-ob-bundle-2   1/1     Running       0          4m1s
ob-oceanbase-cluster-ob-bundle-3   1/1     Terminating   0          88s

# powerfooI @ dev in ~/kubeblocks on git:main o [14:58:31]
$ kubectl get ops 
NAME                                           TYPE                CLUSTER                STATUS    PROGRESS   AGE
ob-oceanbase-cluster-horizontalscaling-4lvd4   HorizontalScaling   ob-oceanbase-cluster   Succeed   1/1        117s
ob-oceanbase-cluster-horizontalscaling-kc5ng   HorizontalScaling   ob-oceanbase-cluster   Succeed   1/1        40s

# powerfooI @ dev in ~/kubeblocks on git:main o [15:02:37] 
$ kubectl get pods  
NAME                               READY   STATUS    RESTARTS   AGE
ob-oceanbase-cluster-ob-bundle-0   1/1     Running   0          8m44s
ob-oceanbase-cluster-ob-bundle-1   1/1     Running   0          8m39s
ob-oceanbase-cluster-ob-bundle-2   1/1     Running   0          8m34s
ahjing99 commented 1 year ago

I think this might be a new problem introduced since 0.7.0-alpha.11, because I was using kbcli/kubeblocks version alpha.11 in the original steps. And today ,when I tried with alpha.13, I still see the same error

➜ ~ kbcli version Kubernetes: v1.27.3-gke.100 KubeBlocks: 0.7.0-alpha.13 kbcli: 0.7.0-alpha.13

➜  ~ kbcli cluster hscale ob-oceanbase-cluster  --components ob-bundle  --replicas 3
Please type the name again(separate with white space when more than one): ob-oceanbase-cluster
OpsRequest ob-oceanbase-cluster-horizontalscaling-dhdvj created successfully, you can view the progress:
    kbcli cluster describe-ops ob-oceanbase-cluster-horizontalscaling-dhdvj -n default
➜  ~ k get pod
NAME                               READY   STATUS    RESTARTS   AGE
ob-oceanbase-cluster-ob-bundle-0   1/1     Running   0          4m43s
ob-oceanbase-cluster-ob-bundle-1   1/1     Running   0          4m3s
ob-oceanbase-cluster-ob-bundle-2   1/1     Running   0          3m29s
ob-oceanbase-cluster-ob-bundle-3   1/1     Running   0          2m35s

➜  ~ k get ops
NAME                                           TYPE                CLUSTER                STATUS    PROGRESS   AGE
ob-oceanbase-cluster-horizontalscaling-bds6q   HorizontalScaling   ob-oceanbase-cluster   Succeed   1/1        5m54s
ob-oceanbase-cluster-horizontalscaling-dhdvj   HorizontalScaling   ob-oceanbase-cluster   Running   -/-        3m22s

➜  ~ k describe cluster ob-oceanbase-cluster
Name:         ob-oceanbase-cluster
Namespace:    default
Labels:       app.kubernetes.io/instance=ob
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=oceanbase-cluster
              app.kubernetes.io/version=4.2.0.0-100010032023083021
              clusterdefinition.kubeblocks.io/name=oceanbase
              clusterversion.kubeblocks.io/name=oceanbase-4.2.0.0-100010032023083021
              helm.sh/chart=oceanbase-cluster-0.7.0-alpha.13
Annotations:  kubeblocks.io/ops-request: [{"name":"ob-oceanbase-cluster-horizontalscaling-dhdvj","type":"HorizontalScaling"}]
              meta.helm.sh/release-name: ob
              meta.helm.sh/release-namespace: default
API Version:  apps.kubeblocks.io/v1alpha1
Kind:         Cluster
Metadata:
  Creation Timestamp:  2023-09-19T07:50:22Z
  Finalizers:
    cluster.kubeblocks.io/finalizer
  Generation:  4
  Managed Fields:
    API Version:  apps.kubeblocks.io/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:meta.helm.sh/release-name:
          f:meta.helm.sh/release-namespace:
        f:labels:
          .:
          f:app.kubernetes.io/instance:
          f:app.kubernetes.io/managed-by:
          f:app.kubernetes.io/name:
          f:app.kubernetes.io/version:
          f:helm.sh/chart:
      f:spec:
        .:
        f:clusterDefinitionRef:
        f:clusterVersionRef:
        f:componentSpecs:
          .:
          k:{"name":"ob-bundle"}:
            .:
            f:componentDefRef:
            f:monitor:
            f:name:
            f:noCreatePDB:
            f:serviceAccountName:
            f:volumeClaimTemplates:
        f:terminationPolicy:
    Manager:      helm
    Operation:    Update
    Time:         2023-09-19T07:50:22Z
    API Version:  apps.kubeblocks.io/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          f:kubeblocks.io/ops-request:
        f:finalizers:
          .:
          v:"cluster.kubeblocks.io/finalizer":
        f:labels:
          f:clusterdefinition.kubeblocks.io/name:
          f:clusterversion.kubeblocks.io/name:
      f:spec:
        f:componentSpecs:
          k:{"name":"ob-bundle"}:
            f:replicas:
            f:resources:
        f:monitor:
        f:resources:
          .:
          f:cpu:
          f:memory:
        f:storage:
          .:
          f:size:
    Manager:      manager
    Operation:    Update
    Time:         2023-09-19T07:55:02Z
    API Version:  apps.kubeblocks.io/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        .:
        f:clusterDefGeneration:
        f:components:
          .:
          f:ob-bundle:
            .:
            f:phase:
            f:podsReady:
            f:podsReadyTime:
        f:conditions:
        f:observedGeneration:
        f:phase:
    Manager:         manager
    Operation:       Update
    Subresource:     status
    Time:            2023-09-19T07:55:02Z
  Resource Version:  11678
  UID:               6a8865a5-9ff4-4b14-94fc-a3d41ea08897
Spec:
  Cluster Definition Ref:  oceanbase
  Cluster Version Ref:     oceanbase-4.2.0.0-100010032023083021
  Component Specs:
    Component Def Ref:  ob-bundle
    Monitor:            false
    Name:               ob-bundle
    No Create PDB:      false
    Replicas:           3
    Resources:
    Service Account Name:  obcluster-observer-sa
    Volume Claim Templates:
      Name:  data-file
      Spec:
        Access Modes:
          ReadWriteOnce
        Resources:
          Requests:
            Storage:  50Gi
      Name:           data-log
      Spec:
        Access Modes:
          ReadWriteOnce
        Resources:
          Requests:
            Storage:  50Gi
      Name:           log
      Spec:
        Access Modes:
          ReadWriteOnce
        Resources:
          Requests:
            Storage:  20Gi
  Monitor:
  Resources:
    Cpu:     0
    Memory:  0
  Storage:
    Size:              0
  Termination Policy:  Delete
Status:
  Cluster Def Generation:  1
  Components:
    Ob - Bundle:
      Phase:            Running
      Pods Ready:       true
      Pods Ready Time:  2023-09-19T07:53:06Z
  Conditions:
    Last Transition Time:  2023-09-19T07:50:22Z
    Message:               The operator has started the provisioning of Cluster: ob-oceanbase-cluster
    Observed Generation:   4
    Reason:                PreCheckSucceed
    Status:                True
    Type:                  ProvisioningStarted
    Last Transition Time:  2023-09-19T07:55:02Z
    Message:               port probe-http-port not found
    Reason:                ApplyResourcesFailed
    Status:                False
    Type:                  ApplyResources
    Last Transition Time:  2023-09-19T07:53:06Z
    Message:               all pods of components are ready, waiting for the probe detection successful
    Reason:                AllReplicasReady
    Status:                True
    Type:                  ReplicasReady
    Last Transition Time:  2023-09-19T07:53:06Z
    Message:               Cluster: ob-oceanbase-cluster is ready, current phase is Running
    Reason:                ClusterReady
    Status:                True
    Type:                  Ready
  Observed Generation:     3
  Phase:                   Running
Events:
  Type     Reason                    Age                    From                Message
  ----     ------                    ----                   ----                -------
  Normal   ComponentPhaseTransition  8m40s                  cluster-controller  Create a new component
  Normal   ApplyResourcesSucceed     6m32s (x3 over 8m40s)  cluster-controller  Successfully applied for resources
  Warning  ReplicasNotReady          6m32s                  cluster-controller  pods are not ready in Components: [ob-bundle], refer to related component message in Cluster.status.components
  Warning  ComponentsNotReady        6m32s                  cluster-controller  pods are unavailable in Components: [ob-bundle], refer to related component message in Cluster.status.components
  Normal   HorizontalScale           6m32s (x2 over 6m32s)  cluster-controller  start horizontal scale component ob-bundle of cluster ob-oceanbase-cluster from 3 to 4
  Normal   ComponentPhaseTransition  6m32s                  cluster-controller  Component is Updating
  Normal   Running                   5m57s (x2 over 6m50s)  cluster-controller  Cluster: ob-oceanbase-cluster is ready, current phase is Running
  Normal   ClusterReady              5m57s (x2 over 6m50s)  cluster-controller  Cluster: ob-oceanbase-cluster is ready, current phase is Running
  Normal   ComponentPhaseTransition  5m57s (x2 over 6m50s)  cluster-controller  Component is Running
  Normal   AllReplicasReady          5m57s (x2 over 6m50s)  cluster-controller  all pods of components are ready, waiting for the probe detection successful
  Normal   PreCheckSucceed           4m1s (x4 over 8m40s)   cluster-controller  The operator has started the provisioning of Cluster: ob-oceanbase-cluster
  Warning  ApplyResourcesFailed      3m40s (x14 over 4m1s)  cluster-controller  port probe-http-port not found
powerfooI commented 1 year ago

It might be a problem of 0.7.x. kbcli version in my environment shows: 

Kubernetes: v1.27.5+k3s1
KubeBlocks: 0.6.1
kbcli: 0.6.1
ahjing99 commented 1 year ago

Foxlake also has the same problem when hsacle in 

Events:
  Type     Reason                    Age                    From                       Message
  ----     ------                    ----                   ----                       -------
  Normal   ComponentPhaseTransition  10m (x2 over 10m)      cluster-controller         Create a new component
  Normal   SysAcctCreate             7m19s                  system-account-controller  Created accounts for cluster: foxlake-xdhvdk, component: foxlake-server, accounts: kbadmin
  Normal   ComponentPhaseTransition  7m17s                  cluster-controller         Component is Updating
  Normal   ApplyResourcesSucceed     7m17s (x2 over 10m)    cluster-controller         Successfully applied for resources
  Warning  ReplicasNotReady          7m17s (x2 over 9m56s)  cluster-controller         pods are not ready in Components: [foxlake-server], refer to related component message in Cluster.status.components
  Warning  ComponentsNotReady        7m17s (x2 over 9m56s)  cluster-controller         pods are unavailable in Components: [foxlake-server], refer to related component message in Cluster.status.components
  Normal   HorizontalScale           7m17s (x2 over 7m18s)  cluster-controller         start horizontal scale component foxlake-server of cluster foxlake-xdhvdk from 1 to 5
  Normal   AllReplicasReady          116s (x2 over 8m35s)   cluster-controller         all pods of components are ready, waiting for the probe detection successful
  Normal   ClusterReady              116s (x2 over 8m35s)   cluster-controller         Cluster: foxlake-xdhvdk is ready, current phase is Running
  Normal   Running                   116s (x2 over 8m35s)   cluster-controller         Cluster: foxlake-xdhvdk is ready, current phase is Running
  Normal   ComponentPhaseTransition  116s (x3 over 9m56s)   cluster-controller         Component is Running
  Normal   PreCheckSucceed           84s (x3 over 10m)      cluster-controller         The operator has started the provisioning of Cluster: foxlake-xdhvdk
  Warning  ApplyResourcesFailed      1s (x16 over 84s)      cluster-controller         port probe-http-port not found