search

apecloud / kubeblocks

KubeBlocks is an open-source control plane software that runs and manages databases, message queues and other stateful applications on K8s.
https://kubeblocks.io
GNU Affero General Public License v3.0
2.06k stars 167 forks source link

[BUG]Mongo cluster status is always updating after upgrade kb from 0.6.4 to 0.7 #5677

Closed ahjing99 closed 10 months ago

ahjing99 commented 10 months ago
  1. Install kb 0.6.4
  2. create clusters 
    
`kbcli cluster create  mongo-kroyiu             --termination-policy=DoNotTerminate             --monitoring-interval=0 --enable-all-logs=false  --cluster-definition=mongodb --cluster-version=mongodb-4.0 --set cpu=100m,memory=0.5Gi,replicas=3,storage=3Gi  --namespace default `

Cluster mongo-kroyiu created

➜ ~ k get cluster NAME CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS AGE mongo-kroyiu mongodb mongodb-4.0 DoNotTerminate Running 11m

4. Upgrade kb to 0.7

➜ ~ kbcli kubeblocks upgrade --version 0.7.0-beta.13 Current KubeBlocks version 0.6.4. Kubernetes version 1.27.3 Kubernetes provider GKE kbcli version 0.7.0-beta.13 Upgrade KubeBlocks from 0.6.4 to 0.7.0-beta.13 Please type 'Yes/yes' to confirm your operation: yes Add and update repo kubeblocks OK Stop KubeBlocks 0.6.4 OK Stop DataProtection OK ⢿ Upgrading KubeBlocks to 0.7.0-beta.13 Transform breaking changes in /Users/mjingz/kubeblocks-0.6.4-0.7.0-beta.13 ⣽ Upgrading KubeBlocks to 0.7.0-beta.13 Transform breaking changes successfully, remove /Users/mjingz/kubeblocks-0.6.4-0.7.0-beta.13 Upgrading KubeBlocks to 0.7.0-beta.13 OK

KubeBlocks has been upgraded to 0.7.0-beta.13 SUCCESSFULLY!

-> Basic commands for cluster: kbcli cluster create -h # help information about creating a database cluster kbcli cluster list # list all database clusters kbcli cluster describe # get cluster information

-> Uninstall KubeBlocks: kbcli kubeblocks uninstall

5. Mongo cluster status is always updating

➜ ~ kbcli cluster describe mongo-kroyiu Name: mongo-kroyiu Created Time: Oct 30,2023 16:36 UTC+0800 NAMESPACE CLUSTER-DEFINITION VERSION STATUS TERMINATION-POLICY default mongodb mongodb-4.0 Updating DoNotTerminate

Endpoints: COMPONENT MODE INTERNAL EXTERNAL mongodb ReadWrite mongo-kroyiu-mongodb.default.svc.cluster.local:27017

Topology: COMPONENT INSTANCE ROLE STATUS AZ NODE CREATED-TIME mongodb mongo-kroyiu-mongodb-0 primary Running us-central1-c gke-yijing-default-pool-3e14ea35-hqtr/10.128.0.30 Oct 30,2023 16:57 UTC+0800 mongodb mongo-kroyiu-mongodb-1 secondary Running us-central1-c gke-yijing-default-pool-3e14ea35-wg54/10.128.0.35 Oct 30,2023 16:56 UTC+0800 mongodb mongo-kroyiu-mongodb-2 secondary Running us-central1-c gke-yijing-default-pool-3e14ea35-hxpl/10.128.0.28 Oct 30,2023 16:56 UTC+0800

Resources Allocation: COMPONENT DEDICATED CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE-SIZE STORAGE-CLASS mongodb false 100m / 100m 512Mi / 512Mi data:3Gi kb-default-sc

Images: COMPONENT TYPE IMAGE mongodb mongodb infracreate-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/mongo:4.0

Data Protection: BACKUP-REPO AUTO-BACKUP BACKUP-SCHEDULE BACKUP-METHOD BACKUP-RETENTION

Show cluster events: kbcli cluster list-events -n default mongo-kroyiu

➜ ~ k describe cluster mongo-kroyiu Name: mongo-kroyiu Namespace: default Labels: clusterdefinition.kubeblocks.io/name=mongodb clusterversion.kubeblocks.io/name=mongodb-4.0 Annotations: API Version: apps.kubeblocks.io/v1alpha1 Kind: Cluster Metadata: Creation Timestamp: 2023-10-30T08:36:41Z Finalizers: cluster.kubeblocks.io/finalizer Generation: 1 Managed Fields: API Version: apps.kubeblocks.io/v1alpha1 Fields Type: FieldsV1 fieldsV1: f:spec: .: f:affinity: .: f:nodeLabels: f:podAntiAffinity: f:tenancy: f:topologyKeys: f:clusterDefinitionRef: f:clusterVersionRef: f:componentSpecs: .: k:{"name":"mongodb"}: .: f:componentDefRef: f:monitor: f:name: f:noCreatePDB: f:replicas: f:resources: .: f:limits: .: f:cpu: f:memory: f:requests: .: f:cpu: f:memory: f:serviceAccountName: f:volumeClaimTemplates: f:terminationPolicy: f:tolerations: Manager: kbcli Operation: Update Time: 2023-10-30T08:36:41Z API Version: apps.kubeblocks.io/v1alpha1 Fields Type: FieldsV1 fieldsV1: f:metadata: f:finalizers: .: v:"cluster.kubeblocks.io/finalizer": f:labels: .: f:clusterdefinition.kubeblocks.io/name: f:clusterversion.kubeblocks.io/name: Manager: manager Operation: Update Time: 2023-10-30T08:36:41Z API Version: apps.kubeblocks.io/v1alpha1 Fields Type: FieldsV1 fieldsV1: f:status: .: f:clusterDefGeneration: f:components: .: f:mongodb: .: f:consensusSetStatus: .: f:followers: f:leader: .: f:accessMode: f:name: f:pod: f:membersStatus: f:phase: f:podsReady: f:podsReadyTime: f:conditions: f:observedGeneration: f:phase: Manager: manager Operation: Update Subresource: status Time: 2023-10-30T08:57:48Z Resource Version: 168712 UID: d6dbf193-cfbe-4eac-a4c9-1946b931b62c Spec: Affinity: Node Labels: Pod Anti Affinity: Preferred Tenancy: SharedNode Topology Keys: Cluster Definition Ref: mongodb Cluster Version Ref: mongodb-4.0 Component Specs: Component Def Ref: mongodb Monitor: false Name: mongodb No Create PDB: false Replicas: 3 Resources: Limits: Cpu: 100m Memory: 512Mi Requests: Cpu: 100m Memory: 512Mi Service Account Name: kb-mongo-kroyiu Volume Claim Templates: Name: data Spec: Access Modes: ReadWriteOnce Resources: Requests: Storage: 3Gi Termination Policy: DoNotTerminate Tolerations: Status: Cluster Def Generation: 2 Components: Mongodb: Consensus Set Status: Followers: Access Mode: Readonly Name: secondary Pod: mongo-kroyiu-mongodb-2 Access Mode: Readonly Name: secondary Pod: mongo-kroyiu-mongodb-1 Leader: Access Mode: ReadWrite Name: primary Pod: mongo-kroyiu-mongodb-0 Members Status: Pod Name: mongo-kroyiu-mongodb-0 Role: Access Mode: ReadWrite Can Vote: true Is Leader: true Name: primary Pod Name: mongo-kroyiu-mongodb-2 Role: Access Mode: Readonly Can Vote: true Is Leader: false Name: secondary Pod Name: mongo-kroyiu-mongodb-1 Role: Access Mode: Readonly Can Vote: true Is Leader: false Name: secondary Phase: Updating Pods Ready: false Pods Ready Time: 2023-10-30T08:38:15Z Conditions: Last Transition Time: 2023-10-30T08:36:41Z Message: The operator has started the provisioning of Cluster: mongo-kroyiu Observed Generation: 1 Reason: PreCheckSucceed Status: True Type: ProvisioningStarted Last Transition Time: 2023-10-30T08:56:46Z Message: Successfully applied for resources Observed Generation: 1 Reason: ApplyResourcesSucceed Status: True Type: ApplyResources Last Transition Time: 2023-10-30T08:55:39Z Message: pods are not ready in Components: [mongodb], refer to related component message in Cluster.status.components Reason: ReplicasNotReady Status: False Type: ReplicasReady Last Transition Time: 2023-10-30T08:55:39Z Message: pods are unavailable in Components: [mongodb], refer to related component message in Cluster.status.components Reason: ComponentsNotReady Status: False Type: Ready Observed Generation: 1 Phase: Updating Events: Type Reason Age From Message

Normal PreCheckSucceed 38m cluster-controller The operator has started the provisioning of Cluster: mongo-kroyiu Warning ApplyResourcesFailed 38m cluster-controller Operation cannot be fulfilled on pods "mongo-kroyiu-mongodb-0": the object has been modified; please apply your changes to the latest version and try again Normal ComponentPhaseTransition 38m cluster-controller Create a new component Normal ApplyResourcesSucceed 38m (x2 over 38m) cluster-controller Successfully applied for resources Normal AllReplicasReady 36m cluster-controller all pods of components are ready, waiting for the probe detection successful Normal Running 36m cluster-controller Cluster: mongo-kroyiu is ready, current phase is Running Normal WaitingForProbeSuccess 36m (x8 over 36m) cluster-controller Waiting for probe success Normal ComponentPhaseTransition 36m cluster-controller Running: true, PodsReady: true, PodsTimedout: false Normal ClusterReady 36m cluster-controller Cluster: mongo-kroyiu is ready, current phase is Running Warning ApplyResourcesFailed 19m (x2 over 19m) cluster-controller BackupPolicy.dataprotection.kubeblocks.io "mongo-kroyiu-mongodb-backup-policy" is invalid: spec.backupMethods: Required value Normal ComponentPhaseTransition 19m (x3 over 19m) cluster-controller component is Updating Warning ReplicasNotReady 19m cluster-controller pods are not ready in Components: [mongodb], refer to related component message in Cluster.status.components Warning ComponentsNotReady 19m cluster-controller pods are unavailable in Components: [mongodb], refer to related component message in Cluster.status.components Normal ApplyResourcesSucceed 18m (x2 over 19m) cluster-controller Successfully applied for resources Warning ApplyResourcesFailed 18m cluster-controller Operation cannot be fulfilled on pods "mongo-kroyiu-mongodb-1": the object has been modified; please apply your changes to the latest version and try again


➜  ~ k logs kubeblocks-6b9684fccc-8xphr -n kb-system >kb.txt
Defaulted container "manager" out of: manager, tools (init), datascript (init)
[kb.txt](https://github.com/apecloud/kubeblocks/files/13203234/kb.txt)
xuriwuyun commented 10 months ago

image image all pods are ready, but rsm.status.InitReplicas is not equal to 3