apecloud / kubeblocks

KubeBlocks is an open-source control plane software that runs and manages databases, message queues and other stateful applications on K8s.
https://kubeblocks.io
GNU Affero General Public License v3.0
2.15k stars 175 forks source link

[BUG]kbcli cluster restore mongodb failed #2785

Closed ahjing99 closed 1 year ago

ahjing99 commented 1 year ago

➜ ~ kbcli version Kubernetes: v1.24.6-aliyun.1 KubeBlocks: 0.5.0-beta.2 kbcli: 0.5.0-beta.2

  1. Create mongodb cluster
    
    kbcli cluster create yjtest1         --termination-policy=Halt         --monitor=false --enable-all-logs=false --cluster-definition=mongodb --set-file=test_create_yjtest1.yaml  --namespace default 
    Warning: cluster version is not specified, use the recently created ClusterVersion mongodb-5.0.14
    Cluster yjtest1 created
    check cluster status

➜ ~ k get cd mongodb -o yaml apiVersion: apps.kubeblocks.io/v1alpha1 kind: ClusterDefinition metadata: annotations: meta.helm.sh/release-name: kb-addon-mongodb meta.helm.sh/release-namespace: default creationTimestamp: "2023-04-20T07:24:14Z" finalizers:

2. backup

kbcli cluster backup yjtest1 --backup-type snapshot --namespace default Backup backup-default-yjtest1-20230420160625 created successfully, you can view the progress: kbcli cluster list-backups --name=backup-default-yjtest1-20230420160625 -n default

➜ ~ k get backup NAME TYPE STATUS TOTAL-SIZE DURATION CREATE-TIME COMPLETION-TIME backup-default-yjtest1-20230420160625 snapshot Completed 20Gi 1m22s 2023-04-20T08:06:25Z 2023-04-20T08:07:46Z

3. Restore

kbcli cluster restore yjtest1-backup --backup backup-default-yjtest1-20230420160625 --namespace default

Cluster yjtest1-backup created check cluster status

➜ ~ k get pod NAME READY STATUS RESTARTS AGE kb-addon-alertmanager-webhook-adaptor-5bd87c4697-mxzxz 2/2 Running 0 47m kb-addon-grafana-696c8c8f94-fchlz 3/3 Running 0 48m kb-addon-prometheus-alertmanager-0 2/2 Running 0 48m kb-addon-prometheus-server-0 2/2 Running 0 48m kb-addon-snapshot-controller-5b7c5df74c-xw4qb 1/1 Running 0 46m kubeblocks-f8dc7d4f7-hpdcp 1/1 Running 0 48m yjtest1-backup-replicaset-0 3/3 Running 0 4m28s yjtest1-backup-replicaset-1 3/3 Running 0 4m28s yjtest1-backup-replicaset-2 3/3 Running 0 4m28s yjtest1-replicaset-0 3/3 Running 0 9m50s yjtest1-replicaset-1 3/3 Running 0 9m50s yjtest1-replicaset-2 3/3 Running 0 9m50s

➜ ~ kbcli cluster list NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME yjtest1 default mongodb mongodb-5.0.14 Halt Running Apr 20,2023 16:02 UTC+0800 yjtest1-backup default mongodb mongodb-5.0.14 Halt Failed Apr 20,2023 16:07 UTC+0800

➜ ~ kbcli cluster describe yjtest1-backup Name: yjtest1-backup Created Time: Apr 20,2023 16:07 UTC+0800 NAMESPACE CLUSTER-DEFINITION VERSION STATUS TERMINATION-POLICY default mongodb mongodb-5.0.14 Failed Halt

Endpoints: COMPONENT MODE INTERNAL EXTERNAL replicaset ReadWrite yjtest1-backup-replicaset.default.svc.cluster.local:27017

Topology: COMPONENT INSTANCE ROLE STATUS AZ NODE CREATED-TIME replicaset yjtest1-backup-replicaset-0 Running cn-hangzhou-g cn-hangzhou.172.16.0.137/172.16.0.137 Apr 20,2023 16:07 UTC+0800 replicaset yjtest1-backup-replicaset-1 Running cn-hangzhou-g cn-hangzhou.172.16.0.136/172.16.0.136 Apr 20,2023 16:07 UTC+0800 replicaset yjtest1-backup-replicaset-2 Running cn-hangzhou-g cn-hangzhou.172.16.0.138/172.16.0.138 Apr 20,2023 16:07 UTC+0800

Resources Allocation: COMPONENT DEDICATED CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE-SIZE STORAGE-CLASS replicaset false 0 / 100m 0 / 512Mi data:20Gi alicloud-disk-topology-alltype

Images: COMPONENT TYPE IMAGE replicaset replicaset mongo:5.0.14

Data Protection: AUTO-BACKUP BACKUP-SCHEDULE TYPE BACKUP-TTL LAST-SCHEDULE RECOVERABLE-TIME Disabled 0 18 0 full 7d

Events(last 5 warnings, see more:kbcli cluster list-events -n default yjtest1-backup): TIME TYPE REASON OBJECT MESSAGE Apr 20,2023 16:11 UTC+0800 Warning RoleProbeTimeout Cluster/yjtest1-backup pod role detection timed out in Component: replicaset Apr 20,2023 16:11 UTC+0800 Warning Failed Cluster/yjtest1-backup Cluster: yjtest1-backup is Failed, check according to the components message

➜ ~ k describe cluster yjtest1-backup Name: yjtest1-backup Namespace: default Labels: clusterdefinition.kubeblocks.io/name=mongodb clusterversion.kubeblocks.io/name=mongodb-5.0.14 Annotations: API Version: apps.kubeblocks.io/v1alpha1 Kind: Cluster Metadata: Creation Timestamp: 2023-04-20T08:07:49Z Finalizers: cluster.kubeblocks.io/finalizer Generation: 1 Managed Fields: API Version: apps.kubeblocks.io/v1alpha1 Fields Type: FieldsV1 fieldsV1: f:spec: .: f:affinity: .: f:podAntiAffinity: f:tenancy: f:clusterDefinitionRef: f:clusterVersionRef: f:componentSpecs: .: k:{"name":"replicaset"}: .: f:componentDefRef: f:monitor: f:name: f:replicas: f:resources: .: f:limits: .: f:cpu: f:memory: f:volumeClaimTemplates: f:terminationPolicy: Manager: kbcli Operation: Update Time: 2023-04-20T08:07:49Z API Version: apps.kubeblocks.io/v1alpha1 Fields Type: FieldsV1 fieldsV1: f:metadata: f:finalizers: .: v:"cluster.kubeblocks.io/finalizer": f:labels: .: f:clusterdefinition.kubeblocks.io/name: f:clusterversion.kubeblocks.io/name: Manager: manager Operation: Update Time: 2023-04-20T08:07:49Z API Version: apps.kubeblocks.io/v1alpha1 Fields Type: FieldsV1 fieldsV1: f:status: .: f:clusterDefGeneration: f:components: .: f:replicaset: .: f:consensusSetStatus: .: f:leader: .: f:accessMode: f:name: f:pod: f:message: .: f:Pod/yjtest1-backup-replicaset-0: f:Pod/yjtest1-backup-replicaset-1: f:Pod/yjtest1-backup-replicaset-2: f:phase: f:podsReady: f:podsReadyTime: f:conditions: f:observedGeneration: f:phase: Manager: manager Operation: Update Subresource: status Time: 2023-04-20T08:11:59Z Resource Version: 818229469 UID: 47982246-b6cb-4804-997b-33a5ebeccf79 Spec: Affinity: Pod Anti Affinity: Preferred Tenancy: SharedNode Cluster Definition Ref: mongodb Cluster Version Ref: mongodb-5.0.14 Component Specs: Component Def Ref: replicaset Monitor: false Name: replicaset Replicas: 3 Resources: Limits: Cpu: 100m Memory: 512Mi Volume Claim Templates: Name: data Spec: Access Modes: ReadWriteOnce Resources: Requests: Storage: 20Gi Termination Policy: Halt Status: Cluster Def Generation: 2 Components: Replicaset: Consensus Set Status: Leader: Access Mode: None Name: Pod: Unknown Message: Pod/yjtest1-backup-replicaset-0: Role probe timeout, check whether the application is available Pod/yjtest1-backup-replicaset-1: Role probe timeout, check whether the application is available Pod/yjtest1-backup-replicaset-2: Role probe timeout, check whether the application is available Phase: Failed Pods Ready: true Pods Ready Time: 2023-04-20T08:11:59Z Conditions: Last Transition Time: 2023-04-20T08:07:49Z Message: The operator has started the provisioning of Cluster: yjtest1-backup Observed Generation: 1 Reason: PreCheckSucceed Status: True Type: ProvisioningStarted Last Transition Time: 2023-04-20T08:07:49Z Message: Successfully applied for resources Observed Generation: 1 Reason: ApplyResourcesSucceed Status: True Type: ApplyResources Last Transition Time: 2023-04-20T08:10:59Z Message: all pods of components are ready, waiting for the probe detection successful Reason: AllReplicasReady Status: True Type: ReplicasReady Last Transition Time: 2023-04-20T08:07:49Z Message: pods are unavailable in Components: [replicaset], refer to related component message in Cluster.status.components Reason: ComponentsNotReady Status: False Type: Ready Observed Generation: 1 Phase: Failed Events: Type Reason Age From Message


Normal PreCheckSucceed 5m39s cluster-controller The operator has started the provisioning of Cluster: yjtest1-backup Normal ApplyResourcesSucceed 5m39s cluster-controller Successfully applied for resources Normal AllReplicasReady 2m29s cluster-controller all pods of components are ready, waiting for the probe detection successful Warning RoleProbeTimeout 89s stateful-set-controller pod role detection timed out in Component: replicaset Warning Failed 89s cluster-controller Cluster: yjtest1-backup is Failed, check according to the components message

kbcli Connect cluster failed ➜ ~ kbcli cluster connect yjtest1-backup error: failed to find the instance to connect, please check cluster status

All 3 pods can be connected ➜ ~ k exec -it yjtest1-backup-replicaset-2 -c mongodb sh kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.

mongo

MongoDB shell version v5.0.14 connecting to: mongodb://127.0.0.1:27017/?compressors=disabled&gssapiServiceName=mongodb Implicit session: session { "id" : UUID("c99b8440-eb64-47a2-b518-cfb6737dc081") } MongoDB server version: 5.0.14

Warning: the "mongo" shell has been superseded by "mongosh", which delivers improved usability and compatibility.The "mongo" shell has been deprecated and will be removed in an upcoming release. For installation instructions, see https://docs.mongodb.com/mongodb-shell/install/

Welcome to the MongoDB shell. For interactive help, type "help". For more comprehensive documentation, see https://docs.mongodb.com/ Questions? Try the MongoDB Developer Community Forums https://community.mongodb.com

xuriwuyun commented 1 year ago

MongoDB backup type support snapshot and full, but restore only support full now.