search

apecloud / kubeblocks

KubeBlocks is an open-source control plane software that runs and manages databases, message queues and other stateful applications on K8s.
https://kubeblocks.io
GNU Affero General Public License v3.0
1.77k stars 156 forks source link

[BUG] cluster delete error configmaps mongo-jucowe-mongodb-mongodb-config not found #6559

Open JashBook opened 5 months ago

JashBook commented 5 months ago

Describe the bug cluster status is always deleting：error configmaps mongo-jucowe-mongodb-mongodb-config not found

kbcli version
Kubernetes: v1.26.6-gke.1700
KubeBlocks: 0.8.2-beta.1
kbcli: 0.8.1

ref： https://github.com/apecloud/kubeblocks/actions/runs/7694417877/job/20966398748

To Reproduce Steps to reproduce the behavior:

  1. create cluster
  2. delete cluster
  3. See error 
    
kubectl get all -l app.kubernetes.io/instance=mongo-jucowe
NAME                                                         COMPLETIONS   DURATION   AGE
job.batch/dp-backup-0-mongo-jucowe-20240129162701-fcc14002   0/1           10h        10h

NAME TYPE CLUSTER STATUS PROGRESS AGE opsrequest.apps.kubeblocks.io/backup-default-mongo-jucowe-20240130000259 Backup mongo-jucowe Failed -/- 11h

NAME CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS AGE cluster.apps.kubeblocks.io/mongo-jucowe mongodb mongodb-6.0 WipeOut Deleting 11h

describe cluster

kubectl describe cluster mongo-jucowe Name: mongo-jucowe Namespace: default Labels: app.kubernetes.io/instance=mongo-jucowe clusterdefinition.kubeblocks.io/name=mongodb clusterversion.kubeblocks.io/name=mongodb-6.0 Annotations: API Version: apps.kubeblocks.io/v1alpha1 Kind: Cluster Metadata: Creation Timestamp: 2024-01-29T15:20:45Z Deletion Grace Period Seconds: 0 Deletion Timestamp: 2024-01-30T02:37:56Z Finalizers: cluster.kubeblocks.io/finalizer Generation: 9 Spec: Affinity: Pod Anti Affinity: Preferred Tenancy: SharedNode Cluster Definition Ref: mongodb Cluster Version Ref: mongodb-6.0 Component Specs: Class Def Ref: Class:
Component Def Ref: mongodb Enabled Logs: running Monitor: true Name: mongodb No Create PDB: false Replicas: 2 Resources: Limits: Cpu: 200m Memory: 644245094400m Requests: Cpu: 200m Memory: 644245094400m Rsm Transform Policy: ToSts Service Account Name: kb-mongo-jucowe Volume Claim Templates: Name: data Spec: Access Modes: ReadWriteOnce Resources: Requests: Storage: 5Gi Monitor: Resources: Cpu: 0 Memory: 0 Storage: Size: 0 Termination Policy: WipeOut Status: Cluster Def Generation: 3 Components: Mongodb: Phase: Running Pods Ready: true Pods Ready Time: 2024-01-29T16:31:09Z Conditions: Last Transition Time: 2024-01-29T15:20:45Z Message: The operator has started the provisioning of Cluster: mongo-jucowe Observed Generation: 8 Reason: PreCheckSucceed Status: True Type: ProvisioningStarted Last Transition Time: 2024-01-30T02:37:57Z Message: Operation cannot be fulfilled on replicatedstatemachines.workloads.kubeblocks.io "mongo-jucowe-mongodb": the object has been modified; please apply your changes to the latest version and try again Reason: ApplyResourcesFailed Status: False Type: ApplyResources Last Transition Time: 2024-01-29T16:31:09Z Message: all pods of components are ready, waiting for the probe detection successful Reason: AllReplicasReady Status: True Type: ReplicasReady Last Transition Time: 2024-01-29T16:31:09Z Message: Cluster: mongo-jucowe is ready, current phase is Running Reason: ClusterReady Status: True Type: Ready Observed Generation: 8 Phase: Deleting Events: Type Reason Age From Message

Normal ApplyResourcesSucceed 41m (x10 over 11h) cluster-controller Successfully applied for resources Normal DeletingCR 4m21s (x1381 over 29m) cluster-controller Deleting : mongo-jucowe

logs kubeblocks

2024-01-30T02:38:14.793Z ERROR Reconciler error {"controller": "configuration", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Configuration", "Configuration": {"name":"mongo-jucowe-mongodb","namespace":"default"}, "namespace": "default", "name": "mongo-jucowe-mongodb", "reconcileID": "85e13bad-2aa4-44a1-8709-fd2c02a3e82e", "error": "configmaps \"mongo-jucowe-mongodb-mongodb-config\" not found", "errorCauses": [{"error": "configmaps \"mongo-jucowe-mongodb-mongodb-config\" not found"}, {"error": "configmaps \"mongo-jucowe-mongodb-mongodb-config\" not found"}]} sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).reconcileHandler /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.2/pkg/internal/controller/controller.go:324 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItem /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.2/pkg/internal/controller/controller.go:265 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).Start.func2.2 /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.2/pkg/internal/controller/controller.go:226 2024-01-30T02:38:35.298Z INFO ConfigurationReconcile failed to run configuration reconcile task. {"controller": "configuration", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Configuration", "Configuration": {"name":"mongo-jucowe-mongodb","namespace":"default"}, "namespace": "default", "name": "mongo-jucowe-mongodb", "reconcileID": "30c542e6-5d45-4624-8bd2-42a463847109", "configuration": {"name":"mongo-jucowe-mongodb","namespace":"default"}} 2024-01-30T02:38:35.298Z ERROR Reconciler error {"controller": "configuration", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Configuration", "Configuration": {"name":"mongo-jucowe-mongodb","namespace":"default"}, "namespace": "default", "name": "mongo-jucowe-mongodb", "reconcileID": "30c542e6-5d45-4624-8bd2-42a463847109", "error": "configmaps \"mongo-jucowe-mongodb-mongodb-config\" not found", "errorCauses": [{"error": "configmaps \"mongo-jucowe-mongodb-mongodb-config\" not found"}, {"error": "configmaps \"mongo-jucowe-mongodb-mongodb-config\" not found"}]} sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).reconcileHandler /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.2/pkg/internal/controller/controller.go:324 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItem /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.2/pkg/internal/controller/controller.go:265 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).Start.func2.2 /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.2/pkg/internal/controller/controller.go:226 2024-01-30T02:39:16.288Z INFO ConfigurationReconcile failed to run configuration reconcile task. {"controller": "configuration", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Configuration", "Configuration": {"name":"mongo-jucowe-mongodb","namespace":"default"}, "namespace": "default", "name": "mongo-jucowe-mongodb", "reconcileID": "3d6761ba-83e3-4ecc-b1e5-192683334081", "configuration": {"name":"mongo-jucowe-mongodb","namespace":"default"}} 2024-01-30T02:39:16.288Z ERROR Reconciler error {"controller": "configuration", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Configuration", "Configuration": {"name":"mongo-jucowe-mongodb","namespace":"default"}, "namespace": "default", "name": "mongo-jucowe-mongodb", "reconcileID": "3d6761ba-83e3-4ecc-b1e5-192683334081", "error": "configmaps \"mongo-jucowe-mongodb-mongodb-config\" not found", "errorCauses": [{"error": "configmaps \"mongo-jucowe-mongodb-mongodb-config\" not found"}, {"error": "configmaps \"mongo-jucowe-mongodb-mongodb-config\" not found"}]} sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).reconcileHandler /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.2/pkg/internal/controller/controller.go:324 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItem /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.2/pkg/internal/controller/controller.go:265 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).Start.func2.2 /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.2/pkg/internal/controller/controller.go:226 2024-01-30T02:40:38.232Z INFO ConfigurationReconcile failed to run configuration reconcile task. {"controller": "configuration", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Configuration", "Configuration": {"name":"mongo-jucowe-mongodb","namespace":"default"}, "namespace": "default", "name": "mongo-jucowe-mongodb", "reconcileID": "74b08f55-4370-4bec-aa41-ac824a3b64ff", "configuration": {"name":"mongo-jucowe-mongodb","namespace":"default"}} 2024-01-30T02:40:38.233Z ERROR Reconciler error {"controller": "configuration", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Configuration", "Configuration": {"name":"mongo-jucowe-mongodb","namespace":"default"}, "namespace": "default", "name": "mongo-jucowe-mongodb", "reconcileID": "74b08f55-4370-4bec-aa41-ac824a3b64ff", "error": "configmaps \"mongo-jucowe-mongodb-mongodb-config\" not found", "errorCauses": [{"error": "configmaps \"mongo-jucowe-mongodb-mongodb-config\" not found"}, {"error": "configmaps \"mongo-jucowe-mongodb-mongodb-config\" not found"}]} sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).reconcileHandler /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.2/pkg/internal/controller/controller.go:324 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItem /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.2/pkg/internal/controller/controller.go:265 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).Start.func2.2 /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.2/pkg/internal/controller/controller.go:226 2024-01-30T02:43:22.097Z INFO ConfigurationReconcile failed to run configuration reconcile task. {"controller": "configuration", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Configuration", "Configuration": {"name":"mongo-jucowe-mongodb","namespace":"default"}, "namespace": "default", "name": "mongo-jucowe-mongodb", "reconcileID": "cdb5e1f5-33de-4bf6-afc7-e98b0c274722", "configuration": {"name":"mongo-jucowe-mongodb","namespace":"default"}} 2024-01-30T02:43:22.097Z ERROR Reconciler error {"controller": "configuration", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Configuration", "Configuration": {"name":"mongo-jucowe-mongodb","namespace":"default"}, "namespace": "default", "name": "mongo-jucowe-mongodb", "reconcileID": "cdb5e1f5-33de-4bf6-afc7-e98b0c274722", "error": "configmaps \"mongo-jucowe-mongodb-mongodb-config\" not found", "errorCauses": [{"error": "configmaps \"mongo-jucowe-mongodb-mongodb-config\" not found"}, {"error": "configmaps \"mongo-jucowe-mongodb-mongodb-config\" not found"}]} sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).reconcileHandler /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.2/pkg/internal/controller/controller.go:324 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItem /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.2/pkg/internal/controller/controller.go:265 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).Start.func2.2 /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.2/pkg/internal/controller/controller.go:226 2024-01-30T02:48:49.806Z INFO ConfigurationReconcile failed to run configuration reconcile task. {"controller": "configuration", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Configuration", "Configuration": {"name":"mongo-jucowe-mongodb","namespace":"default"}, "namespace": "default", "name": "mongo-jucowe-mongodb", "reconcileID": "18e90c1b-9450-47b1-8390-8dbb9f1d710b", "configuration": {"name":"mongo-jucowe-mongodb","namespace":"default"}} 2024-01-30T02:48:49.806Z ERROR Reconciler error {"controller": "configuration", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Configuration", "Configuration": {"name":"mongo-jucowe-mongodb","namespace":"default"}, "namespace": "default", "name": "mongo-jucowe-mongodb", "reconcileID": "18e90c1b-9450-47b1-8390-8dbb9f1d710b", "error": "configmaps \"mongo-jucowe-mongodb-mongodb-config\" not found", "errorCauses": [{"error": "configmaps \"mongo-jucowe-mongodb-mongodb-config\" not found"}, {"error": "configmaps \"mongo-jucowe-mongodb-mongodb-config\" not found"}]} sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).reconcileHandler /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.2/pkg/internal/controller/controller.go:324 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItem /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.2/pkg/internal/controller/controller.go:265 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).Start.func2.2 /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.2/pkg/internal/controller/controller.go:226 2024-01-30T02:59:45.195Z INFO ConfigurationReconcile failed to run configuration reconcile task. {"controller": "configuration", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Configuration", "Configuration": {"name":"mongo-jucowe-mongodb","namespace":"default"}, "namespace": "default", "name": "mongo-jucowe-mongodb", "reconcileID": "c3f66466-80cf-46af-a951-b4473466bd0c", "configuration": {"name":"mongo-jucowe-mongodb","namespace":"default"}} 2024-01-30T02:59:45.195Z ERROR Reconciler error {"controller": "configuration", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Configuration", "Configuration": {"name":"mongo-jucowe-mongodb","namespace":"default"}, "namespace": "default", "name": "mongo-jucowe-mongodb", "reconcileID": "c3f66466-80cf-46af-a951-b4473466bd0c", "error": "configmaps \"mongo-jucowe-mongodb-mongodb-config\" not found", "errorCauses": [{"error": "configmaps \"mongo-jucowe-mongodb-mongodb-config\" not found"}, {"error": "configmaps \"mongo-jucowe-mongodb-mongodb-config\" not found"}]} sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).reconcileHandler /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.2/pkg/internal/controller/controller.go:324 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItem /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.2/pkg/internal/controller/controller.go:265 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2 /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.2/pkg/internal/controller/controller.go:226



**Expected behavior**
A clear and concise description of what you expected to happen.

**Screenshots**
If applicable, add screenshots to help explain your problem.

**Desktop (please complete the following information):**
 - OS: [e.g. iOS]
 - Browser [e.g. chrome, safari]
 - Version [e.g. 22]

**Additional context**
Add any other context about the problem here.
github-actions[bot] commented 4 months ago

This issue has been marked as stale because it has been open for 30 days with no activity