percona / percona-server-mongodb-operator

Percona Operator for MongoDB
https://www.percona.com/doc/kubernetes-operator-for-psmongodb/
Apache License 2.0
327 stars 140 forks source link

psmdb-operator crashes when try to delete psmdb-db #1675

Closed evsigneevk closed 1 week ago

evsigneevk commented 1 week ago

Report

After installation of operator and db into a kubernetes cluster via helm-charts I wanted to delete pasmdb-db. psmdb switches to sttoping state and the operator in CrashLoopBackOff state.

psmdb config file ```yaml backup: enabled: false finalizers: - percona.com/delete-psmdb-pods-in-order - percona.com/delete-psmdb-pvc image: tag: 5.0.28-24 replsets: rs0: affinity: antiAffinityTopologyKey: kubernetes.io/hostname arbiter: enabled: false size: 1 expose: enabled: true name: rs0 nonvoting: enabled: false size: 3 podDisruptionBudget: maxUnavailable: 1 resources: limits: cpu: 1 memory: 1G requests: cpu: 300m memory: 0.5G sidecars: - args: - --discovering-mode - --collect-all - --compatible-mode - --mongodb.uri=$(MONGODB_URI) env: - name: EXPORTER_USER valueFrom: secretKeyRef: key: MONGODB_CLUSTER_MONITOR_USER name: staging-psmdb-db-secrets - name: EXPORTER_PASS valueFrom: secretKeyRef: key: MONGODB_CLUSTER_MONITOR_PASSWORD name: staging-psmdb-db-secrets - name: MONGODB_URI value: mongodb://$(EXPORTER_USER):$(EXPORTER_PASS)@127.0.0.1:27017/admin?replicaSet=rs0 image: percona/mongodb_exporter:0.41.1 name: mongodb-exporter ports: - name: metrics containerPort: 9216 size: 3 volumeSpec: pvc: accessModes: - ReadWriteOnce resources: requests: storage: 2Gi storageClassName: standard sharding: balancer: enabled: false configrs: size: 1 enabled: false mongos: size: 1 systemUsers: MONGODB_CLUSTER_ADMIN_PASSWORD: edited MONGODB_CLUSTER_ADMIN_USER: cluster-admin MONGODB_CLUSTER_MONITOR_PASSWORD: edited MONGODB_CLUSTER_MONITOR_USER: monitoring MONGODB_DATABASE_ADMIN_PASSWORD: edited MONGODB_DATABASE_ADMIN_USER: db-admin MONGODB_USER_ADMIN_USER: userAdmin MONGODB_USER_ADMIN_PASSWORD: edited updateStrategy: SmartUpdate upgradeOptions: apply: disabled schedule: 0 2 * * * setFCV: false versionServiceEndpoint: https://check.percona.com ```

More about the problem

Error message ``` INFO Observed a panic in reconciler: runtime error: invalid memory address or nil pointer dereference {"controller": "psmdb-controller", "object": {"name":"threel-psmdb-db", panic: runtime error: invalid memory address or nil pointer dereference [recovered] panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1bc9277] goroutine 120 [running]: sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1() /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:111 +0x1e5 panic({0x1f247c0?, 0x3b1fa60?}) /usr/local/go/src/runtime/panic.go:770 +0x132 github.com/percona/percona-server-mongodb-operator/pkg/psmdb.containerArgs({0x29d78c0, 0xc001008cc0}, 0xc000a48008, 0xc000b0b208, {0xc0007e46f0, 0xc0007e47e0, {0x0, 0x0, 0x0}}, 0x0) /go/src/github.com/percona/percona-server-mongodb-operator/pkg/psmdb/container.go:191 +0x297 github.com/percona/percona-server-mongodb-operator/pkg/psmdb.container({_, _}, _, _, {_, _}, {0xc0007e46f0, 0xc0007e47e0, {0x0, 0x0, ...}}, ...) /go/src/github.com/percona/percona-server-mongodb-operator/pkg/psmdb/container.go:108 +0xa85 github.com/percona/percona-server-mongodb-operator/pkg/psmdb.StatefulSpec({_, _}, _, _, _, {_, _}, {0x0, {0x0, 0x0}}, ...) /go/src/github.com/percona/percona-server-mongodb-operator/pkg/psmdb/statefulset.go:146 +0x140a github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).getStatefulsetFromReplset(0xc000924630, {0x29d78c0, 0xc001008cc0}, 0xc000a48008, 0xc000 /go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/statefulset.go:115 +0x725 github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).reconcileStatefulSet(0xc000924630, {0x29d78c0, 0xc001008cc0}, 0xc000a48008, 0xc000b0b20 /go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/statefulset.go:36 +0x1e8 github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).reconcileReplsets(0xc000924630, {0x29d78c0, 0xc001008cc0}, 0xc000a48008, {0xc0007fa080, /go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:491 +0x45e github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).Reconcile(0xc000924630, {0x29d78c0, 0xc001008cc0}, {{{0xc000b0dc00?, 0x5?}, {0xc000b0db /go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:426 +0x1bcc sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0x29dc6a8?, {0x29d78c0?, 0xc001008cc0?}, {{{0xc000b0dc00?, 0xb?}, {0xc000b0dbf0?, 0x0?}}}) /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:114 +0xb7 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc00082c420, {0x29d78f8, 0xc000941cc0}, {0x1ff38a0, 0xc000668a40}) /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:311 +0x3bc sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc00082c420, {0x29d78f8, 0xc000941cc0}) /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:261 +0x1be sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2() /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:222 +0x79 created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2 in goroutine 104 /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:218 +0x486 ```

Steps to reproduce

  1. install the operator and db replicaset via helm
  2. uninstall the psmdb-db helm chart

Versions

  1. Kubernetes - Server Version: v1.30.4-gke.1348000
  2. Operator - 1.17.0
  3. Database - 5.0.28-24/7.0.12-7

Anything else?

No response

evsigneevk commented 1 week ago

the issue can be closed. the nature of the problem is helm that doesn't update CRDs. fresh installation (manual upgrading CRDs) works fine