Closed evsigneevk closed 1 week ago
After installation of operator and db into a kubernetes cluster via helm-charts I wanted to delete pasmdb-db. psmdb switches to sttoping state and the operator in CrashLoopBackOff state.
sttoping
CrashLoopBackOff
No response
the issue can be closed. the nature of the problem is helm that doesn't update CRDs. fresh installation (manual upgrading CRDs) works fine
Report
After installation of operator and db into a kubernetes cluster via helm-charts I wanted to delete pasmdb-db. psmdb switches to
sttoping
state and the operator inCrashLoopBackOff
state.psmdb config file
```yaml backup: enabled: false finalizers: - percona.com/delete-psmdb-pods-in-order - percona.com/delete-psmdb-pvc image: tag: 5.0.28-24 replsets: rs0: affinity: antiAffinityTopologyKey: kubernetes.io/hostname arbiter: enabled: false size: 1 expose: enabled: true name: rs0 nonvoting: enabled: false size: 3 podDisruptionBudget: maxUnavailable: 1 resources: limits: cpu: 1 memory: 1G requests: cpu: 300m memory: 0.5G sidecars: - args: - --discovering-mode - --collect-all - --compatible-mode - --mongodb.uri=$(MONGODB_URI) env: - name: EXPORTER_USER valueFrom: secretKeyRef: key: MONGODB_CLUSTER_MONITOR_USER name: staging-psmdb-db-secrets - name: EXPORTER_PASS valueFrom: secretKeyRef: key: MONGODB_CLUSTER_MONITOR_PASSWORD name: staging-psmdb-db-secrets - name: MONGODB_URI value: mongodb://$(EXPORTER_USER):$(EXPORTER_PASS)@127.0.0.1:27017/admin?replicaSet=rs0 image: percona/mongodb_exporter:0.41.1 name: mongodb-exporter ports: - name: metrics containerPort: 9216 size: 3 volumeSpec: pvc: accessModes: - ReadWriteOnce resources: requests: storage: 2Gi storageClassName: standard sharding: balancer: enabled: false configrs: size: 1 enabled: false mongos: size: 1 systemUsers: MONGODB_CLUSTER_ADMIN_PASSWORD: edited MONGODB_CLUSTER_ADMIN_USER: cluster-admin MONGODB_CLUSTER_MONITOR_PASSWORD: edited MONGODB_CLUSTER_MONITOR_USER: monitoring MONGODB_DATABASE_ADMIN_PASSWORD: edited MONGODB_DATABASE_ADMIN_USER: db-admin MONGODB_USER_ADMIN_USER: userAdmin MONGODB_USER_ADMIN_PASSWORD: edited updateStrategy: SmartUpdate upgradeOptions: apply: disabled schedule: 0 2 * * * setFCV: false versionServiceEndpoint: https://check.percona.com ```More about the problem
Error message
``` INFO Observed a panic in reconciler: runtime error: invalid memory address or nil pointer dereference {"controller": "psmdb-controller", "object": {"name":"threel-psmdb-db", panic: runtime error: invalid memory address or nil pointer dereference [recovered] panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1bc9277] goroutine 120 [running]: sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1() /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:111 +0x1e5 panic({0x1f247c0?, 0x3b1fa60?}) /usr/local/go/src/runtime/panic.go:770 +0x132 github.com/percona/percona-server-mongodb-operator/pkg/psmdb.containerArgs({0x29d78c0, 0xc001008cc0}, 0xc000a48008, 0xc000b0b208, {0xc0007e46f0, 0xc0007e47e0, {0x0, 0x0, 0x0}}, 0x0) /go/src/github.com/percona/percona-server-mongodb-operator/pkg/psmdb/container.go:191 +0x297 github.com/percona/percona-server-mongodb-operator/pkg/psmdb.container({_, _}, _, _, {_, _}, {0xc0007e46f0, 0xc0007e47e0, {0x0, 0x0, ...}}, ...) /go/src/github.com/percona/percona-server-mongodb-operator/pkg/psmdb/container.go:108 +0xa85 github.com/percona/percona-server-mongodb-operator/pkg/psmdb.StatefulSpec({_, _}, _, _, _, {_, _}, {0x0, {0x0, 0x0}}, ...) /go/src/github.com/percona/percona-server-mongodb-operator/pkg/psmdb/statefulset.go:146 +0x140a github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).getStatefulsetFromReplset(0xc000924630, {0x29d78c0, 0xc001008cc0}, 0xc000a48008, 0xc000 /go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/statefulset.go:115 +0x725 github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).reconcileStatefulSet(0xc000924630, {0x29d78c0, 0xc001008cc0}, 0xc000a48008, 0xc000b0b20 /go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/statefulset.go:36 +0x1e8 github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).reconcileReplsets(0xc000924630, {0x29d78c0, 0xc001008cc0}, 0xc000a48008, {0xc0007fa080, /go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:491 +0x45e github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).Reconcile(0xc000924630, {0x29d78c0, 0xc001008cc0}, {{{0xc000b0dc00?, 0x5?}, {0xc000b0db /go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:426 +0x1bcc sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0x29dc6a8?, {0x29d78c0?, 0xc001008cc0?}, {{{0xc000b0dc00?, 0xb?}, {0xc000b0dbf0?, 0x0?}}}) /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:114 +0xb7 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc00082c420, {0x29d78f8, 0xc000941cc0}, {0x1ff38a0, 0xc000668a40}) /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:311 +0x3bc sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc00082c420, {0x29d78f8, 0xc000941cc0}) /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:261 +0x1be sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2() /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:222 +0x79 created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2 in goroutine 104 /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:218 +0x486 ```Steps to reproduce
Versions
Anything else?
No response