search

percona / percona-xtradb-cluster-operator

Percona Operator for MySQL based on Percona XtraDB Cluster
https://www.percona.com/doc/kubernetes-operator-for-pxc/index.html
Apache License 2.0
511 stars 187 forks source link

“Observed a panic in reconciler” on the backup restore request #1719

Open sekot opened 4 months ago

sekot commented 4 months ago

Report

restoring a backup from another cluster no longer works, the operator panic and restart. the job is never launch.

More about the problem

the restore.yaml

apiVersion: pxc.percona.com/v1
kind: PerconaXtraDBClusterRestore
metadata:
  name: check-backup-pxc-restore-manual-qyp3i
  namespace: pxc-check-backup
spec:
  pxcCluster: percona-cluster
  backupSource:
    destination: azure://mysql80-pxc-backups/percona-cluster-2024-05-31-00:00:29-full
    azure:
      container: mysql80-pxc-backups
      credentialsSecret: percona-azure-secret-check-backup

the operator logs :

2024-05-31T16:14:02.290Z    INFO    Password expiration policy updated  {"controller": "pxc-controller", "namespace": "pxc-check-backup", "name": "percona-cluster", "reconcileID": "f62cc541-995a-4499-8253-d58aaf4c2b5b", "user": "root"}
percona-xtradb-cluster-operator-854d568785-qcxt7 percona-xtradb-cluster-operator 2024-05-31T16:16:16.876Z   INFO    backup restore request  {"controller": "pxcrestore-controller", "namespace": "pxc-check-backup", "name": "check-backup-pxc-restore-manual-qyp3i", "reconcileID": "1076b45e-e86d-4031-93f4-10de9627614b"}
percona-xtradb-cluster-operator-854d568785-qcxt7 percona-xtradb-cluster-operator 2024-05-31T16:16:16.893Z   INFO    Observed a panic in reconciler: runtime error: invalid memory address or nil pointer dereference    {"controller": "pxcrestore-controller", "namespace": "pxc-check-backup", "name": "check-backup-pxc-restore-manual-qyp3i", "reconcileID": "1076b45e-e86d-4031-93f4-10de9627614b"}
percona-xtradb-cluster-operator-854d568785-qcxt7 percona-xtradb-cluster-operator panic: runtime error: invalid memory address or nil pointer dereference [recovered]
percona-xtradb-cluster-operator-854d568785-qcxt7 percona-xtradb-cluster-operator    panic: runtime error: invalid memory address or nil pointer dereference
percona-xtradb-cluster-operator-854d568785-qcxt7 percona-xtradb-cluster-operator [signal SIGSEGV: segmentation violation code=0x1 addr=0x58 pc=0x1634635]
percona-xtradb-cluster-operator-854d568785-qcxt7 percona-xtradb-cluster-operator 
percona-xtradb-cluster-operator-854d568785-qcxt7 percona-xtradb-cluster-operator goroutine 197 [running]:
percona-xtradb-cluster-operator-854d568785-qcxt7 percona-xtradb-cluster-operator sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1()
percona-xtradb-cluster-operator-854d568785-qcxt7 percona-xtradb-cluster-operator    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/internal/controller/controller.go:116 +0x1e5
percona-xtradb-cluster-operator-854d568785-qcxt7 percona-xtradb-cluster-operator panic({0x1aa5fe0?, 0x2e707b0?})
percona-xtradb-cluster-operator-854d568785-qcxt7 percona-xtradb-cluster-operator    /usr/local/go/src/runtime/panic.go:914 +0x21f
percona-xtradb-cluster-operator-854d568785-qcxt7 percona-xtradb-cluster-operator github.com/percona/percona-xtradb-cluster-operator/pkg/apis/pxc/v1.(*PXCBackupStatus).GetStorageType(...)
percona-xtradb-cluster-operator-854d568785-qcxt7 percona-xtradb-cluster-operator    /go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/apis/pxc/v1/pxc_backup_types.go:140
percona-xtradb-cluster-operator-854d568785-qcxt7 percona-xtradb-cluster-operator github.com/percona/percona-xtradb-cluster-operator/pkg/pxc/backup.RestoreJob(0xc000bf7d40, 0xc000740fc0, 0xc000e98000, {0xc00089bf40, 0x44}, 0x0)
percona-xtradb-cluster-operator-854d568785-qcxt7 percona-xtradb-cluster-operator    /go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/pxc/backup/restore.go:140 +0x75
percona-xtradb-cluster-operator-854d568785-qcxt7 percona-xtradb-cluster-operator github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcrestore.(*azure).Job(0xc000704600?)
percona-xtradb-cluster-operator-854d568785-qcxt7 percona-xtradb-cluster-operator    /go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcrestore/restorer.go:191 +0x32
percona-xtradb-cluster-operator-854d568785-qcxt7 percona-xtradb-cluster-operator github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcrestore.(*ReconcilePerconaXtraDBClusterRestore).validate(0x1a8a960?, {0x204fa08, 0xc000bb9980}, 0xc000bf7d40, 0x204f998?, 0xc000e98000?)
percona-xtradb-cluster-operator-854d568785-qcxt7 percona-xtradb-cluster-operator    /go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcrestore/restore.go:80 +0x4b
percona-xtradb-cluster-operator-854d568785-qcxt7 percona-xtradb-cluster-operator github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcrestore.(*ReconcilePerconaXtraDBClusterRestore).Reconcile(0xc000704600, {0x204fa08, 0xc000bb9980}, {{{0xc0008f6580?, 0x5?}, {0xc000c3acf0?, 0xc000f5ed08?}}})
percona-xtradb-cluster-operator-854d568785-qcxt7 percona-xtradb-cluster-operator    /go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcrestore/controller.go:190 +0xf14
percona-xtradb-cluster-operator-854d568785-qcxt7 percona-xtradb-cluster-operator sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0x2053448?, {0x204fa08?, 0xc000bb9980?}, {{{0xc0008f6580?, 0xb?}, {0xc000c3acf0?, 0x0?}}})
percona-xtradb-cluster-operator-854d568785-qcxt7 percona-xtradb-cluster-operator    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/internal/controller/controller.go:119 +0xb7
percona-xtradb-cluster-operator-854d568785-qcxt7 percona-xtradb-cluster-operator sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc00014f720, {0x204fa40, 0xc0004d0f50}, {0x1b4eb40?, 0xc00005ec40?})
percona-xtradb-cluster-operator-854d568785-qcxt7 percona-xtradb-cluster-operator    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/internal/controller/controller.go:316 +0x3cc
percona-xtradb-cluster-operator-854d568785-qcxt7 percona-xtradb-cluster-operator sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc00014f720, {0x204fa40, 0xc0004d0f50})
percona-xtradb-cluster-operator-854d568785-qcxt7 percona-xtradb-cluster-operator    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/internal/controller/controller.go:266 +0x1af
percona-xtradb-cluster-operator-854d568785-qcxt7 percona-xtradb-cluster-operator sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()
percona-xtradb-cluster-operator-854d568785-qcxt7 percona-xtradb-cluster-operator    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/internal/controller/controller.go:227 +0x79
percona-xtradb-cluster-operator-854d568785-qcxt7 percona-xtradb-cluster-operator created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2 in goroutine 105
percona-xtradb-cluster-operator-854d568785-qcxt7 percona-xtradb-cluster-operator    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/internal/controller/controller.go:223 +0x565

Steps to reproduce

try to restore from another cluster with a different storage account

  1. install a cluster with an azure storage account for backup
  2. launch backup
  3. install another cluster with a different azure storage account
  4. add the secret of the first azure storage account
  5. restore the backup with this secret and backup of step 2

Versions

  1. Kubernetes 1.27
  2. Operator 1.14
  3. Database mysql 5.7 & 8.0

Anything else?

it worked with operator 1.13

same issue here with s3 : https://forums.percona.com/t/observed-a-panic-in-reconciler-on-the-backup-restore-request-perconaxtradbclusterrestore-pxc-percona-com/30192/1

nicolasgrieco commented 4 months ago

I also have the same issue.

sekot commented 4 months ago

ok backup section in cr must be present. so opérator panic is sade :) and I wonder if the backup section should be mandatory to restore.