search

percona / percona-server-mongodb-operator

Percona Operator for MongoDB
https://www.percona.com/doc/kubernetes-operator-for-psmongodb/
Apache License 2.0
350 stars 143 forks source link

/lib64/libc.so.6: version `GLIBC_2.34' not found Error when restoring physical backup on ARM Servers #1656

Open kv-dhyey-moliya opened 1 month ago

kv-dhyey-moliya commented 1 month ago

Report

I am seeing following error on mongodb-operator logs and when restoring backups on ARM Servers.

2024-09-19T06:57:35.443Z    ERROR   Reconciler error    {"controller": "psmdbrestore-controller", "object": {"name":"restore-2024-09-19-03","namespace":"mongodb"}, "namespace": "mongodb", "name": "restore-2024-09-19-03", "reconcileID": "add65b59-601a-4ad0-81fa-0ade99c22894", "error": "reconcile physical restore: resync config stderr: /opt/percona/pbm: /lib64/libc.so.6: version `GLIBC_2.32' not found (required by /opt/percona/pbm)\n/opt/percona/pbm: /lib64/libc.so.6: version `GLIBC_2.34' not found (required by /opt/percona/pbm)\n stdout: : command terminated with exit code 1", "errorVerbose": "command terminated with exit code 1\nresync config stderr: /opt/percona/pbm: /lib64/libc.so.6: version `GLIBC_2.32' not found (required by /opt/percona/pbm)\n/opt/percona/pbm: /lib64/libc.so.6: version `GLIBC_2.34' not found (required by /opt/percona/pbm)\n stdout: \ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore.(*ReconcilePerconaServerMongoDBRestore).reconcilePhysicalRestore\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore/physical.go:134\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore.(*ReconcilePerconaServerMongoDBRestore).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore/perconaservermongodbrestore_controller.go:200\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:222\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_arm64.s:1222\nreconcile physical restore\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore.(*ReconcilePerconaServerMongoDBRestore).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore/perconaservermongodbrestore_controller.go:202\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:222\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_arm64.s:1222"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:324
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:261
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:222

More about the problem

This error only happens with ARM Servers, and does not happen with AMD64 Servers.

Steps to reproduce

  1. Create a MongoDB cluster with ARM Architecture nodegroup (I used t4g.xlarge)
  2. Create a manual backup on S3 storage using type: physical.
  3. Restore the backup. The error will be logged in operator logs.
  4. Not able to perform any further operations (stop/start/backup/restore) on the cluster after this point.

Versions

  1. Kubernetes - 1.29
  2. Operator - 1.17.0
  3. Database - 7.0.12-7
  4. PBM - 2.5.0 and 2.6.0

I tried with pbm image percona/percona-backup-mongodb with versions 2.5.0-multi and 2.6.0-multi. Both versions have this issue.

Anything else?

Workaround to this error,

  1. Manually delete the whole cluster (data will persist on the PVC and backups will persist on S3 storage)
  2. Re-deploy the cluster, but with AMD64 Architecture nodegroup (I used t3.xlarge)
  3. Perform the restore operation again. It will succeed without the error.
  4. Pause the cluster, and change the nodegroup to ARM Architecture (I used t4g.xlarge)
  5. Resume the cluster.
hors commented 1 month ago

Hi @kv-dhyey-moliya, did you use 7.0.12-7-multi image as well? Like psmdb was started on ARM nodes. Am I right?

kv-dhyey-moliya commented 1 month ago

Yes I used "multi" images, with ARM nodes.

kv-dhyey-moliya commented 1 month ago

The db works perfectly on ARM, and logical restore also works perfectly, only problem is physical restore there I am getting this error.

hors commented 1 month ago

@kv-dhyey-moliya It is a pbm issue. PBM arm was built using el9 and glibc 2.34. PSMDB image was built using el9 and glibc 2.28. When the operator performs a physical restore, it copies the pbm binary from the PBM image to the psmdb container (the operator runs special pods for restore). That is why it can't work with psmdb container. Logical backup and restore should work. It will be fixed in the next release of PBM 2.7.0. I think we will have a release next month.

kv-dhyey-moliya commented 1 month ago

Thanks for the detailed update. 🙌