search

percona / percona-server-mongodb-operator

Percona Operator for MongoDB
https://www.percona.com/doc/kubernetes-operator-for-psmongodb/
Apache License 2.0
319 stars 138 forks source link

Error for setting up backup #1591

Closed winston0410 closed 1 month ago

winston0410 commented 1 month ago

Report

I am trying to setup backup for my Percona MongoDB. I notice percona-mongodb-operator has error log once I have setup the backup through CRD.

More about the problem

I am seeing the following log in percona-mongodb-operator:

2024-07-15 10:32:10.581 {"level":"info","ts":1721035930.5811331,"msg":"Configured PBM storage","controller":"psmdb-controller","object":{"name":"mongodb-cluster","namespace":"psmdb"},"namespace":"psmdb","name":"mongodb-cluster","reconcileID":"4a52e94c-4b22-4fa9-8f19-f518b4deea3e"}
2024-07-15 10:31:59.527 {"level":"info","ts":1721035919.5271955,"msg":"Setting PBM config","controller":"psmdb-controller","object":{"name":"mongodb-cluster","namespace":"psmdb"},"namespace":"psmdb","name":"mongodb-cluster","reconcileID":"4a52e94c-4b22-4fa9-8f19-f518b4deea3e","backup":"mongodb-cluster"}
2024-07-15 10:31:59.518 {"level":"info","ts":1721035919.5187798,"msg":"Configuring PBM with storage","controller":"psmdb-controller","object":{"name":"mongodb-cluster","namespace":"psmdb"},"namespace":"psmdb","name":"mongodb-cluster","reconcileID":"4a52e94c-4b22-4fa9-8f19-f518b4deea3e","storage":"azure-blob"}
2024-07-15 11:07:54.216 {"level":"error","ts":1721038074.215949,"msg":"Reconciler error","controller":"psmdb-controller","object":{"name":"mongodb-cluster","namespace":"psmdb"},"namespace":"psmdb","name":"mongodb-cluster","reconcileID":"6185461f-7820-4c21-a998-aa44130c905d","error":"reconcile backup tasks: there is no storage  in cluster mongodb-cluster for weekly task","errorVerbose":"there is no storage  in cluster mongodb-cluster for weekly task\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).reconcileBackupTasks\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/backup.go:46\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:395\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:222\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1695\nreconcile backup tasks\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:397\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:222\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1695","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:324\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:222"}
2024-07-15 10:56:58.633 {"level":"error","ts":1721037418.6333563,"msg":"Reconciler error","controller":"psmdb-controller","object":{"name":"mongodb-cluster","namespace":"psmdb"},"namespace":"psmdb","name":"mongodb-cluster","reconcileID":"ae4e3f3c-8387-48e2-9afb-38c4b338131f","error":"reconcile backup tasks: there is no storage  in cluster mongodb-cluster for weekly task","errorVerbose":"there is no storage  in cluster mongodb-cluster for weekly task\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).reconcileBackupTasks\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/backup.go:46\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:395\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:222\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1695\nreconcile backup tasks\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:397\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:222\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1695","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:324\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:222"}
2024-07-15 10:51:30.730 {"level":"error","ts":1721037090.730234,"msg":"Reconciler error","controller":"psmdb-controller","object":{"name":"mongodb-cluster","namespace":"psmdb"},"namespace":"psmdb","name":"mongodb-cluster","reconcileID":"24dabe0a-6e9d-4352-99f1-3549ecc265dd","error":"reconcile backup tasks: there is no storage  in cluster mongodb-cluster for weekly task","errorVerbose":"there is no storage  in cluster mongodb-cluster for weekly task\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).reconcileBackupTasks\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/backup.go:46\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:395\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:222\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1695\nreconcile backup tasks\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:397\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:222\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1695","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:324\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:222"}

Steps to reproduce

  1. kubectl apply -f https://raw.githubusercontent.com/percona/percona-server-mongodb-operator/v1.16.1/deploy/cw-bundle.yaml

  2. Apply the following CRD:

    ---
apiVersion: v1
kind: Namespace
metadata:
name: psmdb
labels:
pod-security.kubernetes.io/enforce: baseline
---
apiVersion: v1
kind: Secret
metadata:
name: azure-blob-secret
namespace: psmdb
type: Opaque
stringData:
AZURE_STORAGE_ACCOUNT_NAME: "$AZURE_STORAGE_ACCOUNT_NAME"
AZURE_STORAGE_ACCOUNT_KEY: "$AZURE_STORAGE_ACCOUNT_KEY"
---
apiVersion: v1
kind: Secret
metadata:
name: mongodb-user-secret
namespace: psmdb
type: Opaque
stringData:
MONGODB_BACKUP_USER: "$MONGODB_BACKUP_USER"
MONGODB_BACKUP_PASSWORD: "$MONGODB_BACKUP_PASSWORD"
MONGODB_DATABASE_ADMIN_USER: "$MONGODB_DATABASE_ADMIN_USER"
MONGODB_DATABASE_ADMIN_PASSWORD: "$MONGODB_DATABASE_ADMIN_PASSWORD"
MONGODB_CLUSTER_ADMIN_USER: "$MONGODB_CLUSTER_ADMIN_USER"
MONGODB_CLUSTER_ADMIN_PASSWORD: "$MONGODB_CLUSTER_ADMIN_PASSWORD"
MONGODB_CLUSTER_MONITOR_USER: "$MONGODB_CLUSTER_MONITOR_USER"
MONGODB_CLUSTER_MONITOR_PASSWORD: "$MONGODB_CLUSTER_MONITOR_PASSWORD"
MONGODB_USER_ADMIN_USER: "$MONGODB_USER_ADMIN_USER"
MONGODB_USER_ADMIN_PASSWORD: "$MONGODB_USER_ADMIN_PASSWORD"
---
apiVersion: psmdb.percona.com/v1
kind: PerconaServerMongoDB
metadata:
name: mongodb-cluster
namespace: psmdb
finalizers:
- delete-psmdb-pods-in-order
spec:
platform: kubernetes
crVersion: 1.16.1
image: percona/percona-server-mongodb:7.0.11-multi
imagePullPolicy: IfNotPresent
initImage: percona/percona-server-mongodb-operator:1.16.1
clusterServiceDNSMode: Internal
multiCluster:
enabled: false
secrets:
users: mongodb-user-secret
encryptionKey: mongodb-encryption-secret
unsafeFlags:
tls: false
# set this to false, and replsets.size to the number of node, if we have a bigger node pool
replsetSize: true
tls:
mode: preferTLS
allowInvalidCertificates: false
upgradeOptions:
apply: Disabled

replsets:
- name: rs0
  size: 1
  serviceAccountName: default
  livenessProbe:
    failureThreshold: 4
    initialDelaySeconds: 60
    periodSeconds: 30
    timeoutSeconds: 10
    startupDelaySeconds: 7200
  readinessProbe:
    failureThreshold: 8
    initialDelaySeconds: 10
    periodSeconds: 3
    successThreshold: 1
    timeoutSeconds: 2
  volumeSpec:
    persistentVolumeClaim:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: managed-csi
      resources:
        requests:
          # NOTE it has to exactly match the size of the PV
          storage: 5Gi
  expose:
    enabled: true
    exposeType: ClusterIP
  nonvoting:
    enabled: false
    size: 1
  arbiter:
    enabled: false
    size: 1
backup:
enabled: true
image: percona/percona-backup-mongodb:2.5.0-multi
resources:
  limits:
    cpu: 300m
    memory: 1Gi
  requests:
    cpu: 100m
    memory: 0.5Gi
containerSecurityContext:
  privileged: false
storages:
  azure-blob:
    type: azure
    azure:
      container: supportassistant
      prefix: psmdb
      credentialsSecret: azure-blob-secret
pitr:
  enabled: false
tasks:
  - name: weekly
    enabled: true
    keep: 3
    schedule: 0 0 * * 0
    compressionType: gzip
    type: physical
  3. check log in kubectl logs deploy/percona-server-mongodb-operator -n psmdb-operator

Versions

  1. Kubernetes Client Version: v1.29.1 Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 Server Version: v1.28.9
  2. Operator 1.16.1
  3. Database percona/percona-server-mongodb:7.0.11-multi

Anything else?

No response

hors commented 1 month ago

Hi @winston0410, as you can see from the log message you need to set the storage name for schedule backup (task) e.g. https://github.com/percona/percona-server-mongodb-operator/blob/main/deploy/cr.yaml#L613

winston0410 commented 1 month ago

thanks! That resolved my issue