openshift / oadp-operator

OADP Operator
Apache License 2.0
77 stars 71 forks source link

Bug: Need help to diagnostic a PV backup problem. #616

Closed lebrisg closed 2 years ago

lebrisg commented 2 years ago

Contact Details

lebrisg@hotmail.com

Describe bug

I'm testing the OADP-operator with a simple pacman application with a single PV. Everything is backed up fine except the PV and Restic doesn't show any error. The PV is stored through NFS (/nfs 192.168.0.0/16(rw,no_wdelay,no_root_squash)).

apiVersion: oadp.openshift.io/v1alpha1
kind: DataProtectionApplication
metadata:
  name: dpa-sample
spec:
  configuration:
    velero:
      defaultPlugins:
      - openshift
      - aws
    restic:
      enable: true
  backupLocations:
    - name: default
      velero:
        provider: aws
        default: true
        objectStorage:
          bucket: mybucket 
          prefix: velero 
        config:
          profile: default
          region: us-east-1 
          insecureSkipTLSVerify: "false"
          s3ForcePathStyle: "true"
          s3Url: https://s3.example.com
        credential:
          name: cloud-credentials
          key: cloud
oc exec $(oc get pods -n openshift-adp -o name | grep velero) -- ./velero backup describe pacman --details
Defaulted container "velero" out of: velero, openshift-velero-plugin (init), velero-plugin-for-aws (init)
Name:         pacman
Namespace:    openshift-adp
Labels:       velero.io/storage-location=dpa-sample-1
Annotations:  kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"velero.io/v1","kind":"Backup","metadata":{"annotations":{},"name":"pacman","namespace":"openshift-adp"},"spec":{"defaultVolumesToRestic":true,"includedNamespaces":["pacman"],"storageLocation":"dpa-sample-1","ttl":"1h0m0s"}}
..
Velero-Native Snapshots: <none included>

Restic Backups:
  Completed:
    pacman/pacman-backend-5db4bfc66d-wtp6x: mongodb
s3cmd ls s3://mybucket/velero/backups/pacman/*
2022-03-26 13:34           29  s3://mybucket/velero/backups/pacman/pacman-csi-volumesnapshotcontents.json.gz
2022-03-26 13:34           29  s3://mybucket/velero/backups/pacman/pacman-csi-volumesnapshots.json.gz
2022-03-26 13:34        10508  s3://mybucket/velero/backups/pacman/pacman-logs.gz
2022-03-26 13:34          933  s3://mybucket/velero/backups/pacman/pacman-podvolumebackups.json.gz
2022-03-26 13:34          901  s3://mybucket/velero/backups/pacman/pacman-resource-list.json.gz
2022-03-26 13:34           29  s3://mybucket/velero/backups/pacman/pacman-volumesnapshots.json.gz
2022-03-26 13:34       457083  s3://mybucket/velero/backups/pacman/pacman.tar.gz
2022-03-26 13:34         2461  s3://mybucket/velero/backups/pacman/velero-backup.json

What happened?

The PV should be backed up.

OADP Version

1.0.1 (Stable)

OpenShift Version

4.10

Velero pod logs

time="2022-03-26T13:28:30Z" level=info msg="setting log-level to INFO" logSource="pkg/cmd/server/server.go:175"
time="2022-03-26T13:28:30Z" level=info msg="Starting Velero server v1.7.1-OADP (-)" logSource="pkg/cmd/server/server.go:177"
time="2022-03-26T13:28:30Z" level=info msg="No feature flags enabled" logSource="pkg/cmd/server/server.go:181"
..
time="2022-03-26T13:34:37Z" level=info msg="Getting items for resource" backup=openshift-adp/pacman group=v1 logSource="pkg/backup/item_collector.go:170" resource=persistentvolumes
time="2022-03-26T13:34:37Z" level=info msg="Skipping resource because it's cluster-scoped and only specific namespaces are included in the backup" backup=openshift-adp/pacman group=v1 logSource="pkg/backup/item_collector.go:197" resource=persistentvolumes
..
time="2022-03-26T13:34:42Z" level=info msg="Backing up item" backup=openshift-adp/pacman logSource="pkg/backup/item_backupper.go:121" name=pvc-9f8ed44f-316d-4042-91d2-602dd69eeb49 namespace= resource=persistentvolumes
time="2022-03-26T13:34:42Z" level=info msg="Executing custom action" backup=openshift-adp/pacman logSource="pkg/backup/item_backupper.go:327" name=pvc-9f8ed44f-316d-4042-91d2-602dd69eeb49 namespace= resource=persistentvolumes
time="2022-03-26T13:34:42Z" level=info msg="[pv-backup] Returning pv object as is since this is not a migration activity" backup=openshift-adp/pacman cmd=/plugins/velero-plugins logSource="/remote-source/src/github.com/konveyor/openshift-velero-plugin/velero-plugins/persistentvolume/backup.go:32" pluginName=velero-plugins
time="2022-03-26T13:34:42Z" level=info msg="Executing takePVSnapshot" backup=openshift-adp/pacman logSource="pkg/backup/item_backupper.go:405" name=pvc-9f8ed44f-316d-4042-91d2-602dd69eeb49 namespace= resource=persistentvolumes
time="2022-03-26T13:34:42Z" level=info msg="Skipping snapshot of persistent volume because volume is being backed up with restic." backup=openshift-adp/pacman logSource="pkg/backup/item_backupper.go:423" name=pvc-9f8ed44f-316d-4042-91d2-602dd69eeb49 namespace= persistentVolume=pvc-9f8ed44f-316d-4042-91d2-602dd69eeb49 resource=persistentvolumes

Restic pod logs

time="2022-03-26T13:34:43Z" level=info msg="Backup starting" backup=openshift-adp/pacman controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:192" name=pacman-qnbfc namespace=openshift-adp
time="2022-03-26T13:34:43Z" level=info msg="Looking for most recent completed pod volume backup for this PVC" backup=openshift-adp/pacman controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:358" name=pacman-qnbfc namespace=openshift-adp pvcUID=9f8ed44f-316d-4042-91d2-602dd69eeb49
time="2022-03-26T13:34:43Z" level=info msg="No completed pod volume backup found for PVC" backup=openshift-adp/pacman controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:388" name=pacman-qnbfc namespace=openshift-adp pvcUID=9f8ed44f-316d-4042-91d2-602dd69eeb49
time="2022-03-26T13:34:43Z" level=info msg="No parent snapshot found for PVC, not using --parent flag for this backup" backup=openshift-adp/pacman controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:294" name=pacman-qnbfc namespace=openshift-adp
time="2022-03-26T13:34:49Z" level=info msg="Backup completed" backup=openshift-adp/pacman controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:348" name=pacman-qnbfc namespace=openshift-adp

Operator pod logs

No response

New issue

dymurray commented 2 years ago

@lebrisg Please rerun the s3 command with the following:

$ s3cmd ls s3://mybucket/velero/

You should see a folder restic which contains all the PV data for an NFS backup.

lebrisg commented 2 years ago

You can close this issue.

kaovilai commented 2 years ago

Glad it's resolved.