search

migtools / volume-snapshot-mover

https://migtools.github.io/volume-snapshot-mover/developer
Apache License 2.0
17 stars 13 forks source link

fix: OADP-2066 Add backup name to folder structure of Datamover backup #254

Closed mateusoliveira43 closed 1 year ago

mateusoliveira43 commented 1 year ago

Description

When creating a Datamover backup, files are organized in the following structure:

├── bucketName
│   └── OADPNameSpace
│        └── VSCName-pvc
│            └──files

This PRs adds on more level with the backup name:

├── bucketName
│   └── OADPNameSpace
│       └── BackupName
│           └── VSCName-pvc
│                └──files

Examples before and after in IBM Screenshot from 2023-07-12 15-37-41 Screenshot from 2023-07-13 14-39-38

Related issues

Fixes https://github.com/migtools/volume-snapshot-mover/issues/234

How to test

Install VolSync stable-0.7 and OADP stable-1.2 operators from OperatorHub.

Create Velero and DataMover credentials secrets.

Use VolumeSnapshotMover (controller) image created for this PR (it was generated running docker image build --tag quay.io/msouzaol/volume-snapshot-mover:pr254-2 -f Dockerfile.ubi .) in OADP DPA. Ex.:

apiVersion: oadp.openshift.io/v1alpha1
kind: DataProtectionApplication
metadata:
  name: oadp-vsm-example
  namespace: openshift-adp
spec:
  backupLocations:
    - velero:
        backupSyncPeriod: 2m0s
        config:
          profile: default
          region: <region>
          s3ForcePathStyle: 'true'
          s3Url: <URL>
        credential:
          key: cloud
          name: cloud-credentials
        default: true
        objectStorage:
          bucket: <name>
          prefix: velero
        provider: aws
  configuration:
    restic:
      enable: false
    velero:
      defaultPlugins:
        - openshift
        - aws
        - csi
        - vsm
      featureFlags:
        - EnableCSI
  features:
    dataMover:
      credentialName: dm-credentials
      enable: true
      maxConcurrentBackupVolumes: '2'
  unsupportedOverrides:
    dataMoverImageFqin: 'quay.io/msouzaol/volume-snapshot-mover:pr254-2'

Run backups in a application (I suggest using one from bookbag). Example for mysql-persistent

# Normal
velero backup create --include-namespaces mysql-persistent
# Schedule
velero schedule create example-schedule --schedule="0 3 * * *" --include-namespaces mysql-persistent
velero backup create --from-schedule example-schedule
openshift-ci[bot] commented 1 year ago

Hi @mateusoliveira43. Thanks for your PR.

I'm waiting for a migtools member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
mateusoliveira43 commented 1 year ago

/hold

test with empty backup name or trailing slash in backup name

weshayutin commented 1 year ago

works for me :) Thank you @mateusoliveira43 , very well done.

mysql-persistent-wes-pathpatch-1   Completed   0        0          2023-07-14 20:16:31 +0000 UTC   29d       dpa-sample-1       <none>
No resources found

Total Snapshots:  0
No resources found in openshift-adp namespace.
Total OADP Snapshots:  0
Total SnapshotContents:  2
No resources found

Total VSB:  0
No resources found
Completed:  0
No resources found
InProgress:  0
No resources found
SnapshotBackupDone:  0

VSB STATUS
Completed:  0
Processing:  0
Queued:  0
No resources found

Total ReplicationSources:  0

Protected NAMESPACE
No resources found in openshift-adp namespace.
PVC objects:  0
No resources found in openshift-adp namespace.
ReplicationSource:  0

***** VOLUME SNAPSHOT CONTENTS ******
Name:  snapcontent-099f84a0-a23c-45b2-a76e-fe1cd6800b69  ReadyToUse:  true  creationTime:  2023-07-14T20:16:47Z
Name:  snapcontent-bb2e07bf-bee2-4849-a135-78386c625554  ReadyToUse:  true  creationTime:  2023-07-14T20:16:36Z

remote storage:

[whayutin@thinkdoe oadp_tmp]$ s3cmd ls s3://cvpbucketuswest2/openshift-adp/
                          DIR  s3://cvpbucketuswest2/openshift-adp/snapcontent-8d3177e2-3da2-47e1-a1e7-9e0492bead20-pvc/
                          DIR  s3://cvpbucketuswest2/openshift-adp/snapcontent-ba56d7dd-15ad-4ab9-bde1-6b04f9dd7a08-pvc/
[whayutin@thinkdoe oadp_tmp]$ s3cmd ls s3://cvpbucketuswest2/openshift-adp/
                          DIR  s3://cvpbucketuswest2/openshift-adp/mysql-persistent-wes-pathpatch-1/
                          DIR  s3://cvpbucketuswest2/openshift-adp/snapcontent-8d3177e2-3da2-47e1-a1e7-9e0492bead20-pvc/
                          DIR  s3://cvpbucketuswest2/openshift-adp/snapcontent-ba56d7dd-15ad-4ab9-bde1-6b04f9dd7a08-pvc/
[whayutin@thinkdoe oadp_tmp]$ s3cmd ls s3://cvpbucketuswest2/openshift-adp/
                          DIR  s3://cvpbucketuswest2/openshift-adp/mysql-persistent-wes-pathpatch-1/
                          DIR  s3://cvpbucketuswest2/openshift-adp/snapcontent-8d3177e2-3da2-47e1-a1e7-9e0492bead20-pvc/
                          DIR  s3://cvpbucketuswest2/openshift-adp/snapcontent-ba56d7dd-15ad-4ab9-bde1-6b04f9dd7a08-pvc/
[whayutin@thinkdoe oadp_tmp]$ s3cmd ls s3://cvpbucketuswest2/openshift-adp/mysql-persistent-wes-pathpatch-1/
                          DIR  s3://cvpbucketuswest2/openshift-adp/mysql-persistent-wes-pathpatch-1/snapcontent-099f84a0-a23c-45b2-a76e-fe1cd6800b69-pvc/
                          DIR  s3://cvpbucketuswest2/openshift-adp/mysql-persistent-wes-pathpatch-1/snapcontent-bb2e07bf-bee2-4849-a135-78386c625554-pvc/

s3cmd ls s3://cvpbucketuswest2/openshift-adp/mysql-persistent-wes-pathpatch-1/snapcontent-099f84a0-a23c-45b2-a76e-fe1cd6800b69-pvc/
                          DIR  s3://cvpbucketuswest2/openshift-adp/mysql-persistent-wes-pathpatch-1/snapcontent-099f84a0-a23c-45b2-a76e-fe1cd6800b69-pvc/data/
                          DIR  s3://cvpbucketuswest2/openshift-adp/mysql-persistent-wes-pathpatch-1/snapcontent-099f84a0-a23c-45b2-a76e-fe1cd6800b69-pvc/index/
                          DIR  s3://cvpbucketuswest2/openshift-adp/mysql-persistent-wes-pathpatch-1/snapcontent-099f84a0-a23c-45b2-a76e-fe1cd6800b69-pvc/keys/
                          DIR  s3://cvpbucketuswest2/openshift-adp/mysql-persistent-wes-pathpatch-1/snapcontent-099f84a0-a23c-45b2-a76e-fe1cd6800b69-pvc/snapshots/
2023-07-14 20:17          155  s3://cvpbucketuswest2/openshift-adp/mysql-persistent-wes-pathpatch-1/snapcontent-099f84a0-a23c-45b2-a76e-fe1cd6800b69-pvc/config
[whayutin@thinkdoe oadp_tmp]$ s3cmd ls s3://cvpbucketuswest2/openshift-adp/mysql-persistent-wes-pathpatch-1/snapcontent-099f84a0-a23c-45b2-a76e-fe1cd6800b69-pvc/snapshots/
2023-07-14 20:17          183  s3://cvpbucketuswest2/openshift-adp/mysql-persistent-wes-pathpatch-1/snapcontent-099f84a0-a23c-45b2-a76e-fe1cd6800b69-pvc/snapshots/decb7378c2899ece96933bb36ac30046a73c93734a918c3093c65e2a76ba9c4e
weshayutin commented 1 year ago

I don't see anything wrong with this change but I have not tested it.

If there is CI covering this, then I would be happy, if not can someone manually test it as well?

lolz.. CI .. I WISH!!!

openshift-ci[bot] commented 1 year ago

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: kaovilai, mateusoliveira43, shawn-hurley, shubham-pampattiwar

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files: - ~~[OWNERS](https://github.com/migtools/volume-snapshot-mover/blob/master/OWNERS)~~ [kaovilai,shawn-hurley,shubham-pampattiwar] Approvers can indicate their approval by writing `/approve` in a comment Approvers can cancel approval by writing `/approve cancel` in a comment
mateusoliveira43 commented 1 year ago

/unhold

I believe it is hard to hit a bug with the change (user would have to create the VSB by hand with snapcontent name at a specific time without label)

mateusoliveira43 commented 1 year ago

/cherry-pick oadp-1.2

openshift-cherrypick-robot commented 1 year ago

@mateusoliveira43: only migtools org members may request cherry picks. You can still do the cherry-pick manually.

In response to [this](https://github.com/migtools/volume-snapshot-mover/pull/254#issuecomment-1638725148): >/cherry-pick oadp-1.2 Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
kaovilai commented 1 year ago

/cherry-pick oadp-1.2

openshift-cherrypick-robot commented 1 year ago

@kaovilai: new pull request created: #255

In response to [this](https://github.com/migtools/volume-snapshot-mover/pull/254#issuecomment-1638727386): > >/cherry-pick oadp-1.2 > > Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.