search

openshift / openshift-velero-plugin

General Velero plugin for backup and restore of openshift workloads.
Apache License 2.0
47 stars 37 forks source link

OADP-205: Implement registry inside openshift-velero-plugin #145

Closed kaovilai closed 2 years ago

kaovilai commented 2 years ago

Design doc https://github.com/openshift/oadp-operator/pull/737 OADP Implementaiton https://github.com/openshift/oadp-operator/pull/743

To test this PR, you will need to install operator from https://github.com/openshift/oadp-operator/pull/743 with the following command

operator-sdk run bundle quay.io/tkaovila/oadp-operator-bundle:velero-plugin-registry --namespace openshift-adp

Use dpa with unsupportedOverrides

apiVersion: oadp.openshift.io/v1alpha1
kind: DataProtectionApplication
metadata:
  name: velero-sample
spec:
  unsupportedOverrides:
    openshiftPluginImageFqin: "quay.io/tkaovila/openshift-velero-plugin:velero-plugin-registry"

Notable changes: https://github.com/openshift/openshift-velero-plugin/blob/1600327cb3f6f9f60ade880aef8fe16d34e6fb04/velero-plugins/imagestream/restore.go#L49-L56

https://github.com/openshift/openshift-velero-plugin/blob/1600327cb3f6f9f60ade880aef8fe16d34e6fb04/velero-plugins/imagestream/shared.go#L107-L121

Setting transport name for containers/image/copy https://github.com/openshift/openshift-velero-plugin/blob/1600327cb3f6f9f60ade880aef8fe16d34e6fb04/velero-plugins/imagecopy/imagestream.go#L93-L116

OADP-205

kaovilai commented 2 years ago

restore failing time="2022-06-28T08:35:25Z" level=info msg="[imagecopy] Error copying image: reading config blob sha256:c4c25c3dcef13b15393af66f8ffdf93417ef7e504bc49c459370947dbbc04021: Error fetching blob: invalid status code from registry 307 (Temporary Redirect)" cmd=/plugins/velero-plugins logSource="/opt/app-root/pkg/mod/github.com/bombsimon/logrusr/v3@v3.0.0/logrusr.go:108" pluginName=velero-plugins restore=openshift-adp/rst-backup

trace containers/image/Image.ConfigBlob probably something around udistribution.newImageSource. api info

kaovilai commented 2 years ago

Interesting PRs that might resolve restore https://github.com/containers/image/pull/645 https://github.com/containers/image/pull/1237

kaovilai commented 2 years ago

Edit again: ... oops more typo bugs.

Looks like it worked on first tag copy.

time="2022-06-29T19:50:40Z" level=info msg="[imagecopy] Copying tag: \"oadp-registry\"" backup=openshift-adp/backup cmd=/plugins/velero-plugins logSource="/opt/app-root/pkg/mod/github.com/bombsimon/logrusr/v3@v3.0.0/logrusr.go:108" pluginName=velero-plugins
time="2022-06-29T19:50:40Z" level=info msg="[imagecopy] copying image from BSL registry: udistribution-s3-ae04794d-604f-4c71-b779-10131c62ab00" backup=openshift-adp/backup cmd=/plugins/velero-plugins logSource="/opt/app-root/pkg/mod/github.com/bombsimon/logrusr/v3@v3.0.0/logrusr.go:108" pluginName=velero-plugins
time="2022-06-29T19:50:40Z" level=info msg="[imagecopy] copying image to BSL registry: udistribution-s3-ae04794d-604f-4c71-b779-10131c62ab00" backup=openshift-adp/backup cmd=/plugins/velero-plugins logSource="/opt/app-root/pkg/mod/github.com/bombsimon/logrusr/v3@v3.0.0/logrusr.go:108" pluginName=velero-plugins
time="2022-06-29T19:50:40Z" level=info msg="[imagecopy] copying from: udistribution-s3-ae04794d-604f-4c71-b779-10131c62ab00://imagestream-test/python-sample@sha256:487730a3e642d07ce5a6ff11eebccec47cc7a0bd599727c6bc1b1ff129f4fa64" backup=openshift-adp/backup cmd=/plugins/velero-plugins logSource="/opt/app-root/pkg/mod/github.com/bombsimon/logrusr/v3@v3.0.0/logrusr.go:108" pluginName=velero-plugins
time="2022-06-29T19:50:40Z" level=info msg="[imagecopy] copying to: udistribution-s3-ae04794d-604f-4c71-b779-10131c62ab00://imagestream-test/python-sample:oadp-registry" backup=openshift-adp/backup cmd=/plugins/velero-plugins logSource="/opt/app-root/pkg/mod/github.com/bombsimon/logrusr/v3@v3.0.0/logrusr.go:108" pluginName=velero-plugins
time="2022-06-29T19:50:40Z" level=info msg="copying image: udistribution-s3-ae04794d-604f-4c71-b779-10131c62ab00://imagestream-test/python-sample@sha256:487730a3e642d07ce5a6ff11eebccec47cc7a0bd599727c6bc1b1ff129f4fa64; will attempt up to 7 times..." backup=openshift-adp/backup cmd=/plugins/velero-plugins logSource="/opt/app-root/pkg/mod/github.com/bombsimon/logrusr/v3@v3.0.0/logrusr.go:108" pluginName=velero-plugins

Second tag copy have this issue.

time="2022-06-29T19:50:42Z" level=info msg="[imagecopy] Copying tag: \"placeholder\"" backup=openshift-adp/backup cmd=/plugins/velero-plugins logSource="/opt/app-root/pkg/mod/github.com/bombsimon/logrusr/v3@v3.0.0/logrusr.go:108" pluginName=velero-plugins
time="2022-06-29T19:50:42Z" level=info msg="[imagecopy] copying image from BSL registry: udistribution-s3-ae04794d-604f-4c71-b779-10131c62ab00" backup=openshift-adp/backup cmd=/plugins/velero-plugins logSource="/opt/app-root/pkg/mod/github.com/bombsimon/logrusr/v3@v3.0.0/logrusr.go:108" pluginName=velero-plugins
time="2022-06-29T19:50:42Z" level=info msg="[imagecopy] copying image to BSL registry: udistribution-s3-ae04794d-604f-4c71-b779-10131c62ab00" backup=openshift-adp/backup cmd=/plugins/velero-plugins logSource="/opt/app-root/pkg/mod/github.com/bombsimon/logrusr/v3@v3.0.0/logrusr.go:108" pluginName=velero-plugins
time="2022-06-29T19:50:42Z" level=info msg="[imagecopy] copying from: udistribution-s3-ae04794d-604f-4c71-b779-10131c62ab00://imagestream-test/python-sample@sha256:b170df4fefd479a44c3617a7909a37a9343cbc47ad1ce319c436dc20f0ea7855" backup=openshift-adp/backup cmd=/plugins/velero-plugins logSource="/opt/app-root/pkg/mod/github.com/bombsimon/logrusr/v3@v3.0.0/logrusr.go:108" pluginName=velero-plugins
time="2022-06-29T19:50:42Z" level=info msg="[imagecopy] copying to: udistribution-s3-ae04794d-604f-4c71-b779-10131c62ab00://imagestream-test/python-sample:placeholder" backup=openshift-adp/backup cmd=/plugins/velero-plugins logSource="/opt/app-root/pkg/mod/github.com/bombsimon/logrusr/v3@v3.0.0/logrusr.go:108" pluginName=velero-plugins
time="2022-06-29T19:50:42Z" level=info msg="copying image: udistribution-s3-ae04794d-604f-4c71-b779-10131c62ab00://imagestream-test/python-sample@sha256:b170df4fefd479a44c3617a7909a37a9343cbc47ad1ce319c436dc20f0ea7855; will attempt up to 7 times..." backup=openshift-adp/backup cmd=/plugins/velero-plugins logSource="/opt/app-root/pkg/mod/github.com/bombsimon/logrusr/v3@v3.0.0/logrusr.go:108" pluginName=velero-plugins
time="2022-06-29T19:50:42Z" level=info msg="attempt #1 failed, waiting 5s and then retrying" backup=openshift-adp/backup cmd=/plugins/velero-plugins logSource="/opt/app-root/pkg/mod/github.com/bombsimon/logrusr/v3@v3.0.0/logrusr.go:108" pluginName=velero-plugins
...
time="2022-06-29T19:52:27Z" level=info msg="[imagecopy] Error copying image: initializing source udistribution-s3-ae04794d-604f-4c71-b779-10131c62ab00://imagestream-test/python-sample@sha256:b170df4fefd479a44c3617a7909a37a9343cbc47ad1ce319c436dc20f0ea7855: reading manifest sha256:b170df4fefd479a44c3617a7909a37a9343cbc47ad1ce319c436dc20f0ea7855 in docker.io/imagestream-test/python-sample: manifest unknown: manifest unknown" backup=openshift-adp/backup cmd=/plugins/velero-plugins logSource="/opt/app-root/pkg/mod/github.com/bombsimon/logrusr/v3@v3.0.0/logrusr.go:108" pluginName=velero-plugins
time="2022-06-29T19:52:27Z" level=info msg="1 errors encountered backup up item" backup=openshift-adp/backup logSource="pkg/backup/backup.go:413" name=python-sample

manual trace https://github.com/kaovilai/openshift-velero-plugin/blob/a40559b95441ca0f169709b73f5c2749350c8412/velero-plugins/imagecopy/imagestream.go#L193 https://github.com/containers/image/blob/3c83b65b71650f25c11d9b8585f304bd8299dd00/copy/copy.go#L219 containers/image/copy/copy.go:217 srcRef.NewImageSource(ctx, options.SourceCtx) https://github.com/kaovilai/udistribution/blob/2b5e16ac1f8efa0bbcdc513ae7103d4f56f3befa/pkg/image/udistribution/docker_transport.go#L198

kaovilai commented 2 years ago

Resolved outstanding issues. Backup and restore works as intended at least for single image manifest

time="2022-06-29T20:36:31Z" level=info msg="[imagecopy] Copying tag: \"oadp-registry\"" cmd=/plugins/velero-plugins logSource="/opt/app-root/pkg/mod/github.com/bombsimon/logrusr/v3@v3.0.0/logrusr.go:108" pluginName=velero-plugins restore=openshift-adp/backup-rst
time="2022-06-29T20:36:31Z" level=info msg="[imagecopy] copying image from BSL registry: udistribution-s3-f85eb12f-7a1f-47bf-a89f-39a8a24ef7ab" cmd=/plugins/velero-plugins logSource="/opt/app-root/pkg/mod/github.com/bombsimon/logrusr/v3@v3.0.0/logrusr.go:108" pluginName=velero-plugins restore=openshift-adp/backup-rst
time="2022-06-29T20:36:31Z" level=info msg="[imagecopy] copying from: udistribution-s3-f85eb12f-7a1f-47bf-a89f-39a8a24ef7ab://imagestream-test/python-sample@sha256:487730a3e642d07ce5a6ff11eebccec47cc7a0bd599727c6bc1b1ff129f4fa64" cmd=/plugins/velero-plugins logSource="/opt/app-root/pkg/mod/github.com/bombsimon/logrusr/v3@v3.0.0/logrusr.go:108" pluginName=velero-plugins restore=openshift-adp/backup-rst
time="2022-06-29T20:36:31Z" level=info msg="[imagecopy] copying to: docker://image-registry.openshift-image-registry.svc:5000/imagestream-test/python-sample:oadp-registry" cmd=/plugins/velero-plugins logSource="/opt/app-root/pkg/mod/github.com/bombsimon/logrusr/v3@v3.0.0/logrusr.go:108" pluginName=velero-plugins restore=openshift-adp/backup-rst
time="2022-06-29T20:36:31Z" level=info msg="copying image: udistribution-s3-f85eb12f-7a1f-47bf-a89f-39a8a24ef7ab://imagestream-test/python-sample@sha256:487730a3e642d07ce5a6ff11eebccec47cc7a0bd599727c6bc1b1ff129f4fa64; will attempt up to 7 times..." cmd=/plugins/velero-plugins logSource="/opt/app-root/pkg/mod/github.com/bombsimon/logrusr/v3@v3.0.0/logrusr.go:108" pluginName=velero-plugins restore=openshift-adp/backup-rst
time="2022-06-29T20:36:32Z" level=info msg="[imagecopy] Copying tag: \"placeholder\"" cmd=/plugins/velero-plugins logSource="/opt/app-root/pkg/mod/github.com/bombsimon/logrusr/v3@v3.0.0/logrusr.go:108" pluginName=velero-plugins restore=openshift-adp/backup-rst
time="2022-06-29T20:36:32Z" level=info msg="[imagecopy] copying image from BSL registry: udistribution-s3-f85eb12f-7a1f-47bf-a89f-39a8a24ef7ab" cmd=/plugins/velero-plugins logSource="/opt/app-root/pkg/mod/github.com/bombsimon/logrusr/v3@v3.0.0/logrusr.go:108" pluginName=velero-plugins restore=openshift-adp/backup-rst
time="2022-06-29T20:36:32Z" level=info msg="[imagecopy] copying from: udistribution-s3-f85eb12f-7a1f-47bf-a89f-39a8a24ef7ab://imagestream-test/python-sample@sha256:b170df4fefd479a44c3617a7909a37a9343cbc47ad1ce319c436dc20f0ea7855" cmd=/plugins/velero-plugins logSource="/opt/app-root/pkg/mod/github.com/bombsimon/logrusr/v3@v3.0.0/logrusr.go:108" pluginName=velero-plugins restore=openshift-adp/backup-rst
time="2022-06-29T20:36:32Z" level=info msg="[imagecopy] copying to: docker://image-registry.openshift-image-registry.svc:5000/imagestream-test/python-sample:placeholder" cmd=/plugins/velero-plugins logSource="/opt/app-root/pkg/mod/github.com/bombsimon/logrusr/v3@v3.0.0/logrusr.go:108" pluginName=velero-plugins restore=openshift-adp/backup-rst
time="2022-06-29T20:36:32Z" level=info msg="copying image: udistribution-s3-f85eb12f-7a1f-47bf-a89f-39a8a24ef7ab://imagestream-test/python-sample@sha256:b170df4fefd479a44c3617a7909a37a9343cbc47ad1ce319c436dc20f0ea7855; will attempt up to 7 times..." cmd=/plugins/velero-plugins logSource="/opt/app-root/pkg/mod/github.com/bombsimon/logrusr/v3@v3.0.0/logrusr.go:108" pluginName=velero-plugins restore=openshift-adp/backup-rst
time="2022-06-29T20:36:34Z" level=info msg="[imagecopy] copied at least one local image: true" cmd=/plugins/velero-plugins logSource="/opt/app-root/pkg/mod/github.com/bombsimon/logrusr/v3@v3.0.0/logrusr.go:108" pluginName=velero-plugins restore=openshift-adp/backup-rst
time="2022-06-29T20:36:34Z" level=info msg="[imagecopy] copied at least one local image by tag: true" cmd=/plugins/velero-plugins logSource="/opt/app-root/pkg/mod/github.com/bombsimon/logrusr/v3@v3.0.0/logrusr.go:108" pluginName=velero-plugins restore=openshift-adp/backup-rst
time="2022-06-29T20:36:34Z" level=info msg="Skipping restore of ImageStream: python-sample because a registered plugin discarded it" logSource="pkg/restore/restore.go:1156" restore=openshift-adp/backup-rst
kaovilai commented 2 years ago

Just noticed (again?).. currently registry secret the plugin is fetching from is created from OADP-Operator. Could also move secret parsing to openshift-velero-plugin if wanted in the future.

dymurray commented 2 years ago

@kaovilai I do like the idea of enhancing this to do the parsing in the plugin rather than the operator. Something to explore as an enhancement.

kaovilai commented 2 years ago

manual test on aws passing. merging now.