search

openshift / openshift-velero-plugin

General Velero plugin for backup and restore of openshift workloads.
Apache License 2.0
47 stars 37 forks source link

[OADP-459] Caches getters, move MigrationRegistry annotation from common to imagestream plugin #133

Closed kaovilai closed 2 years ago

kaovilai commented 2 years ago

See https://github.com/openshift/oadp-operator/blob/master/docs/config/custom_plugin_images.md on how to use custom images for testing

snippet of DPA with image built from this PR

apiVersion: oadp.openshift.io/v1alpha1
kind: DataProtectionApplication
metadata:
  name: dpa-sample
spec:
  unsupportedOverrides:
    openshiftPluginImageFqin: ghcr.io/kaovilai/openshift-velero-plugin:registrytmp

OADP-459

kaovilai commented 2 years ago

Initial testing indicates this PR works

$ cat /tmp/openshift.io/velero-plugin/<uid>/openshift-adp/velero-sample-1/oadp-registry-config/registry.txt 
<registryhostname>sh-4.4$
sseago commented 2 years ago

Looking at the different things being done in the common backup/restore plugins, the main thing that all pods, deployments, etc. need that we could cache is probably the result of the GetServerVersion and GetRegistryInfo calls. It looks like the OADP registry route (anything done with the MigrationRegistry annotation) is only used by the ImageStream plugin, so I'm thinking that we should probably move the part of common backup/restore plugins which set this to the imagestream backup/restore plugins, since there's no need to do that part at all for 10K deployments, etc.

kaovilai commented 2 years ago

using plugin built from this PR latest commit backup of parks-app with restify DC scaled to 700 pods ❯ velero backup create test-parksapp-pods --include-namespaces=parks-app --include-resources=pod -n openshift-adp;

Started:    2022-04-28 02:09:29 -0400 EDT
Completed:  2022-04-28 02:10:00 -0400 EDT

Expiration:  2022-05-28 02:09:29 -0400 EDT

Total items to be backed up:  704
Items backed up:              704

about 30 seconds or so.

using instead quay.io/konveyor/openshift-velero-plugin:latest yields

Started:    2022-04-28 02:13:45 -0400 EDT
Completed:  2022-04-28 02:16:30 -0400 EDT

Expiration:  2022-05-28 02:13:45 -0400 EDT

Total items to be backed up:  704
Items backed up:              704

almost 3 minutes.

kaovilai commented 2 years ago

see log difference https://gist.github.com/kaovilai/ecb7940bfe1ca10bd55b24620580d8b5

kaovilai commented 2 years ago

This PR is being updated to use structs in memory instead of writing to disk.

kaovilai commented 2 years ago

As of https://github.com/openshift/openshift-velero-plugin/pull/133/commits/2b131ef86a7b9d93ab2aaf94496c176d0e246509 31 seconds. exactly the same as when writing to disk

Started:    2022-04-28 12:07:44 -0400 EDT
Completed:  2022-04-28 12:08:15 -0400 EDT

Expiration:  2022-05-28 12:07:37 -0400 EDT

Total items to be backed up:  704
Items backed up:              704
kaovilai commented 2 years ago

Results from Travis in slack

pre-fix: ~35 minutes was for 10k deployments. ~4 minutes for 1k deployments post-fix: haven't tested 10k yet, but 1k took 30 seconds, which is about the same as velero on non-image resources (e.g. configmaps, secrets, etc.) that I've tested