search

vmware-tanzu / velero

Backup and migrate Kubernetes applications and their persistent volumes
https://velero.io
Apache License 2.0
8.69k stars 1.4k forks source link

Unable to backup workloads without using annotations #3927

Closed abarry-gn closed 3 years ago

abarry-gn commented 3 years ago

What steps did you take and what happened: 1- Backup workload without annotation: velero backup create hello-world-without-annotation --include-namespaces hello-world3 2- Backup description for more details: velero backup describe hello-world-without-annotation

Name:         hello-world-without-annotation
Namespace:    velero
Labels:       velero.io/storage-location=default
Annotations:  velero.io/source-cluster-k8s-gitversion=v1.18.10
              velero.io/source-cluster-k8s-major-version=1
              velero.io/source-cluster-k8s-minor-version=18

Phase:  Failed (run `velero backup logs hello-world-without-annotation` for more information)

Errors:    0
Warnings:  0

Namespaces:
  Included:  hello-world3
  Excluded:  <none>

Resources:
  Included:        *
  Excluded:        <none>
  Cluster-scoped:  auto

Label selector:  <none>

Storage Location:  default

Velero-Native Snapshot PVs:  auto

TTL:  720h0m0s

Hooks:  <none>

Backup Format Version:  1.1.0

Started:    2021-07-02 12:03:58 +0000 UTC
Completed:  2021-07-02 12:03:59 +0000 UTC

Expiration:  2021-08-01 12:03:58 +0000 UTC

Velero-Native Snapshots: <none included>

3- Logs of the backup: velero backup logs hello-world-without-annotation

time="2021-07-02T12:03:58Z" level=info msg="Setting up backup temp file" backup=velero/hello-world-without-annotation logSource="pkg/controller/backup_controller.go:534"
time="2021-07-02T12:03:58Z" level=info msg="Setting up plugin manager" backup=velero/hello-world-without-annotation logSource="pkg/controller/backup_controller.go:541"
time="2021-07-02T12:03:58Z" level=info msg="Getting backup item actions" backup=velero/hello-world-without-annotation logSource="pkg/controller/backup_controller.go:545"
time="2021-07-02T12:03:58Z" level=info msg="Setting up backup store to check for backup existence" backup=velero/hello-world-without-annotation logSource="pkg/controller/backup_controller.go:551"
time="2021-07-02T12:03:59Z" level=info msg="Writing backup version file" backup=velero/hello-world-without-annotation logSource="pkg/backup/backup.go:212"
time="2021-07-02T12:03:59Z" level=info msg="Including namespaces: hello-world3" backup=velero/hello-world-without-annotation logSource="pkg/backup/backup.go:218"
time="2021-07-02T12:03:59Z" level=info msg="Excluding namespaces: <none>" backup=velero/hello-world-without-annotation logSource="pkg/backup/backup.go:219"
time="2021-07-02T12:03:59Z" level=info msg="Including resources: *" backup=velero/hello-world-without-annotation logSource="pkg/backup/backup.go:222"
time="2021-07-02T12:03:59Z" level=info msg="Excluding resources: <none>" backup=velero/hello-world-without-annotation logSource="pkg/backup/backup.go:223"
time="2021-07-02T12:03:59Z" level=info msg="Backing up all pod volumes using restic: false" backup=velero/hello-world-without-annotation logSource="pkg/backup/backup.go:224"

What did you expect to happen:

We expect backup to succeed without using annotations.

Anything else you would like to add:

When we use annotations for the same workload the backup succeeds.

Environment:

Client:
        Version: v1.5.3
        Git commit: 123109a3bcac11dbb6783d2758207bac0d0817cb
Server:
        Version: v1.5.3

Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.10", GitCommit:"62876fc6d93e891aa7fbe19771e6a6c03773b0f7", GitTreeState:"clean", BuildDate:"2020-10-15T01:43:56Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}

Bellow the list of files backed up with and without annotations (we see that the files size are different):

2021-07-02 12:04:36         29 backups/hello-world-without-annotation/hello-world-without-annotation-csi-volumesnapshotcontents.json.gz
2021-07-02 12:04:36         29 backups/hello-world-without-annotation/hello-world-without-annotation-csi-volumesnapshots.json.gz
2021-07-02 12:04:35        354 backups/hello-world-without-annotation/hello-world-without-annotation-logs.gz
2021-07-02 12:04:36         29 backups/hello-world-without-annotation/hello-world-without-annotation-podvolumebackups.json.gz
2021-07-02 12:04:36         27 backups/hello-world-without-annotation/hello-world-without-annotation-resource-list.json.gz
2021-07-02 12:04:36         29 backups/hello-world-without-annotation/hello-world-without-annotation-volumesnapshots.json.gz
2021-07-02 12:04:35        115 backups/hello-world-without-annotation/hello-world-without-annotation.tar.gz
2021-07-02 12:04:35       2126 backups/hello-world-without-annotation/velero-backup.json
2021-07-02 12:08:50        213 restores/hello-world-without-annotation-20210702120802/restore-hello-world-without-annotation-20210702120802-logs.gz
2021-07-02 12:08:50        118 restores/hello-world-without-annotation-20210702120802/restore-hello-world-without-annotation-20210702120802-results.gz
2021-07-02 12:22:38        213 restores/hello-world-without-annotation-20210702122150/restore-hello-world-without-annotation-20210702122150-logs.gz
2021-07-02 12:22:38        118 restores/hello-world-without-annotation-20210702122150/restore-hello-world-without-annotation-20210702122150-results.gz
2021-07-02 12:23:22        213 restores/hello-world-without-annotation-20210702122234/restore-hello-world-without-annotation-20210702122234-logs.gz
2021-07-02 12:23:22        118 restores/hello-world-without-annotation-20210702122234/restore-hello-world-without-annotation-20210702122234-results.

2021-07-02 09:28:37         29 backups/hello-world-annotation/hello-world-annotation-csi-volumesnapshotcontents.json.gz
2021-07-02 09:28:37         29 backups/hello-world-annotation/hello-world-annotation-csi-volumesnapshots.json.gz
2021-07-02 09:28:37       4480 backups/hello-world-annotation/hello-world-annotation-logs.gz
2021-07-02 09:28:37        969 backups/hello-world-annotation/hello-world-annotation-podvolumebackups.json.gz
2021-07-02 09:28:37        571 backups/hello-world-annotation/hello-world-annotation-resource-list.json.gz
2021-07-02 09:28:37         29 backups/hello-world-annotation/hello-world-annotation-volumesnapshots.json.gz
2021-07-02 09:28:37      13632 backups/hello-world-annotation/hello-world-annotation.tar.gz
2021-07-02 09:28:37       2188 backups/hello-world-annotation/velero-backup.json
2021-07-02 09:43:47       1775 restores/hello-world-annotation-20210702093328/restore-hello-world-annotation-20210702093328-logs.gz
2021-07-02 09:43:47        178 restores/hello-world-annotation-20210702093328/restore-hello-world-annotation-20210702093328-results.gz
2021-07-02 09:52:02       1757 restores/hello-world-annotation-20210702094831/restore-hello-world-annotation-20210702094831-logs.gz
2021-07-02 09:52:02        178 restores/hello-world-annotation-20210702094831/restore-hello-world-annotation-20210702094831-results.gz
2021-07-02 12:21:20       1827 restores/hello-world-annotation-20210702122031/restore-hello-world-annotation-20210702122031-logs.gz
2021-07-02 12:21:20        257 restores/hello-world-annotation-20210702122031/restore-hello-world-annotation-20210702122031-results.gz

Furthermore, even if we see a size difference, we tried to restore the backup:

the namespace and pods are restored but the persistent volume was not. We tried to create a persistent volume with the same name and redo the restore and it fails with 'volumes already exists'

Once again with annotations everything goes fine.

Vote on this issue!

This is an invitation to the Velero community to vote on issues, you can see the project's top voted issues listed here.
Use the "reaction smiley face" up to the right of this comment to vote.

abarry-gn commented 3 years ago

Hello @eleanor-millman, Orange team here, as we discussed in our previous meeting we have tried again to backup workloads without using annotations and here is the github issue (and the details).

dsu-igeek commented 3 years ago

@barryboubakar What is the cloud platform used here (AWS, vSphere, Azure, etc)? What annotations are you using when it is successful? Are you trying to back up using Restic or are you using a volume snapshot plugin?

abarry-gn commented 3 years ago

Hi @dsu-igeek ,

[1] - We are using velero to backlog kubernetes worklods. The kubernetes cluster is deployed on vSphere (VMs).

[2] - We use the following command to annotate a pod:

kubectl -n foo annotate pod/sample backup.velero.io/backup-volumes=<VOLUME_NAME>

e.g. kubectl -n hellow-world-ns annotate pod/hello-world backup.velero.io/backup-volumes=data

[3] - We are using Restic to backup the workloads

codegold79 commented 3 years ago

@barryboubakar, thanks for the clarification. Now that we know the issue is with using Restic, I am able to focus on which part of the logs are relevant.

In particular, this part of the logs is probably what you don't like seeing: 

time="2021-07-02T12:03:59Z" level=info msg="Backing up all pod volumes using restic: false" backup=velero/hello-world-without-annotation logSource="pkg/backup/backup.go:224"

Let's see if pod volumes are being skipped due to a flag being set. With what flags was Velero installed on the cluster?

I apologize ahead of time that despite working for VMware, I don't have any experience with vSphere storage. Let me know if some of my suggestions won't work with vSphere.

Here are some restic-related flags I've used in the past with the velero install command:

--use-restic \
--use-volume-snapshots=false \
--default-volumes-to-restic

Given those flags, specifically the --default-volumes-to-restic, I don't need to pod annotations to backup all the pod volumes. Here is the docs for that flag: https://velero.io/docs/v1.6/customize-installation/#default-pod-volume-backup-to-restic.

codegold79 commented 3 years ago

@barryboubakar, you wrote,

We tried to create a persistent volume with the same name and redo the restore and it fails with 'volumes already exists'

I'm confused why this error is happening. We had a PR that checks for PVs before attempting to restore them: https://github.com/vmware-tanzu/velero/pull/1011

Might need to do more digging through restic pod, restic daemon set logs and podvolumebackup manifest to find the root of this problem. Let's cover this later after we figure out the backup problem.

abarry-gn commented 3 years ago

Hi @codegold79 Indeed, we were using all the flags you mentioned except --default-volumes-to-restic. Now that I've added it to velero install command everything is working fine.

Thank you very much for the help.

codegold79 commented 3 years ago

@barryboubakar, glad to have helped! Thank you for the update.

@dsu-igeek, I think this issue can be closed.

zubron commented 3 years ago

Thanks for the update, @barryboubakar! And thanks for providing help, @codegold79 :)

I'm going to close this issue, but let us know if you need any further assistance.