search

vmware-tanzu / velero

Backup and migrate Kubernetes applications and their persistent volumes
https://velero.io
Apache License 2.0
8.78k stars 1.41k forks source link

Restore is getting fail PartiallyFailed with restic fail please give solution #5325

Closed banyalasuresh closed 1 year ago

banyalasuresh commented 2 years ago

What steps did you take and what happened: [A clear and concise description of what the bug is, and what commands you ran.)

I have setup velero on one of my EKS clusters on AWS, storing the backups on AWS S3bucket inside sub folder per namespace. I would like to store the backups/schedules that are created within subfolders for each namespace. example: Bucket name One folder for each namespace within the bucket, and the schedules per namespace to be stored in the respective folders. I have created sub folder in my EKS cluster using below commands but its showing like PHASE is UNKONWN last VALIDATION is UNKONWN and backups is getting fail please assist me to push backups inside the AWS S3bucket subfolders. thanks you so much in advance seeking help please let me know if need any additional information thank you so much. Command to create sub folder $ velero backup-location create subfolder1 \ --provider aws \ --bucket my-bucket-name \ --prefix subfolder1

What did you expect to happen: my Requirement One folder for each namespace within the bucket, and the schedules per namespace to be stored in the respective folders.

please find the logs below:

time="2022-09-08T09:04:18Z" level=error msg="Error getting a backup store" backup-storage-location=test2 controller=backup-storage-location error="rpc error: code = Unknown desc = config has invalid keys [credentialsFile]; valid keys are [region s3Url publicUrl kmsKeyId s3ForcePathStyle signatureVersion profile serverSideEncryption insecureSkipTLSVerify bucket prefix]" error.file="/go/src/github.com/vmware-tanzu/velero-plugin-for-aws/vendor/github.com/vmware-tanzu/velero/pkg/plugin/framework/validation.go:50" error.function=github.com/vmware-tanzu/velero-plugin-for-aws/vendor/github.com/vmware-tanzu/velero/pkg/plugin/framework.validateConfigKeys logSource="pkg/controller/backup_storage_location_controller.go:100"

The following information will help us better understand what's going on:

If you are using velero v1.7.0+:
Please use velero debug --backup <backupname> --restore <restorename> to generate the support bundle, and attach to this issue, more options please refer to velero debug --help

If you are using earlier versions:
Please provide the output of the following commands (Pasting long output into a GitHub gist or other pastebin is fine.)

Anything else you would like to add: [Miscellaneous information that will assist in solving the issue.]

Environment:

Vote on this issue!

This is an invitation to the Velero community to vote on issues, you can see the project's top voted issues listed here.
Use the "reaction smiley face" up to the right of this comment to vote.

qiuming-best commented 2 years ago

@banyalasuresh probably you are using restic do the backup velero backup-location create subfolder1 --provider aws --bucket my-bucket-name --prefix subfolder1 which will create subfolder1 with in my-bucket-name, all meta data will stored in my-bucket-name/subfolder1/backups/$BackupName, all volume backups stored in my-bucket-name/subfolder1/restic/$VolumeNamespace

I'm not sure whether volume data that is stored in the related namespace which is meet your requirement: One folder for each namespace within the bucket, and the schedules per namespace to be stored in the respective folders

For this error: config has invalid keys [credentialsFile]; valid keys are [region s3Url publicUrl kmsKeyId s3ForcePathStyle signatureVersion profile serverSideEncryption insecureSkipTLSVerify bucket prefix]" maybe there is something wrong with your credentials files or you can refer to this

sureshbanyal commented 2 years ago

@qiuming-best thanks for responding, Using this command $ velero backup-location create subfolder1 --provider aws --bucket my-bucket-name --prefix subfolder1 it is creating subfolder in server but not in inside s3 bucket and in server when i check backup location using command $ velero get backup-location its showing like below

NAME PROVIDER BUCKET/PREFIX PHASE LAST VALIDATED ACCESS MODE DEFAULT default aws mybucket-name Unavailable 2022-09-13 06:55:22 +0000 UTC ReadWrite true subfolder2 aws mybucket-name/subfolder2 Unknown Unknown ReadWrite subfolder3 aws mybucket-name/subfolder3 Unknown Unknown ReadWrite subfolder5 aws mybucket-name/subfolder5 Unknown Unknown ReadWrite subfolder6 aws mybucket-name/subfolder6 Unknown Unknown ReadWrite

as per above output could you suggest issue is only with Credentials File i am suffering with this issue from 30 days but unable to resolve the issue and unabel to store my backups in subfolder inside s3 bucket. I am requesting please help me to resolve as soon as possible many thanks, looking for solution please assist me.

qiuming-best commented 2 years ago

@sureshbanyal could you try a new version of velero-plugin-for-aws? such as velero-plugin-for-aws:v1.5.0

sureshbanyal commented 2 years ago

@qiuming-best Sure could you please provide the process link so i can follow that and update the new plugin. velero-plugin-for-aws:v1.5.0 "please provide the upgrade document velero-plugin-for-aws steps. i am having little confusion please help me with the document thank you sooo much @qiuming-best

qiuming-best commented 2 years ago

here is the upgrade doc

And currently, our newest version is v1.9.1, and aws plugin version is v1.5.0

Another way is to uninstall velero, download and install velero v1.9.1 client, and then reinstall it with command like below velero install \ --plugins velero/velero-plugin-for-aws:v1.5.0 \ --provider aws \ --bucket velero-xxx \ --backup-location-config region=minio \ --snapshot-location-config region=minio \ --secret-file ./credentials \ --use-restic \ --default-volumes-to-restic

banyalasuresh commented 2 years ago
@qiuming-best Thanks for Document as per my k8s cluster version : v1.21.0, i have to use velero version v1.7.1. For this velero version plugin should be v1.3.0 Plugin Version Velero Version
v1.5.x v1.9.x
v1.4.x v1.8.x
v1.3.x v1.7.x
v1.2.x v1.6.x
v1.1.x v1.5.x
v1.1.x v1.4.x
v1.0.x v1.3.x
v1.0.x v1.2.0

I have added in velero this plug in : v1.3.0 as per my velero version : v1.7.1 but still backups are getting failed i have verified. My credentials are good when i trouble shoot but still i am getting same error in logs LOGS time="2022-09-18T10:45:07Z" level=error msg="Error getting backup store for this location" backupLocation=subfolder6 controller=backup-sync error="unable to get credentials: unable to get key for secret: Secret \"\" not found" error.file="/go/src/github.com/vmware-tanzu/velero/internal/credentials/file_store.go:69" error.function="github.com/vmware-tanzu/velero/internal/credentials.(namespacedFileStore).Path" logSource="pkg/controller/backup_sync_controller.go:175" time="2022-09-18T10:45:07Z" level=error msg="Error getting backup store for this location" backupLocation=wptestsree controller=backup-sync error="unable to get credentials: unable to get key for secret: Secret \"\" not found" error.file="/go/src/github.com/vmware-tanzu/velero/internal/credentials/file_store.go:69" error.function="github.com/vmware-tanzu/velero/internal/credentials.(namespacedFileStore).Path" logSource="pkg/controller/backup_sync_controller.go:175" time="2022-09-18T10:45:07Z" level=error msg="Error getting backup store for this location" backupLocation=wptestsree1 controller=backup-sync error="rpc error: code = Unknown desc = config has invalid keys [credentialsFile]; valid keys are [region s3Url publicUrl kmsKeyId s3ForcePathStyle signatureVersion profile serverSideEncryption insecureSkipTLSVerify bucket prefix]" error.file="/go/src/github.com/vmware-tanzu/velero-plugin-for-aws/vendor/github.com/vmware-tanzu/velero/pkg/plugin/framework/validation.go:50" error.function=github.com/vmware-tanzu/velero-plugin-for-aws/vendor/github.com/vmware-tanzu/velero/pkg/plugin/framework.validateConfigKeys logSource="pkg/controller/backup_sync_controller.go:175"

Please assist me how can i resolve the above issue thank you so much.

ywk253100 commented 2 years ago

@banyalasuresh Why cannot you use the latest version of Velero? Velero v1.9 supports k8s v1.21.0.

And could you double-check the plugin version you are using by running the command kubectl -n velero get pod velero-xxx? The credentialsFile should be supported by aws plugin v1.2.0+

banyalasuresh commented 2 years ago

@ywk253100 @qiuming-best Thank you so much for suggestion i have installed our newest version is v1.9.1, and aws plugin version is v1.5.0 in my eks cluster i am abele to create sub folder and push backups in to subfolders, for testing i have annotate the pod in my namespace then i deleted name space and all resources in namespace then i restore from my backups but while restoring my restoration is getting fail and pods and replicas were not running automatically. please help me to resolve the issue and find the logs thank you so much. When i describe restore i found below pleas give solution for restoration successfully. Restic Restores: Failed: test-ns/test-bvsr-8fc89d8c55-9gjd4: test-pvc ? One Question does it required to annotate the pods to get restore successfully ?...

banyalasuresh commented 2 years ago

@qiuming-best could you please assist me on restic restore, Restore is getting fail. please find below.


NAME               BACKUP    STATUS            STARTED                         COMPLETED                       ERRORS   WARNINGS   CREATED                         SELECTOR
test-ns-restore    test-ns   PartiallyFailed   2022-09-20 22:52:57 +0000 UTC   2022-09-20 22:53:30 +0000 UTC   1        0          2022-09-20 22:52:56 +0000 UTC   <none>
test-ns-restore1   test-ns   PartiallyFailed   2022-09-21 09:30:03 +0000 UTC   2022-09-21 09:30:34 +0000 UTC   1        0          2022-09-21 09:30:02 +0000 UTC   <none> ```
banyalasuresh commented 2 years ago

when i describe the restore please find below information and assist me to resolve the issue.

: exit status 1
  Cluster:    <none>
  Namespaces: <none>

Backup:  test-ns

Namespaces:
  Included:  all namespaces found in the backup
  Excluded:  <none>

Resources:
  Included:        *
  Excluded:        nodes, events, events.events.k8s.io, backups.velero.io, restores.velero.io, resticrepositories.velero.io
  Cluster-scoped:  auto

Namespace mappings:  <none>

Label selector:  <none>

Restore PVs:  auto

Restic Restores:
  Failed:
    test-ns/test-mdir-6fc89d8c55-9gjd4: test-pvc

Existing Resource Policy:   <none>

Preserve Service NodePorts:  auto
stale[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

pavmohan1 commented 1 year ago

Using k describe backup may be useful in finding out the failure reason

stale[bot] commented 1 year ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

stale[bot] commented 1 year ago

Closing the stale issue.