search

vmware-tanzu / velero

Backup and migrate Kubernetes applications and their persistent volumes
https://velero.io
Apache License 2.0
8.66k stars 1.39k forks source link

Restore from backup from another region in Azure #5474

Open naama-connecteam opened 1 year ago

naama-connecteam commented 1 year ago

We’re trying to use Velero to restore a backup across different regions in Azure. We copied the snapshots manually to a new storage account (in a new resource group) we created in the region where we want to restore, and manually changed the references to the snapshots location in <BACKUP_NAME>-volumesnapshots.json file, specifically the field “providerSnapshotID”. The change in the json is from MC_k8s-env-<SRC_ENV>-production_env-<SRC_ENV>-production_<SRC_REGION> to MC_k8s-env-<DEST_ENV>-production_env-<DEST_ENV>-production_<DEST_REGION>.

When we created the restore, we got this error for all the volumes:

error executing PVAction for persistentvolumes/pvc-XXX: rpc error: code = Unknown desc = compute.DisksClient#CreateOrUpdate: Failure sending request: StatusCode=404 -- Original Error: Code="ResourceGroupNotFound" Message="Resource group 'MC_<RESOURCEGROUP_NAME>' could not be found."

The resource group name in the error doesn’t exist. It is a combination of the new resourcegroup name and the old region: MC_k8s-env-<DEST_ENV>-production_env-<DEST_ENV>-production_<SRC_REGION> which we cannot find in any of the json files and we don’t know how it is generated.

Also I can't find any error when I run velero restore logs backup-mongo-XXX, or look at the pod's logs.

qiuming-best commented 1 year ago

@naama-connecteam Did you use CSI or Azure plugin to do the volume snapshot? if so, the providerSnapshotID filed should be generated by Azure volume snapshot, which format would be like this /subscriptions/2261f3e7-d159-48fe-95a3-0e6a96exxx/resourceGroups/MC_xxx-test_eastus/providers/Microsoft.Compute/snapshots/restore-82ca08c2-3972-4511-acdc-b0e74559c24-8cd76dd9-95d2-4353-b78e-xxxx", the providerSnapshotID is generated by the provider's snapshot service, Which is a random character string and Velero retrieved it, I'm not sure about the format you combined with is correct

slawekww commented 1 year ago

Is it any option to use Velero and backup volume snapshots into two different Azure resource groups with two different locations?

Once I've used Azure CSI instead of Velero CSI plugin, it is possible to create snapshots into two different Azure resource groups but location of snapshots are always the same as AKS location. If it is not supported by Velero (and Azure maybe) and Azure snapshots are copied manually to different Azure location, what should be changed in Velero stored backups to use copied Azure snapshots?

stefanopulze commented 3 months ago

Any news about this topic?