We tried doing a restore including 3 EBS snapshots today. Velero's logs indicate they restored successfully. There are PVs and PVCs created in the namespace as they should, and the PVs have volume IDs. However, the volume IDs did not exist in EC2 anywhere, and kubernetes failed to attach them (volume not found errors). When we got to the bottom of it, our velero IAM policy didn't have permission for kms:ReEncrypt* on the key used to encrypt the volumes, and an asynchronous error was generated, and the volumes never finished creating.
It would be really nice if velero could check the status of EBS volume restores and only mark the restore complete/successful if the EBS volume gets created, and display any errors related to lack of permissions if possible.
skriss
commented
4 years ago
We tried doing a restore including 3 EBS snapshots today. Velero's logs indicate they restored successfully. There are PVs and PVCs created in the namespace as they should, and the PVs have volume IDs. However, the volume IDs did not exist in EC2 anywhere, and kubernetes failed to attach them (volume not found errors). When we got to the bottom of it, our velero IAM policy didn't have permission for
kms:ReEncrypt*on the key used to encrypt the volumes, and an asynchronous error was generated, and the volumes never finished creating.It would be really nice if velero could check the status of EBS volume restores and only mark the restore complete/successful if the EBS volume gets created, and display any errors related to lack of permissions if possible.