AWS/EKS: velero backup partiallyFailed - error 403

[DavidRisaro]

What steps did you take and what happened: [A clear and concise description of what the bug is, and what commands you ran.)

I'm running a scheduled backup and a manual one (velero create backup all-namespaces), and it always (partially) fails on 8 specific pods, all in different namespaces, with error

time="2022-12-06T19:03:29Z" level=info msg="1 errors encountered backup up item" backup=velero/all-namespaces-20221206190325 logSource="pkg/backup/backup.go:413" name=prometheus-grafana-5674f4884b-7jgxd
time="2022-12-06T19:03:29Z" level=error msg="Error backing up item" backup=velero/all-namespaces-20221206190325 error="error getting volume info: rpc error: code = Unknown desc = UnauthorizedOperation: You are not authorized to perform this operation.\n\tstatus code: 403, request id: 3688a339-4214-424f-bf0e-c41ca1601c1a" logSource="pkg/backup/backup.go:417" name=prometheus-grafana-5674f4884b-7jgxd

Every other pod, volume, etc appears to backup perfectly, or at least I wasn't able to find an error yet.

Velero has full access to S3, so I don't think the issue comes from there.

What did you expect to happen: Backup finish correctly with all resources backed up.

The following information will help us better understand what's going on:

bundle-2022-12-07-10-13-39.tar.gz

Environment:

• Velero version (use velero version): Client & Server: Version: v1.9.2
• Velero features (use velero client config get features): features:
• Kubernetes version (use kubectl version): Client Version: v1.25.4 Kustomize Version: v4.5.7 Server Version: v1.22.15-eks-fb459a0
• Kubernetes installer & version:
• Cloud provider or hardware configuration: AWS
• OS (e.g. from /etc/os-release): PRETTY_NAME="Ubuntu 22.04.1 LTS"

Vote on this issue!

This is an invitation to the Velero community to vote on issues, you can see the project's top voted issues listed here.
Use the "reaction smiley face" up to the right of this comment to vote.

• :+1: for "I would like to see this bug fixed as soon as possible"
• :-1: for "There are more important bugs to focus on right now"

[ywk253100]

Did you follow the guide to set proper permission for Velero? Seems Velero has no permission to do the snapshot for volumes

[DavidRisaro]

Did you follow the guide to set proper permission for Velero? Seems Velero has no permission to do the snapshot for volumes

Yes, I created the user with all the privileges. Something I forgot was that a couple weeks back (when I first tested this) it finished completely.

PS: I close the thread by accident, sorry.