The velero-plugin-for-aws supports tagging the objects in S3, as advertised here and implemented here. This requires the s3:PutObjectTagging permission for the bucket, which the current velero policy here does not contain.
[x] ✋ I have searched the open/closed issues and my issue is not listed.
The following steps show not only how to reproduce the behavior, but also how the fix in this PR resolves the issue:
Clone the repo above, which contains the minimal reproducible example for the issue.
Run tofu init and tofu apply to create and configure the required resources (VPC, EKS, EBS CSI driver, Snapshot Controller, Velero, Pod w/ PVC and data, etc.).
Wait until everything is set up and the ebs-pvc-pod has run and saved data to the volume.
Run velero backup create ebs-test-broken --include-namespaces=default --snapshot-move-data.
See the error in the logs described below, and note that velero backup describe ebs-test-broken shows the backup failed.
Switch the use_fixed_velero_policy variable in terraform.tfvars to true in the chrisRedwine/velero-mre repo.
Run tofu apply
Wait for the 3 velero pods to restart so that they use the fixed IRSA.
Run velero backup create ebs-test-fixed --include-namespaces=default --snapshot-move-data.
Notice no errors in the logs, and that velero backup describe ebs-test-fixed shows the backup succeeded.
(Make sure to clean up with tofu destroy afterwards)
Expected behavior
The command should succeed without any errors.
Actual behavior
The command fails with the following permissions issue (taken from the velero pod logs):
level=error msg="Error uploading log file" backup=ebs-test bucket=velero-backups-velero-mre error="rpc error: code = Unknown desc = error putting object backups/backups/ebs-test/ebs-test-logs.gz: operation error S3: PutObject, https response error StatusCode: 403, RequestID: <redacted>, HostID: <redacted>, api error AccessDenied: User: arn:aws:sts::<redacted>:assumed-role/velero-irsa-velero-mre/<redacted> is not authorized to perform: s3:PutObjectTagging on resource: \"arn:aws:s3:::velero-backups-velero-mre/backups/backups/ebs-test/ebs-test-logs.gz\" because no identity-based policy allows the s3:PutObjectTagging action" error.file="/go/src/velero-plugin-for-aws/velero-plugin-for-aws/object_store.go:279" error.function="main.(*ObjectStore).PutObject" logSource="pkg/persistence/object_store.go:256" prefix=backups
level=error msg="backup failed" backuprequest=velero/ebs-test controller=backup error="rpc error: code = Unknown desc = error putting object backups/backups/ebs-test/velero-backup.json: operation error S3: PutObject, https response error StatusCode: 403, RequestID: <redacted>, HostID: <redacted>, api error AccessDenied: User: arn:aws:sts::<redacted>:assumed-role/velero-irsa-velero-mre/<redacted> is not authorized to perform: s3:PutObjectTagging on resource: \"arn:aws:s3:::velero-backups-velero-mre/backups/backups/ebs-test/velero-backup.json\" because no identity-based policy allows the s3:PutObjectTagging action" logSource="pkg/controller/backup_controller.go:287"
Notice that the IRSA role lacks the s3:PutObjectTagging permission, which it needs because of this code.
Description
The
velero-plugin-for-aws
supports tagging the objects in S3, as advertised here and implemented here. This requires thes3:PutObjectTagging
permission for the bucket, which the current velero policy here does not contain.Versions
Module version [Required]:
5.44.0
Terraform version: OpenTofu
1.7.3
Provider version(s): AWS
5.66.0
Reproduction Code [Required]
Probably a bit overkill, but: https://github.com/chrisRedwine/velero-mre
The following steps show not only how to reproduce the behavior, but also how the fix in this PR resolves the issue:
tofu init
andtofu apply
to create and configure the required resources (VPC, EKS, EBS CSI driver, Snapshot Controller, Velero, Pod w/ PVC and data, etc.).ebs-pvc-pod
has run and saved data to the volume.velero backup create ebs-test-broken --include-namespaces=default --snapshot-move-data
.velero backup describe ebs-test-broken
shows the backup failed.use_fixed_velero_policy
variable interraform.tfvars
totrue
in thechrisRedwine/velero-mre
repo.tofu apply
velero
pods to restart so that they use the fixed IRSA.velero backup create ebs-test-fixed --include-namespaces=default --snapshot-move-data
.velero backup describe ebs-test-fixed
shows the backup succeeded.tofu destroy
afterwards)Expected behavior
The command should succeed without any errors.
Actual behavior
The command fails with the following permissions issue (taken from the
velero
pod logs):Notice that the IRSA role lacks the
s3:PutObjectTagging
permission, which it needs because of this code.Additional context
velero-plugin-for-aws
repo.