[Support]: IONOS S3 connection - InvalidAccessKeyId

[Ahmadre]

What steps did you take and what happened:

I needed to connect to the IONOS S3 Bucket and it's working fine if you try it with Postman, but If I try to put the credentials into the values.yaml I always get:

Error-Message

Click To Expand

```bash time="2021-03-12T23:58:58Z" level=warning msg="Invalid backup locations detected: available/unavailable/unknown: 2/2/0, Backup location \"saverso\" is unavailable: rpc error: code = Unknown desc = InvalidAccessKeyId: \n\tstatus code: 403, request id: tx0000000000000034c4f0d-00604c0042-9d6bb8b-fra1b, host id: ; Backup location \"default\" is unavailable: rpc error: code = Unknown desc = InvalidAccessKeyId: \n\tstatus code: 403, request id: tx0000000000000034a8f9e-00604c0042-9d3682b-fra1b, host id: )" controller=backupstoragelocation logSource="pkg/controller/backupstoragelocation_controller.go:157" ```

In the following your can the value I setup:

Value.yaml

Click To Expand

```yaml image: repository: velero/velero tag: main pullPolicy: IfNotPresent initContainers: - name: velero-plugin-for-aws image: velero/velero-plugin-for-aws:v1.1.0 imagePullPolicy: IfNotPresent volumeMounts: - mountPath: /target name: plugins configuration: provider: aws backupStorageLocation: name: backups bucket: saverso config: region: s3-de-central s3Url: "https://s3-de-central.profitbricks.com:443" volumeSnapshotLocation: name: backups config: region: s3-de-central credentials: name: credentials secretContents: cloud: | [default] aws_access_key_id=xxx aws_secret_access_key=xxxxxx schedules: saversobackup: schedule: "0 3 * * *" template: includeClusterResources: true storageLocation: backups ttl: "240h0m0s" includedResources: - '*' excludedNamespaces: - kube-node-lease - kube-public - kube-system - default - nginx-ingress ```

Stacktrace

Click To Expand

```bash time="2021-03-12T18:50:31Z" level=info msg="setting log-level to INFO" logSource="pkg/cmd/server/server.go:172" time="2021-03-12T18:50:31Z" level=info msg="Starting Velero server main (2cddda84c507de1cf62ab34648dfb1a109aaa813)" logSource="pkg/cmd/server/server.go:174" time="2021-03-12T18:50:31Z" level=info msg="No feature flags enabled" logSource="pkg/cmd/server/server.go:178" time="2021-03-12T18:50:31Z" level=info msg="registering plugin" command=/velero kind=BackupItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/crd-remap-version time="2021-03-12T18:50:31Z" level=info msg="registering plugin" command=/velero kind=BackupItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/pod time="2021-03-12T18:50:31Z" level=info msg="registering plugin" command=/velero kind=BackupItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/pv time="2021-03-12T18:50:31Z" level=info msg="registering plugin" command=/velero kind=BackupItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/service-account time="2021-03-12T18:50:31Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/add-pv-from-pvc time="2021-03-12T18:50:31Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/add-pvc-from-pod time="2021-03-12T18:50:31Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/change-pvc-node-selector time="2021-03-12T18:50:31Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/change-storage-class time="2021-03-12T18:50:31Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/cluster-role-bindings time="2021-03-12T18:50:31Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/crd-preserve-fields time="2021-03-12T18:50:31Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/init-restore-hook time="2021-03-12T18:50:31Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/job time="2021-03-12T18:50:31Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/pod time="2021-03-12T18:50:31Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/restic time="2021-03-12T18:50:31Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/role-bindings time="2021-03-12T18:50:31Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/service time="2021-03-12T18:50:31Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/service-account time="2021-03-12T18:50:31Z" level=info msg="registering plugin" command=/plugins/velero-plugin-for-aws kind=VolumeSnapshotter logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/aws time="2021-03-12T18:50:31Z" level=info msg="registering plugin" command=/plugins/velero-plugin-for-aws kind=ObjectStore logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/aws time="2021-03-12T18:50:32Z" level=info msg="Checking existence of namespace." logSource="pkg/cmd/server/server.go:376" namespace=velero time="2021-03-12T18:50:32Z" level=info msg="Namespace exists" logSource="pkg/cmd/server/server.go:382" namespace=velero I0312 18:50:34.829574 1 request.go:645] Throttling request took 1.025398027s, request: GET:https://10.233.0.1:443/apis/crd.projectcalico.org/v1?timeout=32s time="2021-03-12T18:50:36Z" level=info msg="Checking existence of Velero custom resource definitions" logSource="pkg/cmd/server/server.go:411" time="2021-03-12T18:50:36Z" level=info msg="All Velero custom resource definitions exist" logSource="pkg/cmd/server/server.go:445" time="2021-03-12T18:50:36Z" level=warning msg="Velero restic daemonset not found; restic backups/restores will not work until it's created" logSource="pkg/cmd/server/server.go:496" time="2021-03-12T18:50:36Z" level=info msg="Starting controllers" logSource="pkg/cmd/server/server.go:555" time="2021-03-12T18:50:36Z" level=info msg="Starting metric server at address [:8085]" logSource="pkg/cmd/server/server.go:562" time="2021-03-12T18:50:36Z" level=info msg="Backup sync period is 1m0s" logSource="pkg/controller/backup_sync_controller.go:77" time="2021-03-12T18:50:36Z" level=info msg="Waiting for informer caches to sync" logSource="pkg/cmd/server/server.go:800" time="2021-03-12T18:50:39Z" level=info msg="Creating schedule velero-saversobackup" controller=schedule logSource="pkg/controller/schedule_controller.go:105" time="2021-03-12T18:50:39Z" level=info msg="Done waiting for informer caches to sync" logSource="pkg/cmd/server/server.go:803" time="2021-03-12T18:50:39Z" level=info msg="Informer cache synced" informer="*v1.ResticRepository" logSource="pkg/cmd/server/server.go:814" time="2021-03-12T18:50:39Z" level=info msg="Informer cache synced" informer="*v1.Backup" logSource="pkg/cmd/server/server.go:814" time="2021-03-12T18:50:39Z" level=info msg="Informer cache synced" informer="*v1.VolumeSnapshotLocation" logSource="pkg/cmd/server/server.go:814" time="2021-03-12T18:50:39Z" level=info msg="Informer cache synced" informer="*v1.Restore" logSource="pkg/cmd/server/server.go:814" time="2021-03-12T18:50:39Z" level=info msg="Informer cache synced" informer="*v1.Schedule" logSource="pkg/cmd/server/server.go:814" time="2021-03-12T18:50:39Z" level=info msg="Informer cache synced" informer="*v1.DeleteBackupRequest" logSource="pkg/cmd/server/server.go:814" time="2021-03-12T18:50:39Z" level=info msg="Informer cache synced" informer="*v1.PodVolumeBackup" logSource="pkg/cmd/server/server.go:814" time="2021-03-12T18:50:39Z" level=info msg="Server starting..." logSource="pkg/cmd/server/server.go:872" time="2021-03-12T18:50:39Z" level=info msg="Starting controller" controller=backup-sync logSource="pkg/controller/generic_controller.go:76" time="2021-03-12T18:50:39Z" level=info msg="Starting controller" controller=backup logSource="pkg/controller/generic_controller.go:76" time="2021-03-12T18:50:39Z" level=info msg="Starting controller" controller=gc logSource="pkg/controller/generic_controller.go:76" time="2021-03-12T18:50:39Z" level=info msg="Starting controller" controller=backup-deletion logSource="pkg/controller/generic_controller.go:76" time="2021-03-12T18:50:39Z" level=info msg="Starting controller" controller=restore logSource="pkg/controller/generic_controller.go:76" time="2021-03-12T18:50:39Z" level=info msg="Backup has expired" backup=velero/velero-saversobackup-20210311191333 controller=gc expiration="2021-03-12 18:13:33 +0000 UTC" logSource="pkg/controller/gc_controller.go:135" time="2021-03-12T18:50:39Z" level=info msg="Starting controller" controller=schedule logSource="pkg/controller/generic_controller.go:76" time="2021-03-12T18:50:39Z" level=info msg="Checking for expired DeleteBackupRequests" controller=backup-deletion logSource="pkg/controller/backup_deletion_controller.go:596" time="2021-03-12T18:50:39Z" level=info msg="Done checking for expired DeleteBackupRequests" controller=backup-deletion logSource="pkg/controller/backup_deletion_controller.go:624" time="2021-03-12T18:50:39Z" level=info msg="Starting controller" controller=restic-repo logSource="pkg/controller/generic_controller.go:76" time="2021-03-12T18:50:39Z" level=info msg="Creating a new deletion request" backup=velero/velero-saversobackup-20210311191333 controller=gc expiration="2021-03-12 18:13:33 +0000 UTC" logSource="pkg/controller/gc_controller.go:173" time="2021-03-12T18:50:39Z" level=info msg="Validating backup storage location" backup-storage-location=default controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:114" time="2021-03-12T18:50:39Z" level=error msg="Error listing backups in backup store" backupLocation=default controller=backup-sync error="rpc error: code = Unknown desc = InvalidAccessKeyId: \n\tstatus code: 403, request id: tx00000000000000305c39e-00604bb7ff-9d67586-fra1b, host id: " error.file="/go/src/github.com/vmware-tanzu/velero-plugin-for-aws/velero-plugin-for-aws/object_store.go:331" error.function="main.(*ObjectStore).ListCommonPrefixes" logSource="pkg/controller/backup_sync_controller.go:182" time="2021-03-12T18:50:39Z" level=info msg="Backup storage location is invalid, marking as unavailable" backup-storage-location=default controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:117" time="2021-03-12T18:50:39Z" level=info msg="Validating backup storage location" backup-storage-location=saverso controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:114" time="2021-03-12T18:50:39Z" level=error msg="Error listing backups in backup store" backupLocation=saverso controller=backup-sync error="rpc error: code = Unknown desc = InvalidAccessKeyId: \n\tstatus code: 403, request id: tx00000000000000305c3b8-00604bb7ff-9d67586-fra1b, host id: " error.file="/go/src/github.com/vmware-tanzu/velero-plugin-for-aws/velero-plugin-for-aws/object_store.go:331" error.function="main.(*ObjectStore).ListCommonPrefixes" logSource="pkg/controller/backup_sync_controller.go:182" time="2021-03-12T18:50:39Z" level=info msg="Backup storage location is invalid, marking as unavailable" backup-storage-location=saverso controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:117" time="2021-03-12T18:50:39Z" level=info msg="Validating backup storage location" backup-storage-location=s3 controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:114" time="2021-03-12T18:50:39Z" level=info msg="Backup storage location valid, marking as available" backup-storage-location=s3 controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:121" time="2021-03-12T18:50:39Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, Backup storage location \"default\" is unavailable: rpc error: code = Unknown desc = InvalidAccessKeyId: \n\tstatus code: 403, request id: tx00000000000000305c39f-00604bb7ff-9d67586-fra1b, host id: ; Backup storage location \"saverso\" is unavailable: rpc error: code = Unknown desc = InvalidAccessKeyId: \n\tstatus code: 403, request id: tx00000000000000304687c-00604bb7ff-9db4ea4-fra1b, host id: )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T18:50:39Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T18:50:39Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T18:50:39Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T18:50:39Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T18:50:39Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T18:50:39Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T18:50:39Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T18:50:39Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T18:50:39Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T18:50:39Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T18:50:39Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T18:50:39Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T18:50:39Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T18:50:39Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T18:50:39Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T18:50:39Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T18:50:39Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T18:50:39Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T18:50:39Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T18:50:39Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T18:50:39Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T18:50:39Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T18:50:39Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T18:50:40Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T18:50:40Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T18:50:40Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T18:50:40Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T18:50:40Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T18:50:40Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T18:50:40Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T18:50:40Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T18:50:41Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T18:50:41Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T18:50:41Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T18:50:41Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T18:50:41Z" level=info msg="Backup has expired" backup=velero/velero-saversobackup-20210311193853 controller=gc expiration="2021-03-12 18:38:53 +0000 UTC" logSource="pkg/controller/gc_controller.go:135" time="2021-03-12T18:50:41Z" level=info msg="Creating a new deletion request" backup=velero/velero-saversobackup-20210311193853 controller=gc expiration="2021-03-12 18:38:53 +0000 UTC" logSource="pkg/controller/gc_controller.go:173" time="2021-03-12T18:50:41Z" level=info msg="Removing existing deletion requests for backup" backup=velero-saversobackup-20210311191333 controller=backup-deletion logSource="pkg/controller/backup_deletion_controller.go:473" name=velero-saversobackup-20210311191333-7m9ql namespace=velero time="2021-03-12T18:50:41Z" level=info msg="Backup has expired" backup=velero/velero-saversobackup-20210311194813 controller=gc expiration="2021-03-12 18:48:13 +0000 UTC" logSource="pkg/controller/gc_controller.go:135" time="2021-03-12T18:50:41Z" level=info msg="Creating a new deletion request" backup=velero/velero-saversobackup-20210311194813 controller=gc expiration="2021-03-12 18:48:13 +0000 UTC" logSource="pkg/controller/gc_controller.go:173" time="2021-03-12T18:50:42Z" level=info msg="Backup has expired" backup=velero/velero-saversobackup-20210311190359 controller=gc expiration="2021-03-12 18:03:59 +0000 UTC" logSource="pkg/controller/gc_controller.go:135" time="2021-03-12T18:50:42Z" level=info msg="Creating a new deletion request" backup=velero/velero-saversobackup-20210311190359 controller=gc expiration="2021-03-12 18:03:59 +0000 UTC" logSource="pkg/controller/gc_controller.go:173" time="2021-03-12T18:50:42Z" level=info msg="Backup has expired" backup=velero/velero-saversobackup-20210311192114 controller=gc expiration="2021-03-12 18:21:14 +0000 UTC" logSource="pkg/controller/gc_controller.go:135" time="2021-03-12T18:50:42Z" level=info msg="Creating a new deletion request" backup=velero/velero-saversobackup-20210311192114 controller=gc expiration="2021-03-12 18:21:14 +0000 UTC" logSource="pkg/controller/gc_controller.go:173" time="2021-03-12T18:50:42Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T18:50:42Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T18:50:42Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T18:50:42Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T18:50:42Z" level=info msg="Removing PV snapshots" backup=velero-saversobackup-20210311191333 controller=backup-deletion logSource="pkg/controller/backup_deletion_controller.go:333" name=velero-saversobackup-20210311191333-7m9ql namespace=velero time="2021-03-12T18:50:42Z" level=info msg="Removing restic snapshots" backup=velero-saversobackup-20210311191333 controller=backup-deletion logSource="pkg/controller/backup_deletion_controller.go:359" name=velero-saversobackup-20210311191333-7m9ql namespace=velero time="2021-03-12T18:50:42Z" level=info msg="Removing backup from backup storage" backup=velero-saversobackup-20210311191333 controller=backup-deletion logSource="pkg/controller/backup_deletion_controller.go:367" name=velero-saversobackup-20210311191333-7m9ql namespace=velero time="2021-03-12T18:50:42Z" level=info msg="Removing restores" backup=velero-saversobackup-20210311191333 controller=backup-deletion logSource="pkg/controller/backup_deletion_controller.go:389" name=velero-saversobackup-20210311191333-7m9ql namespace=velero time="2021-03-12T18:50:42Z" level=info msg="Removing existing deletion requests for backup" backup=velero-saversobackup-20210311193853 controller=backup-deletion logSource="pkg/controller/backup_deletion_controller.go:473" name=velero-saversobackup-20210311193853-qtwhq namespace=velero time="2021-03-12T18:50:42Z" level=info msg="Removing PV snapshots" backup=velero-saversobackup-20210311193853 controller=backup-deletion logSource="pkg/controller/backup_deletion_controller.go:333" name=velero-saversobackup-20210311193853-qtwhq namespace=velero time="2021-03-12T18:50:42Z" level=info msg="Removing restic snapshots" backup=velero-saversobackup-20210311193853 controller=backup-deletion logSource="pkg/controller/backup_deletion_controller.go:359" name=velero-saversobackup-20210311193853-qtwhq namespace=velero time="2021-03-12T18:50:42Z" level=info msg="Removing backup from backup storage" backup=velero-saversobackup-20210311193853 controller=backup-deletion logSource="pkg/controller/backup_deletion_controller.go:367" name=velero-saversobackup-20210311193853-qtwhq namespace=velero time="2021-03-12T18:50:43Z" level=info msg="Removing restores" backup=velero-saversobackup-20210311193853 controller=backup-deletion logSource="pkg/controller/backup_deletion_controller.go:389" name=velero-saversobackup-20210311193853-qtwhq namespace=velero time="2021-03-12T18:50:43Z" level=info msg="Removing existing deletion requests for backup" backup=velero-saversobackup-20210311194813 controller=backup-deletion logSource="pkg/controller/backup_deletion_controller.go:473" name=velero-saversobackup-20210311194813-d97n6 namespace=velero time="2021-03-12T18:50:43Z" level=info msg="Backup has expired" backup=velero/velero-saversobackup-20210311194813 controller=gc expiration="2021-03-12 18:48:13 +0000 UTC" logSource="pkg/controller/gc_controller.go:135" time="2021-03-12T18:50:43Z" level=info msg="Backup already has a pending deletion request" backup=velero/velero-saversobackup-20210311194813 controller=gc expiration="2021-03-12 18:48:13 +0000 UTC" logSource="pkg/controller/gc_controller.go:168" time="2021-03-12T18:50:43Z" level=info msg="Removing PV snapshots" backup=velero-saversobackup-20210311194813 controller=backup-deletion logSource="pkg/controller/backup_deletion_controller.go:333" name=velero-saversobackup-20210311194813-d97n6 namespace=velero time="2021-03-12T18:50:43Z" level=info msg="Removing restic snapshots" backup=velero-saversobackup-20210311194813 controller=backup-deletion logSource="pkg/controller/backup_deletion_controller.go:359" name=velero-saversobackup-20210311194813-d97n6 namespace=velero time="2021-03-12T18:50:43Z" level=info msg="Removing backup from backup storage" backup=velero-saversobackup-20210311194813 controller=backup-deletion logSource="pkg/controller/backup_deletion_controller.go:367" name=velero-saversobackup-20210311194813-d97n6 namespace=velero time="2021-03-12T18:50:43Z" level=info msg="Removing restores" backup=velero-saversobackup-20210311194813 controller=backup-deletion logSource="pkg/controller/backup_deletion_controller.go:389" name=velero-saversobackup-20210311194813-d97n6 namespace=velero time="2021-03-12T18:50:43Z" level=info msg="Removing existing deletion requests for backup" backup=velero-saversobackup-20210311190359 controller=backup-deletion logSource="pkg/controller/backup_deletion_controller.go:473" name=velero-saversobackup-20210311190359-m92c4 namespace=velero time="2021-03-12T18:50:43Z" level=info msg="Removing PV snapshots" backup=velero-saversobackup-20210311190359 controller=backup-deletion logSource="pkg/controller/backup_deletion_controller.go:333" name=velero-saversobackup-20210311190359-m92c4 namespace=velero time="2021-03-12T18:50:43Z" level=info msg="Removing restic snapshots" backup=velero-saversobackup-20210311190359 controller=backup-deletion logSource="pkg/controller/backup_deletion_controller.go:359" name=velero-saversobackup-20210311190359-m92c4 namespace=velero time="2021-03-12T18:50:43Z" level=info msg="Removing backup from backup storage" backup=velero-saversobackup-20210311190359 controller=backup-deletion logSource="pkg/controller/backup_deletion_controller.go:367" name=velero-saversobackup-20210311190359-m92c4 namespace=velero time="2021-03-12T18:50:43Z" level=info msg="Removing restores" backup=velero-saversobackup-20210311190359 controller=backup-deletion logSource="pkg/controller/backup_deletion_controller.go:389" name=velero-saversobackup-20210311190359-m92c4 namespace=velero time="2021-03-12T18:50:43Z" level=info msg="Removing existing deletion requests for backup" backup=velero-saversobackup-20210311192114 controller=backup-deletion logSource="pkg/controller/backup_deletion_controller.go:473" name=velero-saversobackup-20210311192114-rfvw9 namespace=velero time="2021-03-12T18:50:44Z" level=info msg="Removing PV snapshots" backup=velero-saversobackup-20210311192114 controller=backup-deletion logSource="pkg/controller/backup_deletion_controller.go:333" name=velero-saversobackup-20210311192114-rfvw9 namespace=velero time="2021-03-12T18:50:44Z" level=info msg="Removing restic snapshots" backup=velero-saversobackup-20210311192114 controller=backup-deletion logSource="pkg/controller/backup_deletion_controller.go:359" name=velero-saversobackup-20210311192114-rfvw9 namespace=velero time="2021-03-12T18:50:44Z" level=info msg="Removing backup from backup storage" backup=velero-saversobackup-20210311192114 controller=backup-deletion logSource="pkg/controller/backup_deletion_controller.go:367" name=velero-saversobackup-20210311192114-rfvw9 namespace=velero time="2021-03-12T18:50:44Z" level=info msg="Removing restores" backup=velero-saversobackup-20210311192114 controller=backup-deletion logSource="pkg/controller/backup_deletion_controller.go:389" name=velero-saversobackup-20210311192114-rfvw9 namespace=velero time="2021-03-12T18:50:44Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T18:50:44Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T18:50:44Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T18:50:44Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T18:50:50Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T18:50:50Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T18:50:50Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T18:50:50Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T18:51:00Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T18:51:00Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T18:51:00Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T18:51:00Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T18:51:09Z" level=error msg="Error listing backups in backup store" backupLocation=default controller=backup-sync error="rpc error: code = Unknown desc = InvalidAccessKeyId: \n\tstatus code: 403, request id: tx0000000000000030489c5-00604bb81d-9db4ea4-fra1b, host id: " error.file="/go/src/github.com/vmware-tanzu/velero-plugin-for-aws/velero-plugin-for-aws/object_store.go:331" error.function="main.(*ObjectStore).ListCommonPrefixes" logSource="pkg/controller/backup_sync_controller.go:182" time="2021-03-12T18:51:09Z" level=error msg="Error listing backups in backup store" backupLocation=saverso controller=backup-sync error="rpc error: code = Unknown desc = InvalidAccessKeyId: \n\tstatus code: 403, request id: tx00000000000000302c78c-00604bb81d-9d6bb8b-fra1b, host id: " error.file="/go/src/github.com/vmware-tanzu/velero-plugin-for-aws/velero-plugin-for-aws/object_store.go:331" error.function="main.(*ObjectStore).ListCommonPrefixes" logSource="pkg/controller/backup_sync_controller.go:182" time="2021-03-12T18:51:20Z" level=info msg="Validating backup storage location" backup-storage-location=backups controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:114" time="2021-03-12T18:51:20Z" level=info msg="Backup storage location valid, marking as available" backup-storage-location=backups controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:121" time="2021-03-12T18:51:20Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T18:51:20Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T18:51:20Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T18:51:20Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T18:51:39Z" level=error msg="Error listing backups in backup store" backupLocation=default controller=backup-sync error="rpc error: code = Unknown desc = InvalidAccessKeyId: \n\tstatus code: 403, request id: tx00000000000000304a9c0-00604bb83b-9db4ea4-fra1b, host id: " error.file="/go/src/github.com/vmware-tanzu/velero-plugin-for-aws/velero-plugin-for-aws/object_store.go:331" error.function="main.(*ObjectStore).ListCommonPrefixes" logSource="pkg/controller/backup_sync_controller.go:182" time="2021-03-12T18:51:40Z" level=info msg="Validating backup storage location" backup-storage-location=default controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:114" time="2021-03-12T18:51:40Z" level=error msg="Error listing backups in backup store" backupLocation=saverso controller=backup-sync error="rpc error: code = Unknown desc = InvalidAccessKeyId: \n\tstatus code: 403, request id: tx00000000000000304a9d7-00604bb83c-9db4ea4-fra1b, host id: " error.file="/go/src/github.com/vmware-tanzu/velero-plugin-for-aws/velero-plugin-for-aws/object_store.go:331" error.function="main.(*ObjectStore).ListCommonPrefixes" logSource="pkg/controller/backup_sync_controller.go:182" time="2021-03-12T18:51:40Z" level=info msg="Backup storage location is invalid, marking as unavailable" backup-storage-location=default controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:117" time="2021-03-12T18:51:40Z" level=info msg="Validating backup storage location" backup-storage-location=saverso controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:114" time="2021-03-12T18:51:40Z" level=info msg="Backup storage location is invalid, marking as unavailable" backup-storage-location=saverso controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:117" time="2021-03-12T18:51:40Z" level=info msg="Validating backup storage location" backup-storage-location=s3 controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:114" time="2021-03-12T18:51:40Z" level=info msg="Backup storage location valid, marking as available" backup-storage-location=s3 controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:121" time="2021-03-12T18:51:40Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, Backup storage location \"default\" is unavailable: rpc error: code = Unknown desc = InvalidAccessKeyId: \n\tstatus code: 403, request id: tx0000000000000030127f1-00604bb83c-9d3682b-fra1b, host id: ; Backup storage location \"saverso\" is unavailable: rpc error: code = Unknown desc = InvalidAccessKeyId: \n\tstatus code: 403, request id: tx000000000000003060544-00604bb83c-9d67586-fra1b, host id: )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T18:51:40Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T18:51:40Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T18:51:40Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T18:52:01Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T18:52:01Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T18:52:01Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T18:52:01Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T18:52:10Z" level=error msg="Error listing backups in backup store" backupLocation=default controller=backup-sync error="rpc error: code = Unknown desc = InvalidAccessKeyId: \n\tstatus code: 403, request id: tx000000000000003014759-00604bb85a-9d3682b-fra1b, host id: " error.file="/go/src/github.com/vmware-tanzu/velero-plugin-for-aws/velero-plugin-for-aws/object_store.go:331" error.function="main.(*ObjectStore).ListCommonPrefixes" logSource="pkg/controller/backup_sync_controller.go:182" time="2021-03-12T18:52:10Z" level=error msg="Error listing backups in backup store" backupLocation=saverso controller=backup-sync error="rpc error: code = Unknown desc = InvalidAccessKeyId: \n\tstatus code: 403, request id: tx00000000000000304c95b-00604bb85a-9db4ea4-fra1b, host id: " error.file="/go/src/github.com/vmware-tanzu/velero-plugin-for-aws/velero-plugin-for-aws/object_store.go:331" error.function="main.(*ObjectStore).ListCommonPrefixes" logSource="pkg/controller/backup_sync_controller.go:182" time="2021-03-12T18:52:40Z" level=error msg="Error listing backups in backup store" backupLocation=default controller=backup-sync error="rpc error: code = Unknown desc = InvalidAccessKeyId: \n\tstatus code: 403, request id: tx0000000000000030165e5-00604bb878-9d3682b-fra1b, host id: " error.file="/go/src/github.com/vmware-tanzu/velero-plugin-for-aws/velero-plugin-for-aws/object_store.go:331" error.function="main.(*ObjectStore).ListCommonPrefixes" logSource="pkg/controller/backup_sync_controller.go:182" time="2021-03-12T18:52:40Z" level=error msg="Error listing backups in backup store" backupLocation=saverso controller=backup-sync error="rpc error: code = Unknown desc = InvalidAccessKeyId: \n\tstatus code: 403, request id: tx00000000000000304e7e2-00604bb878-9db4ea4-fra1b, host id: " error.file="/go/src/github.com/vmware-tanzu/velero-plugin-for-aws/velero-plugin-for-aws/object_store.go:331" error.function="main.(*ObjectStore).ListCommonPrefixes" logSource="pkg/controller/backup_sync_controller.go:182" time="2021-03-12T18:52:40Z" level=info msg="Validating backup storage location" backup-storage-location=backups controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:114" time="2021-03-12T18:52:40Z" level=info msg="Backup storage location valid, marking as available" backup-storage-location=backups controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:121" time="2021-03-12T18:52:40Z" level=info msg="Validating backup storage location" backup-storage-location=default controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:114" time="2021-03-12T18:52:40Z" level=info msg="Backup storage location is invalid, marking as unavailable" backup-storage-location=default controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:117" time="2021-03-12T18:52:40Z" level=info msg="Validating backup storage location" backup-storage-location=saverso controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:114" time="2021-03-12T18:52:40Z" level=info msg="Backup storage location is invalid, marking as unavailable" backup-storage-location=saverso controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:117" time="2021-03-12T18:52:40Z" level=info msg="Validating backup storage location" backup-storage-location=s3 controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:114" time="2021-03-12T18:52:40Z" level=info msg="Backup storage location valid, marking as available" backup-storage-location=s3 controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:121" time="2021-03-12T18:52:41Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, Backup storage location \"default\" is unavailable: rpc error: code = Unknown desc = InvalidAccessKeyId: \n\tstatus code: 403, request id: tx0000000000000030325cd-00604bb878-9d6bb8b-fra1b, host id: ; Backup storage location \"saverso\" is unavailable: rpc error: code = Unknown desc = InvalidAccessKeyId: \n\tstatus code: 403, request id: tx00000000000000304e83c-00604bb878-9db4ea4-fra1b, host id: )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T18:52:41Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T18:52:41Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T18:52:41Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T18:52:41Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T18:53:10Z" level=error msg="Error listing backups in backup store" backupLocation=default controller=backup-sync error="rpc error: code = Unknown desc = InvalidAccessKeyId: \n\tstatus code: 403, request id: tx0000000000000030505e1-00604bb896-9db4ea4-fra1b, host id: " error.file="/go/src/github.com/vmware-tanzu/velero-plugin-for-aws/velero-plugin-for-aws/object_store.go:331" error.function="main.(*ObjectStore).ListCommonPrefixes" logSource="pkg/controller/backup_sync_controller.go:182" time="2021-03-12T18:53:10Z" level=error msg="Error listing backups in backup store" backupLocation=saverso controller=backup-sync error="rpc error: code = Unknown desc = InvalidAccessKeyId: \n\tstatus code: 403, request id: tx00000000000000306611f-00604bb896-9d67586-fra1b, host id: " error.file="/go/src/github.com/vmware-tanzu/velero-plugin-for-aws/velero-plugin-for-aws/object_store.go:331" error.function="main.(*ObjectStore).ListCommonPrefixes" logSource="pkg/controller/backup_sync_controller.go:182" time="2021-03-12T18:53:40Z" level=error msg="Error listing backups in backup store" backupLocation=default controller=backup-sync error="rpc error: code = Unknown desc = InvalidAccessKeyId: \n\tstatus code: 403, request id: tx000000000000003036519-00604bb8b4-9d6bb8b-fra1b, host id: " error.file="/go/src/github.com/vmware-tanzu/velero-plugin-for-aws/velero-plugin-for-aws/object_store.go:331" error.function="main.(*ObjectStore).ListCommonPrefixes" logSource="pkg/controller/backup_sync_controller.go:182" time="2021-03-12T18:53:40Z" level=info msg="Validating backup storage location" backup-storage-location=default controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:114" time="2021-03-12T18:53:40Z" level=error msg="Error listing backups in backup store" backupLocation=saverso controller=backup-sync error="rpc error: code = Unknown desc = InvalidAccessKeyId: \n\tstatus code: 403, request id: tx00000000000000305277d-00604bb8b4-9db4ea4-fra1b, host id: " error.file="/go/src/github.com/vmware-tanzu/velero-plugin-for-aws/velero-plugin-for-aws/object_store.go:331" error.function="main.(*ObjectStore).ListCommonPrefixes" logSource="pkg/controller/backup_sync_controller.go:182" time="2021-03-12T18:53:41Z" level=info msg="Backup storage location is invalid, marking as unavailable" backup-storage-location=default controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:117" time="2021-03-12T18:53:41Z" level=info msg="Validating backup storage location" backup-storage-location=saverso controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:114" time="2021-03-12T18:53:41Z" level=info msg="Backup storage location is invalid, marking as unavailable" backup-storage-location=saverso controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:117" time="2021-03-12T18:53:41Z" level=info msg="Validating backup storage location" backup-storage-location=s3 controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:114" time="2021-03-12T18:53:41Z" level=info msg="Backup storage location valid, marking as available" backup-storage-location=s3 controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:121" time="2021-03-12T18:53:41Z" level=info msg="Validating backup storage location" backup-storage-location=backups controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:114" time="2021-03-12T18:53:41Z" level=info msg="Backup storage location valid, marking as available" backup-storage-location=backups controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:121" time="2021-03-12T18:53:41Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, Backup storage location \"default\" is unavailable: rpc error: code = Unknown desc = InvalidAccessKeyId: \n\tstatus code: 403, request id: tx0000000000000030682c5-00604bb8b5-9d67586-fra1b, host id: ; Backup storage location \"saverso\" is unavailable: rpc error: code = Unknown desc = InvalidAccessKeyId: \n\tstatus code: 403, request id: tx0000000000000030527af-00604bb8b5-9db4ea4-fra1b, host id: )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T18:53:41Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T18:53:41Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T18:53:41Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T18:53:41Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T18:54:11Z" level=error msg="Error listing backups in backup store" backupLocation=default controller=backup-sync error="rpc error: code = Unknown desc = InvalidAccessKeyId: \n\tstatus code: 403, request id: tx0000000000000030547e2-00604bb8d3-9db4ea4-fra1b, host id: " error.file="/go/src/github.com/vmware-tanzu/velero-plugin-for-aws/velero-plugin-for-aws/object_store.go:331" error.function="main.(*ObjectStore).ListCommonPrefixes" logSource="pkg/controller/backup_sync_controller.go:182" time="2021-03-12T18:54:11Z" level=error msg="Error listing backups in backup store" backupLocation=saverso controller=backup-sync error="rpc error: code = Unknown desc = InvalidAccessKeyId: \n\tstatus code: 403, request id: tx00000000000000301c5e8-00604bb8d3-9d3682b-fra1b, host id: " error.file="/go/src/github.com/vmware-tanzu/velero-plugin-for-aws/velero-plugin-for-aws/object_store.go:331" error.function="main.(*ObjectStore).ListCommonPrefixes" logSource="pkg/controller/backup_sync_controller.go:182" time="2021-03-12T18:54:41Z" level=error msg="Error listing backups in backup store" backupLocation=default controller=backup-sync error="rpc error: code = Unknown desc = InvalidAccessKeyId: \n\tstatus code: 403, request id: tx00000000000000306c130-00604bb8f1-9d67586-fra1b, host id: " error.file="/go/src/github.com/vmware-tanzu/velero-plugin-for-aws/velero-plugin-for-aws/object_store.go:331" error.function="main.(*ObjectStore).ListCommonPrefixes" logSource="pkg/controller/backup_sync_controller.go:182" time="2021-03-12T18:54:41Z" level=info msg="Validating backup storage location" backup-storage-location=backups controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:114" time="2021-03-12T18:54:41Z" level=error msg="Error listing backups in backup store" backupLocation=saverso controller=backup-sync error="rpc error: code = Unknown desc = InvalidAccessKeyId: \n\tstatus code: 403, request id: tx000000000000003056612-00604bb8f1-9db4ea4-fra1b, host id: " error.file="/go/src/github.com/vmware-tanzu/velero-plugin-for-aws/velero-plugin-for-aws/object_store.go:331" error.function="main.(*ObjectStore).ListCommonPrefixes" logSource="pkg/controller/backup_sync_controller.go:182" time="2021-03-12T18:54:41Z" level=info msg="Backup storage location valid, marking as available" backup-storage-location=backups controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:121" time="2021-03-12T18:54:41Z" level=info msg="Validating backup storage location" backup-storage-location=default controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:114" time="2021-03-12T18:54:41Z" level=info msg="Backup storage location is invalid, marking as unavailable" backup-storage-location=default controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:117" time="2021-03-12T18:54:41Z" level=info msg="Validating backup storage location" backup-storage-location=saverso controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:114" time="2021-03-12T18:54:41Z" level=info msg="Backup storage location is invalid, marking as unavailable" backup-storage-location=saverso controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:117" time="2021-03-12T18:54:41Z" level=info msg="Validating backup storage location" backup-storage-location=s3 controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:114" time="2021-03-12T18:54:41Z" level=info msg="Backup storage location valid, marking as available" backup-storage-location=s3 controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:121" time="2021-03-12T18:54:41Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, Backup storage location \"default\" is unavailable: rpc error: code = Unknown desc = InvalidAccessKeyId: \n\tstatus code: 403, request id: tx00000000000000301e43e-00604bb8f1-9d3682b-fra1b, host id: ; Backup storage location \"saverso\" is unavailable: rpc error: code = Unknown desc = InvalidAccessKeyId: \n\tstatus code: 403, request id: tx00000000000000301e456-00604bb8f1-9d3682b-fra1b, host id: )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T18:54:41Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T18:54:41Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T18:54:41Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T18:54:41Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T18:54:45Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T18:54:45Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T18:54:45Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T18:54:45Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T18:55:11Z" level=error msg="Error listing backups in backup store" backupLocation=default controller=backup-sync error="rpc error: code = Unknown desc = InvalidAccessKeyId: \n\tstatus code: 403, request id: tx000000000000003058295-00604bb90f-9db4ea4-fra1b, host id: " error.file="/go/src/github.com/vmware-tanzu/velero-plugin-for-aws/velero-plugin-for-aws/object_store.go:331" error.function="main.(*ObjectStore).ListCommonPrefixes" logSource="pkg/controller/backup_sync_controller.go:182" time="2021-03-12T18:55:11Z" level=error msg="Error listing backups in backup store" backupLocation=saverso controller=backup-sync error="rpc error: code = Unknown desc = InvalidAccessKeyId: \n\tstatus code: 403, request id: tx00000000000000302009a-00604bb90f-9d3682b-fra1b, host id: " error.file="/go/src/github.com/vmware-tanzu/velero-plugin-for-aws/velero-plugin-for-aws/object_store.go:331" error.function="main.(*ObjectStore).ListCommonPrefixes" logSource="pkg/controller/backup_sync_controller.go:182" time="2021-03-12T18:55:41Z" level=error msg="Error listing backups in backup store" backupLocation=default controller=backup-sync error="rpc error: code = Unknown desc = InvalidAccessKeyId: \n\tstatus code: 403, request id: tx00000000000000302227d-00604bb92d-9d3682b-fra1b, host id: " error.file="/go/src/github.com/vmware-tanzu/velero-plugin-for-aws/velero-plugin-for-aws/object_store.go:331" error.function="main.(*ObjectStore).ListCommonPrefixes" logSource="pkg/controller/backup_sync_controller.go:182" time="2021-03-12T18:55:41Z" level=error msg="Error listing backups in backup store" backupLocation=saverso controller=backup-sync error="rpc error: code = Unknown desc = InvalidAccessKeyId: \n\tstatus code: 403, request id: tx000000000000003022283-00604bb92d-9d3682b-fra1b, host id: " error.file="/go/src/github.com/vmware-tanzu/velero-plugin-for-aws/velero-plugin-for-aws/object_store.go:331" error.function="main.(*ObjectStore).ListCommonPrefixes" logSource="pkg/controller/backup_sync_controller.go:182" time="2021-03-12T18:55:41Z" level=info msg="Validating backup storage location" backup-storage-location=backups controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:114" time="2021-03-12T18:55:41Z" level=info msg="Backup storage location valid, marking as available" backup-storage-location=backups controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:121" time="2021-03-12T18:55:41Z" level=info msg="Validating backup storage location" backup-storage-location=default controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:114" time="2021-03-12T18:55:41Z" level=info msg="Backup storage location is invalid, marking as unavailable" backup-storage-location=default controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:117" time="2021-03-12T18:55:42Z" level=info msg="Validating backup storage location" backup-storage-location=saverso controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:114" time="2021-03-12T18:55:42Z" level=info msg="Backup storage location is invalid, marking as unavailable" backup-storage-location=saverso controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:117" time="2021-03-12T18:55:42Z" level=info msg="Validating backup storage location" backup-storage-location=s3 controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:114" time="2021-03-12T18:55:42Z" level=info msg="Backup storage location valid, marking as available" backup-storage-location=s3 controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:121" time="2021-03-12T18:55:42Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, Backup storage location \"default\" is unavailable: rpc error: code = Unknown desc = InvalidAccessKeyId: \n\tstatus code: 403, request id: tx0000000000000030222c4-00604bb92d-9d3682b-fra1b, host id: ; Backup storage location \"saverso\" is unavailable: rpc error: code = Unknown desc = InvalidAccessKeyId: \n\tstatus code: 403, request id: tx00000000000000305a4ee-00604bb92e-9db4ea4-fra1b, host id: )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T18:55:42Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T18:55:42Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T18:55:42Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T18:55:42Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" I0312 18:55:44.153343 1 request.go:645] Throttling request took 1.025660374s, request: GET:https://10.233.0.1:443/apis/discovery.k8s.io/v1beta1?timeout=32s time="2021-03-12T18:56:11Z" level=error msg="Error listing backups in backup store" backupLocation=default controller=backup-sync error="rpc error: code = Unknown desc = InvalidAccessKeyId: \n\tstatus code: 403, request id: tx00000000000000305c52d-00604bb94b-9db4ea4-fra1b, host id: " error.file="/go/src/github.com/vmware-tanzu/velero-plugin-for-aws/velero-plugin-for-aws/object_store.go:331" error.function="main.(*ObjectStore).ListCommonPrefixes" logSource="pkg/controller/backup_sync_controller.go:182" time="2021-03-12T18:56:12Z" level=error msg="Error listing backups in backup store" backupLocation=saverso controller=backup-sync error="rpc error: code = Unknown desc = InvalidAccessKeyId: \n\tstatus code: 403, request id: tx00000000000000305c535-00604bb94c-9db4ea4-fra1b, host id: " error.file="/go/src/github.com/vmware-tanzu/velero-plugin-for-aws/velero-plugin-for-aws/object_store.go:331" error.function="main.(*ObjectStore).ListCommonPrefixes" logSource="pkg/controller/backup_sync_controller.go:182" time="2021-03-12T18:56:42Z" level=error msg="Error listing backups in backup store" backupLocation=default controller=backup-sync error="rpc error: code = Unknown desc = InvalidAccessKeyId: \n\tstatus code: 403, request id: tx000000000000003025fc6-00604bb96a-9d3682b-fra1b, host id: " error.file="/go/src/github.com/vmware-tanzu/velero-plugin-for-aws/velero-plugin-for-aws/object_store.go:331" error.function="main.(*ObjectStore).ListCommonPrefixes" logSource="pkg/controller/backup_sync_controller.go:182" time="2021-03-12T18:56:42Z" level=info msg="Validating backup storage location" backup-storage-location=backups controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:114" time="2021-03-12T18:56:42Z" level=error msg="Error listing backups in backup store" backupLocation=saverso controller=backup-sync error="rpc error: code = Unknown desc = InvalidAccessKeyId: \n\tstatus code: 403, request id: tx000000000000003041f98-00604bb96a-9d6bb8b-fra1b, host id: " error.file="/go/src/github.com/vmware-tanzu/velero-plugin-for-aws/velero-plugin-for-aws/object_store.go:331" error.function="main.(*ObjectStore).ListCommonPrefixes" logSource="pkg/controller/backup_sync_controller.go:182" time="2021-03-12T18:56:42Z" level=info msg="Backup storage location valid, marking as available" backup-storage-location=backups controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:121" time="2021-03-12T18:56:42Z" level=info msg="Validating backup storage location" backup-storage-location=default controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:114" time="2021-03-12T18:56:42Z" level=info msg="Backup storage location is invalid, marking as unavailable" backup-storage-location=default controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:117" time="2021-03-12T18:56:42Z" level=info msg="Validating backup storage location" backup-storage-location=saverso controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:114" time="2021-03-12T18:56:42Z" level=info msg="Backup storage location is invalid, marking as unavailable" backup-storage-location=saverso controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:117" time="2021-03-12T18:56:42Z" level=info msg="Validating backup storage location" backup-storage-location=s3 controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:114" time="2021-03-12T18:56:42Z" level=info msg="Backup storage location valid, marking as available" backup-storage-location=s3 controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:121" time="2021-03-12T18:56:42Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, Backup storage location \"default\" is unavailable: rpc error: code = Unknown desc = InvalidAccessKeyId: \n\tstatus code: 403, request id: tx000000000000003025ff8-00604bb96a-9d3682b-fra1b, host id: ; Backup storage location \"saverso\" is unavailable: rpc error: code = Unknown desc = InvalidAccessKeyId: \n\tstatus code: 403, request id: tx000000000000003073d5a-00604bb96a-9d67586-fra1b, host id: )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T18:56:42Z" level=info msg="Validating backup storage location" backup-storage-location=s3 controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:114" time="2021-03-12T18:56:42Z" level=info msg="Backup storage location valid, marking as available" backup-storage-location=s3 controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:121" time="2021-03-12T18:56:42Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T18:56:42Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T18:56:42Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T18:56:42Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T18:56:42Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T18:57:12Z" level=error msg="Error listing backups in backup store" backupLocation=default controller=backup-sync error="rpc error: code = Unknown desc = InvalidAccessKeyId: \n\tstatus code: 403, request id: tx000000000000003027be8-00604bb988-9d3682b-fra1b, host id: " error.file="/go/src/github.com/vmware-tanzu/velero-plugin-for-aws/velero-plugin-for-aws/object_store.go:331" error.function="main.(*ObjectStore).ListCommonPrefixes" logSource="pkg/controller/backup_sync_controller.go:182" time="2021-03-12T18:57:12Z" level=error msg="Error listing backups in backup store" backupLocation=saverso controller=backup-sync error="rpc error: code = Unknown desc = InvalidAccessKeyId: \n\tstatus code: 403, request id: tx000000000000003043bad-00604bb988-9d6bb8b-fra1b, host id: " error.file="/go/src/github.com/vmware-tanzu/velero-plugin-for-aws/velero-plugin-for-aws/object_store.go:331" error.function="main.(*ObjectStore).ListCommonPrefixes" logSource="pkg/controller/backup_sync_controller.go:182" time="2021-03-12T18:57:42Z" level=error msg="Error listing backups in backup store" backupLocation=default controller=backup-sync error="rpc error: code = Unknown desc = InvalidAccessKeyId: \n\tstatus code: 403, request id: tx000000000000003061e7a-00604bb9a6-9db4ea4-fra1b, host id: " error.file="/go/src/github.com/vmware-tanzu/velero-plugin-for-aws/velero-plugin-for-aws/object_store.go:331" error.function="main.(*ObjectStore).ListCommonPrefixes" logSource="pkg/controller/backup_sync_controller.go:182" time="2021-03-12T18:57:42Z" level=info msg="Validating backup storage location" backup-storage-location=saverso controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:114" time="2021-03-12T18:57:42Z" level=error msg="Error listing backups in backup store" backupLocation=saverso controller=backup-sync error="rpc error: code = Unknown desc = InvalidAccessKeyId: \n\tstatus code: 403, request id: tx000000000000003045c50-00604bb9a6-9d6bb8b-fra1b, host id: " error.file="/go/src/github.com/vmware-tanzu/velero-plugin-for-aws/velero-plugin-for-aws/object_store.go:331" error.function="main.(*ObjectStore).ListCommonPrefixes" logSource="pkg/controller/backup_sync_controller.go:182" time="2021-03-12T18:57:42Z" level=info msg="Backup storage location is invalid, marking as unavailable" backup-storage-location=saverso controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:117" time="2021-03-12T18:57:42Z" level=info msg="Validating backup storage location" backup-storage-location=s3 controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:114" time="2021-03-12T18:57:42Z" level=info msg="Backup storage location valid, marking as available" backup-storage-location=s3 controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:121" time="2021-03-12T18:57:42Z" level=info msg="Validating backup storage location" backup-storage-location=backups controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:114" time="2021-03-12T18:57:42Z" level=info msg="Backup storage location valid, marking as available" backup-storage-location=backups controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:121" time="2021-03-12T18:57:43Z" level=info msg="Validating backup storage location" backup-storage-location=default controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:114" time="2021-03-12T18:57:43Z" level=info msg="Backup storage location is invalid, marking as unavailable" backup-storage-location=default controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:117" time="2021-03-12T18:57:43Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, Backup storage location \"saverso\" is unavailable: rpc error: code = Unknown desc = InvalidAccessKeyId: \n\tstatus code: 403, request id: tx000000000000003029ca2-00604bb9a6-9d3682b-fra1b, host id: ; Backup storage location \"default\" is unavailable: rpc error: code = Unknown desc = InvalidAccessKeyId: \n\tstatus code: 403, request id: tx000000000000003045c92-00604bb9a7-9d6bb8b-fra1b, host id: )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T18:57:43Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T18:57:43Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T18:57:43Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T18:57:43Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T18:58:12Z" level=error msg="Error listing backups in backup store" backupLocation=default controller=backup-sync error="rpc error: code = Unknown desc = InvalidAccessKeyId: \n\tstatus code: 403, request id: tx000000000000003048226-00604bb9c4-9d6bb8b-fra1b, host id: " error.file="/go/src/github.com/vmware-tanzu/velero-plugin-for-aws/velero-plugin-for-aws/object_store.go:331" error.function="main.(*ObjectStore).ListCommonPrefixes" logSource="pkg/controller/backup_sync_controller.go:182" time="2021-03-12T18:58:12Z" level=error msg="Error listing backups in backup store" backupLocation=saverso controller=backup-sync error="rpc error: code = Unknown desc = InvalidAccessKeyId: \n\tstatus code: 403, request id: tx000000000000003064475-00604bb9c4-9db4ea4-fra1b, host id: " error.file="/go/src/github.com/vmware-tanzu/velero-plugin-for-aws/velero-plugin-for-aws/object_store.go:331" error.function="main.(*ObjectStore).ListCommonPrefixes" logSource="pkg/controller/backup_sync_controller.go:182" time="2021-03-12T18:58:43Z" level=error msg="Error listing backups in backup store" backupLocation=default controller=backup-sync error="rpc error: code = Unknown desc = InvalidAccessKeyId: \n\tstatus code: 403, request id: tx000000000000003066a5d-00604bb9e3-9db4ea4-fra1b, host id: " error.file="/go/src/github.com/vmware-tanzu/velero-plugin-for-aws/velero-plugin-for-aws/object_store.go:331" error.function="main.(*ObjectStore).ListCommonPrefixes" logSource="pkg/controller/backup_sync_controller.go:182" time="2021-03-12T18:58:43Z" level=info msg="Validating backup storage location" backup-storage-location=backups controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:114" time="2021-03-12T18:58:43Z" level=error msg="Error listing backups in backup store" backupLocation=saverso controller=backup-sync error="rpc error: code = Unknown desc = InvalidAccessKeyId: \n\tstatus code: 403, request id: tx00000000000000307c5bc-00604bb9e3-9d67586-fra1b, host id: " error.file="/go/src/github.com/vmware-tanzu/velero-plugin-for-aws/velero-plugin-for-aws/object_store.go:331" error.function="main.(*ObjectStore).ListCommonPrefixes" logSource="pkg/controller/backup_sync_controller.go:182" time="2021-03-12T18:58:43Z" level=info msg="Backup storage location valid, marking as available" backup-storage-location=backups controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:121" time="2021-03-12T18:58:43Z" level=info msg="Validating backup storage location" backup-storage-location=default controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:114" time="2021-03-12T18:58:43Z" level=info msg="Backup storage location is invalid, marking as unavailable" backup-storage-location=default controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:117" time="2021-03-12T18:58:43Z" level=info msg="Validating backup storage location" backup-storage-location=saverso controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:114" time="2021-03-12T18:58:43Z" level=info msg="Backup storage location is invalid, marking as unavailable" backup-storage-location=saverso controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:117" time="2021-03-12T18:58:43Z" level=info msg="Validating backup storage location" backup-storage-location=s3 controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:114" time="2021-03-12T18:58:43Z" level=info msg="Backup storage location valid, marking as available" backup-storage-location=s3 controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:121" time="2021-03-12T18:58:43Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, Backup storage location \"default\" is unavailable: rpc error: code = Unknown desc = InvalidAccessKeyId: \n\tstatus code: 403, request id: tx000000000000003066ab7-00604bb9e3-9db4ea4-fra1b, host id: ; Backup storage location \"saverso\" is unavailable: rpc error: code = Unknown desc = InvalidAccessKeyId: \n\tstatus code: 403, request id: tx00000000000000307c617-00604bb9e3-9d67586-fra1b, host id: )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T18:58:43Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T18:58:43Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T18:58:43Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T18:58:43Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T18:59:13Z" level=error msg="Error listing backups in backup store" backupLocation=default controller=backup-sync error="rpc error: code = Unknown desc = InvalidAccessKeyId: \n\tstatus code: 403, request id: tx000000000000003068d61-00604bba01-9db4ea4-fra1b, host id: " error.file="/go/src/github.com/vmware-tanzu/velero-plugin-for-aws/velero-plugin-for-aws/object_store.go:331" error.function="main.(*ObjectStore).ListCommonPrefixes" logSource="pkg/controller/backup_sync_controller.go:182" time="2021-03-12T18:59:13Z" level=error msg="Error listing backups in backup store" backupLocation=saverso controller=backup-sync error="rpc error: code = Unknown desc = InvalidAccessKeyId: \n\tstatus code: 403, request id: tx00000000000000307e8a7-00604bba01-9d67586-fra1b, host id: " error.file="/go/src/github.com/vmware-tanzu/velero-plugin-for-aws/velero-plugin-for-aws/object_store.go:331" error.function="main.(*ObjectStore).ListCommonPrefixes" logSource="pkg/controller/backup_sync_controller.go:182" time="2021-03-12T18:59:43Z" level=error msg="Error listing backups in backup store" backupLocation=default controller=backup-sync error="rpc error: code = Unknown desc = InvalidAccessKeyId: \n\tstatus code: 403, request id: tx000000000000003081035-00604bba1f-9d67586-fra1b, host id: " error.file="/go/src/github.com/vmware-tanzu/velero-plugin-for-aws/velero-plugin-for-aws/object_store.go:331" error.function="main.(*ObjectStore).ListCommonPrefixes" logSource="pkg/controller/backup_sync_controller.go:182" time="2021-03-12T18:59:43Z" level=info msg="Validating backup storage location" backup-storage-location=backups controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:114" time="2021-03-12T18:59:43Z" level=error msg="Error listing backups in backup store" backupLocation=saverso controller=backup-sync error="rpc error: code = Unknown desc = InvalidAccessKeyId: \n\tstatus code: 403, request id: tx00000000000000304f2d2-00604bba1f-9d6bb8b-fra1b, host id: " error.file="/go/src/github.com/vmware-tanzu/velero-plugin-for-aws/velero-plugin-for-aws/object_store.go:331" error.function="main.(*ObjectStore).ListCommonPrefixes" logSource="pkg/controller/backup_sync_controller.go:182" time="2021-03-12T18:59:43Z" level=info msg="Backup storage location valid, marking as available" backup-storage-location=backups controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:121" time="2021-03-12T18:59:43Z" level=info msg="Validating backup storage location" backup-storage-location=default controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:114" time="2021-03-12T18:59:43Z" level=info msg="Backup storage location is invalid, marking as unavailable" backup-storage-location=default controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:117" time="2021-03-12T18:59:43Z" level=info msg="Validating backup storage location" backup-storage-location=saverso controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:114" time="2021-03-12T18:59:43Z" level=info msg="Backup storage location is invalid, marking as unavailable" backup-storage-location=saverso controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:117" time="2021-03-12T18:59:43Z" level=info msg="Validating backup storage location" backup-storage-location=s3 controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:114" time="2021-03-12T18:59:43Z" level=info msg="Backup storage location valid, marking as available" backup-storage-location=s3 controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:121" time="2021-03-12T18:59:44Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, Backup storage location \"default\" is unavailable: rpc error: code = Unknown desc = InvalidAccessKeyId: \n\tstatus code: 403, request id: tx0000000000000030810b3-00604bba1f-9d67586-fra1b, host id: ; Backup storage location \"saverso\" is unavailable: rpc error: code = Unknown desc = InvalidAccessKeyId: \n\tstatus code: 403, request id: tx00000000000000304f3a9-00604bba1f-9d6bb8b-fra1b, host id: )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T18:59:44Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T18:59:44Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T18:59:44Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T18:59:44Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T19:00:13Z" level=error msg="Error listing backups in backup store" backupLocation=default controller=backup-sync error="rpc error: code = Unknown desc = InvalidAccessKeyId: \n\tstatus code: 403, request id: tx00000000000000308390a-00604bba3d-9d67586-fra1b, host id: " error.file="/go/src/github.com/vmware-tanzu/velero-plugin-for-aws/velero-plugin-for-aws/object_store.go:331" error.function="main.(*ObjectStore).ListCommonPrefixes" logSource="pkg/controller/backup_sync_controller.go:182" time="2021-03-12T19:00:13Z" level=error msg="Error listing backups in backup store" backupLocation=saverso controller=backup-sync error="rpc error: code = Unknown desc = InvalidAccessKeyId: \n\tstatus code: 403, request id: tx000000000000003035bd5-00604bba3d-9d3682b-fra1b, host id: " error.file="/go/src/github.com/vmware-tanzu/velero-plugin-for-aws/velero-plugin-for-aws/object_store.go:331" error.function="main.(*ObjectStore).ListCommonPrefixes" logSource="pkg/controller/backup_sync_controller.go:182" time="2021-03-12T19:00:43Z" level=error msg="Error listing backups in backup store" backupLocation=default controller=backup-sync error="rpc error: code = Unknown desc = InvalidAccessKeyId: \n\tstatus code: 403, request id: tx000000000000003070742-00604bba5b-9db4ea4-fra1b, host id: " error.file="/go/src/github.com/vmware-tanzu/velero-plugin-for-aws/velero-plugin-for-aws/object_store.go:331" error.function="main.(*ObjectStore).ListCommonPrefixes" logSource="pkg/controller/backup_sync_controller.go:182" time="2021-03-12T19:00:44Z" level=info msg="Validating backup storage location" backup-storage-location=backups controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:114" time="2021-03-12T19:00:44Z" level=error msg="Error listing backups in backup store" backupLocation=saverso controller=backup-sync error="rpc error: code = Unknown desc = InvalidAccessKeyId: \n\tstatus code: 403, request id: tx000000000000003054522-00604bba5c-9d6bb8b-fra1b, host id: " error.file="/go/src/github.com/vmware-tanzu/velero-plugin-for-aws/velero-plugin-for-aws/object_store.go:331" error.function="main.(*ObjectStore).ListCommonPrefixes" logSource="pkg/controller/backup_sync_controller.go:182" time="2021-03-12T19:00:44Z" level=info msg="Backup storage location valid, marking as available" backup-storage-location=backups controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:121" time="2021-03-12T19:00:44Z" level=info msg="Validating backup storage location" backup-storage-location=default controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:114" time="2021-03-12T19:00:44Z" level=info msg="Backup storage location is invalid, marking as unavailable" backup-storage-location=default controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:117" time="2021-03-12T19:00:44Z" level=info msg="Validating backup storage location" backup-storage-location=saverso controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:114" time="2021-03-12T19:00:44Z" level=info msg="Backup storage location is invalid, marking as unavailable" backup-storage-location=saverso controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:117" time="2021-03-12T19:00:44Z" level=info msg="Validating backup storage location" backup-storage-location=s3 controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:114" time="2021-03-12T19:00:44Z" level=info msg="Backup storage location valid, marking as available" backup-storage-location=s3 controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:121" time="2021-03-12T19:00:44Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, Backup storage location \"default\" is unavailable: rpc error: code = Unknown desc = InvalidAccessKeyId: \n\tstatus code: 403, request id: tx000000000000003070793-00604bba5c-9db4ea4-fra1b, host id: ; Backup storage location \"saverso\" is unavailable: rpc error: code = Unknown desc = InvalidAccessKeyId: \n\tstatus code: 403, request id: tx0000000000000030862f5-00604bba5c-9d67586-fra1b, host id: )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T19:00:44Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T19:00:44Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T19:00:44Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T19:00:44Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" I0312 19:00:47.827275 1 request.go:645] Throttling request took 1.025188579s, request: GET:https://10.233.0.1:443/apis/coordination.k8s.io/v1?timeout=32s time="2021-03-12T19:01:14Z" level=error msg="Error listing backups in backup store" backupLocation=default controller=backup-sync error="rpc error: code = Unknown desc = InvalidAccessKeyId: \n\tstatus code: 403, request id: tx000000000000003072e62-00604bba7a-9db4ea4-fra1b, host id: " error.file="/go/src/github.com/vmware-tanzu/velero-plugin-for-aws/velero-plugin-for-aws/object_store.go:331" error.function="main.(*ObjectStore).ListCommonPrefixes" logSource="pkg/controller/backup_sync_controller.go:182" time="2021-03-12T19:01:14Z" level=error msg="Error listing backups in backup store" backupLocation=saverso controller=backup-sync error="rpc error: code = Unknown desc = InvalidAccessKeyId: \n\tstatus code: 403, request id: tx0000000000000030889a9-00604bba7a-9d67586-fra1b, host id: " error.file="/go/src/github.com/vmware-tanzu/velero-plugin-for-aws/velero-plugin-for-aws/object_store.go:331" error.function="main.(*ObjectStore).ListCommonPrefixes" logSource="pkg/controller/backup_sync_controller.go:182" time="2021-03-12T19:01:44Z" level=error msg="Error listing backups in backup store" backupLocation=default controller=backup-sync error="rpc error: code = Unknown desc = InvalidAccessKeyId: \n\tstatus code: 403, request id: tx000000000000003075885-00604bba98-9db4ea4-fra1b, host id: " error.file="/go/src/github.com/vmware-tanzu/velero-plugin-for-aws/velero-plugin-for-aws/object_store.go:331" error.function="main.(*ObjectStore).ListCommonPrefixes" logSource="pkg/controller/backup_sync_controller.go:182" time="2021-03-12T19:01:44Z" level=info msg="Validating backup storage location" backup-storage-location=default controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:114" time="2021-03-12T19:01:44Z" level=error msg="Error listing backups in backup store" backupLocation=saverso controller=backup-sync error="rpc error: code = Unknown desc = InvalidAccessKeyId: \n\tstatus code: 403, request id: tx000000000000003059667-00604bba98-9d6bb8b-fra1b, host id: " error.file="/go/src/github.com/vmware-tanzu/velero-plugin-for-aws/velero-plugin-for-aws/object_store.go:331" error.function="main.(*ObjectStore).ListCommonPrefixes" logSource="pkg/controller/backup_sync_controller.go:182" time="2021-03-12T19:01:44Z" level=info msg="Backup storage location is invalid, marking as unavailable" backup-storage-location=default controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:117" time="2021-03-12T19:01:44Z" level=info msg="Validating backup storage location" backup-storage-location=saverso controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:114" time="2021-03-12T19:01:44Z" level=info msg="Backup storage location is invalid, marking as unavailable" backup-storage-location=saverso controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:117" time="2021-03-12T19:01:44Z" level=info msg="Validating backup storage location" backup-storage-location=s3 controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:114" time="2021-03-12T19:01:44Z" level=info msg="Backup storage location valid, marking as available" backup-storage-location=s3 controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:121" time="2021-03-12T19:01:44Z" level=info msg="Validating backup storage location" backup-storage-location=backups controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:114" time="2021-03-12T19:01:44Z" level=info msg="Backup storage location valid, marking as available" backup-storage-location=backups controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:121" time="2021-03-12T19:01:44Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, Backup storage location \"default\" is unavailable: rpc error: code = Unknown desc = InvalidAccessKeyId: \n\tstatus code: 403, request id: tx0000000000000030758c5-00604bba98-9db4ea4-fra1b, host id: ; Backup storage location \"saverso\" is unavailable: rpc error: code = Unknown desc = InvalidAccessKeyId: \n\tstatus code: 403, request id: tx0000000000000030596b7-00604bba98-9d6bb8b-fra1b, host id: )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T19:01:44Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T19:01:44Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T19:01:44Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T19:01:44Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T19:02:14Z" level=error msg="Error listing backups in backup store" backupLocation=default controller=backup-sync error="rpc error: code = Unknown desc = InvalidAccessKeyId: \n\tstatus code: 403, request id: tx00000000000000305bea5-00604bbab6-9d6bb8b-fra1b, host id: " error.file="/go/src/github.com/vmware-tanzu/velero-plugin-for-aws/velero-plugin-for-aws/object_store.go:331" error.function="main.(*ObjectStore).ListCommonPrefixes" logSource="pkg/controller/backup_sync_controller.go:182" time="2021-03-12T19:02:14Z" level=error msg="Error listing backups in backup store" backupLocation=saverso controller=backup-sync error="rpc error: code = Unknown desc = InvalidAccessKeyId: \n\tstatus code: 403, request id: tx00000000000000308dc24-00604bbab6-9d67586-fra1b, host id: " error.file="/go/src/github.com/vmware-tanzu/velero-plugin-for-aws/velero-plugin-for-aws/object_store.go:331" error.function="main.(*ObjectStore).ListCommonPrefixes" logSource="pkg/controller/backup_sync_controller.go:182" time="2021-03-12T19:02:44Z" level=error msg="Error listing backups in backup store" backupLocation=default controller=backup-sync error="rpc error: code = Unknown desc = InvalidAccessKeyId: \n\tstatus code: 403, request id: tx00000000000000307a828-00604bbad4-9db4ea4-fra1b, host id: " error.file="/go/src/github.com/vmware-tanzu/velero-plugin-for-aws/velero-plugin-for-aws/object_store.go:331" error.function="main.(*ObjectStore).ListCommonPrefixes" logSource="pkg/controller/backup_sync_controller.go:182" time="2021-03-12T19:02:44Z" level=error msg="Error listing backups in backup store" backupLocation=saverso controller=backup-sync error="rpc error: code = Unknown desc = InvalidAccessKeyId: \n\tstatus code: 403, request id: tx000000000000003042637-00604bbad4-9d3682b-fra1b, host id: " error.file="/go/src/github.com/vmware-tanzu/velero-plugin-for-aws/velero-plugin-for-aws/object_store.go:331" error.function="main.(*ObjectStore).ListCommonPrefixes" logSource="pkg/controller/backup_sync_controller.go:182" time="2021-03-12T19:02:44Z" level=info msg="Validating backup storage location" backup-storage-location=backups controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:114" time="2021-03-12T19:02:44Z" level=info msg="Backup storage location valid, marking as available" backup-storage-location=backups controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:121" time="2021-03-12T19:02:45Z" level=info msg="Validating backup storage location" backup-storage-location=default controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:114" time="2021-03-12T19:02:45Z" level=info msg="Backup storage location is invalid, marking as unavailable" backup-storage-location=default controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:117" time="2021-03-12T19:02:45Z" level=info msg="Validating backup storage location" backup-storage-location=saverso controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:114" time="2021-03-12T19:02:45Z" level=info msg="Backup storage location is invalid, marking as unavailable" backup-storage-location=saverso controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:117" time="2021-03-12T19:02:45Z" level=info msg="Validating backup storage location" backup-storage-location=s3 controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:114" time="2021-03-12T19:02:45Z" level=info msg="Backup storage location valid, marking as available" backup-storage-location=s3 controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:121" time="2021-03-12T19:02:45Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, Backup storage location \"default\" is unavailable: rpc error: code = Unknown desc = InvalidAccessKeyId: \n\tstatus code: 403, request id: tx00000000000000305e643-00604bbad5-9d6bb8b-fra1b, host id: ; Backup storage location \"saverso\" is unavailable: rpc error: code = Unknown desc = InvalidAccessKeyId: \n\tstatus code: 403, request id: tx0000000000000030426b0-00604bbad5-9d3682b-fra1b, host id: )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T19:02:45Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T19:02:45Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T19:02:45Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T19:02:45Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T19:03:15Z" level=error msg="Error listing backups in backup store" backupLocation=default controller=backup-sync error="rpc error: code = Unknown desc = InvalidAccessKeyId: \n\tstatus code: 403, request id: tx000000000000003092d2b-00604bbaf3-9d67586-fra1b, host id: " error.file="/go/src/github.com/vmware-tanzu/velero-plugin-for-aws/velero-plugin-for-aws/object_store.go:331" error.function="main.(*ObjectStore).ListCommonPrefixes" logSource="pkg/controller/backup_sync_controller.go:182" time="2021-03-12T19:03:15Z" level=error msg="Error listing backups in backup store" backupLocation=saverso controller=backup-sync error="rpc error: code = Unknown desc = InvalidAccessKeyId: \n\tstatus code: 403, request id: tx000000000000003092d31-00604bbaf3-9d67586-fra1b, host id: " error.file="/go/src/github.com/vmware-tanzu/velero-plugin-for-aws/velero-plugin-for-aws/object_store.go:331" error.function="main.(*ObjectStore).ListCommonPrefixes" logSource="pkg/controller/backup_sync_controller.go:182" time="2021-03-12T19:03:45Z" level=error msg="Error listing backups in backup store" backupLocation=default controller=backup-sync error="rpc error: code = Unknown desc = InvalidAccessKeyId: \n\tstatus code: 403, request id: tx0000000000000030953b9-00604bbb11-9d67586-fra1b, host id: " error.file="/go/src/github.com/vmware-tanzu/velero-plugin-for-aws/velero-plugin-for-aws/object_store.go:331" error.function="main.(*ObjectStore).ListCommonPrefixes" logSource="pkg/controller/backup_sync_controller.go:182" time="2021-03-12T19:03:45Z" level=info msg="Validating backup storage location" backup-storage-location=backups controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:114" time="2021-03-12T19:03:45Z" level=error msg="Error listing backups in backup store" backupLocation=saverso controller=backup-sync error="rpc error: code = Unknown desc = InvalidAccessKeyId: \n\tstatus code: 403, request id: tx000000000000003063661-00604bbb11-9d6bb8b-fra1b, host id: " error.file="/go/src/github.com/vmware-tanzu/velero-plugin-for-aws/velero-plugin-for-aws/object_store.go:331" error.function="main.(*ObjectStore).ListCommonPrefixes" logSource="pkg/controller/backup_sync_controller.go:182" time="2021-03-12T19:03:45Z" level=info msg="Backup storage location valid, marking as available" backup-storage-location=backups controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:121" time="2021-03-12T19:03:45Z" level=info msg="Validating backup storage location" backup-storage-location=default controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:114" time="2021-03-12T19:03:45Z" level=info msg="Backup storage location is invalid, marking as unavailable" backup-storage-location=default controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:117" time="2021-03-12T19:03:45Z" level=info msg="Validating backup storage location" backup-storage-location=saverso controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:114" time="2021-03-12T19:03:45Z" level=info msg="Backup storage location is invalid, marking as unavailable" backup-storage-location=saverso controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:117" time="2021-03-12T19:03:45Z" level=info msg="Validating backup storage location" backup-storage-location=s3 controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:114" time="2021-03-12T19:03:45Z" level=info msg="Backup storage location valid, marking as available" backup-storage-location=s3 controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:121" time="2021-03-12T19:03:45Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, Backup storage location \"default\" is unavailable: rpc error: code = Unknown desc = InvalidAccessKeyId: \n\tstatus code: 403, request id: tx0000000000000030636a9-00604bbb11-9d6bb8b-fra1b, host id: ; Backup storage location \"saverso\" is unavailable: rpc error: code = Unknown desc = InvalidAccessKeyId: \n\tstatus code: 403, request id: tx000000000000003047721-00604bbb11-9d3682b-fra1b, host id: )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T19:03:45Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T19:03:45Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T19:03:45Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T19:03:45Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T19:04:15Z" level=error msg="Error listing backups in backup store" backupLocation=default controller=backup-sync error="rpc error: code = Unknown desc = InvalidAccessKeyId: \n\tstatus code: 403, request id: tx000000000000003097a34-00604bbb2f-9d67586-fra1b, host id: " error.file="/go/src/github.com/vmware-tanzu/velero-plugin-for-aws/velero-plugin-for-aws/object_store.go:331" error.function="main.(*ObjectStore).ListCommonPrefixes" logSource="pkg/controller/backup_sync_controller.go:182" time="2021-03-12T19:04:15Z" level=error msg="Error listing backups in backup store" backupLocation=saverso controller=backup-sync error="rpc error: code = Unknown desc = InvalidAccessKeyId: \n\tstatus code: 403, request id: tx000000000000003049d0c-00604bbb2f-9d3682b-fra1b, host id: " error.file="/go/src/github.com/vmware-tanzu/velero-plugin-for-aws/velero-plugin-for-aws/object_store.go:331" error.function="main.(*ObjectStore).ListCommonPrefixes" logSource="pkg/controller/backup_sync_controller.go:182" time="2021-03-12T19:04:45Z" level=error msg="Error listing backups in backup store" backupLocation=default controller=backup-sync error="rpc error: code = Unknown desc = InvalidAccessKeyId: \n\tstatus code: 403, request id: tx000000000000003067e4b-00604bbb4d-9d6bb8b-fra1b, host id: " error.file="/go/src/github.com/vmware-tanzu/velero-plugin-for-aws/velero-plugin-for-aws/object_store.go:331" error.function="main.(*ObjectStore).ListCommonPrefixes" logSource="pkg/controller/backup_sync_controller.go:182" time="2021-03-12T19:04:45Z" level=info msg="Validating backup storage location" backup-storage-location=backups controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:114" time="2021-03-12T19:04:45Z" level=error msg="Error listing backups in backup store" backupLocation=saverso controller=backup-sync error="rpc error: code = Unknown desc = InvalidAccessKeyId: \n\tstatus code: 403, request id: tx000000000000003067e65-00604bbb4d-9d6bb8b-fra1b, host id: " error.file="/go/src/github.com/vmware-tanzu/velero-plugin-for-aws/velero-plugin-for-aws/object_store.go:331" error.function="main.(*ObjectStore).ListCommonPrefixes" logSource="pkg/controller/backup_sync_controller.go:182" time="2021-03-12T19:04:45Z" level=info msg="Backup storage location valid, marking as available" backup-storage-location=backups controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:121" time="2021-03-12T19:04:45Z" level=info msg="Validating backup storage location" backup-storage-location=default controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:114" time="2021-03-12T19:04:45Z" level=info msg="Backup storage location is invalid, marking as unavailable" backup-storage-location=default controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:117" time="2021-03-12T19:04:46Z" level=info msg="Validating backup storage location" backup-storage-location=saverso controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:114" time="2021-03-12T19:04:46Z" level=info msg="Backup storage location is invalid, marking as unavailable" backup-storage-location=saverso controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:117" time="2021-03-12T19:04:46Z" level=info msg="Validating backup storage location" backup-storage-location=s3 controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:114" time="2021-03-12T19:04:46Z" level=info msg="Backup storage location valid, marking as available" backup-storage-location=s3 controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:121" time="2021-03-12T19:04:46Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, Backup storage location \"default\" is unavailable: rpc error: code = Unknown desc = InvalidAccessKeyId: \n\tstatus code: 403, request id: tx0000000000000030840cb-00604bbb4d-9db4ea4-fra1b, host id: ; Backup storage location \"saverso\" is unavailable: rpc error: code = Unknown desc = InvalidAccessKeyId: \n\tstatus code: 403, request id: tx0000000000000030840e4-00604bbb4e-9db4ea4-fra1b, host id: )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T19:04:46Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T19:04:46Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T19:04:46Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T19:04:46Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T19:05:16Z" level=error msg="Error listing backups in backup store" backupLocation=default controller=backup-sync error="rpc error: code = Unknown desc = InvalidAccessKeyId: \n\tstatus code: 403, request id: tx00000000000000306a1e8-00604bbb6b-9d6bb8b-fra1b, host id: " error.file="/go/src/github.com/vmware-tanzu/velero-plugin-for-aws/velero-plugin-for-aws/object_store.go:331" error.function="main.(*ObjectStore).ListCommonPrefixes" logSource="pkg/controller/backup_sync_controller.go:182" time="2021-03-12T19:05:16Z" level=error msg="Error listing backups in backup store" backupLocation=saverso controller=backup-sync error="rpc error: code = Unknown desc = InvalidAccessKeyId: \n\tstatus code: 403, request id: tx00000000000000304e23a-00604bbb6c-9d3682b-fra1b, host id: " error.file="/go/src/github.com/vmware-tanzu/velero-plugin-for-aws/velero-plugin-for-aws/object_store.go:331" error.function="main.(*ObjectStore).ListCommonPrefixes" logSource="pkg/controller/backup_sync_controller.go:182" time="2021-03-12T19:05:46Z" level=error msg="Error listing backups in backup store" backupLocation=default controller=backup-sync error="rpc error: code = Unknown desc = InvalidAccessKeyId: \n\tstatus code: 403, request id: tx00000000000000306c732-00604bbb8a-9d6bb8b-fra1b, host id: " error.file="/go/src/github.com/vmware-tanzu/velero-plugin-for-aws/velero-plugin-for-aws/object_store.go:331" error.function="main.(*ObjectStore).ListCommonPrefixes" logSource="pkg/controller/backup_sync_controller.go:182" time="2021-03-12T19:05:46Z" level=info msg="Validating backup storage location" backup-storage-location=backups controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:114" time="2021-03-12T19:05:46Z" level=error msg="Error listing backups in backup store" backupLocation=saverso controller=backup-sync error="rpc error: code = Unknown desc = InvalidAccessKeyId: \n\tstatus code: 403, request id: tx00000000000000309e4c8-00604bbb8a-9d67586-fra1b, host id: " error.file="/go/src/github.com/vmware-tanzu/velero-plugin-for-aws/velero-plugin-for-aws/object_store.go:331" error.function="main.(*ObjectStore).ListCommonPrefixes" logSource="pkg/controller/backup_sync_controller.go:182" time="2021-03-12T19:05:46Z" level=info msg="Backup storage location valid, marking as available" backup-storage-location=backups controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:121" time="2021-03-12T19:05:46Z" level=info msg="Validating backup storage location" backup-storage-location=default controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:114" time="2021-03-12T19:05:46Z" level=info msg="Backup storage location is invalid, marking as unavailable" backup-storage-location=default controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:117" time="2021-03-12T19:05:46Z" level=info msg="Validating backup storage location" backup-storage-location=saverso controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:114" time="2021-03-12T19:05:46Z" level=info msg="Backup storage location is invalid, marking as unavailable" backup-storage-location=saverso controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:117" time="2021-03-12T19:05:46Z" level=info msg="Validating backup storage location" backup-storage-location=s3 controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:114" time="2021-03-12T19:05:46Z" level=info msg="Backup storage location valid, marking as available" backup-storage-location=s3 controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:121" time="2021-03-12T19:05:46Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, Backup storage location \"default\" is unavailable: rpc error: code = Unknown desc = InvalidAccessKeyId: \n\tstatus code: 403, request id: tx0000000000000030889ce-00604bbb8a-9db4ea4-fra1b, host id: ; Backup storage location \"saverso\" is unavailable: rpc error: code = Unknown desc = InvalidAccessKeyId: \n\tstatus code: 403, request id: tx00000000000000306c7c2-00604bbb8a-9d6bb8b-fra1b, host id: )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T19:05:46Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T19:05:46Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T19:05:46Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T19:05:46Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" I0312 19:05:51.501445 1 request.go:645] Throttling request took 1.025289456s, request: GET:https://10.233.0.1:443/apis/rbac.authorization.k8s.io/v1?timeout=32s time="2021-03-12T19:06:16Z" level=error msg="Error listing backups in backup store" backupLocation=default controller=backup-sync error="rpc error: code = Unknown desc = InvalidAccessKeyId: \n\tstatus code: 403, request id: tx00000000000000306ec0a-00604bbba8-9d6bb8b-fra1b, host id: " error.file="/go/src/github.com/vmware-tanzu/velero-plugin-for-aws/velero-plugin-for-aws/object_store.go:331" error.function="main.(*ObjectStore).ListCommonPrefixes" logSource="pkg/controller/backup_sync_controller.go:182" time="2021-03-12T19:06:16Z" level=error msg="Error listing backups in backup store" backupLocation=saverso controller=backup-sync error="rpc error: code = Unknown desc = InvalidAccessKeyId: \n\tstatus code: 403, request id: tx0000000000000030a098d-00604bbba8-9d67586-fra1b, host id: " error.file="/go/src/github.com/vmware-tanzu/velero-plugin-for-aws/velero-plugin-for-aws/object_store.go:331" error.function="main.(*ObjectStore).ListCommonPrefixes" logSource="pkg/controller/backup_sync_controller.go:182" time="2021-03-12T19:06:46Z" level=error msg="Error listing backups in backup store" backupLocation=default controller=backup-sync error="rpc error: code = Unknown desc = InvalidAccessKeyId: \n\tstatus code: 403, request id: tx000000000000003070e76-00604bbbc6-9d6bb8b-fra1b, host id: " error.file="/go/src/github.com/vmware-tanzu/velero-plugin-for-aws/velero-plugin-for-aws/object_store.go:331" error.function="main.(*ObjectStore).ListCommonPrefixes" logSource="pkg/controller/backup_sync_controller.go:182" time="2021-03-12T19:06:46Z" level=info msg="Validating backup storage location" backup-storage-location=backups controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:114" time="2021-03-12T19:06:46Z" level=info msg="Backup storage location valid, marking as available" backup-storage-location=backups controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:121" time="2021-03-12T19:06:46Z" level=error msg="Error listing backups in backup store" backupLocation=saverso controller=backup-sync error="rpc error: code = Unknown desc = InvalidAccessKeyId: \n\tstatus code: 403, request id: tx000000000000003070eb2-00604bbbc6-9d6bb8b-fra1b, host id: " error.file="/go/src/github.com/vmware-tanzu/velero-plugin-for-aws/velero-plugin-for-aws/object_store.go:331" error.function="main.(*ObjectStore).ListCommonPrefixes" logSource="pkg/controller/backup_sync_controller.go:182" time="2021-03-12T19:06:46Z" level=info msg="Validating backup storage location" backup-storage-location=default controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:114" time="2021-03-12T19:06:46Z" level=info msg="Backup storage location is invalid, marking as unavailable" backup-storage-location=default controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:117" time="2021-03-12T19:06:47Z" level=info msg="Validating backup storage location" backup-storage-location=saverso controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:114" time="2021-03-12T19:06:47Z" level=info msg="Backup storage location is invalid, marking as unavailable" backup-storage-location=saverso controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:117" time="2021-03-12T19:06:47Z" level=info msg="Validating backup storage location" backup-storage-location=s3 controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:114" time="2021-03-12T19:06:47Z" level=info msg="Backup storage location valid, marking as available" backup-storage-location=s3 controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:121" time="2021-03-12T19:06:47Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, Backup storage location \"default\" is unavailable: rpc error: code = Unknown desc = InvalidAccessKeyId: \n\tstatus code: 403, request id: tx0000000000000030a2c3e-00604bbbc6-9d67586-fra1b, host id: ; Backup storage location \"saverso\" is unavailable: rpc error: code = Unknown desc = InvalidAccessKeyId: \n\tstatus code: 403, request id: tx00000000000000308d128-00604bbbc7-9db4ea4-fra1b, host id: )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T19:06:47Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T19:06:47Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T19:06:47Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T19:06:47Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T19:07:16Z" level=error msg="Error listing backups in backup store" backupLocation=default controller=backup-sync error="rpc error: code = Unknown desc = InvalidAccessKeyId: \n\tstatus code: 403, request id: tx00000000000000308f591-00604bbbe4-9db4ea4-fra1b, host id: " error.file="/go/src/github.com/vmware-tanzu/velero-plugin-for-aws/velero-plugin-for-aws/object_store.go:331" error.function="main.(*ObjectStore).ListCommonPrefixes" logSource="pkg/controller/backup_sync_controller.go:182" time="2021-03-12T19:07:16Z" level=error msg="Error listing backups in backup store" backupLocation=saverso controller=backup-sync error="rpc error: code = Unknown desc = InvalidAccessKeyId: \n\tstatus code: 403, request id: tx0000000000000030573a3-00604bbbe4-9d3682b-fra1b, host id: " error.file="/go/src/github.com/vmware-tanzu/velero-plugin-for-aws/velero-plugin-for-aws/object_store.go:331" error.function="main.(*ObjectStore).ListCommonPrefixes" logSource="pkg/controller/backup_sync_controller.go:182" time="2021-03-12T19:07:47Z" level=error msg="Error listing backups in backup store" backupLocation=default controller=backup-sync error="rpc error: code = Unknown desc = InvalidAccessKeyId: \n\tstatus code: 403, request id: tx000000000000003075c29-00604bbc03-9d6bb8b-fra1b, host id: " error.file="/go/src/github.com/vmware-tanzu/velero-plugin-for-aws/velero-plugin-for-aws/object_store.go:331" error.function="main.(*ObjectStore).ListCommonPrefixes" logSource="pkg/controller/backup_sync_controller.go:182" time="2021-03-12T19:07:47Z" level=info msg="Validating backup storage location" backup-storage-location=backups controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:114" time="2021-03-12T19:07:47Z" level=error msg="Error listing backups in backup store" backupLocation=saverso controller=backup-sync error="rpc error: code = Unknown desc = InvalidAccessKeyId: \n\tstatus code: 403, request id: tx000000000000003059c93-00604bbc03-9d3682b-fra1b, host id: " error.file="/go/src/github.com/vmware-tanzu/velero-plugin-for-aws/velero-plugin-for-aws/object_store.go:331" error.function="main.(*ObjectStore).ListCommonPrefixes" logSource="pkg/controller/backup_sync_controller.go:182" time="2021-03-12T19:07:47Z" level=info msg="Backup storage location valid, marking as available" backup-storage-location=backups controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:121" time="2021-03-12T19:07:47Z" level=info msg="Validating backup storage location" backup-storage-location=default controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:114" time="2021-03-12T19:07:47Z" level=info msg="Backup storage location is invalid, marking as unavailable" backup-storage-location=default controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:117" time="2021-03-12T19:07:47Z" level=info msg="Validating backup storage location" backup-storage-location=saverso controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:114" time="2021-03-12T19:07:47Z" level=info msg="Backup storage location is invalid, marking as unavailable" backup-storage-location=saverso controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:117" time="2021-03-12T19:07:47Z" level=info msg="Validating backup storage location" backup-storage-location=s3 controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:114" time="2021-03-12T19:07:47Z" level=info msg="Backup storage location valid, marking as available" backup-storage-location=s3 controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:121" time="2021-03-12T19:07:47Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, Backup storage location \"default\" is unavailable: rpc error: code = Unknown desc = InvalidAccessKeyId: \n\tstatus code: 403, request id: tx000000000000003075c8f-00604bbc03-9d6bb8b-fra1b, host id: ; Backup storage location \"saverso\" is unavailable: rpc error: code = Unknown desc = InvalidAccessKeyId: \n\tstatus code: 403, request id: tx0000000000000030a7a49-00604bbc03-9d67586-fra1b, host id: )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T19:07:47Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T19:07:47Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T19:07:47Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T19:07:47Z" level=warning msg="Unavailable backup storage locations detected: available/unavailable/unknown: 2/2/0, )" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:169" time="2021-03-12T19:08:17Z" level=error msg="Error listing backups in backup store" backupLocation=default controller=backup-sync error="rpc error: code = Unknown desc = InvalidAccessKeyId: \n\tstatus code: 403, request id: tx0000000000000030aa2ce-00604bbc21-9d67586-fra1b, host id: " error.file="/go/src/github.com/vmware-tanzu/velero-plugin-for-aws/velero-plugin-for-aws/object_store.go:331" error.function="main.(*ObjectStore).ListCommonPrefixes" logSource="pkg/controller/backup_sync_controller.go:182" time="2021-03-12T19:08:17Z" level=error msg="Error listing backups in backup store" backupLocation=saverso controller=backup-sync error="rpc error: code = Unknown desc = InvalidAccessKeyId: \n\tstatus code: 403, request id: tx0000000000000030aa2e1-00604bbc21-9d67586-fra1b, host id: " error.file="/go/src/github.com/vmware-tanzu/velero-plugin-for-aws/velero-plugin-for-aws/object_store.go:331" error.function="main.(*ObjectStore).ListCommonPrefixes" logSource="pkg/controller/backup_sync_controller.go:182" ```

Anything else you would like to add:

Here you can see my test from Postman:

It's working perfectly, but I need to provide the Service Name, but I don't see in the actual values.yaml any way to configure that...maybe that's is missing here in my connection...

Environment:

• helm version (use helm version):

    $ version.BuildInfo{Version:"v3.5.0", GitCommit:"32c22239423b3b4ba6706d450bd044baffdcf9e6", GitTreeState:"dirty", GoVersion:"go1.15.6"}

• helm chart version and app version:

NAME    NAMESPACE       REVISION        UPDATED                                 STATUS          CHART           APP VERSION
velero  velero          33              2021-03-12 19:50:09.444304 +0100 CET    deployed        velero-2.15.0   1.5.3  

• Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.15", GitCommit:"73dd5c840662bb066a146d0871216333181f4b64", GitTreeState:"clean", BuildDate:"2021-01-13T13:22:41Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.15", GitCommit:"73dd5c840662bb066a146d0871216333181f4b64", GitTreeState:"archive", BuildDate:"2021-01-13T19:24:02Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}

• Kubernetes installer & version:

Managed K8S-Cluster

• Cloud provider or hardware configuration:

IONOS by 1&1 - Cloud (Intel Skylake):

• S3 Bucket Reference for IONOS:

[IONOS S3 Bucket API](https://devops.ionos.com/api/s3/?hstc=94004290.61c6314c7e913008fbf3920436be8227.1613413417846.1615495428227.1615540550167.14&hssc=94004290.2.1615540550167&hsfp=3606176375#overview)__

[Ahmadre]

I discovered one of my bockup-volumes from the schedules got:

status:
  expiration: '2021-03-22T23:43:05Z'
  formatVersion: 1.1.0
  phase: FailedValidation
  validationErrors:
    - >-
      provider aws has more than one possible volume snapshot location, and none
      were specified explicitly or as a default
  version: 1

Am I doing something wrong here?

[Ahmadre]

SOLVED:

I had multiple backup locations when I tried several options with helm upgrade --install :)

Just had to delete the unavailables and leave just one backup location I setup in my schedules: