search

vmware-tanzu / velero

Backup and migrate Kubernetes applications and their persistent volumes
https://velero.io
Apache License 2.0
8.81k stars 1.41k forks source link

"BackupStorageLocation \"default\" is unavailable: rpc error: code = Unknown desc = NoCredentialProviders: no valid providers in chain. #6439

Closed mscott83 closed 1 year ago

mscott83 commented 1 year ago

What steps did you take and what happened: Brand new install of Velero using the following installation command:

velero install --provider aws --plugins velero/velero-plugin-for-aws:v1.7.0 --bucket my-velero-backups --backup-location-config region=eu-west-2 --snapshot-location-config region=eu-west-2 --secret-file ./credentials-velero

credentials-velero contains the following:

[default]
aws_access_key=AKIA...
aws_secret_access_key=a_secret_key

After the installation completes, the "default" BackupStorageLocation exists but shows as "Unavailable". I've checked the the credentials exist in the "cloud-credentials" secret, and they are valid. I don't know how to troubleshoot this further.

What did you expect to happen: Everything installs and works normally

The following information will help us better understand what's going on:

Defaulted container "velero" out of: velero, velero-velero-plugin-for-aws (init)
time="2023-06-29T16:49:49Z" level=info msg="setting log-level to INFO" logSource="pkg/cmd/server/server.go:186"
time="2023-06-29T16:49:49Z" level=info msg="Starting Velero server v1.11.0 (0da2baa908c88ec3c45da15001f6a4b0bda64ae2-dirty)" logSource="pkg/cmd/server/server.go:188"
time="2023-06-29T16:49:49Z" level=info msg="1 feature flags enabled []" logSource="pkg/cmd/server/server.go:190"
time="2023-06-29T16:49:49Z" level=info msg="registering plugin" command=/velero kind=BackupItemAction logSource="pkg/plugin/clientmgmt/process/registry.go:101" name=velero.io/crd-remap-version
time="2023-06-29T16:49:49Z" level=info msg="registering plugin" command=/velero kind=BackupItemAction logSource="pkg/plugin/clientmgmt/process/registry.go:101" name=velero.io/pod
time="2023-06-29T16:49:49Z" level=info msg="registering plugin" command=/velero kind=BackupItemAction logSource="pkg/plugin/clientmgmt/process/registry.go:101" name=velero.io/pv
time="2023-06-29T16:49:49Z" level=info msg="registering plugin" command=/velero kind=BackupItemAction logSource="pkg/plugin/clientmgmt/process/registry.go:101" name=velero.io/service-account
time="2023-06-29T16:49:49Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/process/registry.go:101" name=velero.io/add-pv-from-pvc
time="2023-06-29T16:49:49Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/process/registry.go:101" name=velero.io/add-pvc-from-pod
time="2023-06-29T16:49:49Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/process/registry.go:101" name=velero.io/admission-webhook-configuration
time="2023-06-29T16:49:49Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/process/registry.go:101" name=velero.io/apiservice
time="2023-06-29T16:49:49Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/process/registry.go:101" name=velero.io/change-image-name
time="2023-06-29T16:49:49Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/process/registry.go:101" name=velero.io/change-pvc-node-selector
time="2023-06-29T16:49:49Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/process/registry.go:101" name=velero.io/change-storage-class
time="2023-06-29T16:49:49Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/process/registry.go:101" name=velero.io/cluster-role-bindings
time="2023-06-29T16:49:49Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/process/registry.go:101" name=velero.io/crd-preserve-fields
time="2023-06-29T16:49:49Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/process/registry.go:101" name=velero.io/init-restore-hook
time="2023-06-29T16:49:49Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/process/registry.go:101" name=velero.io/job
time="2023-06-29T16:49:49Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/process/registry.go:101" name=velero.io/pod
time="2023-06-29T16:49:49Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/process/registry.go:101" name=velero.io/pod-volume-restore
time="2023-06-29T16:49:49Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/process/registry.go:101" name=velero.io/role-bindings
time="2023-06-29T16:49:49Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/process/registry.go:101" name=velero.io/secret
time="2023-06-29T16:49:49Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/process/registry.go:101" name=velero.io/service
time="2023-06-29T16:49:49Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/process/registry.go:101" name=velero.io/service-account
time="2023-06-29T16:49:49Z" level=info msg="registering plugin" command=/plugins/velero-plugin-for-aws kind=VolumeSnapshotter logSource="pkg/plugin/clientmgmt/process/registry.go:101" name=velero.io/aws
time="2023-06-29T16:49:49Z" level=info msg="registering plugin" command=/plugins/velero-plugin-for-aws kind=ObjectStore logSource="pkg/plugin/clientmgmt/process/registry.go:101" name=velero.io/aws
time="2023-06-29T16:49:50Z" level=info msg="Metrics server is starting to listen" addr=":8080" logSource="/go/pkg/mod/github.com/bombsimon/logrusr/v3@v3.0.0/logrusr.go:108" logger=controller-runtime.metrics
time="2023-06-29T16:49:50Z" level=info msg="Checking existence of namespace." logSource="pkg/cmd/server/server.go:423" namespace=velero
time="2023-06-29T16:49:50Z" level=info msg="Namespace exists" logSource="pkg/cmd/server/server.go:429" namespace=velero
I0629 16:49:52.063524       1 request.go:690] Waited for 1.046298187s due to client-side throttling, not priority and fairness, request: GET:https://10.43.0.1:443/apis/policy/v1?timeout=32s
time="2023-06-29T16:49:53Z" level=info msg="Checking existence of Velero custom resource definitions" logSource="pkg/cmd/server/server.go:458"
time="2023-06-29T16:49:53Z" level=info msg="All Velero custom resource definitions exist" logSource="pkg/cmd/server/server.go:492"
time="2023-06-29T16:49:53Z" level=warning msg="Velero node agent not found; pod volume backups/restores will not work until it's created" logSource="pkg/cmd/server/server.go:558"
time="2023-06-29T16:49:53Z" level=warning msg="Velero node agent not found; pod volume backups/restores will not work until it's created" logSource="pkg/cmd/server/server.go:567"
time="2023-06-29T16:49:54Z" level=info msg="Starting controllers" logSource="pkg/cmd/server/server.go:620"
time="2023-06-29T16:49:54Z" level=info msg="Starting metric server at address [:8085]" logSource="pkg/cmd/server/server.go:625"
time="2023-06-29T16:49:54Z" level=info msg="Server starting..." logSource="pkg/cmd/server/server.go:948"
time="2023-06-29T16:49:54Z" level=info msg="Starting server" addr="{\"IP\":\"::\",\"Port\":8080,\"Zone\":\"\"}" kind=metrics logSource="/go/pkg/mod/github.com/bombsimon/logrusr/v3@v3.0.0/logrusr.go:108" path=/metrics
time="2023-06-29T16:49:54Z" level=info msg="Starting EventSource" controller=backupstoragelocation controllerGroup=velero.io controllerKind=BackupStorageLocation logSource="/go/pkg/mod/github.com/bombsimon/logrusr/v3@v3.0.0/logrusr.go:108" source="kind source: *v1.BackupStorageLocation"
time="2023-06-29T16:49:54Z" level=info msg="Starting EventSource" controller=backup controllerGroup=velero.io controllerKind=Backup logSource="/go/pkg/mod/github.com/bombsimon/logrusr/v3@v3.0.0/logrusr.go:108" source="kind source: *v1.Backup"
time="2023-06-29T16:49:54Z" level=info msg="Starting EventSource" controller=backupstoragelocation controllerGroup=velero.io controllerKind=BackupStorageLocation logSource="/go/pkg/mod/github.com/bombsimon/logrusr/v3@v3.0.0/logrusr.go:108" source="kind source: *v1.BackupStorageLocationList"
time="2023-06-29T16:49:54Z" level=info msg="Starting EventSource" controller=backupstoragelocation controllerGroup=velero.io controllerKind=BackupStorageLocation logSource="/go/pkg/mod/github.com/bombsimon/logrusr/v3@v3.0.0/logrusr.go:108" source="kind source: *v1.BackupStorageLocation"
time="2023-06-29T16:49:54Z" level=info msg="Starting Controller" controller=backup controllerGroup=velero.io controllerKind=Backup logSource="/go/pkg/mod/github.com/bombsimon/logrusr/v3@v3.0.0/logrusr.go:108"
time="2023-06-29T16:49:54Z" level=info msg="Starting EventSource" controller=backup controllerGroup=velero.io controllerKind=Backup logSource="/go/pkg/mod/github.com/bombsimon/logrusr/v3@v3.0.0/logrusr.go:108" source="kind source: *v1.Backup"
time="2023-06-29T16:49:54Z" level=info msg="Starting Controller" controller=backupstoragelocation controllerGroup=velero.io controllerKind=BackupStorageLocation logSource="/go/pkg/mod/github.com/bombsimon/logrusr/v3@v3.0.0/logrusr.go:108"
time="2023-06-29T16:49:54Z" level=info msg="Starting EventSource" controller=backupstoragelocation controllerGroup=velero.io controllerKind=BackupStorageLocation logSource="/go/pkg/mod/github.com/bombsimon/logrusr/v3@v3.0.0/logrusr.go:108" source="kind source: *v1.BackupStorageLocationList"
time="2023-06-29T16:49:54Z" level=info msg="Starting EventSource" controller=downloadrequest controllerGroup=velero.io controllerKind=DownloadRequest logSource="/go/pkg/mod/github.com/bombsimon/logrusr/v3@v3.0.0/logrusr.go:108" source="kind source: *v1.DownloadRequest"
time="2023-06-29T16:49:54Z" level=info msg="Starting EventSource" controller=serverstatusrequest controllerGroup=velero.io controllerKind=ServerStatusRequest logSource="/go/pkg/mod/github.com/bombsimon/logrusr/v3@v3.0.0/logrusr.go:108" source="kind source: *v1.ServerStatusRequest"
time="2023-06-29T16:49:54Z" level=info msg="Starting Controller" controller=downloadrequest controllerGroup=velero.io controllerKind=DownloadRequest logSource="/go/pkg/mod/github.com/bombsimon/logrusr/v3@v3.0.0/logrusr.go:108"
time="2023-06-29T16:49:54Z" level=info msg="Starting Controller" controller=serverstatusrequest controllerGroup=velero.io controllerKind=ServerStatusRequest logSource="/go/pkg/mod/github.com/bombsimon/logrusr/v3@v3.0.0/logrusr.go:108"
time="2023-06-29T16:49:54Z" level=info msg="Starting EventSource" controller=backup controllerGroup=velero.io controllerKind=Backup logSource="/go/pkg/mod/github.com/bombsimon/logrusr/v3@v3.0.0/logrusr.go:108" source="kind source: *v1.Backup"
time="2023-06-29T16:49:54Z" level=info msg="Starting EventSource" controller=deletebackuprequest controllerGroup=velero.io controllerKind=DeleteBackupRequest logSource="/go/pkg/mod/github.com/bombsimon/logrusr/v3@v3.0.0/logrusr.go:108" source="kind source: *v1.DeleteBackupRequest"
time="2023-06-29T16:49:54Z" level=info msg="Starting EventSource" controller=backup controllerGroup=velero.io controllerKind=Backup logSource="/go/pkg/mod/github.com/bombsimon/logrusr/v3@v3.0.0/logrusr.go:108" source="kind source: *v1.Backup"
time="2023-06-29T16:49:54Z" level=info msg="Starting Controller" controller=backup controllerGroup=velero.io controllerKind=Backup logSource="/go/pkg/mod/github.com/bombsimon/logrusr/v3@v3.0.0/logrusr.go:108"
time="2023-06-29T16:49:54Z" level=info msg="Starting EventSource" controller=backuprepository controllerGroup=velero.io controllerKind=BackupRepository logSource="/go/pkg/mod/github.com/bombsimon/logrusr/v3@v3.0.0/logrusr.go:108" source="kind source: *v1.BackupRepository"
time="2023-06-29T16:49:54Z" level=info msg="Starting EventSource" controller=backup controllerGroup=velero.io controllerKind=Backup logSource="/go/pkg/mod/github.com/bombsimon/logrusr/v3@v3.0.0/logrusr.go:108" source="kind source: *v1.BackupList"
time="2023-06-29T16:49:54Z" level=info msg="Starting EventSource" controller=restore controllerGroup=velero.io controllerKind=Restore logSource="/go/pkg/mod/github.com/bombsimon/logrusr/v3@v3.0.0/logrusr.go:108" source="kind source: *v1.Restore"
time="2023-06-29T16:49:54Z" level=info msg="Starting Controller" controller=backup controllerGroup=velero.io controllerKind=Backup logSource="/go/pkg/mod/github.com/bombsimon/logrusr/v3@v3.0.0/logrusr.go:108"
time="2023-06-29T16:49:54Z" level=info msg="Starting EventSource" controller=backup controllerGroup=velero.io controllerKind=Backup logSource="/go/pkg/mod/github.com/bombsimon/logrusr/v3@v3.0.0/logrusr.go:108" source="kind source: *v1.BackupList"
time="2023-06-29T16:49:54Z" level=info msg="Starting EventSource" controller=deletebackuprequest controllerGroup=velero.io controllerKind=DeleteBackupRequest logSource="/go/pkg/mod/github.com/bombsimon/logrusr/v3@v3.0.0/logrusr.go:108" source="kind source: *v1.DeleteBackupRequestList"
time="2023-06-29T16:49:54Z" level=info msg="Starting Controller" controller=backupstoragelocation controllerGroup=velero.io controllerKind=BackupStorageLocation logSource="/go/pkg/mod/github.com/bombsimon/logrusr/v3@v3.0.0/logrusr.go:108"
time="2023-06-29T16:49:54Z" level=info msg="Starting Controller" controller=restore controllerGroup=velero.io controllerKind=Restore logSource="/go/pkg/mod/github.com/bombsimon/logrusr/v3@v3.0.0/logrusr.go:108"
time="2023-06-29T16:49:54Z" level=info msg="Starting EventSource" controller=schedule controllerGroup=velero.io controllerKind=Schedule logSource="/go/pkg/mod/github.com/bombsimon/logrusr/v3@v3.0.0/logrusr.go:108" source="kind source: *v1.Schedule"
time="2023-06-29T16:49:54Z" level=info msg="Starting Controller" controller=deletebackuprequest controllerGroup=velero.io controllerKind=DeleteBackupRequest logSource="/go/pkg/mod/github.com/bombsimon/logrusr/v3@v3.0.0/logrusr.go:108"
time="2023-06-29T16:49:54Z" level=info msg="Starting EventSource" controller=restore controllerGroup=velero.io controllerKind=Restore logSource="/go/pkg/mod/github.com/bombsimon/logrusr/v3@v3.0.0/logrusr.go:108" source="kind source: *v1.Restore"
time="2023-06-29T16:49:54Z" level=info msg="Starting EventSource" controller=schedule controllerGroup=velero.io controllerKind=Schedule logSource="/go/pkg/mod/github.com/bombsimon/logrusr/v3@v3.0.0/logrusr.go:108" source="kind source: *v1.ScheduleList"
time="2023-06-29T16:49:54Z" level=info msg="Starting EventSource" controller=backuprepository controllerGroup=velero.io controllerKind=BackupRepository logSource="/go/pkg/mod/github.com/bombsimon/logrusr/v3@v3.0.0/logrusr.go:108" source="kind source: *v1.BackupRepositoryList"
time="2023-06-29T16:49:54Z" level=info msg="Starting Controller" controller=backup controllerGroup=velero.io controllerKind=Backup logSource="/go/pkg/mod/github.com/bombsimon/logrusr/v3@v3.0.0/logrusr.go:108"
time="2023-06-29T16:49:54Z" level=info msg="Starting EventSource" controller=backuprepository controllerGroup=velero.io controllerKind=BackupRepository logSource="/go/pkg/mod/github.com/bombsimon/logrusr/v3@v3.0.0/logrusr.go:108" source="kind source: *v1.BackupStorageLocation"
time="2023-06-29T16:49:54Z" level=info msg="Starting Controller" controller=schedule controllerGroup=velero.io controllerKind=Schedule logSource="/go/pkg/mod/github.com/bombsimon/logrusr/v3@v3.0.0/logrusr.go:108"
time="2023-06-29T16:49:54Z" level=info msg="Starting EventSource" controller=restore controllerGroup=velero.io controllerKind=Restore logSource="/go/pkg/mod/github.com/bombsimon/logrusr/v3@v3.0.0/logrusr.go:108" source="kind source: *v1.RestoreList"
time="2023-06-29T16:49:54Z" level=info msg="Starting Controller" controller=backuprepository controllerGroup=velero.io controllerKind=BackupRepository logSource="/go/pkg/mod/github.com/bombsimon/logrusr/v3@v3.0.0/logrusr.go:108"
time="2023-06-29T16:49:54Z" level=info msg="Starting Controller" controller=restore controllerGroup=velero.io controllerKind=Restore logSource="/go/pkg/mod/github.com/bombsimon/logrusr/v3@v3.0.0/logrusr.go:108"
time="2023-06-29T16:49:54Z" level=info msg="Starting workers" controller=backup controllerGroup=velero.io controllerKind=Backup logSource="/go/pkg/mod/github.com/bombsimon/logrusr/v3@v3.0.0/logrusr.go:108" worker count=1
time="2023-06-29T16:49:54Z" level=info msg="Starting workers" controller=backup controllerGroup=velero.io controllerKind=Backup logSource="/go/pkg/mod/github.com/bombsimon/logrusr/v3@v3.0.0/logrusr.go:108" worker count=1
time="2023-06-29T16:49:54Z" level=info msg="Starting workers" controller=backup controllerGroup=velero.io controllerKind=Backup logSource="/go/pkg/mod/github.com/bombsimon/logrusr/v3@v3.0.0/logrusr.go:108" worker count=1
time="2023-06-29T16:49:54Z" level=info msg="Starting workers" controller=backupstoragelocation controllerGroup=velero.io controllerKind=BackupStorageLocation logSource="/go/pkg/mod/github.com/bombsimon/logrusr/v3@v3.0.0/logrusr.go:108" worker count=1
time="2023-06-29T16:49:54Z" level=info msg="Starting workers" controller=backup controllerGroup=velero.io controllerKind=Backup logSource="/go/pkg/mod/github.com/bombsimon/logrusr/v3@v3.0.0/logrusr.go:108" worker count=1
time="2023-06-29T16:49:54Z" level=info msg="Starting workers" controller=backuprepository controllerGroup=velero.io controllerKind=BackupRepository logSource="/go/pkg/mod/github.com/bombsimon/logrusr/v3@v3.0.0/logrusr.go:108" worker count=1
time="2023-06-29T16:49:54Z" level=info msg="Starting workers" controller=restore controllerGroup=velero.io controllerKind=Restore logSource="/go/pkg/mod/github.com/bombsimon/logrusr/v3@v3.0.0/logrusr.go:108" worker count=1
time="2023-06-29T16:49:54Z" level=info msg="Starting workers" controller=deletebackuprequest controllerGroup=velero.io controllerKind=DeleteBackupRequest logSource="/go/pkg/mod/github.com/bombsimon/logrusr/v3@v3.0.0/logrusr.go:108" worker count=1
time="2023-06-29T16:49:54Z" level=info msg="Starting workers" controller=restore controllerGroup=velero.io controllerKind=Restore logSource="/go/pkg/mod/github.com/bombsimon/logrusr/v3@v3.0.0/logrusr.go:108" worker count=1
time="2023-06-29T16:49:54Z" level=info msg="Starting workers" controller=backupstoragelocation controllerGroup=velero.io controllerKind=BackupStorageLocation logSource="/go/pkg/mod/github.com/bombsimon/logrusr/v3@v3.0.0/logrusr.go:108" worker count=1
time="2023-06-29T16:49:54Z" level=info msg="Starting workers" controller=schedule controllerGroup=velero.io controllerKind=Schedule logSource="/go/pkg/mod/github.com/bombsimon/logrusr/v3@v3.0.0/logrusr.go:108" worker count=1
time="2023-06-29T16:49:54Z" level=info msg="Starting workers" controller=downloadrequest controllerGroup=velero.io controllerKind=DownloadRequest logSource="/go/pkg/mod/github.com/bombsimon/logrusr/v3@v3.0.0/logrusr.go:108" worker count=1
time="2023-06-29T16:49:54Z" level=info msg="Starting workers" controller=serverstatusrequest controllerGroup=velero.io controllerKind=ServerStatusRequest logSource="/go/pkg/mod/github.com/bombsimon/logrusr/v3@v3.0.0/logrusr.go:108" worker count=10
time="2023-06-29T16:50:00Z" level=error msg="Error getting backup store for this location" backupLocation=velero/default controller=backup-sync error="rpc error: code = Unknown desc = NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors" error.file="/go/src/velero-plugin-for-aws/velero-plugin-for-aws/volume_snapshotter.go:62" error.function=main.getSession logSource="pkg/controller/backup_sync_controller.go:100"
time="2023-06-29T16:50:00Z" level=error msg="Error getting a backup store" backup-storage-location=velero/default controller=backup-storage-location error="rpc error: code = Unknown desc = NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors" error.file="/go/src/velero-plugin-for-aws/velero-plugin-for-aws/volume_snapshotter.go:62" error.function=main.getSession logSource="pkg/controller/backup_storage_location_controller.go:148"
time="2023-06-29T16:50:00Z" level=info msg="BackupStorageLocation is invalid, marking as unavailable" backup-storage-location=velero/default controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:131"
time="2023-06-29T16:50:00Z" level=error msg="Current BackupStorageLocations available/unavailable/unknown: 0/0/1, BackupStorageLocation \"default\" is unavailable: rpc error: code = Unknown desc = NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors)" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:192"
time="2023-06-29T16:51:00Z" level=error msg="Error getting backup store for this location" backupLocation=velero/default controller=backup-sync error="rpc error: code = Unknown desc = NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors" error.file="/go/src/velero-plugin-for-aws/velero-plugin-for-aws/volume_snapshotter.go:62" error.function=main.getSession logSource="pkg/controller/backup_sync_controller.go:100"
time="2023-06-29T16:51:10Z" level=error msg="Error getting a backup store" backup-storage-location=velero/default controller=backup-storage-location error="rpc error: code = Unknown desc = NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors" error.file="/go/src/velero-plugin-for-aws/velero-plugin-for-aws/volume_snapshotter.go:62" error.function=main.getSession logSource="pkg/controller/backup_storage_location_controller.go:148"
time="2023-06-29T16:51:10Z" level=info msg="BackupStorageLocation is invalid, marking as unavailable" backup-storage-location=velero/default controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:131"
time="2023-06-29T16:51:10Z" level=error msg="Current BackupStorageLocations available/unavailable/unknown: 0/1/0, BackupStorageLocation \"default\" is unavailable: rpc error: code = Unknown desc = NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors)" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:192"
time="2023-06-29T16:52:00Z" level=error msg="Error getting backup store for this location" backupLocation=velero/default controller=backup-sync error="rpc error: code = Unknown desc = NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors" error.file="/go/src/velero-plugin-for-aws/velero-plugin-for-aws/volume_snapshotter.go:62" error.function=main.getSession logSource="pkg/controller/backup_sync_controller.go:100"
time="2023-06-29T16:52:20Z" level=error msg="Error getting a backup store" backup-storage-location=velero/default controller=backup-storage-location error="rpc error: code = Unknown desc = NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors" error.file="/go/src/velero-plugin-for-aws/velero-plugin-for-aws/volume_snapshotter.go:62" error.function=main.getSession logSource="pkg/controller/backup_storage_location_controller.go:148"
time="2023-06-29T16:52:20Z" level=info msg="BackupStorageLocation is invalid, marking as unavailable" backup-storage-location=velero/default controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:131"
time="2023-06-29T16:52:20Z" level=error msg="Current BackupStorageLocations available/unavailable/unknown: 0/1/0, BackupStorageLocation \"default\" is unavailable: rpc error: code = Unknown desc = NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors)" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:192"
time="2023-06-29T16:53:00Z" level=error msg="Error getting backup store for this location" backupLocation=velero/default controller=backup-sync error="rpc error: code = Unknown desc = NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors" error.file="/go/src/velero-plugin-for-aws/velero-plugin-for-aws/volume_snapshotter.go:62" error.function=main.getSession logSource="pkg/controller/backup_sync_controller.go:100"
time="2023-06-29T16:53:30Z" level=error msg="Error getting a backup store" backup-storage-location=velero/default controller=backup-storage-location error="rpc error: code = Unknown desc = NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors" error.file="/go/src/velero-plugin-for-aws/velero-plugin-for-aws/volume_snapshotter.go:62" error.function=main.getSession logSource="pkg/controller/backup_storage_location_controller.go:148"
time="2023-06-29T16:53:30Z" level=info msg="BackupStorageLocation is invalid, marking as unavailable" backup-storage-location=velero/default controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:131"
time="2023-06-29T16:53:30Z" level=error msg="Current BackupStorageLocations available/unavailable/unknown: 0/1/0, BackupStorageLocation \"default\" is unavailable: rpc error: code = Unknown desc = NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors)" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:192"
time="2023-06-29T16:54:00Z" level=error msg="Error getting backup store for this location" backupLocation=velero/default controller=backup-sync error="rpc error: code = Unknown desc = NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors" error.file="/go/src/velero-plugin-for-aws/velero-plugin-for-aws/volume_snapshotter.go:62" error.function=main.getSession logSource="pkg/controller/backup_sync_controller.go:100"
time="2023-06-29T16:54:40Z" level=error msg="Error getting a backup store" backup-storage-location=velero/default controller=backup-storage-location error="rpc error: code = Unknown desc = NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors" error.file="/go/src/velero-plugin-for-aws/velero-plugin-for-aws/volume_snapshotter.go:62" error.function=main.getSession logSource="pkg/controller/backup_storage_location_controller.go:148"
time="2023-06-29T16:54:40Z" level=info msg="BackupStorageLocation is invalid, marking as unavailable" backup-storage-location=velero/default controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:131"
time="2023-06-29T16:54:40Z" level=error msg="Current BackupStorageLocations available/unavailable/unknown: 0/1/0, BackupStorageLocation \"default\" is unavailable: rpc error: code = Unknown desc = NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors)" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:192"
I0629 16:54:59.860873       1 request.go:690] Waited for 1.042760848s due to client-side throttling, not priority and fairness, request: GET:https://10.43.0.1:443/apis/argoproj.io/v1alpha1?timeout=32s
time="2023-06-29T16:55:00Z" level=error msg="Error getting backup store for this location" backupLocation=velero/default controller=backup-sync error="rpc error: code = Unknown desc = NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors" error.file="/go/src/velero-plugin-for-aws/velero-plugin-for-aws/volume_snapshotter.go:62" error.function=main.getSession logSource="pkg/controller/backup_sync_controller.go:100"
time="2023-06-29T16:55:50Z" level=error msg="Error getting a backup store" backup-storage-location=velero/default controller=backup-storage-location error="rpc error: code = Unknown desc = NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors" error.file="/go/src/velero-plugin-for-aws/velero-plugin-for-aws/volume_snapshotter.go:62" error.function=main.getSession logSource="pkg/controller/backup_storage_location_controller.go:148"
time="2023-06-29T16:55:50Z" level=info msg="BackupStorageLocation is invalid, marking as unavailable" backup-storage-location=velero/default controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:131"
time="2023-06-29T16:55:50Z" level=error msg="Current BackupStorageLocations available/unavailable/unknown: 0/1/0, BackupStorageLocation \"default\" is unavailable: rpc error: code = Unknown desc = NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors)" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:192"
time="2023-06-29T16:56:00Z" level=error msg="Error getting backup store for this location" backupLocation=velero/default controller=backup-sync error="rpc error: code = Unknown desc = NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors" error.file="/go/src/velero-plugin-for-aws/velero-plugin-for-aws/volume_snapshotter.go:62" error.function=main.getSession logSource="pkg/controller/backup_sync_controller.go:100"
time="2023-06-29T16:57:00Z" level=error msg="Error getting a backup store" backup-storage-location=velero/default controller=backup-storage-location error="rpc error: code = Unknown desc = NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors" error.file="/go/src/velero-plugin-for-aws/velero-plugin-for-aws/volume_snapshotter.go:62" error.function=main.getSession logSource="pkg/controller/backup_storage_location_controller.go:148"
time="2023-06-29T16:57:00Z" level=info msg="BackupStorageLocation is invalid, marking as unavailable" backup-storage-location=velero/default controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:131"
time="2023-06-29T16:57:00Z" level=error msg="Error getting backup store for this location" backupLocation=velero/default controller=backup-sync error="rpc error: code = Unknown desc = NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors" error.file="/go/src/velero-plugin-for-aws/velero-plugin-for-aws/volume_snapshotter.go:62" error.function=main.getSession logSource="pkg/controller/backup_sync_controller.go:100"
time="2023-06-29T16:57:00Z" level=error msg="Current BackupStorageLocations available/unavailable/unknown: 0/1/0, BackupStorageLocation \"default\" is unavailable: rpc error: code = Unknown desc = NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors)" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:192"

Anything else you would like to add:

Environment:

Vote on this issue!

This is an invitation to the Velero community to vote on issues, you can see the project's top voted issues listed here.
Use the "reaction smiley face" up to the right of this comment to vote.

draghuram commented 1 year ago

Can you copy the output of "describe" of the "default" BSL? Usually, there should be a message stating the reason for any issues (such as authentication problem).

draghuram commented 1 year ago

BTW, the key should be "aws_access_key_id". In your snippet above, it is "aws_access_key". May be that is the issue here.

mscott83 commented 1 year ago

@draghuram you win the prize, I should have used "aws_access_key_id". When I corrected that and re-installed, it worked fine.

Amazing that you can spend hours troubleshooting something and fail to spot the most basic config error. Thank you so much for your help!

draghuram commented 1 year ago

Glad to be of help. And it is true that fresh eyes can see what you miss after staring at it for hours!