Access to MinIO through credentials file not working

[jonkerj]

What steps did you take and what happened: I am trying to use Velero targetting minio to backup my k8s cluster. Whatever I do, velero complains (repeatedly) about this in the logs:

time="2021-04-15T17:19:49Z" level=info msg="No backup locations were ready to be verified" controller=backupstoragelocation logSource="pkg/controller/backupstoragelocation_controller.go:120"
time="2021-04-15T17:19:49Z" level=error msg="Current backup storage locations available/unavailable/unknown: 0/0/1)" controller=backupstoragelocation logSource="pkg/controller/backupstoragelocation_controller.go:154"               time="2021-04-15T17:19:50Z" level=info msg="Checking for existing backup locations ready to be verified; there needs to be at least 1 backup location available" controller=backupstoragelocation logSource="pkg/controller/backupstora
gelocation_controller.go:58"
time="2021-04-15T17:19:56Z" level=error msg="Error getting a backup store" backupstoragelocation=default controller=backupstoragelocation error="rpc error: code = Unknown desc = NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors" error.file="/go/src/github.com/vmware-tanzu/velero-plugin-for-aws/velero-plugin-for-aws/volume_snapshotter.go:59" error.function=main.getSession logS
ource="pkg/controller/backupstoragelocation_controller.go:87"

The BSL is not getting verified and all Backups fail.

I have installed Velero using Helm (chart v2.16.0 / app v1.5.3) using the following values:

deployRestic: false
snapshotsEnabled: false
configuration:
  provider: aws
  backupStorageLocation:
    provider: aws
    bucket: velero
    config:
      publicUrl: http://SOME.FQDN:9000
      region: minio
      s3ForcePathStyle: "true"
      s3Url: http://SOME.FQDN:9000
credentials:
  useSecret: true
  existingSecret: minio-credentials
schedules:
  backup:
    schedule: '@every 24h'
    template:
      ttl: 732h
      includeClusterResources: true
initContainers:
- name: velero-plugin-for-aws
  image: velero/velero-plugin-for-aws:v1.1.0
  imagePullPolicy: IfNotPresent
  volumeMounts:
  - mountPath: /target
    name: plugins

I've preconfigured a secret:

apiVersion: v1
kind: Secret
metadata:
  name: minio-credentials
  namespace: velero
stringData:
  cloud: |
    [default]
    aws_access_key_id=<secret>
    aws_secret_key=<not going to tell>

To double check, kubectl exec -n velero deploy/velero -- cat /credentials/cloud yields the above data.

And here is the thing: when I switch from credential file based secrets to env vars, it does work. Config is identical, except for:

credentials:
  useSecret: false
  extraSecretRef: velero-env-vars

With the following secret:

apiVersion: v1
kind: Secret
metadata:
  name: velero-env-vars
  namespace: velero
stringData:
  AWS_ACCESS_KEY_ID: <secret>
  AWS_SECRET_ACCESS_KEY: <not going to tell>

What did you expect to happen: I would expect velero to be able to access Minio using the credentials file.

Anything else you would like to add: I tried several things:

• install using velero install
• switch to image tag v1.5.4 Without result

Unfortunately, issues with seemingly similar problems (#2279, #2278) were closed without resolution.

Environment:

• Velero version (use velero version): client v1.5.3, server v1.5.3
• Velero features (use velero client config get features):
• Kubernetes version (use kubectl version): client v1.20.2 server v1.20.5
• Kubernetes installer & version: terraform/talos v0.9.1
• Cloud provider or hardware configuration: qemu
• OS (e.g. from /etc/os-release): n/a (talos)

Vote on this issue!

This is an invitation to the Velero community to vote on issues, you can see the project's top voted issues listed here.
Use the "reaction smiley face" up to the right of this comment to vote.

• :+1: for "I would like to see this bug fixed as soon as possible"
• :-1: for "There are more important bugs to focus on right now"

[jenting]

@jonkerj Your pre-configured Secret is incorrect, the key is a typo. It should be aws_secret_access_key, not aws_secret_key.

• WAS

  apiVersion: v1
  kind: Secret
  metadata:
  name: minio-credentials
  namespace: velero
  stringData:
  cloud: |
  [default]
  aws_access_key_id=<secret>
  aws_secret_key=<not going to tell>

• IS

  apiVersion: v1
  kind: Secret
  metadata:
  name: minio-credentials
  namespace: velero
  stringData:
  cloud: |
  [default]
  aws_access_key_id=<secret>
  aws_secret_access_key=<not going to tell>

[ejschoen]

I can report the same behavior, without the typo in the key. I was able to make progress, however, by creating the cloud-credentials secret outside of "velero install" (i.e., in the 00-minio-deployment.yaml file), but still providing the secret name to velero install. It reported the secret already existing, and proceeded. The only other difference, and I don't think this should be significant, is that I made the secret with stringData instead of data (which velero install does when it creates the secret).

[johnlinvc]

I have the same issue when using Minio as backupstoragelocation and using openebs as default provider. What I found is that AWS_SHARED_CREDENTIALS_FILE env var is only set when using AWS provider 0. The workaround I have is adding it to the extraEnvVars like below.

credentials:
  secretContents:
    cloud: |
      [default]
      aws_access_key_id=<secret>
      aws_secret_key=<not going to tell>
    extraEnvVars:
      AWS_SHARED_CREDENTIALS_FILE: /credentials/cloud

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale[bot]]

Closing the stale issue.

[kub3let]

I tried setting up velero today using ansible & minio, came up with the same problem. existingSecret would not work, had to use extraSecretRef so this issue still exists.