"restic repository is not ready" error on backups with Minio and Restic

[irizzant]

What steps did you take and what happened: I have a problem running a backup using Minio as storage backend.

I run the backup with:

velero backup create test --default-volumes-to-restic=true --ttl=24h0m0s

to backup the whole cluster and the backup starts correctly. After the backup completes I see:

NAME                           STATUS            ERRORS   WARNINGS   CREATED                          EXPIRES   STORAGE LOCATION   SELECTOR
test                           PartiallyFailed   140      1          2021-04-07 19:00:15 +0200 CEST   4h        default            <none>

The logs reports:

time="2021-04-07T21:00:29Z" level=error msg="Error backing up item" backup=velero/test error="restic repository is not ready: " error.file="/go/src/github.com/vmware-tanzu/velero/pkg/restic/repository_ensurer.go:144" error.function="github.com/vmware-tanzu/velero/pkg/restic.(*repositoryEnsurer).EnsureRepo" logSource="pkg/backup/backup.go:431" name=restic-vfm7f

I'm not sure what's happening here.

The backup storage location is available:

NAME      PROVIDER   BUCKET/PREFIX   PHASE       LAST VALIDATED                   ACCESS MODE
default   aws        k8s-backups     Available   2021-04-08 14:50:22 +0200 CEST   ReadWrite

What did you expect to happen: The backup completes successfully

The output of the following commands will help us better understand what's going on: (Pasting long output into a GitHub gist or other pastebin is fine.)

• kubectl logs deployment/velero -n velero logs-velero.txt

• velero backup describe <backupname> or kubectl get backup/<backupname> -n velero -o yaml details.txt

• velero backup logs <backupname> backup-logs.txt

• velero restore describe <restorename> or kubectl get restore/<restorename> -n velero -o yaml

• velero restore logs <restorename>

Anything else you would like to add: [Miscellaneous information that will assist in solving the issue.]

Environment:

• Velero version (use velero version):

  Client:
      Version: v1.5.3
      Git commit: 123109a3bcac11dbb6783d2758207bac0d0817cb
  Server:
      Version: v1.5.3

• Velero features (use velero client config get features): features: <NOT SET>
• Kubernetes version (use kubectl version):

  Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.0", GitCommit:"af46c47ce925f4c4ad5cc8d1fca46c7b77d13b38", GitTreeState:"clean", BuildDate:"2020-12-08T17:59:43Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}
  Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.7", GitCommit:"1dd5338295409edcfff11505e7bb246f0d325d15", GitTreeState:"clean", BuildDate:"2021-01-13T13:15:20Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}

• Kubernetes installer & version: kubespray last release
• Cloud provider or hardware configuration:
• OS (e.g. from /etc/os-release):

Vote on this issue!

This is an invitation to the Velero community to vote on issues, you can see the project's top voted issues listed here.
Use the "reaction smiley face" up to the right of this comment to vote.

• :+1: for "I would like to see this bug fixed as soon as possible"
• :-1: for "There are more important bugs to focus on right now"

[zubron]

Hi @irizzant - Sorry you're having issues with this, and thank you for providing all the logs. Unfortunately, it looks like some of the restic logs that would give us some more information are only printed at the debug log level. Could you please change your Velero deployment to increase the log level using these instructions and run another backup and share the logs? This will give us more details, including the stdout and stderr when running restic commands. Thanks!

[irizzant]

Hi @zubron thanks for the update. Here the updated logs: kubectl logs deployment/velero -n velero logs-velero.txt velero backup logs <backupname> I get An error occurred: timed out waiting for download URL