Restic Swift Error - "TooManyRequests"

[sashokbg]

Output of restic version

restic 0.13.1 compiled with go1.18.1 on linux/amd64

How did you run restic exactly?

I ran my "openrc.sh" to authenticate with Open Stack Swift.

My "env | grep OS"

OS_AUTH_URL=https://auth.cloud.ovh.net/v3
OS_IDENTITY_API_VERSION=3
OS_USER_DOMAIN_NAME=Default
OS_PROJECT_DOMAIN_NAME=Default
OS_TENANT_ID=REDACTED
OS_TENANT_NAME=REDACTED
OS_USERNAME=REDACTED
OS_PASSWORD=VERY REDACTED
OS_REGION_NAME=DE

. restic -r swift:home-cloud-backup:/backup1 backup cluster

Load(<key/e9ec344cac>, 0, 0) returned error, retrying after 552.330144ms: conn.ObjectOpen: TooManyRequests
Load(<key/e9ec344cac>, 0, 0) returned error, retrying after 1.080381816s: conn.ObjectOpen: TooManyRequests
Load(<key/e9ec344cac>, 0, 0) returned error, retrying after 1.31013006s: conn.ObjectOpen: TooManyRequests

Doing a simple "swift upload home-cloud-backup dir" works with no problems.

What backend/server/service did you use to store the repository?

Swift Cloud Archive on OVH

Expected behavior

Upload the backup to the swift backend

Actual behavior

Restic managed to upload 2 files (see image below), so it looks like it is not authentication related.

Steps to reproduce the behavior

• Create an OVH Account
• Create a user with "storage" permissions
• Download the openrc.sh file
• source openrc.sh -> enter password
• run restic init
• run restic backup

Do you have any idea what may have caused this?

Unfortunately no, but I think it might be related to running too many parallel connections.

Do you have an idea how to solve the issue?

No

Did restic help you today? Did it make you happy in any way?

Yes, restic and k8up would be the perfect solution form my home-cloud solution. It is open source and free as in freedom :) projects that make it possible to be independent from big enterprises and also learn a thing or two on the way ;)

[sashokbg]

Hello, I tried to enable debug logs as described in the tutorial https://restic.readthedocs.io/en/latest/090_participating.html After building locally the problem seems to disappear and my backups are correctly uploaded.

Here is the output of my "./restic version" local:

restic 0.13.1 (v0.13.0-126-g26c33332) compiled with go1.18.1 on linux/amd64

What is strange is that the version seems to be the same (probably different build ?)

[MichaelEischer]

Getting a TooManyRequests error right when loading the repository keys is odd. At that point restic has barely issued any request at all to the backend. Are you able to still replicate the error using the regular restic 0.13.1 build? Otherwise I'm tempted to say that the backend did something strange...

[sashokbg]

Hello @MichaelEischer, this morning everything works fine with both official release and compiled version. I will contact my cloud provider (OVH) to see if the problem is related to their API quotas.

[sashokbg]

Hello, @MichaelEischer

I found where the error is comming from. I am using OVH Cloud Archive which is the equivalent of S3 glacier. When uplading files they are in "frozen" state and to download them you first need to wait for them to become available. This is done automatically when triggering the first download. As the doc states - the API returns 429 (too many requests) when unfreezing a file - https://docs.ovh.com/gb/en/storage/pca/unlock/.

I have also found a great article on the topic posted on duplicity's forum - https://forum.duplicacy.com/t/cold-storage-compatibility-with-ovh-cloud-public-archive/2001/2

My question is - is it possible to store metadata files such as config on a different location ? Let's say we add a feature to put metadata in one bucket (swift container) and actual backup data on another ? Do you think there is any other way to go around this limitation ?

Related to #3235 & discussion #3202

[Kulgar]

I'd also like to see this issue adressed :-)

[MichaelEischer]

Let's close this in favor the cold storage discussion issue https://github.com/restic/restic/issues/3202 .