Backup to Linode ObjectStorage fails

[alwinzwets]

This might be related to issue #743. Also I'm not sure whether this is a Barman or a Linode issue. I'm trying to setup a backup strategy to Linode's Object Storage. I am using barman via cloudnative-pg 1.23.2. This fails with the following error:

{"level":"info","ts":"2024-07-04T09:09:09Z","msg":"Starting barman-cloud-backup","backupName":"backup-example","backupNamespace":"backup-example","logging_pod":"database-cluster-1","options":["--user","postgres","--name","backup-20240704090909","--gzip","--encryption","AES256","--jobs","2","--endpoint-url","https://nl-ams-1.linodeobjects.com","--cloud-provider","aws-s3","s3://bucketname/backups","database-cluster"]}
{"level":"info","ts":"2024-07-04T09:09:16Z","logger":"barman-cloud-backup","msg":"2024-07-04 09:09:16,434 [24983] ERROR: Backup failed uploading data (An error occurred (InvalidArgument) when calling the CreateMultipartUpload operation: Unknown)","pipe":"stderr","logging_pod":"database-cluster-1"}
{"level":"info","ts":"2024-07-04T09:09:16Z","logger":"barman-cloud-backup","msg":"2024-07-04 09:09:16,449 [24983] ERROR: Backup failed uploading backup.info file (An error occurred (InvalidArgument) when calling the PutObject operation: Unknown)","pipe":"stderr","logging_pod":"database-cluster-1"}

To reproduce, use the following configuration:

backup:
    barmanObjectStore:
      data:
        compression: gzip
        encryption: AES256
        jobs: 2
      destinationPath: s3://bucketname/backups
      endpointURL: https://nl-ams-1.linodeobjects.com
      s3Credentials:
        accessKeyId:
          key: ACCESS_KEY_ID
          name: database-cluster-backup-s3-creds
        secretAccessKey:
          key: ACCESS_SECRET_KEY
          name: database-cluster-backup-s3-creds
      wal:
        compression: gzip
        encryption: AES256
        maxParallel: 1
    retentionPolicy: 30d
    target: prefer-standby

The same configuration with an AWS or Wasabi Object Storage works as expected, for example with the following configuration:

backup:
    barmanObjectStore:
      data:
        compression: gzip
        encryption: AES256
        jobs: 2
      destinationPath: s3://bucketname/backups
      endpointURL: https://s3.eu-central-1.wasabisys.com
      s3Credentials:
        accessKeyId:
          key: ACCESS_KEY_ID
          name: database-cluster-backup-s3-creds
        secretAccessKey:
          key: ACCESS_SECRET_KEY
          name: database-cluster-backup-s3-creds
      wal:
        compression: gzip
        encryption: AES256
        maxParallel: 1
    retentionPolicy: 30d
    target: prefer-standby

The fact that the backup works with Wasabi and S3 made me reason the issue could be on Linode's side, so I filed a support ticket with them, referencing #743, with the following response:

... Thank you for passing along those detail for us. It's important to note that our Object Storage does support multipart uploads and is S3 compatible.

After reading through the thread you shared, it looks like this may be an issue with how the barman code interacts with our buckets. Since this is a third party source, our team won't be able to make any changes on that front. With that in mind, I would recommend bringing your findings to barmans support and/or submitting a bug request to see if they can help provide potential workarounds:

https://pgbarman.org/support/ ...

Any ideas what could be the problem?

Thanks,

Alwin

[gcalacoci]

Hi Alwin,

It seems that Linode object storage is not fully compatible with s3.

Barman uses boto3 lib (official aws python library) to execute multipart uploads into s3 buckets. As a result, it is able to work, out of the box, with all the services that are fully S3-compatible, the fact that is working with wasabi (and obviously s3) as you reported, shows that.

Barman officially supports s3 only and is not possible for us to test Barman on all the "s3 compatible" services (that are a lot).

Said so, looking at the error you reported seems that something is possibly evaluated by Linode as not valid, so the first thing that comes to my mind is: is the URL to Linode a valid endpoint?

[alwinzwets]

Hi Giulio,

The endpoint is as per the Linode docs and this setup does work in other applications (like Cyberduck for example) so I think the problem is not in the configuration. If I change the endpoint and/or the key/secret the backup fails as expected with an AccessDenied so at least there is some communication with the Linode platform.

Thanks for pointing out the use of boto3. I'll try to raise this info again with Linode. Hopefully they are willing to test the boto3 <>Linode integration on their end, this indeed sounds like the main suspect.

Thanks, Alwin

[alwinzwets]

Response from Linode:

Evening Alwin,

We do support Boto3 with our S3 compatible storage, and some of our oldest public communications are four years old. A thought came to mind that, perhaps, our implemented version of Boto3 may differ from Barman's compatibility requirement. So, I'll be checking with our developers to ask if they could confirm the version we're utilizing, but also suggest you doublecheck your version and our documents.

Here's a link to our Boto3 documentation which was updated in April, and another page with a search for Boto3.

Using the AWS SDK for Python (boto3) with Object Storage | Linode Docs Linode Questions matching boto3 I'll be leaving this support ticket open while we investigate this a bit further, but at this immediate moment I can say that Boto3 is implemented and we're compatible.

Best regards,