search

EnterpriseDB / barman

Barman - Backup and Recovery Manager for PostgreSQL
https://www.pgbarman.org/
GNU General Public License v3.0
2.15k stars 193 forks source link

Backups to Cloudflare R2 (S3 Compatible) storage fail #954

Open andrewheberle opened 5 months ago

andrewheberle commented 5 months ago

I'm backing up to S3 compatible storage (Cloudflare R2) from Cloud Native PG (CNPG) which uses barman-cloud-backup.

Backups are failing with an error of:

ERROR: Upload error: An error occurred (InvalidPart) when calling the CompleteMultipartUpload operation: All non-trailing parts must have the same length. (worker 1)

The above error is surfaced from barman-cloud-backup when attempting to complete the weekly full backup.

$ barman-cloud-backup --version
barman-cloud-backup 3.10.1

I've come across what seems like a similar issue when using Cloudflare R2 as a backend for a Docker registry that was resolved by setting a fixed chunk size for multipart uploads, however I don't know enough about the barman codebase to know if this is the same issue.

https://community.cloudflare.com/t/all-non-trailing-parts-must-have-the-same-length/552190/7

And this discussion here:

https://github.com/distribution/distribution/pull/3940#issuecomment-1638356120

andrewheberle commented 5 months ago

Just an update here, I had "snappy" compression enabled for my base backups however once I changed the settings to use no compression the backups have succeeded so far.

Other backups of very small databases seem OK (ie < 100MB) but this one uncompressed is about 600MB

maxpain commented 1 month ago

the same problem on R2

martinmarques commented 4 weeks ago

Could this be related to this issue: https://github.com/EnterpriseDB/barman/issues/957

We support S3-compatible object stores, and though providers outside AWS claim 100% compatibility, as you can see in the case of Linode, it's not always fully compatible.

It would be good to check this for anything Cloudflare doesn't yet support.