Backblaze B2: The XML you provided was not well-formed or did not validate against our published schema

[cbenard]

I opened an issue with offen/docker-volume-backup about a sudden error I'm receiving nightly with no application or configuration changes, using Backblaze B2.

He's using minio-go and asked me to open an issue here. His application does not enable the XML writer tracing, so I cannot see what XML is being sent. This started on January 5 and has happened 3 nights in a row, after working for a year prior without changes.

He suspects an error/change with B2, but I don't see anything on their Blog to announce any changes. Are you able to add any input/support?

The error from B2 is:

The XML you provided was not well-formed or did not validate against our published schema

Log output of the failed run was:

time=2024-01-07T09:00:10.996Z level=INFO msg="Stopping 1 container(s) labeled `docker-volume-backup.stop-during-backup=true` out of 5 running container(s)."
time=2024-01-07T09:01:01.876Z level=INFO msg="Created backup of `/backup` at `/tmp/backup-2024-01-07T09-00-08.tar.gz`."
time=2024-01-07T09:01:16.346Z level=INFO msg="Restarted 1 container(s) and the matching service(s)."
time=2024-01-07T09:01:20.113Z level=INFO msg="Encrypted backup using given passphrase, saving as `/tmp/backup-2024-01-07T09-00-08.tar.gz.gpg`."
time=2024-01-07T09:01:27.181Z level=INFO msg="Uploaded a copy of backup `/tmp/backup-2024-01-07T09-00-08.tar.gz.gpg` to bucket `backup-cbenard`." storage=S3
time=2024-01-07T09:01:27.365Z level=ERROR msg="Fatal error running backup: pruneBackups: error pruning backups: The XML you provided was not well-formed or did not validate against our published schema"
time=2024-01-07T09:01:27.368Z level=INFO msg="Removed tar file `/tmp/backup-2024-01-07T09-00-08.tar.gz`."
time=2024-01-07T09:01:27.376Z level=INFO msg="Removed GPG file `/tmp/backup-2024-01-07T09-00-08.tar.gz.gpg`."

I'm not sure if the error is in the ListObjects or the RemoveObjects call. In the linked issue above, I repro'd using s3cmd to ListObjects and RemoveObjects without issue.

# s3cmd ls --recursive "s3://my-bucket-name/sub/path/backup-*"`
2023-11-07 09:01     36380038  s3://my-bucket-name/sub/path/backup-2023-11-07T09-00-05.tar.gz.gpg
2023-11-08 09:00     36385764  s3://my-bucket-name/sub/path/backup-2023-11-08T09-00-05.tar.gz.gpg
... and so on

# s3cmd rm s3://my-bucket-name/sub/path/backup-2023-11-07T09-00-05.tar.gz.gpg s3://my-bucket-name/sub/path/backup-2023-11-08T09-00-05.tar.gz.gpg
delete: 's3://my-bucket-name/sub/path/backup-2023-11-07T09-00-05.tar.gz.gpg'
delete: 's3://my-bucket-name/sub/path/backup-2023-11-08T09-00-05.tar.gz.gpg'

Thank you.

[m90]

It's probably relevant to note this error is returned from a method that calls ListObjects and RemoveObjects APIs.

[harshavardhana]

Have you tested the tool with AWS S3 and MinIO ?

[m90]

S3 and MinIO still work as B2 previously did, yes (using these myself, not using B2 though).

[cbenard]

The tool creates tarballs and uploads them and prunes the older ones older than a threshold.

I don't have a way to repro with either of the ones listed. I don't have an account with AWS, nor would it have the accumulated files.

I don't have a way to run storage for MinIO currently which is why I am using offsite storage.

I don't know of a way to find out if B2 changed something, but I also cannot see the XML that @m90's tool is sending in the POST.

[klauspost]

ListObjects doesn't send XML, so I'd guess the problem is with their implementation of RemoveObjects.

Since you say it happens "all of a sudden", my guess is it is a bug introduced by Backblaze. I suggest you reach out to them.

If you find more info and it is a bug on our side feel free to open a new issue - but if it works with AWS there isn't really more we can do here.