Backups to AWS S3 fail: readfrom tcp [..]: http: ContentLength=859699 with Body length 0

[stevenwhayes]

I'd like to try this out but I'm not sure how to invoke this. What is the SSL cert supposed to be? The utility will not start without one.

[stevenwhayes]

Ok I think I get the SSL piece now, that is PBS standard stuff

Maybe incorporate some of the discussion here into your README

https://forum.proxmox.com/threads/s3-backup-support.152901/

[stevenwhayes]

here's the AWS IAM policy that worked for me if you want to add it in to the README:

{ "Version": "2012-10-17", "Statement": [ { "Sid": "AllowSpecificBucketAccess", "Effect": "Allow", "Action": [ "s3:ListBucket", "s3:GetObject", "s3:PutObject" ], "Resource": [ "arn:aws:s3:::yourbucketname", "arn:aws:s3:::yourbucketname/*" ] }, { "Sid": "AllowListAllMyBuckets", "Effect": "Allow", "Action": "s3:ListAllMyBuckets", "Resource": "*" } ] }

[stevenwhayes]

I am hitting a "real" error now, none of the backups will complete. They all fail with something like this

ERROR: backup write data failed: command error: write_data upload error: pipelined request failed: Put "http://mybucket.s3.dualstack.us-east-2.amazonaws.com/chunks/6b/cb/52c4d184c27f6e999bb37e22c032adc7d65eb85769a5beac2f8b963d60c9": readfrom tcp :37040->3.5.131.60:80: http: ContentLength=2167529 with Body length 0{"data":{"digest":"6bcb52c4d184c27f6e999bb37e22c032adc7d65eb85769a5beac2f8b963d60c9","offset":0,"size":4194304,"chunk_is_known":false}}

[abbbi]

can you provide a complete log of the proxy, with debug info? Which Proxmox version are you using this with? Personally i have not verified this proxy still works with Proxmox 8.3

But judging from the error message, there seems to be a general problem for the proxy not beeing able to upload data to the bucket for some reason.

[stevenwhayes]

I'm on proxmox pve 8.2.2. I tried a couple more things, one using IAM credentials with full admin on the account to rule anything out there. It still errored. I tried a different VM, and interestingly enough it failed the first time, but then completed the second time. I have not tried all my VM's yet, do you know if there are any limits to disk size or things like that? I've attached the debug log, it only contains the fail not the success I was not capturing the log on that one. pmoxs3proxy-debug.zip

I'd really like to get this tool working, it is a very good idea and fills a very needed gap with proxmox, thank you for writing it.

[abbbi]

hi,

there is really not much to see in the Log. It fails while writing an non-existant junk to the bucket. During upload it is verified the object does not already exist and if not, upload is attempted. It fails here:

https://github.com/tizbac/pmoxs3backuproxy/blob/master/cmd/pmoxs3backuproxy/main.go#L942

it seems to have successfully written 14717 chunks before, so the general functionality seems to be given. It also detects some chunks already existant. There are no limits afaik, the chunks are usually pretty small files.

Unfortunately the error output does not give any detail upon why the write operation failed. You could enable logging on the AWS bucket and check why it fails to write sometimes. Maybe some Quotas beeing hit or an issue with the uplink connection? The error message:

readfrom tcp xx.xx.xx.10:39972->3.5.128.159:80: http: ContentLength=859699 with Body length 0

gives me the impression it might be some underlying networking issue that makes the write operation fail. Is your uplink saturated during backup? As the problem seems to appear sporadically, too, i have the gut feeling this might be an issue with the uplink.

You could try to setup an minio instance locally and see if you experience the same troubles while backing up to a local s3 instance?

[abbbi]

ive pushed the following commit to branch errcode, which i think should then give more detail on why the upload failed:

https://github.com/tizbac/pmoxs3backuproxy/commit/cdd17144e5270f8e496c0d36dea9bc8ae7cb45dd

beware its untested. Could you try with the version from this branch and see if it returns an detailed error message in cases it fails to upload the chunks? It may help to find out what exactly fails.

[stevenwhayes]

I gave that version a try, the backup made more progress than earlier today but ultimately error'ed out. I've attached the log. I'm sure it would work with minio locally, unfortunately that does not help my use case. I've tried to backup proxmox direct to S3 with FUSE as well and it has similar but worse issues. If I can get a baseline backup to complete, seems like there is better chance the incrementals will then complete, but I fear this just may not be possible without being on a very fast connection to S3 or maybe even from inside AWS. If the underlying S3 protocols allow retry / retransmit maybe that could help? pmoxs3backuproxy-testing.zip

[abbbi]

not much to see from the advanced log output.. I still think your uplink is at fault here. Of course if a baseline backup makes it through, incrementals are more likely to work because less data is transferred. You could try to see if loggin on the S3 bucket logs some more information.

[stevenwhayes]

I have good Internet, gigabit up and down but it is residential. One thing I'm noticing is the fail messages are always reporting an HTTP connection. HTTP tends to be subject to more aggressive traffic shaping, especially on residential networks. If the connection can be forced to HTTPS, that may make a difference. I understand if you don't want to spend more time on this though, I'm not much of a go coder but I can try to figure that out.

[abbbi]

the proxy supports ssl:

  -usessl
        Enable SSL connection to the endpoint, for use with cloud S3 providers