s3 put failing with 400 with >5GB file

[ericjs]

I've been using this for quite a while successfully. (Thanks for the great tool!) However I've been running into a put that keeps failing with:

400 Bad Request failed to upload partNumber=1

The file is greater then 5GB (the first such file I've tried to upload). The command looks like

aws put --md5 "--max-time=100000" "wrycan-shorterm-backup/gex-qa/xms-repo-base-454845-d.svnbackup.gz" "xms-repo-base-454845-d.svnbackup.gz"

(I kept boosting max-time at first thinking it was timing out. But the error actually occurs after about 2 minutes.)

I have the output from running it with -vv but I'm not sure if I should paste it here, wrt the size of it, or if anything sensitive is logged. But the line that looks notable to me is

• ALPN, server did not agree to a protocol

[ericjs]

One additional piece of information. By using higher compression, I was able to get my file down to 4.4GB and do a successful put. The put took a little over a minute. As I mentioned the put with the 5GB file fails after about 2 minutes (not exactly 2, a little over last time I checked the timing). Maybe that is a clue.

[timkay]

Eric,

Thanks for the feedback. It's been 6 years since I made any substantive changes to "aws" and 3 years since the ticket before yours. I'm thrilled and surprised that there is still interest in this project. (I do still use it myself.)

To address your report, I confirmed there is a failure with multi-part upload, which is a feature needed to upload files bigger than 5GB. Unfortunately, the problem has to do with Amazon changing their API yet again, involving how authorization credentials are provided. I suppose it's reasonable for API changes over a long period of time. Unfortunately, Amazon was never willing to help me support this project, even back in the beginning, when it was the only command-line tool available, and the top-rated community contributed project. The new feature velocity of AWS, their API changes, and their unwillingness to support this effort makes it difficult for me to keep it working.

I am willing to continue to support the project if it doesn't require substantial code changes, but I'm afraid fixing this one is to involved.

...Tim

[ericjs]

Hi Tim, Well that's understandable. Thanks for the candid reply, some projects seem to just stop responding when they hit that phase. I suppose long term I should start looking for a replacement but for now I'll just do something to break my data into smaller files and probably use this until amazon breaks it more seriously. Any opinions on what's a good replacement cli tool?

[timkay]

Eric,

Yeah, I'm disappointed too. I put a lot of energy into it, supporting many features I don't even use, based on requests from the commuunity. Amazon was a hindrance, not a help. I even started getting a monthly charge of hundreds of dollars because of the sanity check. The check pulls an S3 file connection/test, which happened every time somebody ran "aws", except for users who turned off sanity checks. I owned that bucket and was getting charged. I made the bucket private; sanity check still worked because the header of the 404 page still had the Date: header. However, I still kept getting charged. Apparently, if somebody pulls an object from your private bucket (even if the object doesn't exist), AWS charges you. I contacted AWS support and asked them to change ownership to themselves. They said it wasn't technically possible. I told them that I could simply delete the bucket, but then "aws" would start failing for 10's of thousands of their customers. They still said it wasn't technically possible. I reached out one last time, and they did that change.

For now, I use both AWS and GCP, with a preference for GCP. I am going to need to find a replacement too, preferably one that works on both. I know there are good libraries these days, certainly for Python. If I don't find a good command line tool, maybe I'll write a new command line tool that uses one of the libraries. I could keep it compatible possibly.

Let me know if you find something.

...Tim

On Fri, May 24, 2019 at 9:20 AM Eric notifications@github.com wrote:

Hi Tim, Well that's understandable. Thanks for the candid reply, some projects seem to just stop responding when they hit that phase. I suppose long term I should start looking for a replacement but for now I'll just do something to break my data into smaller files and probably use this until amazon breaks it more seriously. Any opinions on what's a good replacement cli tool?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/timkay/aws/issues/114?email_source=notifications&email_token=AAAP46DS76XSXXQYPPEENS3PXAIWRA5CNFSM4HOYVAC2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODWF3PUY#issuecomment-495695827, or mute the thread https://github.com/notifications/unsubscribe-auth/AAAP46HS2262TQCHZ6PKBDDPXAIWRANCNFSM4HOYVACQ .

[ericjs]

Wow that sucks you were getting charged like that, and that they tried to claim they couldn't change it...talk about good deeds not going unpunished! I feel like Amazon obfuscates their services to make it hard to figure out what anything is going to cost you. We've switched to Digital Ocean for most of our servers, but we haven't found a good inexpensive equivalent of S3 (though we haven't looked very recently).

No software is forever. You should feel proud that you've made something that served so many people so well for many years.