jqdoumen
commented
11 years ago I'm having exactly the same issue. For last night's upload of 3 files, 2 failed with this error.
Open echamberlain opened 11 years ago
jqdoumen
commented
11 years ago I'm having exactly the same issue. For last night's upload of 3 files, 2 failed with this error.
ctiwald
commented
11 years ago Having the exact same issue with a 13GB upload.
offlinehacker
commented
11 years ago In progress of fixing ;) On Nov 12, 2012 3:24 PM, "Christopher Tiwald" notifications@github.com wrote:
Having the exact same issue with a 13GB upload.
— Reply to this email directly or view it on GitHubhttps://github.com/uskudnik/amazon-glacier-cmd-interface/issues/95#issuecomment-10289329.
SitronNO
commented
11 years ago I had the same problem with several files. However, if I changed the partsize, it worked. Can some of you try to change the partsize, and see if the file will upload without the 408-error?
vsespb
commented
11 years ago IMHO It's Amazon problem https://forums.aws.amazon.com/thread.jspa?messageID=399111 I experience it with this my client https://github.com/vsespb/mt-aws-glacier when do high concurrency uploads.
vsespb
commented
11 years ago Seems changing tcp_congestion_control from cubic to westwood helps a lot in my case ( I use ADSL) . Note that different Linux kernels have different default tcp_congestion_control. Also different mode should be used for different internet connection technologies.
theRemix
commented
11 years ago Confirmed multiple attempts failed uploading a 6.6G file using $glacier-cmd upload --partsize 8 my_vault my_large_file.tar.bz2 --partsize 32 also did not work, and same results without the --partsize param
... File "/usr/lib64/python2.6/site-packages/glacier-0.2dev-py2.6.egg/glacier/glaciercorecalls.py", line 129, in write data) File "/usr/lib64/python2.6/site-packages/boto-2.8.0-py2.6.egg/boto/glacier/layer1.py", line 637, in upload_part response_headers=response_headers) File "/usr/lib64/python2.6/site-packages/boto-2.8.0-py2.6.egg/boto/glacier/layer1.py", line 84, in make_request raise UnexpectedHTTPResponseError(ok_responses, response) boto.glacier.exceptions.UnexpectedHTTPResponseError: Expected 204, got (408, code=RequestTimeoutException, message=Request timed out.)
does it work with smaller filesizes? i'm going to split it up and test.
theRemix
commented
11 years ago i got something different, socket.error Connection reset by peer
i split up my large files into 100M chunks and uploaded
$glacier-cmd upload --partsize 32 my_vault *
... File "/usr/lib64/python2.6/site-packages/boto-2.8.0-py2.6.egg/boto/glacier/layer1.py", line 79, in make_request data=data) File "/usr/lib64/python2.6/site-packages/boto-2.8.0-py2.6.egg/boto/connection.py", line 932, in make_request return self._mexe(http_request, sender, override_num_retries) File "/usr/lib64/python2.6/site-packages/boto-2.8.0-py2.6.egg/boto/connection.py", line 894, in _mexe raise e socket.error: [Errno 104] Connection reset by peer
i also tested with --partsize 8 and got this same error boto.glacier.exceptions.UnexpectedHTTPResponseError: Expected 204, got (408, code=RequestTimeoutException, message=Request timed out.)
adammulligan
commented
11 years ago FYI: I was getting this problem with pretty much every upload great than 100mb or so, but cloning the latest uskudnik/amazon-glacier-cmd-interface (0b88b042b3) and reinstalling seems to have fixed it entirely and I no longer get 408s. I still think there's an issue on Amazon's end since it's happening with multiple clients, but hopefully this will help some people.
uskudnik
commented
11 years ago Anyone else fixed this problem with https://github.com/uskudnik/amazon-glacier-cmd-interface/commit/0b88b042b3 (although I suspect issue was fixed a couple of commits earlier or Amazon fixed it on its end)?
kmand
commented
11 years ago I just started using this and downloaded the master.zip which is presumably a2d5763f67. I'm using it with bacula, which generates large files to upload. So far I have had a 10GB upload fail, a 50GB succeed and a 38GB fail. The failures give the error message of this thread. The uploads are going at about 1.5GB/hour through my cable modem, so the successful 50 GB took over 30 hours. The 10GB and 38GB failed about half way through.
Any suggestions?
SitronNO
commented
11 years ago I do not understand how a calculation of the clients ETA is suppose to fix the 408-bug? https://github.com/uskudnik/amazon-glacier-cmd-interface/commit/0b88b042b3 - Or am I looking at the wrong code/commit?
And just to make matters worse, I use the Parallel uploads-branch from wvmarle, which does not contain the code.
SitronNO
commented
11 years ago I just tested the latest code (git://github.com/uskudnik/amazon-glacier-cmd-interface.git) on a new client, and I did not get the 408-error as I did with the parallel-uploads-code. I tried over 11 files, the biggest was 37GB. Total of 67GB.
The-Judge
commented
11 years ago I cloned the repo and installed it from a2d5763 . I created a fresh vault and tried to upload a testfile of 785MB to it (I choosed Ubuntu ISO image for it). I'm using this command to do so:
glacier-cmd upload --description "Test Glacier handling with Ubuntu ISO Image" testvault ubuntu-13.04-desktop-amd64.isoThe upload starts and glacier-cmd displays it's progress bar. It uploads fine at first, but at any time this upload crashes with the described error. It already crashed at 30MB, 90MB, 120MB, ... not really foreseeable.
ken107
commented
10 years ago Looks like this problem is with the Amazon API. I'm using the Java API and had to write my own uploader to circumvent this problem. My uploader first downloads the entire list of parts, then it only uploads parts that are missing or has mismatched checksum. This allows to resume the upload from where it got interrupted. EDIT: I found the reason is the SDK doesn't retry on a 408 by default. This can be fixed by using a custom ClientConfiguration.RetryPolicy when constructing the AmazonGlacierClient.
simonbaudry
commented
10 years ago Same issue for me with a file of 1.1 GB !
I solved it by setting partsize parameter to 32.
mraheel83
commented
9 years ago Yesterday I setup glacier-cmd from current source code and I'm still getting this error when I'm uploading 140GB file.
I'm getting this error after 10GB upload out of 140GB. Please advise me how can I fix it.
Error is:
File "/usr/bin/glacier-cmd", line 9, in
Thanks
gburca
commented
9 years ago Does the gburca@85ef4aa tip/branch fix this issue for anyone? See also https://github.com/uskudnik/amazon-glacier-cmd-interface/issues/171#issuecomment-101495322.
SitronNO
commented
9 years ago In the last days I have tried to upload 248 different files, all ranging from 13Mb to 37Gb (total of 450GB). After three days of 408 errors every 1-6 hours, I tried the code from @gburca (85ef4aa6dd58a36a77e4406366ddc27c7813cf5e) It has now uploaded 250GB the last 24 hours without a single error. Will try some more, but this is promising!
My smaller uploads are successful, but this one keeps failing with the following error, any ideas?
Traceback (most recent call last): 295.36 KB/s, average 385.03 KB/s, eta 19:06:42.
load_entry_point('glacier==0.2dev', 'console_scripts', 'glacier-cmd')()
File "/Library/Python/2.7/site-packages/glacier-0.2dev-py2.7.egg/glacier/glacier.py", line 751, in main
args.func(args)
File "/Library/Python/2.7/site-packages/glacier-0.2dev-py2.7.egg/glacier/glacier.py", line 147, in wrapper
return fn(_args, _kwargs)
File "/Library/Python/2.7/site-packages/glacier-0.2dev-py2.7.egg/glacier/glacier.py", line 300, in upload
args.name, args.partsize, args.uploadid, args.resume)
File "/Library/Python/2.7/site-packages/glacier-0.2dev-py2.7.egg/glacier/GlacierWrapper.py", line 62, in wrapper
ret = fn(_args, _kwargs)
File "/Library/Python/2.7/site-packages/glacier-0.2dev-py2.7.egg/glacier/GlacierWrapper.py", line 204, in glacier_connect_wrap
return func(_args, _kwargs)
File "/Library/Python/2.7/site-packages/glacier-0.2dev-py2.7.egg/glacier/GlacierWrapper.py", line 62, in wrapper
ret = fn(_args, _kwargs)
File "/Library/Python/2.7/site-packages/glacier-0.2dev-py2.7.egg/glacier/GlacierWrapper.py", line 257, in sdb_connect_wrap
return func(_args, _kwargs)
File "/Library/Python/2.7/site-packages/glacier-0.2dev-py2.7.egg/glacier/GlacierWrapper.py", line 62, in wrapper
ret = fn(_args, _kwargs)
File "/Library/Python/2.7/site-packages/glacier-0.2dev-py2.7.egg/glacier/GlacierWrapper.py", line 1086, in upload
writer.write(part)
File "/Library/Python/2.7/site-packages/glacier-0.2dev-py2.7.egg/glacier/glaciercorecalls.py", line 129, in write
data)
File "/Library/Python/2.7/site-packages/boto/glacier/layer1.py", line 625, in upload_part
response_headers=response_headers)
File "/Library/Python/2.7/site-packages/boto/glacier/layer1.py", line 83, in make_request
raise UnexpectedHTTPResponseError(ok_responses, response)
boto.glacier.exceptions.UnexpectedHTTPResponseError: Expected 204, got (408, code=RequestTimeoutException, message=Request timed out.)
File "/usr/local/bin/glacier-cmd", line 8, in