Setting signature expiration

[jqdoumen]

Since about two weeks, a large portion of my uploads fail because of what seems to be a signature expiration. Getting errors like :

boto.glacier.exceptions.UnexpectedHTTPResponseError: Expected 204, got (403, code=InvalidSignatureException, message=Signature expired: 20121108T030749Z is now earlier than 20121108T031304Z (20121108T031804Z - 5 min.))

I suspect hick-ups in network connectivity (plausible?). I've also noted that what could be uploaded before in one night, now takes double time.

I've been playing around with the partsize param hoping uploads with smaller parts might be less susceptible to network interruptions, but that does not seem to help.

Is the signature validity period configurable? Seems like it is 5 min. Can this be increased?

The full error message actually is :

File "/usr/local/bin/glacier-cmd", line 8, in load_entry_point('glacier==0.2dev', 'console_scripts', 'glacier-cmd')() File "/Library/Python/2.6/site-packages/glacier-0.2dev-py2.6.egg/glacier/glacier.py", line 751, in main args.func(args) File "/Library/Python/2.6/site-packages/glacier-0.2dev-py2.6.egg/glacier/glacier.py", line 147, in wrapper return fn(_args, _kwargs) File "/Library/Python/2.6/site-packages/glacier-0.2dev-py2.6.egg/glacier/glacier.py", line 300, in upload args.name, args.partsize, args.uploadid, args.resume) File "/Library/Python/2.6/site-packages/glacier-0.2dev-py2.6.egg/glacier/GlacierWrapper.py", line 63, in wrapper ret = fn(_args, _kwargs) File "/Library/Python/2.6/site-packages/glacier-0.2dev-py2.6.egg/glacier/GlacierWrapper.py", line 205, in glacier_connect_wrap return func(_args, _kwargs) File "/Library/Python/2.6/site-packages/glacier-0.2dev-py2.6.egg/glacier/GlacierWrapper.py", line 63, in wrapper ret = fn(_args, _kwargs) File "/Library/Python/2.6/site-packages/glacier-0.2dev-py2.6.egg/glacier/GlacierWrapper.py", line 258, in sdb_connect_wrap return func(_args, _kwargs) File "/Library/Python/2.6/site-packages/glacier-0.2dev-py2.6.egg/glacier/GlacierWrapper.py", line 63, in wrapper ret = fn(_args, _kwargs) File "/Library/Python/2.6/site-packages/glacier-0.2dev-py2.6.egg/glacier/GlacierWrapper.py", line 1092, in upload writer.write(part) File "/Library/Python/2.6/site-packages/glacier-0.2dev-py2.6.egg/glacier/glaciercorecalls.py", line 129, in write data) File "/Library/Python/2.6/site-packages/boto/glacier/layer1.py", line 625, in upload_part response_headers=response_headers) File "/Library/Python/2.6/site-packages/boto/glacier/layer1.py", line 83, in make_request raise UnexpectedHTTPResponseError(ok_responses, response)

[wvmarle]

Did you check your computer's clock?

[jqdoumen]

I have ntp enabled, so the clock should be right. I've just double checked with a clock on the internet, and seems to be correct up till the second.

[jqdoumen]

I've installed the latest version of glacier-cmd, rebooted my machine ... Since four days now every single upload I'm trying results in the above error.

[wvmarle]

Are you using HEAD or another branch?

[wvmarle]

And it does upload part of your archive, right? First number of parts go fine, after which it crashes out?

[jqdoumen]

Yes, it uploads over half of the archive. Actually, the speed seems to be pretty constant, and then all of a sudden I get the error.

On Fri, Nov 9, 2012 at 7:35 AM, wvmarle notifications@github.com wrote:

And it does upload part of your archive, right? First number of parts go fine, after which it crashes out?

— Reply to this email directly or view it on GitHubhttps://github.com/uskudnik/amazon-glacier-cmd-interface/issues/103#issuecomment-10216830.

Jan Doumen

This slow-burning country has more history, art, food and architecture packed into its tiny self than many of its bigger, louder neighbors. A rich and bubbling vat of beer, chocolate, oil paint and bureaucrats.

The Lonely Planet about Belgium

Follow me: [image: Facebook] http://www.facebook.com/jan.doumen [image: Twitter] http://www.twitter.com/DeJeanDOumen [image: LinkedIn]http://be.linkedin.com/in/jandoumen [image: Blogger] http://dejean.blogspot.com/ [image: devianArt]http://dejean.deviantart.com/ [image: Google Plus] https://plus.google.com/114059910845203446151 [image: Tumblr]http://dejean.tumblr.com Contact me: [image: Google Talk] jan.doumen [image: Skype] jan.doumen

[jqdoumen]

I used the version I found here : https://github.com/uskudnik/amazon-glacier-cmd-interface

Is there an easy way to know to find out what branch or version you are actually using?

On Fri, Nov 9, 2012 at 7:32 AM, wvmarle notifications@github.com wrote:

Are you using HEAD or another branch?

— Reply to this email directly or view it on GitHubhttps://github.com/uskudnik/amazon-glacier-cmd-interface/issues/103#issuecomment-10216802.

Jan Doumen

This slow-burning country has more history, art, food and architecture packed into its tiny self than many of its bigger, louder neighbors. A rich and bubbling vat of beer, chocolate, oil paint and bureaucrats.

The Lonely Planet about Belgium

Follow me: [image: Facebook] http://www.facebook.com/jan.doumen [image: Twitter] http://www.twitter.com/DeJeanDOumen [image: LinkedIn]http://be.linkedin.com/in/jandoumen [image: Blogger] http://dejean.blogspot.com/ [image: devianArt]http://dejean.deviantart.com/ [image: Google Plus] https://plus.google.com/114059910845203446151 [image: Tumblr]http://dejean.tumblr.com Contact me: [image: Google Talk] jan.doumen [image: Skype] jan.doumen

[jqdoumen]

I'm now also uploading using another tool, to see whether it is glacier-cmd or something else that is the issue, as I'm more and more assuming it has nothing to do with Amazon settings or the internet connection.

[wvmarle]

That's the master branch aka HEAD. Tonight I'll try to study the signature issues and so more.

It's interesting that you see this happen, and looking at the error message the signature is expired for minutes, not mere seconds what I first expected.

This is not an error from glacier-cmd, this is from boto (the library we use to do the direct communication with Amazon Glacier). So the issue is either a bug in boto, or an incorrect way of communicating to boto. At the moment I don't know how those signatures work, and how/when boto creates them, so I can't say more about it.

[jqdoumen]

OK, thanks !

BTW, yesterday evening, while googling for the error message, I saw that another tool available through git, had had the same issue. If I remember well, they found that the error they captured was actually something else, something with multi-part upload or so. I'll try to find that post back.

On Fri, Nov 9, 2012 at 7:47 AM, wvmarle notifications@github.com wrote:

That's the master branch aka HEAD. Tonight I'll try to study the signature issues and so more.

It's interesting that you see this happen, and looking at the error message the signature is expired for minutes, not mere seconds what I first expected.

This is not an error from glacier-cmd, this is from boto (the library we use to do the direct communication with Amazon Glacier). So the issue is either a bug in boto, or an incorrect way of communicating to boto. At the moment I don't know how those signatures work, and how/when boto creates them, so I can't say more about it.

— Reply to this email directly or view it on GitHubhttps://github.com/uskudnik/amazon-glacier-cmd-interface/issues/103#issuecomment-10217019.

Jan Doumen

This slow-burning country has more history, art, food and architecture packed into its tiny self than many of its bigger, louder neighbors. A rich and bubbling vat of beer, chocolate, oil paint and bureaucrats.

The Lonely Planet about Belgium

Follow me: [image: Facebook] http://www.facebook.com/jan.doumen [image: Twitter] http://www.twitter.com/DeJeanDOumen [image: LinkedIn]http://be.linkedin.com/in/jandoumen [image: Blogger] http://dejean.blogspot.com/ [image: devianArt]http://dejean.deviantart.com/ [image: Google Plus] https://plus.google.com/114059910845203446151 [image: Tumblr]http://dejean.tumblr.com Contact me: [image: Google Talk] jan.doumen [image: Skype] jan.doumen

[jqdoumen]

If this extra info is of any help ... I uploaded for two months every night without any issue. On Nov 1st I downloaded the latest version of glacier I found on the link below, and I started having the issue from time to time (1 in 5 or so). With the version I downloaded the other day, there isn't a single one that still is working.

On Fri, Nov 9, 2012 at 8:09 AM, Jan DOUMEN jan.doumen@gmail.com wrote:

OK, thanks !

BTW, yesterday evening, while googling for the error message, I saw that another tool available through git, had had the same issue. If I remember well, they found that the error they captured was actually something else, something with multi-part upload or so. I'll try to find that post back.

On Fri, Nov 9, 2012 at 7:47 AM, wvmarle notifications@github.com wrote:

That's the master branch aka HEAD. Tonight I'll try to study the signature issues and so more.

It's interesting that you see this happen, and looking at the error message the signature is expired for minutes, not mere seconds what I first expected.

This is not an error from glacier-cmd, this is from boto (the library we use to do the direct communication with Amazon Glacier). So the issue is either a bug in boto, or an incorrect way of communicating to boto. At the moment I don't know how those signatures work, and how/when boto creates them, so I can't say more about it.

— Reply to this email directly or view it on GitHubhttps://github.com/uskudnik/amazon-glacier-cmd-interface/issues/103#issuecomment-10217019.

Jan Doumen

This slow-burning country has more history, art, food and architecture packed into its tiny self than many of its bigger, louder neighbors. A rich and bubbling vat of beer, chocolate, oil paint and bureaucrats.

The Lonely Planet about Belgium

Follow me: [image: Facebook] http://www.facebook.com/jan.doumen [image: Twitter] http://www.twitter.com/DeJeanDOumen [image: LinkedIn]http://be.linkedin.com/in/jandoumen [image: Blogger] http://dejean.blogspot.com/ [image: devianArt]http://dejean.deviantart.com/ [image: Google Plus] https://plus.google.com/114059910845203446151 [image: Tumblr] http://dejean.tumblr.com Contact me: [image: Google Talk] jan.doumen [image: Skype] jan.doumen

Jan Doumen

This slow-burning country has more history, art, food and architecture packed into its tiny self than many of its bigger, louder neighbors. A rich and bubbling vat of beer, chocolate, oil paint and bureaucrats.

The Lonely Planet about Belgium

Follow me: [image: Facebook] http://www.facebook.com/jan.doumen [image: Twitter] http://www.twitter.com/DeJeanDOumen [image: LinkedIn]http://be.linkedin.com/in/jandoumen [image: Blogger] http://dejean.blogspot.com/ [image: devianArt]http://dejean.deviantart.com/ [image: Google Plus] https://plus.google.com/114059910845203446151 [image: Tumblr]http://dejean.tumblr.com Contact me: [image: Google Talk] jan.doumen [image: Skype] jan.doumen

[jqdoumen]

https://github.com/MoriTanosuke/glacieruploader/issues/14

[SitronNO]

@jqdoumen:

Is there an easy way to know to find out what branch or version you are actually using?

Yes, use the commando git branch -a

[jqdoumen]

With my own scripts I do manage to upload files. There are some pickups from time to time, but they are managed :

2012/11/09 09:12:37 bytes 645922816-654311423/ 2012/11/09 09:12:37 Part uploaded, checksum: 05ba9e82f5c744ca97eb81ed1e60dcaa649a23a7556f505a560d37a91e6df444 2012/11/09 09:13:52 bytes 654311424-662700031/ Nov 9, 2012 9:14:03 AM org.apache.http.impl.client.DefaultRequestDirector tryExecute INFO: I/O exception (java.net.SocketException) caught when processing request: Broken pipe Nov 9, 2012 9:14:03 AM org.apache.http.impl.client.DefaultRequestDirector tryExecute INFO: Retrying request 2012/11/09 09:13:52 Part uploaded, checksum: 4e2ea2d512d88e91f9af598d45e29f4ecbecf90ee2a91f2ed0291606ae007fb0 2012/11/09 09:15:15 bytes 662700032-671088639/*

There seems to be some time difference, but of the order of seconds ...

[jqdoumen]

It seems to be getting more weird. While waiting for a solution, I wanted to update my reconciliation logs, so I requested the inventory of two vaults. Nothing seems to happen. So I tried to use the search functionality, nothing seems to happen. I type 'glacier-cmd search' ... wait for a few seconds ... and then I get back to the prompt, without any output. No error message either.

I wanted to analyze further to check if the abort does not always happen on the same moment, either in time or in bytes. The load I started last night now was halted with another error message:

iMac:Attachments DeJean$ glacier-cmd upload --partsize 8 Photos_Scans35mm /Volumes/Photos/Photos/20110319.tar.gz 20110319.tar.gz Wrote 1.0 GB. Rate 105.06 KB/s.Rate 425.27 KB/s, average 105.10 KB/s, eta 02:16:38.
/Library/Python/2.6/site-packages/glacier-0.2dev-py2.6.egg/glacier/glacierexception.py:80: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6 self.message = message /Library/Python/2.6/site-packages/glacier-0.2dev-py2.6.egg/glacier/glacierexception.py:111: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6 exc = self if self.message is None else self.message str: File name given for upload can not be found: 20110319.tar.gz.

Which is, well, strange. Why does he tell me he can't find the file after he already uploaded 1 GB of it.

To make sure nothing is wrong locally on my NAS, I've copied that file to my local disk, and relaunched the upload. Will tell you the result in a few hours.

But, through the AWS console I also noticed that the number of files in the vault has been increasing, even though the uploads seem to be failing. I'll try to get the inventory through another script, to check if the new files were uploaded after all, or if they were partially uploaded (I would expect that latter not being possible).

In the log above he says he wrote 1.0 GB ... well, the file IS 1 GB, more precisely 1092663554 bytes. Could it be that something is going wrong with only the last part, and in particular with the closing of the upload?

[jqdoumen]

With the file first copied local, he uploads till 100% and then ... starts all over again ...

iMac:~ DeJean$ glacier-cmd upload --partsize 8 Photos_Scans35mm 20110319.tar.gz 20110319.tar.gz Wrote 1.0 GB. Rate 108.87 KB/s.Rate 423.27 KB/s, average 108.91 KB/s, eta 11:45:08.
Wrote 8.0 MB of 1.0 GB (0%). Rate 113.01 KB/s, average 113.01 KB/s, eta 14:22:35.

[jqdoumen]

After a closer analysis from my side, I think there are actually three issues. I'll leave this one for signature validity. I'll open separate issues for the continuous loop and for the 'filename not found'.