search

uskudnik / amazon-glacier-cmd-interface

Command line interface for Amazon Glacier
MIT License
374 stars 100 forks source link

Upload resumption #79

Closed wvmarle closed 11 years ago

wvmarle commented 11 years ago

Implemented automatic upload resumption.

Now $ glacier-cmd upload <vault> <archive> --resume will check whether there is such an upload in progress (kept in the bookkeeping db) by checking file name and file size to get the uploadid. After that it works the same as if --uploadid <uploadid> were given.

Changed bookkeeping db to use the archiveid (or uploadid) as key, instead of the file name.

Inventory bookkeeping code is wrong; needs to be rewritten. Disabled here.

Added updatedb command for people upgrading from previous versions. This updates the db to use the new keys, and purges any items it can not recognise (i.e. no archiveid/uploadid key present).

wvmarle commented 11 years ago

OK so this pull request has become a little messy when it comes to patches as in the meantime upstream/master updated, so I updated this branch to prevent conflicts, messed around with local branches... sigh... lots of commits now...

Anyway fixed this mmap issue - tested with smallish files - by mapping only the relevant part of the file, instead of the complete file.

Please test with the big files, I currently don't have big uploads to test it on!

wvmarle commented 11 years ago

Now I'm trying to git rebase -i master this whole mess of commits to make it a single commit that's based on the CURRENT master branch, but other than resolving conflicts and doing several git rebase --continue calls it's not consolidating this mess but only making more of a mess out of it!

WTF?!

wvmarle commented 11 years ago

That should be it. A single commit for the whole shebang. And based off the current master so should apply to master without conflicts. I think. And hope.

Implemented automatic upload resumption. Implemented speed fix (see issue #71). Fixed up bookkeeping of uploads. Changed mmapping of files to allow for big files.

wvmarle commented 11 years ago

Wait merging. It seems other issues have appeared in this code thanks to the rebasing and whatnot happened. Gotta give it a good look.

wvmarle commented 11 years ago

As I thought this needed some minor fix to make it work. This one now good to commit.

wvmarle commented 11 years ago

Conflicts galore - all now included in parallel_uploads branch.