search

uskudnik / amazon-glacier-cmd-interface

Command line interface for Amazon Glacier
MIT License
374 stars 100 forks source link

Mmap #77

Closed wvmarle closed 11 years ago

wvmarle commented 11 years ago

Mapping file with mmap to save on memory use. Should allow blocks larger than available RAM.

Note: I have tested the uploads and resumption, both from file and from stdin, and it works for me. I have not done any memory footprint checks yet.

Sending file over stdin still has memory issues; there seems to be a way around this but it's not easy as you have to address the httpconnection directly. And the key issue here is how to get this connection, as authentication is needed.

See also: http://stackoverflow.com/questions/12871362/sending-content-of-large-file-in-parts-over-http- in-python

And there is this issue of unknown byte size when piping over stdin, and the need of giving a hash of the complete part in advance.

wvmarle commented 11 years ago

Oh, my. So many patches.

And I thought creating a new branch allows me to do just a single patch in the pull request - it's ca167cc that's important.

Obviously been thinking wrong :-)

uskudnik commented 11 years ago

Since you commented about the number of patches - are you done with changes now and I can merge or do I wait?

wvmarle commented 11 years ago

Basically only the last one is a new patch; the rest is identical to what was in my other pull request from my master branch. What I didn't realise is that this new branch would list all those patches again (it actually gives all patches against upstream/master which was far behind my master).

Working on other things now (upload/download resumption) - that's going to be different pull requests.