search

uskudnik / amazon-glacier-cmd-interface

Command line interface for Amazon Glacier
MIT License
374 stars 100 forks source link

Mmapping file - instead of loading it all in memory. Should allow upload... #78

Closed wvmarle closed 11 years ago

wvmarle commented 11 years ago

... of large blocks on small-memory systems.

Hope this pull request gives a single patch now :-)

wvmarle commented 11 years ago

Yes! It works as it should :-)

I'm starting to understand the concept of branches and so. Can keep this kind of pull requests a lot more organised and clear what's going on.

uskudnik commented 11 years ago

:D

Smaller ones are also a lot quicker and easier to merge - I like to at the very least skip through the code and see what changes were made :)

wvmarle commented 11 years ago

It seems you're right.

I just read into mmap a bit deeper - what I didn't realise is that (on 32-bit systems - this should not be an issue on 64-bit systems) it has a 2 GB size limit. So we can not map the complete file in one go, which is exactly what it is trying to do now, and that's why you get this error. Unfortunately this is not clearly mentioned in the Python docs.

There are workarounds (of course). File has to be mapped part by part, no more than 2GB at a time. It means some extra coding.

We will end up with a limit on the part size of 2 GB (half Glacier's limit), effectively limiting the maximum archive size we can handle from 39 TB to 19.5 TB. I don't see that as too much of an issue: people handling files >19.5 TB won't be using tools like this one anyway.

iolsen commented 11 years ago

Sorry, I deleted my comment here and moved it to the commit itself. I thought it might go unnoticed on a closed pull request. In any case, I'm glad it makes sense. I can confirm it's a 32-bit OS where I see this.

wvmarle commented 11 years ago

Less chance that it gets noticed there :-) Other than because I automatically get notifications (like other contributors) for being subscribed to this project. To make sure, simply open a new issue.

Anyway will have a look at it later tonight. Doesn't look too hard to fix.

offlinehacker commented 11 years ago

Don't even try to mmap whole file. Wierd things can happen on different kinds of os and arhitectures. On Oct 16, 2012 4:10 PM, "wvmarle" notifications@github.com wrote:

Less chance that it gets noticed there :-) Other than because I automatically get notifications (like other contributors) for being subscribed to this project. To make sure, simply open a new issue.

Anyway will have a look at it later tonight. Doesn't look too hard to fix.

— Reply to this email directly or view it on GitHubhttps://github.com/uskudnik/amazon-glacier-cmd-interface/pull/78#issuecomment-9489308.

wvmarle commented 11 years ago

Found that out, indeed. Unfortunately the hard way.

Lesson learned for whenever I need this in the future :-)

offlinehacker commented 11 years ago

+1 :)

On Wed, Oct 17, 2012 at 8:19 AM, wvmarle notifications@github.com wrote:

Found that out, indeed. Unfortunately the hard way.

Lesson learned for whenever I need this in the future :-)

— Reply to this email directly or view it on GitHubhttps://github.com/uskudnik/amazon-glacier-cmd-interface/pull/78#issuecomment-9517112.