search

uskudnik / amazon-glacier-cmd-interface

Command line interface for Amazon Glacier
MIT License
374 stars 100 forks source link

mmap.error: [Errno 12] Cannot allocate memory #94

Open stigger opened 11 years ago

stigger commented 11 years ago

Hi. I'm trying to upload a 4G file, but glacier-cmd uploads first 54.8 megabytes, then fails. I tried multiple times, but it fails always after exactly 54.8 megabytes. Here is the stacktrace:

Wrote 54.8 MB. Rate 443.97 KB/s.te 525.79 KB/s, average 452.05 KB/s, eta 20:45:21.                                                                                         
Traceback (most recent call last):                                                                                                                
  File "/usr/bin/glacier-cmd", line 9, in <module>
    load_entry_point('glacier==0.2dev', 'console_scripts', 'glacier-cmd')()
  File "/usr/lib/python2.6/site-packages/glacier-0.2dev-py2.6.egg/glacier/glacier.py", line 751, in main
    args.func(args)
  File "/usr/lib/python2.6/site-packages/glacier-0.2dev-py2.6.egg/glacier/glacier.py", line 147, in wrapper
    return fn(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/glacier-0.2dev-py2.6.egg/glacier/glacier.py", line 300, in upload
    args.name, args.partsize, args.uploadid, args.resume)
  File "/usr/lib/python2.6/site-packages/glacier-0.2dev-py2.6.egg/glacier/GlacierWrapper.py", line 62, in wrapper
    ret = fn(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/glacier-0.2dev-py2.6.egg/glacier/GlacierWrapper.py", line 204, in glacier_connect_wrap
    return func(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/glacier-0.2dev-py2.6.egg/glacier/GlacierWrapper.py", line 62, in wrapper
    ret = fn(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/glacier-0.2dev-py2.6.egg/glacier/GlacierWrapper.py", line 225, in sdb_connect_wrap
    return func(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/glacier-0.2dev-py2.6.egg/glacier/GlacierWrapper.py", line 62, in wrapper
    ret = fn(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/glacier-0.2dev-py2.6.egg/glacier/GlacierWrapper.py", line 925, in upload
    mmapped_file = mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ)
mmap.error: [Errno 12] Cannot allocate memory
offlinehacker commented 11 years ago

How much free memory do you have? And what system do you use?

On Fri, Oct 26, 2012 at 2:16 PM, Vyacheslav Karpukhin < notifications@github.com> wrote:

Hi. I'm trying to upload 4G file, but glacier-cmd uploads first 54.8 megabytes, then fails. I tried multiple times, but it fails always after exactly 54.8 megabytes. Here is the stacktrace:

Wrote 54.8 MB. Rate 443.97 KB/s.te 525.79 KB/s, average 452.05 KB/s, eta 20:45:21. Traceback (most recent call last): File "/usr/bin/glacier-cmd", line 9, in load_entry_point('glacier==0.2dev', 'console_scripts', 'glacier-cmd')() File "/usr/lib/python2.6/site-packages/glacier-0.2dev-py2.6.egg/glacier/glacier.py", line 751, in main args.func(args) File "/usr/lib/python2.6/site-packages/glacier-0.2dev-py2.6.egg/glacier/glacier.py", line 147, in wrapper return fn(_args, _kwargs) File "/usr/lib/python2.6/site-packages/glacier-0.2dev-py2.6.egg/glacier/glacier.py", line 300, in upload args.name, args.partsize, args.uploadid, args.resume) File "/usr/lib/python2.6/site-packages/glacier-0.2dev-py2.6.egg/glacier/GlacierWrapper.py", line 62, in wrapper ret = fn(_args, _kwargs) File "/usr/lib/python2.6/site-packages/glacier-0.2dev-py2.6.egg/glacier/GlacierWrapper.py", line 204, in glacier_connect_wrap return func(_args, _kwargs) File "/usr/lib/python2.6/site-packages/glacier-0.2dev-py2.6.egg/glacier/GlacierWrapper.py", line 62, in wrapper ret = fn(_args, _kwargs) File "/usr/lib/python2.6/site-packages/glacier-0.2dev-py2.6.egg/glacier/GlacierWrapper.py", line 225, in sdb_connect_wrap return func(_args, _kwargs) File "/usr/lib/python2.6/site-packages/glacier-0.2dev-py2.6.egg/glacier/GlacierWrapper.py", line 62, in wrapper ret = fn(_args, _kwargs) File "/usr/lib/python2.6/site-packages/glacier-0.2dev-py2.6.egg/glacier/GlacierWrapper.py", line 925, in upload mmapped_file = mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ) mmap.error: [Errno 12] Cannot allocate memory

— Reply to this email directly or view it on GitHubhttps://github.com/uskudnik/amazon-glacier-cmd-interface/issues/94.

offlinehacker commented 11 years ago

kernel version, distro and architecture.

On Fri, Oct 26, 2012 at 2:42 PM, Jaka Hudoklin jakahudoklin@gmail.comwrote:

How much free memory do you have? And what system do you use?

On Fri, Oct 26, 2012 at 2:16 PM, Vyacheslav Karpukhin < notifications@github.com> wrote:

Hi. I'm trying to upload 4G file, but glacier-cmd uploads first 54.8 megabytes, then fails. I tried multiple times, but it fails always after exactly 54.8 megabytes. Here is the stacktrace:

Wrote 54.8 MB. Rate 443.97 KB/s.te 525.79 KB/s, average 452.05 KB/s, eta 20:45:21. Traceback (most recent call last): File "/usr/bin/glacier-cmd", line 9, in load_entry_point('glacier==0.2dev', 'console_scripts', 'glacier-cmd')() File "/usr/lib/python2.6/site-packages/glacier-0.2dev-py2.6.egg/glacier/glacier.py", line 751, in main args.func(args) File "/usr/lib/python2.6/site-packages/glacier-0.2dev-py2.6.egg/glacier/glacier.py", line 147, in wrapper return fn(_args, _kwargs) File "/usr/lib/python2.6/site-packages/glacier-0.2dev-py2.6.egg/glacier/glacier.py", line 300, in upload args.name, args.partsize, args.uploadid, args.resume) File "/usr/lib/python2.6/site-packages/glacier-0.2dev-py2.6.egg/glacier/GlacierWrapper.py", line 62, in wrapper ret = fn(_args, _kwargs) File "/usr/lib/python2.6/site-packages/glacier-0.2dev-py2.6.egg/glacier/GlacierWrapper.py", line 204, in glacier_connect_wrap return func(_args, _kwargs) File "/usr/lib/python2.6/site-packages/glacier-0.2dev-py2.6.egg/glacier/GlacierWrapper.py", line 62, in wrapper ret = fn(_args, _kwargs) File "/usr/lib/python2.6/site-packages/glacier-0.2dev-py2.6.egg/glacier/GlacierWrapper.py", line 225, in sdb_connect_wrap return func(_args, _kwargs) File "/usr/lib/python2.6/site-packages/glacier-0.2dev-py2.6.egg/glacier/GlacierWrapper.py", line 62, in wrapper ret = fn(_args, _kwargs) File "/usr/lib/python2.6/site-packages/glacier-0.2dev-py2.6.egg/glacier/GlacierWrapper.py", line 925, in upload mmapped_file = mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ) mmap.error: [Errno 12] Cannot allocate memory

— Reply to this email directly or view it on GitHubhttps://github.com/uskudnik/amazon-glacier-cmd-interface/issues/94.

stigger commented 11 years ago

I'm on CentOS 6.3, kernel 2.6.32-279.5.2.el6.i686. I have 2G of RAM, about 834 megabytes of it is free.

wvmarle commented 11 years ago

The problem is shown in the error message.

It's trying to mmap the complete file... should have been changed since but I just check my code and it's not there any more. Too much merging and so I'm afraid. Have to look into that. Sigh. Really have to go through that whole uploading code I'm afraid, it's quite complex now with the resumption and mmapping and whatnot.

offlinehacker commented 11 years ago

It's wierd, where the heck went commit that fixed that!