MemoryError on RaspberryPi

[onlygecko]

Hi,

I would love to use OfflineIMAP for backing up my mails. On my laptop (Ubuntu 14.04 repository package) it run's smoothly, but on my raspberrypi OfflineIMAP crashes with the following error:

Traceback (most recent call last):
  File "/home/pi/opt/offlineimap-6.5.4/offlineimap/threadutil.py", line 156, in run
    Thread.run(self)
  File "/usr/lib/python2.7/threading.py", line 505, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/pi/opt/offlineimap-6.5.4/offlineimap/folder/Base.py", line 330, in copymessageto
    message = self.getmessage(uid)
  File "/home/pi/opt/offlineimap-6.5.4/offlineimap/folder/IMAP.py", line 219, in getmessage
    '(BODY.PEEK[])')
  File "/home/pi/opt/offlineimap-6.5.4/offlineimap/imaplib2.py", line 1150, in uid
    return self._simple_command('UID', command, *args, **kw)
  File "/home/pi/opt/offlineimap-6.5.4/offlineimap/imaplib2.py", line 1595, in _simple_command
    return self._command_complete(self._command(name, *args), kw)
  File "/home/pi/opt/offlineimap-6.5.4/offlineimap/imaplib2.py", line 1360, in _command_complete
    typ, dat = rqb.get_response('command: %s => %%s' % rqb.name)
  File "/home/pi/opt/offlineimap-6.5.4/offlineimap/imaplib2.py", line 173, in get_response
    raise typ(exc_fmt % str(val))
error: command: UID => program error: <type 'exceptions.MemoryError'> - 

Last 1 debug messages logged for Copy message from remote-xyz:Gesendet prior to exception:
thread: Register new thread 'Copy message from remote-xyz:Gesendet' (account 'xyz')
ERROR: Exceptions occurred during the run!
ERROR: command: UID => program error: <type 'exceptions.MemoryError'> - 
Exception in thread xxx.com handler (most likely raised during interpreter shutdown):Exception in thread xxx.com reader (most likely raised during interpreter shutdown):Exception in thread Folder Papierkorb [acc: xyz] (most likely raised during interpreter shutdown):
Traceback (most recent call last):
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 552, in __bootstrap_inner
  File "/usr/lib/python2.7/threading.py", line 552, in __bootstrap_inner
Traceback (most recent call last):

  File "/usr/lib/python2.7/threading.py", line 505, in run
  File "/usr/lib/python2.7/threading.py", line 505, in run
  File "/home/pi/opt/offlineimap-6.5.4/offlineimap/imaplib2.py", line 1657, in _handler  File "/usr/lib/python2.7/threading.py", line 552, in __bootstrap_inner  File "/home/pi/opt/offlineimap-6.5.4/offlineimap/imaplib2.py", line 1773, in _reader

  File "/home/pi/opt/offlineimap-6.5.4/offlineimap/threadutil.py", line 227, in run<type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'Empty'
<type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'exc_info'

  File "/home/pi/opt/offlineimap-6.5.4/offlineimap/threadutil.py", line 156, in run

I've tested the version that comes with raspbian (6.3.4), the current version 6.5.7 and the version that worked on my laptop - 6.5.4.

I found a not so nice blog post about OfflineIMAP, saying that it consumes a lot of RAM. And well, that could be the reason for failing. On my laptop it consumed more than 1GB, But the raspberry pi (model b) has just up to 480MB.

Besides the fact, that the reason for crashing could be something else: Why is OfflineIMAP consuming that much memory?

I would love to help fixing this bug, because the raspberry pi is a great backup server and OfflineIMAP is a great tool for backing up mails.

[nicolas33]

I've never hit such memory consumption. I guess OfflineIMAP is used in a long runtime process which is something I never do. What's your configuration file?

[onlygecko]

The error occurs always at the same folder when I start synchronizing from the beginning. This folder ('Gesendet' - sent) contains 120 mails. The errors occurs at message 96. Is there any additional output so I can look up the 'suspicious' mail?

It's still strange that I get no error on my laptop.

Here is the config file:

[general]
accounts = xyz

[Account xyz]
localrepository = local-xyz
remoterepository = remote-xyz

[Repository local-xyz]
type = Maildir
localfolders = ~/data/mailbackup/xyz

[Repository remote-xyz]
type = IMAP
remotehost = myserver.com
remoteuser = username
remotepass = secret
ssl = yes
cert_fingerprint = aaaaaaaaaaaaaaaf4efefe9597fbd57e09996c58
maxconnections = 2

[nicolas33]

Weird. Comparing results on both the laptop and the R-PI is a good thing. You could also enable debug logs.

[onlygecko]

There were three large mails, each 95MB, in this folder. After deleting them synchronization works fine. Do you think this is an issue which needs to be addressed?

[nicolas33]

They were on the Maildir or on the server?

[nicolas33]

There is no internal limitation from our side regarding mail size. If there's an issue, it should be fixed but I don't get what's the issue exactly (beside that those mails could not be synced).

[chris001]

How about out of memory error. The Raspberry Pi has about 128mb ram if I recall correctly. Each mail was 95mb. OLI loads the entire message into ram, 96mb. Probably this is far more than the R-Pi has available. How does the python implementation on R-P handle requests for more ram than it has available? Probably throws exception?

[onlygecko]

To sum up, and to avoid misunderstandings: ;-)

There is/was a problem with synchronizing my mail account (just) on a Raspberry Pi. I've got an Memory error every time I've tried to sync the account. The behaviour was reproducible. The error occurred while syncing a specific folder. This folder contained 130 mails, 3 of them with a file size of 95MB each. After deleting the large mails, the sync worked perfectly.

This leads to the assumption that offlineimap want's more memory than the raspberry can provide. The raspberry has 480MB RAM available. (512MB minus xxMB for graphics).

And now my question: Is this a problem that can / should be solved? Are very big mails (>50MB) a usual use case? On which platforms (laptops, servers, micro pcs) do you, the developer, focus? Or is it enough to throw a message like: 'There is not enough memory to process message XYZ from Dave'?

What Do you think? What would offlineimap users expect / want?

[chris001]

@onlygecko @nicolas33 The following is my opinion! Offlineimap algorithm implementation should be enhanced to run on far less memory, the 512MB in a R-Pi should be more than adequate. See criticisms here: http://blog.ezyang.com/2012/08/offlineimap-sucks/ "Poor space complexity The memory usage of OfflineIMAP is linear with the number of messages in your inbox. For large mailboxes, this effectively means loading hundreds of thousands of elements into a set and doing expensive operations on it (OfflineIMAP consistently pegs my CPU when I run it). OfflineIMAP should be able to run in constant space, but zero algorithmic thought has been put into this problem space."

For one, the messages should be downloaded with python directly to disk, not memory. Reason, exactly as in this case, you can legitimately have a 95MB email message containing base64 mime encoded attachments, but disk space is more plentiful than ram. Also, most/all of the processing is on the header, and possibly the filename for the local Maildir option, body is ignored except for possible hash fingerprinting, so send it directly to disk.

Indeed, OLI is due for some implementation optimizations, it should run perfectly on a 512mb R-Pi, and consume a relatively constant amount of resources. Possibly 40-50MB of ram?

[nicolas33]

This is all full of good sense.