Open jaredmauch opened 7 years ago
Is this error reproducible on next run for the exact same UID?
if I restart it, it recovers ok. I run offlineimap detached in a screen so don't notice when it fails immediately.
I wonder the error stands because of maxconnections = 4
. You only have one idle folder, so I'd try to reduce this value to 2 (or even one if all the emails are downloaded by offlineimap are from INBOX).
I'm pretty sure that's a IDLE issue where offlineimap does the same job twice: trying to save the same UID twice in the database.
is there a good trace i can enable for this? my mailbox is quite large on the backside (~1.37m messages). I can trigger this usually 1-3x a day, so good debugs should be easy to come by.
IntegrityError: PRIMARY KEY must be unique
means offlineimap tries to create the same UID twice while this should not happen. The root cause is likely that offlineimap downloads the same UID concurrently. This should never happen. Tracking down issues about concurrency is not easy. I don't think the debug logs will help much.
You might keep maxconnections
set to 4 and try v7.0.6 with singlethreadperfolder
enabled (new feature). This might help.
I think this issue and #174 are the same. I bet the printed error is different because of the database backend. SQLite is more strict than legacy text.
Ok, have to build a new package and hand to our OS team. I didn't poke at the code, you don't lock the UID to prevent multiple threads from accessing at once?
There's no "lock per UID" concept in the code.
It's possible to install offlineimap as a user: remove the currently installed offlineimap and download the zip (or git clone
the repository). Then ./offlineimap.py [options]
.
yeah, i'd rather the other users pick up 7.0.6 as well. let me do my thing and i'll get back in the next ~24h
Apologies for the late reply, I'm still seeing the issue with 7.0.6:
prior 2 cycles:
Syncing IP-ENG-OPS: IMAP -> Maildir Copy message UID 615517 (1/3) RemoteNTT:IP-ENG-OPS -> LocalNTT Copy message UID 615518 (2/3) RemoteNTT:IP-ENG-OPS -> LocalNTT Copy message UID 615519 (3/3) RemoteNTT:IP-ENG-OPS -> LocalNTT Account sync NTT: *** Finished account 'NTT' in 0:42 Next refresh in 4.0 minutes Next refresh in 3.0 minutes Thread-73: Syncing INBOX: IMAP -> Maildir Copy message UID 289029 (1/1) RemoteNTT:INBOX -> LocalNTT
Folder IP-ENG-OPS [acc: NTT]: Syncing IP-ENG-OPS: IMAP -> Maildir Copy message UID 615520 (1/3) RemoteNTT:IP-ENG-OPS -> LocalNTT Copy message UID 615521 (2/3) RemoteNTT:IP-ENG-OPS -> LocalNTT Copy message UID 615522 (3/3) RemoteNTT:IP-ENG-OPS -> LocalNTT Thread 'Copy message from RemoteNTT:IP-ENG-OPS' terminated with exception: Traceback (most recent call last): File "/home/jared/offlineimap/offlineimap/threadutil.py", line 163, in run Thread.run(self) File "/usr/lib64/python2.7/threading.py", line 764, in run self.__target(*self.__args, **self.__kwargs) File "/home/jared/offlineimap/offlineimap/folder/Base.py", line 814, in copymessageto message = self.getmessage(uid) File "/home/jared/offlineimap/offlineimap/folder/IMAP.py", line 302, in getmessage data = self._fetch_from_imap(str(uid), 2) File "/home/jared/offlineimap/offlineimap/folder/IMAP.py", line 720, in _fetch_from_imap imapobj.select(self.getfullname(), readonly=True) File "/home/jared/offlineimap/offlineimap/imaplibutil.py", line 64, in select exc_info()[2]) File "/home/jared/offlineimap/offlineimap/imaplibutil.py", line 53, in select result = super(UsefulIMAPMixIn, self).select(mailbox, readonly) File "/home/jared/offlineimap/offlineimap/bundled_imaplib2.py", line 1053, in select typ, dat = rqb.get_response('command: %s => %%s' % rqb.name) File "/home/jared/offlineimap/offlineimap/bundled_imaplib2.py", line 200, in get_response raise typ(exc_fmt % str(val)) OfflineImapError: Server 'mail.us.ntt.net' closed connection, error on SELECT 'IP-ENG-OPS'. Server said: command: EXAMINE => no response after 30.0 secs Last 1 debug messages logged for Copy message from RemoteNTT:IP-ENG-OPS prior to exception: thread: Register new thread 'Copy message from RemoteNTT:IP-ENG-OPS' (account 'NTT') ERROR: Exceptions occurred during the run! ERROR: IMAP server 'RemoteNTT' does not have a message with UID '289023' Traceback: File "/home/jared/offlineimap/offlineimap/folder/Base.py", line 814, in copymessageto message = self.getmessage(uid) File "/home/jared/offlineimap/offlineimap/folder/IMAP.py", line 302, in getmessage data = self._fetch_from_imap(str(uid), 2) File "/home/jared/offlineimap/offlineimap/folder/IMAP.py", line 761, in _fetch_from_imap raise OfflineImapError(reason, severity) ERROR: IMAP server 'RemoteNTT' does not have a message with UID '289027' Traceback: File "/home/jared/offlineimap/offlineimap/folder/Base.py", line 814, in copymessageto message = self.getmessage(uid) File "/home/jared/offlineimap/offlineimap/folder/IMAP.py", line 302, in getmessage data = self._fetch_from_imap(str(uid), 2) File "/home/jared/offlineimap/offlineimap/folder/IMAP.py", line 761, in _fetch_from_imap raise OfflineImapError(reason, severity)
And does the server has UIDs 289023 and 289027 in the RemoteNTT folder?
Seems so for 23, not for 27.
A100 UID FETCH 289023:289030 FLAGS * 52454 FETCH (UID 289023 FLAGS (\Seen $NotJunk NotJunk)) * 52455 FETCH (UID 289024 FLAGS (\Seen $NotJunk NotJunk)) * 52456 FETCH (UID 289025 FLAGS (\Seen $NotJunk NotJunk)) * 52457 FETCH (UID 289026 FLAGS (\Seen $NotJunk NotJunk)) * 52458 FETCH (UID 289028 FLAGS (\Seen $NotJunk NotJunk)) * 52459 FETCH (UID 289029 FLAGS ($NotJunk NotJunk)) * 52460 FETCH (UID 289030 FLAGS (\Seen $NotJunk NotJunk)) * 52499 EXISTS A100 OK Fetch completed.
Weird. Does the next run fail on those UIDs (with and without a full restart)?
due to the error offlineimap stops and i must manually restart it. (i leave it in detached screen and require my password as interactive to launch).
the error does not exist after restart and it proceeds as fine.. it seems if i were to:
while true do; offlineimap; done
with my password in the rc file it would recover without issue. I have other colleagues who use this as a workaround/solution.
Here's another related/similar error/exception:
Syncing Sent Messages: IMAP -> Maildir Skipping Sent Messages (not changed) Folder INBOX [acc: NTT]: Syncing INBOX: IMAP -> Maildir Thread-109: Copy message UID 289063 (1/2) RemoteNTT:INBOX -> LocalNTT Copy message UID 289064 (2/2) RemoteNTT:INBOX -> LocalNTT Folder stats [acc: NTT]: Syncing stats: IMAP -> Maildir Skipping stats (not changed) Folder RANCID [acc: NTT]: Syncing RANCID: IMAP -> Maildir Skipping RANCID (not changed) Folder IP-ENG-OPS [acc: NTT]: Syncing IP-ENG-OPS: IMAP -> Maildir Folder INBOX [acc: NTT]: Copy message UID 401 (1/52494) RemoteNTT:INBOX -> LocalNTT Copy message UID 402 (2/52494) RemoteNTT:INBOX -> LocalNTT Copy message UID 404 (3/52494) RemoteNTT:INBOX -> LocalNTT Copy message UID 410 (4/52494) RemoteNTT:INBOX -> LocalNTT Copy message UID 411 (5/52494) RemoteNTT:INBOX -> LocalNTT Copy message from RemoteNTT:INBOX: ERROR: Copying message 404 [acc: NTT] PRIMARY KEY must be unique Folder INBOX [acc: NTT]: Copy message UID 412 (6/52494) RemoteNTT:INBOX -> LocalNTT Thread 'Copy message from RemoteNTT:INBOX' terminated with exception: Traceback (most recent call last): File "/home/jared/offlineimap/offlineimap/threadutil.py", line 163, in run Thread.run(self) File "/usr/lib64/python2.7/threading.py", line 764, in run self.__target(*self.__args, **self.__kwargs) File "/home/jared/offlineimap/offlineimap/folder/Base.py", line 827, in copymessageto statusfolder.savemessage(new_uid, message, flags, rtime) File "/home/jared/offlineimap/offlineimap/folder/LocalStatusSQLite.py", line 341, in savemessage (uid,flags,mtime,labels)) File "/home/jared/offlineimap/offlineimap/folder/LocalStatusSQLite.py", line 170, in __sql_write self.connection.execute(sql, args) IntegrityError: PRIMARY KEY must be unique Last 4 debug messages logged for Copy message from RemoteNTT:INBOX prior to exception: thread: Register new thread 'Copy message from RemoteNTT:INBOX' (account 'NTT') imap: Returned object from fetching 404: 'Return-Path:Received: from mail1.us.ntt.net (mail1.us.ntt.net [129.250.38.32]) (authenticated user=postmaster@us.ntt.net b...> -b > > > > >>>>> > > > > >>>> > > > > >>> >=20 ' maildir: Write mail 'INBOX:404' with flags set(['S']) maildir: savemessage: returning uid 404 ERROR: Exceptions occurred during the run! ERROR: IMAP server 'RemoteNTT' does not have a message with UID '289046' Traceback: File "/home/jared/offlineimap/offlineimap/folder/Base.py", line 814, in copymessageto message = self.getmessage(uid) File "/home/jared/offlineimap/offlineimap/folder/IMAP.py", line 302, in getmessage data = self._fetch_from_imap(str(uid), 2) File "/home/jared/offlineimap/offlineimap/folder/IMAP.py", line 761, in _fetch_from_imap raise OfflineImapError(reason, severity) ERROR: IMAP server 'RemoteNTT' does not have a message with UID '289048' Traceback: File "/home/jared/offlineimap/offlineimap/folder/Base.py", line 814, in copymessageto message = self.getmessage(uid) File "/home/jared/offlineimap/offlineimap/folder/IMAP.py", line 302, in getmessage data = self._fetch_from_imap(str(uid), 2) File "/home/jared/offlineimap/offlineimap/folder/IMAP.py", line 761, in _fetch_from_imap raise OfflineImapError(reason, severity) ERROR: IMAP server 'RemoteNTT' does not have a message with UID '289050' Traceback: File "/home/jared/offlineimap/offlineimap/folder/Base.py", line 814, in copymessageto message = self.getmessage(uid) File "/home/jared/offlineimap/offlineimap/folder/IMAP.py", line 302, in getmessage data = self._fetch_from_imap(str(uid), 2) File "/home/jared/offlineimap/offlineimap/folder/IMAP.py", line 761, in _fetch_from_imap raise OfflineImapError(reason, severity) ERROR: IMAP server 'RemoteNTT' does not have a message with UID '289063' Traceback: File "/home/jared/offlineimap/offlineimap/folder/Base.py", line 814, in copymessageto message = self.getmessage(uid) File "/home/jared/offlineimap/offlineimap/folder/IMAP.py", line 302, in getmessage data = self._fetch_from_imap(str(uid), 2) File "/home/jared/offlineimap/offlineimap/folder/IMAP.py", line 761, in _fetch_from_imap raise OfflineImapError(reason, severity) ERROR: Copying message 404 [acc: NTT] PRIMARY KEY must be unique Traceback: File "/home/jared/offlineimap/offlineimap/folder/Base.py", line 827, in copymessageto statusfolder.savemessage(new_uid, message, flags, rtime) File "/home/jared/offlineimap/offlineimap/folder/LocalStatusSQLite.py", line 341, in savemessage (uid,flags,mtime,labels)) File "/home/jared/offlineimap/offlineimap/folder/LocalStatusSQLite.py", line 170, in __sql_write self.connection.execute(sql, args)
The closed socket is another issue.
The primary key error is likely a bug in concurrency because we download the same UID more than once.
Forget to say that's related to IDLE, IMHO.
Did you enable singlethreadperfolder
, BTW?
No, but it's now set
[Repository RemoteNTT] type = IMAP singlethreadperfolder =yes remotehost = mail.us.ntt.net ssl = yes cert_fingerprint = 86919ab4738503b01a6ab946748a07814c4592fc remoteuser = jmauch@us.ntt.net idlefolders = ['INBOX'] maxconnections = 4 holdconnectionopen = yes folderfilter = lambda foldername: not re.search('(^Trash$|5YearsAndBeyond)', foldername)
Well, I'm not sure this setting is honored in IDLE mode. I don't think this can harm, though. Thanks for your tests.
I've looked at the code about IDLE: that sucks.
been running for almost 24h with singlethreadperfolder set.
Good to know! I'll wait for you to get more feedback after more time with this setup. Then, I'll add this information to the documentation.
Been running for a week with no need to restart. perhaps this should just be the default option on the codebase, with setting it to no driving someone to debug it further?
I agree. Notice this feature is recent and still marked experimental. According to your positive feedbacks I think we can enable singlethreadperfolder by default for IDLE mode.
@jaredmauch Done. Could you try the current next
branch with the singlethreadperfolder option disabled, please?
Please read unset instead of disabled. IOW, comment the singlethreadperfolder option. Explicitly enabling or disabling the option is still supported in IDLE mode.
yeah, let me pull it onto that host.
btw, when quitting i see this:
Account sync NTT: Next refresh in 3.0 minutes ^CTerminating NOW (this may take a few seconds)... ERROR: Exceptions occurred during the run! ERROR: ERROR in syncfolder for NTT folder INBOX: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/offlineimap/accounts.py", line 537, in syncfolder statusfolder.cachemessagelist() File "/usr/lib/python2.7/site-packages/offlineimap/folder/LocalStatusSQLite.py", line 234, in cach emessagelist flags = set(row[1]) TypeError: 'NoneType' object is not iterable 'NoneType' object is not iterable ERROR: Copying message 82372 [acc: NTT] PRIMARY KEY must be unique Traceback: File "/usr/lib/python2.7/site-packages/offlineimap/folder/Base.py", line 827, in copymessageto statusfolder.savemessage(new_uid, message, flags, rtime) File "/usr/lib/python2.7/site-packages/offlineimap/folder/LocalStatusSQLite.py", line 341, in save message (uid,flags,mtime,labels)) File "/usr/lib/python2.7/site-packages/offlineimap/folder/LocalStatusSQLite.py", line 170, in __sq l_write self.connection.execute(sql, args) ERROR: while syncing INBOX [account NTT] PRIMARY KEY must be unique
btw, when quitting i see this:
Which version?
706
Jared Mauch
On Sep 20, 2016, at 10:55 AM, Nicolas Sebrecht notifications@github.com wrote:
btw, when quitting i see this:
Which version?
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.
With singlethreadperfolder?
Yes, that was with singlethreadperfolder.
with next
i see this:
Folder Drafts [acc: NTT]: Syncing Drafts: IMAP -> Maildir Folder Archive [acc: NTT]: Skipping Archive (not changed) Folder Apple Mail To Do [acc: NTT]: Skipping Apple Mail To Do (not changed) Folder Deleted Messages [acc: NTT]: Syncing Deleted Messages: IMAP -> Maildir Folder Drafts [acc: NTT]: Skipping Drafts (not changed) Folder Deleted Messages [acc: NTT]: Skipping Deleted Messages (not changed) Folder Junk [acc: NTT]: Syncing Junk: IMAP -> Maildir Skipping Junk (not changed) Folder Notes [acc: NTT]: Syncing Notes: IMAP -> Maildir Skipping Notes (not changed) Folder oldinbox [acc: NTT]: Syncing oldinbox: IMAP -> Maildir Skipping oldinbox (not changed) Folder dwdm [acc: NTT]: Syncing dwdm: IMAP -> Maildir Skipping dwdm (not changed) Folder peering [acc: NTT]: Syncing peering: IMAP -> Maildir Skipping peering (not changed) Folder projects and ip-eng-pp [acc: NTT]: Syncing projects and ip-eng-pp: IMAP -> Maildir Skipping projects and ip-eng-pp (not changed) Folder Sent Messages [acc: NTT]: Syncing Sent Messages: IMAP -> Maildir Folder INBOX [acc: NTT]: Syncing INBOX: IMAP -> Maildir Folder Sent Messages [acc: NTT]: Skipping Sent Messages (not changed) Folder INBOX [acc: NTT]: Skipping INBOX (not changed) Folder stats [acc: NTT]: Syncing stats: IMAP -> Maildir Skipping stats (not changed) Folder RANCID [acc: NTT]: Syncing RANCID: IMAP -> Maildir Skipping RANCID (not changed) Folder IP-ENG-OPS [acc: NTT]: Syncing IP-ENG-OPS: IMAP -> Maildir Copy message UID 616485 (1/2) RemoteNTT:IP-ENG-OPS -> LocalNTT Copy message UID 616486 (2/2) RemoteNTT:IP-ENG-OPS -> LocalNTT Thread-306: Copy message UID 24842 (17686/179785) RemoteNTT:IP-ENG-OPS -> LocalNTT Account sync NTT: *** Finished account 'NTT' in 1:01 Next refresh in 4.0 minutes Thread-306: ERROR: Copying message 24842 [acc: NTT] PRIMARY KEY must be unique ERROR: while syncing IP-ENG-OPS [account NTT] PRIMARY KEY must be unique ERROR: ERROR in syncfolder for NTT folder IP-ENG-OPS: Traceback (most recent call last): File "/home/jared/offlineimap/offlineimap/accounts.py", line 584, in syncfolder remotefolder.syncmessagesto(localfolder, statusfolder) File "/home/jared/offlineimap/offlineimap/folder/Base.py", line 1090, in syncmessagesto action(dstfolder, statusfolder) File "/home/jared/offlineimap/offlineimap/folder/Base.py", line 918, in __syncmessagesto_copy self.copymessageto(uid, dstfolder, statusfolder, register=0) File "/home/jared/offlineimap/offlineimap/folder/Base.py", line 827, in copymessageto statusfolder.savemessage(new_uid, message, flags, rtime) File "/home/jared/offlineimap/offlineimap/folder/LocalStatusSQLite.py", line 341, in savemessage (uid,flags,mtime,labels)) File "/home/jared/offlineimap/offlineimap/folder/LocalStatusSQLite.py", line 170, in __sql_write self.connection.execute(sql, args) IntegrityError: PRIMARY KEY must be unique PRIMARY KEY must be unique Account sync NTT: Next refresh in 3.0 minutes Next refresh in 2.0 minutes Next refresh in 1.0 minutes *** Processing account NTT
Ok. So, singlethreadperfolder reduces the exposure to the concurrency issue without fully fixing it.
btw, still running ok in the background after ~3 days with next
Nice. Seems like singlethreadperfolder is greatly improving IDLE. There might be yet another concurrency issue to explain the latest bug report.
I noticed offlineimap died a few days ago (Sept 26th) and left a 5G core file on the system. (Yes, my mailbox is a beast).
Would be great to report the debug trace. I'm not sure what you mean by a 5G "core file". Do you mean the log file?
No. core file of Python running offlineimap was 5GB. Guessing memory leak.
Jared Mauch
On Sep 29, 2016, at 6:19 PM, Nicolas Sebrecht notifications@github.com wrote:
Would be great to report the debug trace. I'm not sure what you mean by a 5G "core file". Do you mean the log file?
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.
The memory leak issue will be discussed in #385.
@jaredmauch, could you please check if the path in #438 resolves the issue for you?
General informations
offlineimap -V
): 7.0.5 + 7.0.6Configuration file offlineimaprc
pythonfile (if any)
Logs, error
Steps to reproduce the error