search

OfflineIMAP / offlineimap

Read/sync your IMAP mailboxes (python2) [LEGACY: move to offlineimap3]
http://www.offlineimap.org
Other
1.78k stars 360 forks source link

KeyError in `folder/Gmail.py` #613

Open nicolasbock opened 5 years ago

nicolasbock commented 5 years ago

General informations

Running offlineimap I am getting the following:

 ERROR: ERROR in syncfolder for nicolasbock.gmail folder [Gmail]/Trash: Traceback (most recent call last):
  File "/usr/lib64/python2.7/site-packages/offlineimap/accounts.py", line 652, in syncfolder
    localfolder.syncmessagesto(remotefolder, statusfolder)
  File "/usr/lib64/python2.7/site-packages/offlineimap/folder/Base.py", line 1116, in syncmessagesto
    action(dstfolder, statusfolder)
  File "/usr/lib64/python2.7/site-packages/offlineimap/folder/Base.py", line 944, in __syncmessagesto_copy
    self.copymessageto(uid, dstfolder, statusfolder, register=0)
  File "/usr/lib64/python2.7/site-packages/offlineimap/folder/GmailMaildir.py", line 227, in copymessageto
    labels = dstfolder.getmessagelabels(uid)
  File "/usr/lib64/python2.7/site-packages/offlineimap/folder/Gmail.py", line 105, in getmessagelabels
    if 'labels' in self.messagelist[uid]:
KeyError: 690815

Is this a known issue? Any tips for debugging it further?

Thanks!

nicolas33 commented 5 years ago

I think this should not happen. It looks like there's a race condition. Is this reproducible? If so, with the same key number?

nicolasbock commented 5 years ago

The UUID changes every time I run the command. And I tried this now with -1 and master and it's still happening. I'll see whether I can get a more precise location in the code.

nicolas33 commented 5 years ago

folder/GmailMaildir.py didn't change for a long time. I think it would be wise to check v7.2.0 (and v7.1.1 if v7.2.0 fails the same way). If this appear to be a regression, we could git bissect this issue.

nicolasbock commented 5 years ago

This problem is present in v7.0.0 and in v6.x the process seems to hang at syncing the All Mail folder.

nicolasbock commented 5 years ago

@nicolas33 I think the problem is here. It seems that the new UUID is not communicated and subsequent code assumes that the old UUID is still there. I might be wrong of course :wink:

nicolas33 commented 5 years ago

Maybe it is worth adding print() debug statements to track how the uid lists change during the sync.

nicolasbock commented 5 years ago

Yes, that's what I did. And right before the call to change_message_uid the old uid is still in messagelist but right after it's not anymore.

nicolasbock commented 5 years ago

I suspect that the problem is here

nicolas33 commented 5 years ago

This is a line of code where we delete the uid after the mail is stored in the maildir. This statement is legal.

I'd say all the current lines of code where we delete uids from the message list are legal. The issue might be more subtle like calling any upper method while it should not in this context or not correctly locking some chunk of code, etc.

Could you share your config file? Could you share more detailed logs (with -d imap,maildir CLI option)?

nicolasbock commented 5 years ago

I don't quite understand how this is supposed to work. In https://github.com/OfflineIMAP/offlineimap/blob/master/offlineimap/folder/GmailMaildir.py#L218 copymessageto is called which removes the uid. Then in https://github.com/OfflineIMAP/offlineimap/blob/master/offlineimap/folder/GmailMaildir.py#L227 getmessagelages is called with the (now deleted) uid. Why would that work?

nicolasbock commented 5 years ago 
[Repository nicolasbock.gmail-remote]
type = Gmail
createfolders = false
folderfilter = lambda folder: folder in ['INBOX',
                                         '[Gmail]/All Mail',
                                         '[Gmail]/Chats',
                                         '[Gmail]/Drafts',
                                         '[Gmail]/Sent Mail',
                                         '[Gmail]/Spam',
                                         '[Gmail]/Starred',
                                         '[Gmail]/Trash']
auth_mechanisms = LOGIN
maxconnections = 5
nicolasbock commented 5 years ago 
 Copy message UID 690892 (1/296) nicolasbock.gmail-local:[Gmail].Trash -> nicolasbock.gmail-remote:[Gmail]/Trash
 nbock 3: starting copymessageto
 nbock 1: 690892 in messagelist: True
 nbock 5: 690892 in messagelist: True
 nbock 6: 690892 in messagelist: True
 []: getmessageheaderlist: called to get X-Keywords
 []: getmessageheaderlist: eoh = 4598
 []: getmessageheaderlist: headers = 'Delivered-To: nicolasbock@gmail.com\nReceived: by 2002:a9d:21:0:0:0:0:0 with SMTP id 30csp5503157ota;\n        Wed, 17 Apr 2019 09:42:28 -0700 (PDT)\nX-Google-Smtp-Source: APXvYqxX3NZJHrFPIj+0rcb8AkvWZ0ZkJL6+w+B8gJcqTKEGC0ZJTh3gjs0PQFFNuYkswiCvza4j\nX-Received: by 2002:a37:59c5:: with SMTP id n188mr64646686qkb.208.1555519348036;\n        Wed, 17 Apr 2019 09:42:28 -0700 (PDT)\nARC-Seal: i=1; a=rsa-sha256; t=1555519348; cv=none;\n        d=google.com; s=arc-20160816;\n        b=VvUdCI2cMCNbRW4UzrrFxTDVxdHwcQQaRaZsVTqU+mWjUqMj6fUpGEj7rvFG8JyCJS\n         sPjkxXPau4K4HzFeNh8tOnJ+4AL3qy7EKfEL3fp9R2ktMjxDqs4osl8y8Nj6aUwKaz8f\n         z4DJoOJWb9VzBkK1Mci8KUm4d/DKK6z8BlzutCarPS6+lH6w10JZ+WJiVW1JF32goRs5\n         0JlY1ITXbE2h2Cr+E5PaeiA90x2Ty1/nA2WcR8Kl8T0AifZ7g+1D0gzwqw4aTKFbNrTZ\n         pbI+A63SS2sM6TS/3JPRjHLkB7XmiyH6Uj7CJK1/yjUGykfncttZELcA6TVV0QqbVGJu\n         5DNQ==\nARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816;\n        h=list-unsubscribe:list-post:list-archive:list-id:precedence\n         :content-transfer-encoding:mime-version:subject:references\n         :in-reply-to:message-id:cc:to:reply-to:from:dkim-signature:date;\n        bh=nc9CK8jr8JqtXdLPxeBkBOKkjFaJmXfGwfnpxlP9+gE=;\n        b=jaiVVDM3cO5pC3HNUf2Y3XWQuhurPVf20I6ehaN1srsWLEOg43dZkbnbvMJ0riXWQM\n         2YHEc1BAJsGrEnHbRSjjlQNbKIOZ8/POyYmxIJdSgDE3ta3QuXQmf3qA3SK1T+AfbRQa\n         2uJ4c0GrhjjP2LVtTsvCJq7oUWufwnbIUEP8MNnqPqhzX24Phe/xhO+t9A3nwo7AXDbi\n         AOLVOpTt/IocY+QtTkcn+CJVCn8JIqiyV41rE/BlpnR04qRBQNwPYxwSwP0Zrk7HMEM/\n         UOBfZfzJeCAJL716MxbSOjKjMsEBNixiIvzQXG18U5J0QdDPHg1jgvGxch4LN7miGPlb\n         gsRw==\nARC-Authentication-Results: i=1; mx.google.com;\n       dkim=pass (test mode) header.i=@github.com header.s=pf2014 header.b=ZA2IhJG2;\n       spf=pass (google.com: domain of noreply@github.com designates 192.30.252.198 as permitted sender) smtp.mailfrom=noreply@github.com;\n       dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=github.com\nReturn-Path: <noreply@github.com>\nReceived: from out-7.smtp.github.com (out-7.smtp.github.com. [192.30.252.198])\n        by mx.google.com with ESMTPS id 50si5749913qtv.225.2019.04.17.09.42.27\n        for <nicolasbock@gmail.com>\n        (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128);\n        Wed, 17 Apr 2019 09:42:28 -0700 (PDT)\nReceived-SPF: pass (google.com: domain of noreply@github.com designates 192.30.252.198 as permitted sender) client-ip=192.30.252.198;\nAuthentication-Results: mx.google.com;\n       dkim=pass (test mode) header.i=@github.com header.s=pf2014 header.b=ZA2IhJG2;\n       spf=pass (google.com: domain of noreply@github.com designates 192.30.252.198 as permitted sender) smtp.mailfrom=noreply@github.com;\n       dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=github.com\nDate: Wed, 17 Apr 2019 09:42:27 -0700\nDKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=github.com;\n\ts=pf2014; t=1555519347;\n\tbh=nc9CK8jr8JqtXdLPxeBkBOKkjFaJmXfGwfnpxlP9+gE=;\n\th=Date:From:Reply-To:To:Cc:In-Reply-To:References:Subject:List-ID:\n\t List-Archive:List-Post:List-Unsubscribe:From;\n\tb=ZA2IhJG2p7mafWxB7+Q31vXHkG8lrtSCr2bklhD9/Ir5gmflqZgFp8PKCfz4jc9Rd\n\t BdAFxWA0QSzMYucse3OihFjrXanxcdo92njY95dCBE6K7xI5J2fhTuBDmHJlgbF5q2\n\t BEeBoEgnYM05w4G7tWzCR53NU67CAOTRzVOudfw0=\nFrom: =?UTF-8?B?VGlsbCBTY2jDpGZlcg==?= <notifications@github.com>\nReply-To: gentoo/gentoo <reply+000cd6a9c12b39e1a7644b332064b13547f6beb1a1bc3f6e92cebac489f392a169ce194abc8b@reply.github.com>\nTo: gentoo/gentoo <gentoo@noreply.github.com>\nCc: Subscribed <subscribed@noreply.github.com>\nMessage-ID: <gentoo/gentoo/pull/11463/c484169036@github.com>\nIn-Reply-To: <gentoo/gentoo/pull/11463@github.com>\nReferences: <gentoo/gentoo/pull/11463@github.com>\nSubject: Re: [gentoo/gentoo] Davical 1.1.8 version bump (#11463)\nMime-Version: 1.0\nContent-Type: multipart/alternative;\n boundary="--==_mimepart_5cb7577399b16_19493fecaf6cd9601666cf";\n charset=UTF-8\nContent-Transfer-Encoding: 7bit\nPrecedence: list\nX-GitHub-Sender: tillschaefer\nX-GitHub-Recipient: nicolasbock\nX-GitHub-Reason: subscribed\nList-ID: gentoo/gentoo <gentoo.gentoo.github.com>\nList-Archive: https://github.com/gentoo/gentoo\nList-Post: <mailto:reply+000cd6a9c12b39e1a7644b332064b13547f6beb1a1bc3f6e92cebac489f392a169ce194abc8b@reply.github.com>\nList-Unsubscribe: <mailto:unsub+000cd6a9c12b39e1a7644b332064b13547f6beb1a1bc3f6e92cebac489f392a169ce194abc8b@reply.github.com>,\n <https://github.com/notifications/unsubscribe/AAzWqWI1bEAMqePWre2mTLiRMBwRTkSjks5vh07zgaJpZM4cEGa_>\nX-Auto-Response-Suppress: All\nX-GitHub-Recipient-Address: nicolasbock@gmail.com\nX-Keywords: gentoo/github'
 [imap]: Write mail '[Gmail]/Trash:690892' with flags set([])
 []: deletemessageheaders: called to delete []
 []: getmessageheader: called to get message-id
 []: getmessageheader: eoh = 7157
 []: getmessageheader: headers = 'Delivered-To: nicolasbock@gmail.com\r\nReceived: by 2002:a9d:21:0:0:0:0:0 with SMTP id 30csp5503157ota;\r\n        Wed, 17 Apr 2019 09:42:28 -0700 (PDT)\r\nX-Google-Smtp-Source: APXvYqxX3NZJHrFPIj+0rcb8AkvWZ0ZkJL6+w+B8gJcqTKEGC0ZJTh3gjs0PQFFNuYkswiCvza4j\r\nX-Received: by 2002:a37:59c5:: with SMTP id n188mr64646686qkb.208.1555519348036;\r\n        Wed, 17 Apr 2019 09:42:28 -0700 (PDT)\r\nARC-Seal: i=1; a=rsa-sha256; t=1555519348; cv=none;\r\n        d=google.com; s=arc-20160816;\r\n        b=VvUdCI2cMCNbRW4UzrrFxTDVxdHwcQQaRaZsVTqU+mWjUqMj6fUpGEj7rvFG8JyCJS\r\n         sPjkxXPau4K4HzFeNh8tOnJ+4AL3qy7EKfEL3fp9R2ktMjxDqs4osl8y8Nj6aUwKaz8f\r\n         z4DJoOJWb9VzBkK1Mci8KUm4d/DKK6z8BlzutCarPS6+lH6w10JZ+WJiVW1JF32goRs5\r\n         0JlY1ITXbE2h2Cr+E5PaeiA90x2Ty1/nA2WcR8Kl8T0AifZ7g+1D0gzwqw4aTKFbNrTZ\r\n         pbI+A63SS2sM6TS/3JPRjHLkB7XmiyH6Uj7CJK1/yjUGykfncttZELcA6TVV0QqbVGJu\r\n         5DNQ==\r\nARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816;\r\n        h=list-unsubscribe:list-post:list-archive:list-id:precedence\r\n         :content-transfer-encoding:mime-version:subject:references\r\n         :in-reply-to:message-id:cc:to:reply-to:from:dkim-signature:date;\r\n        bh=nc9CK8jr8JqtXdLPxeBkBOKkjFaJmXfGwfnpxlP9+gE=;\r\n        b=jaiVVDM3cO5pC3HNUf2Y3XWQuhurPVf20I6ehaN1srsWLEOg43dZkbnbvMJ0riXWQM\r\n         2YHEc1BAJsGrEnHbRSjjlQNbKIOZ8/POyYmxIJdSgDE3ta3QuXQmf3qA3SK1T+AfbRQa\r\n         2uJ4c0GrhjjP2LVtTsvCJq7oUWufwnbIUEP8MNnqPqhzX24Phe/xhO+t9A3nwo7AXDbi\r\n         AOLVOpTt/IocY+QtTkcn+CJVCn8JIqiyV41rE/BlpnR04qRBQNwPYxwSwP0Zrk7HMEM/\r\n         UOBfZfzJeCAJL716MxbSOjKjMsEBNixiIvzQXG18U5J0QdDPHg1jgvGxch4LN7miGPlb\r\n         gsRw==\r\nARC-Authentication-Results: i=1; mx.google.com;\r\n       dkim=pass (test mode) header.i=@github.com header.s=pf2014 header.b=ZA2IhJG2;\r\n       spf=pass (google.com: domain of noreply@github.com designates 192.30.252.198 as permitted sender) smtp.mailfrom=noreply@github.com;\r\n       dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=github.com\r\nReturn-Path: <noreply@github.com>\r\nReceived: from out-7.smtp.github.com (out-7.smtp.github.com. [192.30.252.198])\r\n        by mx.google.com with ESMTPS id 50si5749913qtv.225.2019.04.17.09.42.27\r\n        for <nicolasbock@gmail.com>\r\n        (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128);\r\n        Wed, 17 Apr 2019 09:42:28 -0700 (PDT)\r\nReceived-SPF: pass (google.com: domain of noreply@github.com designates 192.30.252.198 as permitted sender) client-ip=192.30.252.198;\r\nAuthentication-Results: mx.google.com;\r\n       dkim=pass (test mode) header.i=@github.com header.s=pf2014 header.b=ZA2IhJG2;\r\n       spf=pass (google.com: domain of noreply@github.com designates 192.30.252.198 as permitted sender) smtp.mailfrom=noreply@github.com;\r\n       dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=github.com\r\nDate: Wed, 17 Apr 2019 09:42:27 -0700\r\nDKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=github.com;\r\n\ts=pf2014; t=1555519347;\r\n\tbh=nc9CK8jr8JqtXdLPxeBkBOKkjFaJmXfGwfnpxlP9+gE=;\r\n\th=Date:From:Reply-To:To:Cc:In-Reply-To:References:Subject:List-ID:\r\n\t List-Archive:List-Post:List-Unsubscribe:From;\r\n\tb=ZA2IhJG2p7mafWxB7+Q31vXHkG8lrtSCr2bklhD9/Ir5gmflqZgFp8PKCfz4jc9Rd\r\n\t BdAFxWA0QSzMYucse3OihFjrXanxcdo92njY95dCBE6K7xI5J2fhTuBDmHJlgbF5q2\r\n\t BEeBoEgnYM05w4G7tWzCR53NU67CAOTRzVOudfw0=\r\nFrom: =?UTF-8?B?VGlsbCBTY2jDpGZlcg==?= <notifications@github.com>\r\nReply-To: gentoo/gentoo <reply+000cd6a9c12b39e1a7644b332064b13547f6beb1a1bc3f6e92cebac489f392a169ce194abc8b@reply.github.com>\r\nTo: gentoo/gentoo <gentoo@noreply.github.com>\r\nCc: Subscribed <subscribed@noreply.github.com>\r\nMessage-ID: <gentoo/gentoo/pull/11463/c484169036@github.com>\r\nIn-Reply-To: <gentoo/gentoo/pull/11463@github.com>\r\nReferences: <gentoo/gentoo/pull/11463@github.com>\r\nSubject: Re: [gentoo/gentoo] Davical 1.1.8 version bump (#11463)\r\nMime-Version: 1.0\r\nContent-Type: multipart/alternative;\r\n boundary="--==_mimepart_5cb7577399b16_19493fecaf6cd9601666cf";\r\n charset=UTF-8\r\nContent-Transfer-Encoding: 7bit\r\nPrecedence: list\r\nX-GitHub-Sender: tillschaefer\r\nX-GitHub-Recipient: nicolasbock\r\nX-GitHub-Reason: subscribed\r\nList-ID: gentoo/gentoo <gentoo.gentoo.github.com>\r\nList-Archive: https://github.com/gentoo/gentoo\r\nList-Post: <mailto:reply+000cd6a9c12b39e1a7644b332064b13547f6beb1a1bc3f6e92cebac489f392a169ce194abc8b@reply.github.com>\r\nList-Unsubscribe: <mailto:unsub+000cd6a9c12b39e1a7644b332064b13547f6beb1a1bc3f6e92cebac489f392a169ce194abc8b@reply.github.com>,\r\n <https://github.com/notifications/unsubscribe/AAzWqWI1bEAMqePWre2mTLiRMBwRTkSjks5vh07zgaJpZM4cEGa_>\r\nX-Auto-Response-Suppress: All\r\nX-GitHub-Recipient-Address: nicolasbock@gmail.com\r\nX-Keywords: gentoo/github\r\n\r\n\r\n----==_mimepart_5cb7577399b16_19493fecaf6cd9601666cf\r\nContent-Type: text/plain;\r\n charset=UTF-8\r\nContent-Transfer-Encoding: 7bit\r\n\r\nAny progress here? Anything else, that blocks the request?\r\n\r\n-- \r\nYou are receiving this because you are subscribed to this thread.\r\nReply to this email directly or view it on GitHub:\r\nhttps://github.com/gentoo/gentoo/pull/11463#issuecomment-484169036\r\n----==_mimepart_5cb7577399b16_19493fecaf6cd9601666cf\r\nContent-Type: text/html;\r\n charset=UTF-8\r\nContent-Transfer-Encoding: 7bit\r\n\r\n<p>Any progress here? Anything else, that blocks the request?</p>\r\n\r\n<p style="font-size:small;-webkit-text-size-adjust:none;color:#666;">&mdash;<br />You are receiving this because you are subscribed to this thread.<br />Reply to this email directly, <a href="https://github.com/gentoo/gentoo/pull/11463#issuecomment-484169036">view it on GitHub</a>, or <a href="https://github.com/notifications/unsubscribe-auth/AAzWqc02QcRbaAIXSE3xmcUi_hs0hS9tks5vh07zgaJpZM4cEGa_">mute the thread</a>.<img src="https://github.com/notifications/beacon/AAzWqVGhYniRbECnUnJkzadTEK4FmdyKks5vh07zgaJpZM4cEGa_.gif" height="1" width="1" alt="" /></p>\r\n<script type="application/json" data-scope="inboxmarkup">{"api_version":"1.0","publisher":{"api_key":"05dde50f1d1a384dd78767c55493e4bb","name":"GitHub"},"entity":{"external_key":"github/gentoo/gentoo","title":"gentoo/gentoo","subtitle":"GitHub repository","main_image_url":"https://github.githubassets.com/images/email/message_cards/header.png","avatar_image_url":"https://github.githubassets.com/images/email/message_cards/avatar.png","action":{"name":"Open in GitHub","url":"https://github.com/gentoo/gentoo"}},"updates":{"snippets":[{"icon":"PERSON","message":"@tillschaefer in #11463: Any progress here? Anything else, that blocks the request?"}],"action":{"name":"View Pull Request","url":"https://github.com/gentoo/gentoo/pull/11463#issuecomment-484169036"}}}</script>\r\n<script type="application/ld+json">[\r\n{\r\n"@context": "http://schema.org",\r\n"@type": "EmailMessage",\r\n"potentialAction": {\r\n"@type": "ViewAction",\r\n"target": "https://github.com/gentoo/gentoo/pull/11463#issuecomment-484169036",\r\n"url": "https://github.com/gentoo/gentoo/pull/11463#issuecomment-484169036",\r\n"name": "View Pull Request"\r\n},\r\n"description": "View this Pull Request on GitHub",\r\n"publisher": {\r\n"@type": "Organization",\r\n"name": "GitHub",\r\n"url": "https://github.com"\r\n}\r\n}\r\n]</script>\r\n----==_mimepart_5cb7577399b16_19493fecaf6cd9601666cf--\r\n'
 [imap]: savemessage: date: "22-Apr-2019 16:09:18 -0600", content: 'Delivered-To: nicolasbock@gmail.com
Received: by 2002:a9d:21:0:0:0:0:0 with SMTP id 30csp5503157ota;
        Wed, 17 Apr 2019 09:42:28 -0700 (PDT)
..._mimepart_5cb7577399b16_19493fecaf6cd9601666cf--
'
 [imap]: savemessage: returning new UID 706419
 [imap]: __options2hash returning: {'X-GM-LABELS': '(gentoo/github)', 'UID': '706419'}
 nbock 7: 690892 in messagelist: True
 nbock 8: new_uid 706419
 nbock 9: 690892 in messagelist: False
 nbock 10: 690892 in messagelist: False
 nbock 2: 690892 in messagelist: False
 ERROR: while syncing [Gmail].Trash [account nicolasbock.gmail]
  690892
 ['  File "/home/nbock/venv-offlineimap/lib/python2.7/site-packages/offlineimap/folder/Base.py", line 1127, in syncmessagesto\n    action(dstfolder, statusfolder)\n', '  File "/home/nbock/venv-offlineimap/lib/python2.7/site-packages/offlineimap/folder/Base.py", line 954, in __syncmessagesto_copy\n    uid, dstfolder, statusfolder, register=0)\n', '  File "/home/nbock/venv-offlineimap/lib/python2.7/site-packages/offlineimap/folder/GmailMaildir.py", line 233, in copymessageto\n    labels = dstfolder.getmessagelabels(uid)\n', '  File "/home/nbock/venv-offlineimap/lib/python2.7/site-packages/offlineimap/folder/Gmail.py", line 105, in getmessagelabels\n    if \'labels\' in self.messagelist[uid]:\n']
 ERROR: ERROR in syncfolder for nicolasbock.gmail folder [Gmail]/Trash: Traceback (most recent call last):
  File "/home/nbock/venv-offlineimap/lib/python2.7/site-packages/offlineimap/accounts.py", line 652, in syncfolder
    localfolder.syncmessagesto(remotefolder, statusfolder)
  File "/home/nbock/venv-offlineimap/lib/python2.7/site-packages/offlineimap/folder/Base.py", line 1127, in syncmessagesto
    action(dstfolder, statusfolder)
  File "/home/nbock/venv-offlineimap/lib/python2.7/site-packages/offlineimap/folder/Base.py", line 954, in __syncmessagesto_copy
    uid, dstfolder, statusfolder, register=0)
  File "/home/nbock/venv-offlineimap/lib/python2.7/site-packages/offlineimap/folder/GmailMaildir.py", line 233, in copymessageto
    labels = dstfolder.getmessagelabels(uid)
  File "/home/nbock/venv-offlineimap/lib/python2.7/site-packages/offlineimap/folder/Gmail.py", line 105, in getmessagelabels
    if 'labels' in self.messagelist[uid]:
KeyError: 690892

  690892
nicolas33 commented 5 years ago

I don't quite understand how this is supposed to work. In https://github.com/OfflineIMAP/offlineimap/blob/master/offlineimap/folder/GmailMaildir.py#L218 copymessageto is called which removes the uid. Then in https://github.com/OfflineIMAP/offlineimap/blob/master/offlineimap/folder/GmailMaildir.py#L227 getmessagelages is called with the (now deleted) uid. Why would that work?

GmailMaildir#L212 doesn't remove the uid from the messagelist. It changes the local uid once the email is uploaded and assigned a new uid by the server.

Email added in the local maildir does not have uid, yet. They are assigned uid = -1 and are uploaded to the server. In the code, the variable realcopy should be False. I don't get why offlineimap wants to upload emails with already assigned uid in the first place.

I wonder you're moving email locally to the trash folder without renaming the filename of the email. I this something possible in your workflow?

NOTE: mutt, for example, renames the files like this when moving/copying emails to another mail folder: 1556345324.31940_5.<hostname>:2,S.

nicolas33 commented 5 years ago

Thanks for the logs. This makes things much more clear.

nicolas33 commented 5 years ago

https://github.com/OfflineIMAP/offlineimap/issues/613#issuecomment-487259011 updated.

nicolasbock commented 5 years ago

@nicolas33 Thanks for the reply! I use afew to move emails around and notmuch to read them. I haven't had any issues with this setup so far.

Do I understand you correctly that the problem is that the local mail has a UID (based on its filename) but there is no copy of it on the remote? Sorry my understanding of how IMAP works is pretty basic.

nicolas33 commented 5 years ago

Do I understand you correctly that the problem is that the local mail has a UID (based on its filename) but there is no copy of it on the remote? Sorry my understanding of how IMAP works is pretty basic.

Yes, I don't get why offlineimap tries to upload new emails with UID > 0. New emails in the local maildir are assigned the UID = -1.

However, I was wrong about the move over maildirs as the root cause. We have the folder MD5 to protect against this.

To dig into this issue, you might like to debug why these messages (= emails) are assigned a positive UID (likely by GmailMaildir or a parent object). In the offlineimap workflow, positive UIDs are stored in the filename and the status cache once they (the UIDs) are fetched from the IMAP server (at message download time or after a sucessful upload).