k9mail silently stopped to sync mails with IMAP, while other clients work.

[mmueller-kaffeeschluerfercom]

I am facing two problems here: The first is that the IMAP synchronization does not work anymore, and the second is that it fails silently, i.e. there is no notification about the error. This is bad because I rely on k9mail to be notified when I receive mails or when there is an error.

Expected behavior

IMAP mail should sync according to the time schedule I set in the settings. If there is a problem, a notification should appear. Normally, when sync is fired, the status changes to: "Poll Account:Inbox 0/200" and then the 0 is counting up the numbers of new mails in the Inbox.

Actual behavior

The sync is fired periodically like it should. The status hangs a few seconds (around 10 s) on: "Poll Account:Inbox 0/200" an then returns to "Next poll In ... minutes". No new mails are shown. No notification is shown. All other clients show newly arrived mails. I tried "Clear messages" and "Recreate data". After that, the Inbox was empty. Then, the next sync attempt downloaded 4 old messages and then failed again (silently). Every now and then, a single new mail comes through, but very few do... The problem began sometime on october 24. Before this, I never had this problem (using k9mail for some years now). If I switch to the folder view, under the "Inbox" title, there is a grey line saying "NumberFormatException: For".

I then switched on logging and the last entry is an error:

[10-28 13:53:17.798 8020:8040 E/ImapSync] synchronizeMailbox java.lang.NumberFormatException: For input string: "NIL" at java.lang.Integer.parseInt(Integer.java:521) at java.lang.Integer.parseInt(Integer.java:556) at com.fsck.k9.mail.store.imap.ImapList.getNumber(ImapList.java:48) at com.fsck.k9.mail.store.imap.ImapFolder.parseBodyStructure(ImapFolder.java:1028) at com.fsck.k9.mail.store.imap.ImapFolder.parseBodyStructure(ImapFolder.java:981) at com.fsck.k9.mail.store.imap.ImapFolder.handleFetchResponse(ImapFolder.java:885) at com.fsck.k9.mail.store.imap.ImapFolder.fetch(ImapFolder.java:737) at com.fsck.k9.controller.imap.ImapSync.downloadLargeMessages(ImapSync.java:701) at com.fsck.k9.controller.imap.ImapSync.downloadMessages(ImapSync.java:471) at com.fsck.k9.controller.imap.ImapSync.synchronizeMailboxSynchronous(ImapSync.java:258) at com.fsck.k9.controller.imap.ImapSync.sync(ImapSync.java:63) at com.fsck.k9.controller.imap.ImapMessageStore.sync(ImapMessageStore.java:25) at com.fsck.k9.controller.MessagingController.synchronizeMailboxSynchronous(MessagingController.java:742) at com.fsck.k9.controller.MessagingController$9.run(MessagingController.java:726) at com.fsck.k9.controller.MessagingController.runInBackground(MessagingController.java:206) at com.fsck.k9.controller.MessagingController.access$000(MessagingController.java:122) at com.fsck.k9.controller.MessagingController$1.run(MessagingController.java:175) at java.lang.Thread.run(Thread.java:761)

So apparently, k9mail is detecting an error. But why is there no notification about the error? I missed some important mails thinking that no mails arrived... I don't know if I have to file a separate issue for the missing notification...

And could there be a workaround for the problem? Or a fix? As the other clients (Thunderbird, Apple Mail) work, I assume it's a problem of k9mail.

Just FYI, I'm using the davmail Exchange gateway davmail.sourceforge.net but this should not be the problem as all the other clients work like a charm and k9mail did work as well until october 24.

Steps to reproduce

1. Configure an IMAP account in k9mail.
2. Hit "Check mail".
3. No (or not all) new mails are synced.
4. No notification is shown.

Environment

K-9 Mail version: 5.600

Android version: 7.1.2

Account type: IMAP (davmail)

[mmueller-kaffeeschluerfercom]

I have tested master branch. Problem is still there.

I'm not sure why, but some messages generate an ImapList with the String "NIL" on index 6, where the body size should be stored, according to ImapFolder.java:977. Maybe because of messages with empty body? I don't know.

I have have changed ImapList.java to return 0 in case of "NIL" and the problem of not syncing mails seems to be resolved:

@@ -47,6 +47,9 @@ class ImapList extends ArrayList<Object> {
     }

     public int getNumber(int index) {
+        if (index==6 && getString(index).equals("NIL"))
+            return 0;
+
         return Integer.parseInt(getString(index));
     }

I'm not sure if I'm introducing a new problem with that. Is this the right location to solve this problem?

And then, for the second problem (not showing a notification): The error is catched at ImapSync.java:233. How can I fire a notification from there?

[cketti]

RFC 3501 defines the body size (body-fld-octets) as number, with number = 1*DIGIT. So using NIL instead of a series of digits is not standard-conforming.

Now it could be the server sending an invalid response. It could also be a bug in K-9 Mail when parsing the BODYSTRUCTURE response. The easiest way to tell is if you record a debug log that includes the server response.

For error reporting there's #1058.

[mmueller-kaffeeschluerfercom]

I tried to make a debug log but I can't find out how to include the server response. I checked "Log sensitive information" but the server response is sill not in the log. Nevertheless, while debugging in android studio I believe I narrowed down the problem to one server response. I read out the inputStream buffer inside the readResponse in ImapResponseParser.java. The answer is:

* 1378 FETCH (UID 29357 BODYSTRUCTURE ("TEXT" "PLAIN" NIL NIL NIL "7BIT" NIL 0))

So it really seems to be the server sending NIL, isn't it? But then, I wonder why the other email clients (thunderbird and macos mail) accept the NIL if it is not standard...

And I'm thinking about a better way to react on a malformed server response. Now it's just silently stopping to sync. For now, I can see three scenarios:

• Add my modification NIL ==> 0 (but handles only this specific malformation)
• Skip mails generating malformed responses and notify about server problems
• Stop syncing like now but show an error notification

What do you think?

[FAUguy]

Yes, this is a problem with 5.6 and also the new 5.7 update. I've used K-9 Mail for years (with many phones) and never had an IMAP Sync issue until recently. I'm currently using a Samsung S9 Plus and Note 9, and K-9 Mail just doesn't want to sync, keeps saying Sync Disabled. I have it set to Poll every 15 min, and even that doesn't always work. I can do a manual sync, and it will them poll new emails that were sent an hour ago that didn't get delivered. What's interesting is that the included Samsung E-Mail app (which I don't like) works just fine with IMAP Sync.

[doronbehar]

I too experience some sync issues with Gmail - I delete messages on K-9 and they appear on my PC client as unread. Here is my logcat:

``` 01-04 00:19:02.529 14565 14590 I WM-WorkerWrapper: Worker result SUCCESS for Work [ id=5b4e1389-41e1-4a9a-b2fb-47966c949798, tags={ com.fsck.k9.job.MailSyncWorker, MailSync } ] 01-04 00:23:36.520 14565 14578 I com.fsck.k9: Background concurrent copying GC freed 142672(5MB) AllocSpace objects, 0(0B) LOS objects, 49% free, 6MB/12MB, paused 259us total 171.019ms 01-04 00:24:03.704 14565 14588 I WM-WorkerWrapper: Worker result SUCCESS for Work [ id=aa050a96-a589-45c0-adbe-9d52ae060ef8, tags={ com.fsck.k9.job.MailSyncWorker, MailSync } ] 01-04 00:24:04.827 14565 14588 I WM-WorkerWrapper: Worker result SUCCESS for Work [ id=092459f4-6633-47ef-92f5-cb5e7ed9a2c1, tags={ com.fsck.k9.job.MailSyncWorker, MailSync } ] 01-04 00:26:03.462 14565 17368 W Adreno-EGL: : EGL_BAD_ATTRIBUTE 01-04 00:26:03.463 14565 17368 D vndksupport: Loading /vendor/lib/hw/gralloc.msm8974.so from current namespace instead of sphal namespace. 01-04 00:26:36.509 14565 17368 W Adreno-EGL: : EGL_BAD_ATTRIBUTE 01-04 00:26:36.513 14565 17368 D vndksupport: Loading /vendor/lib/hw/gralloc.msm8974.so from current namespace instead of sphal namespace. 01-04 00:28:46.983 14565 14565 W ActivityThread: handleWindowVisibility: no activity for token android.os.BinderProxy@7a74ccb 01-04 00:28:47.201 14565 17368 W Adreno-EGL: : EGL_BAD_ATTRIBUTE 01-04 00:28:47.202 14565 17368 D vndksupport: Loading /vendor/lib/hw/gralloc.msm8974.so from current namespace instead of sphal namespace. 01-04 00:28:48.663 14565 14578 I com.fsck.k9: Background concurrent copying GC freed 19287(1089KB) AllocSpace objects, 0(0B) LOS objects, 54% free, 5MB/11MB, paused 179us total 104.658ms ```

[doronbehar]

@cketti is there any progress with this? Please tell me/us how can we help you help us debug this. I tried deleting the Gmail account and then re-adding it but still the messages deleted in k9 are not deleted in the server - both in other clients and both at mail.google.com - this is my best attempt at trying to narrow down the issue, it seems that read/unread state of messages is synced properly.

EDIT: I can confirm that deleting the account and re-adding it fixed the issue :)

[fbessou]

I also received an email which made synchronization fail. From what I understand, the second body part of the body structure is invalid:

2021-01-10 15:03:08.771 22990-23006/? V/ImapConnection: conn162279409<<<#null# [38, FETCH, [UID, 97136, BODYSTRUCTURE, [[TEXT, PLAIN, [CHARSET, utf-8], NIL, NIL, QUOTED-PRINTABLE, 2038, 41, NIL, NIL, NIL, NIL], [NIL, RELATED, [BOUNDARY, _=_swift_v4_1604479656_95b3ddbef36d89c6fac1d1baa1fd1f92_=_], NIL, NIL, NIL], ALTERNATIVE, [BOUNDARY, _=_swift_v4_1604479656_c82b80e4e05e0795b1edf3b11cdb39c2_=_], NIL, NIL, NIL]]]
2021-01-10 15:03:08.771 22990-23006/? V/ImapFolder: Stored uid '97136' for msgSeq 38 into map
2021-01-10 15:03:08.772 22990-23006/? E/ImapSync: synchronizeMailbox
    java.lang.IndexOutOfBoundsException: Index: 6, Size: 6
        at java.util.ArrayList.get(ArrayList.java:437)
        at com.fsck.k9.mail.store.imap.ImapList.getString(ImapList.java:38)
        at com.fsck.k9.mail.store.imap.ImapList.getNumber(ImapList.java:68)
        at com.fsck.k9.mail.store.imap.ImapFolder.parseBodyStructure(ImapFolder.kt:897)
        at com.fsck.k9.mail.store.imap.ImapFolder.parseBodyStructure(ImapFolder.kt:856)
        at com.fsck.k9.mail.store.imap.ImapFolder.handleFetchResponse(ImapFolder.kt:774)
        at com.fsck.k9.mail.store.imap.ImapFolder.fetch(ImapFolder.kt:622)
        at com.fsck.k9.backend.imap.ImapSync.downloadLargeMessages(ImapSync.kt:577)
        at com.fsck.k9.backend.imap.ImapSync.downloadMessages(ImapSync.kt:380)
        at com.fsck.k9.backend.imap.ImapSync.synchronizeMailboxSynchronous(ImapSync.kt:189)
        at com.fsck.k9.backend.imap.ImapSync.sync(ImapSync.kt:32)
        at com.fsck.k9.backend.imap.ImapBackend.sync(ImapBackend.kt:50)
        at com.fsck.k9.controller.MessagingController.syncFolder(MessagingController.java:657)
        at com.fsck.k9.controller.MessagingController.synchronizeMailboxSynchronous(MessagingController.java:614)
        at com.fsck.k9.controller.MessagingController.lambda$synchronizeMailbox$3$MessagingController(MessagingController.java:599)
        at com.fsck.k9.controller.-$$Lambda$MessagingController$Xx9NETpzWcoXEDzz-o_rzXywOEM.run(Unknown Source:8)
        at com.fsck.k9.controller.MessagingController.runInBackground(MessagingController.java:216)
        at com.fsck.k9.controller.MessagingController.access$000(MessagingController.java:108)
        at com.fsck.k9.controller.MessagingController$1.run(MessagingController.java:157)
        at java.lang.Thread.run(Thread.java:764)

In thunderbird, the message is fetched but is not rendered.

I suppose that the mail structure is broken leading to the imap server not producing a correct bodystructure, or something like that. A solution could be to skip mails with broken descriptions and let the user know about that.

[yaomtc]

Mail is not sending. Here is the notification I have:

Failed to send some messages Errnoexception: android_getaddrinfo failed: EPERM (Operation not permitted)

Also, in looking into this, I noticed I haven't been getting emails in my K-9 inbox for 3 days, while I can see them fine in Thunderbird.

Could this be the same issue, or should I create a separate issue?

[cketti]

@yaomtc This is a different issue.