Not receiving emails from Zoho IMAP

quartznets commented 4 years ago

Operating System (Linux/Mac/Windows/iOS/Android): Android
Delta Chat Version: 1.12.2-fat from F-Droid
Expected behavior: Receive emails from Zoho's IMAP in DeltaChat
Actual behavior: Deltachat doesn't receive emails from Zoho's IMAP (imappro.zoho.com because it's on my own domain), while works fine with other providers (tested other account self-hosted on dockerized mailcow)

Show classic emails option is set to All

Nothing in Contact requests
Steps to reproduce the problem: Login into the Zoho email account with Deltachat, send some emails to it from other email box
Screenshots: Nothing extraordinary to screenshot, just ordinary app window with no chats (except app's default ones)
Logs: I've attached the full deltachat log to this post -> deltachat-log-20200814-001652.txt which was made on a clean instance trying to reproduce the problem However, I'd like to highlight this one here as it's noticable:

08-14 00:16:47.456 11827 11856 W DeltaChat: src/scheduler.rs:147: IMAP IDLE protocol failed to init/complete: io: Error(([42, 32, 70, 76, 65, 71, 83, 32, 40, 92, 65, 110, 115, 119, 101, 114, 101, 100, 32, 92, 70, 108, 97, 103, 103, 101, 100, 32, 92, 68, 101, 108, 101, 116, 101, 100, 32, 92, 83, 101, 101, 110, 32, 92, 68, 114, 97, 102, 116, 32, 92, 42, 41, 13, 10, 42, 32, 79, 75, 32, 91, 80, 69, 82, 77, 65, 78, 69, 78, 84, 70, 76, 65, 71, 83, 32, 40, 92, 42, 32, 92, 65, 110, 115, 119, 101, 114, 101, 100, 32, 92, 70, 108, 97, 103, 103, 101, 100, 32, 92, 68, 101, 108, 101, 116, 101, 100, 32, 92, 83, 101, 101, 110, 32, 92, 68, 114, 97, 102, 116, 41, 93, 32, 80, 101, 114, 109, 97, 110, 101, 110, 116, 32, 102, 108, 97, 103, 115, 13, 10, 65, 48, 48, 48, 51, 32, 79, 75, 32, 91, 82, 69, 65, 68, 45, 87, 82, 73, 84, 69, 93, 32, 83, 69, 76, 69, 67, 84, 32, 99, 111, 109, 112, 108, 101, 116, 101, 100, 13, 10], Alt)) during parsing of [42, 32, 70, 76, 65, 71, 83, 32, 40, 92, 65, 110, 115, 119, 101, 114, 101, 100, 32, 92, 70, 108, 97, 103, 103, 101, 100, 32, 92, 68, 101, 108, 101, 116, 101, 100, 32, 92, 83, 101, 101, 110, 32, 92, 68, 114, 97, 102, 116, 32, 92, 42, 41, 13, 10, 42, 32, 79, 75, 32, 91, 80, 69, 82, 77, 65, 78, 69, 78, 84, 70, 76, 65, 71, 83, 32, 40, 92, 42, 32, 92, 65, 110, 115, 119, 101, 114, 101, 100, 32, 92, 70, 108, 97, 103, 103, 101, 100, 32, 92, 68, 101, 108, 101, 116, 101, 100, 32, 92, 83, 101, 101, 110, 32, 92, 68, 114, 97, 102, 116, 41, 93, 32, 80, 101, 114, 109, 97, 110, 101, 110, 116, 32, 102, 108, 97, 103, 115, 13, 10, 65, 48, 48, 48, 51, 32, 79, 75, 32, 91, 82, 69, 65, 68, 45, 87, 82, 73, 84, 69, 93, 32, 83, 69, 76, 69, 67, 84, 32, 99, 111, 109, 112, 108, 101, 116, 101, 100, 13, 10]

EclecticSoftwareEngineer commented 4 years ago

I can confirm the same behaviour on Android 1.12.2 F-Droid, Zoho, custom domain. Desktop 1.12.0 Debian 64-bit seems to work fine.

gerryfrancis commented 4 years ago

The corresponding thread in the Delta Chat support forum is https://support.delta.chat/t/not-receiving-emails-in-dc-using-zoho/1158 , for reference.

gerryfrancis commented 4 years ago

@EclecticSoftwareEngineer Thank you for confirming. Could you be so kind and compare the settings of your mail account in Android and Desktop, please, to verify that they are the same? Many thanks in advance.

gerryfrancis commented 4 years ago

For reference: https://github.com/deltachat/provider-db/issues/95#issuecomment-583355159

EclecticSoftwareEngineer commented 4 years ago

All account settings on Android and Desktop were the same, except the one which stopped receiving messages (Android 1.12.2) is using imappro.zoho.com, whereas the desktop one is using imap.zoho.com. I notice now that these resolve to different IP addresses.

However, everything has been working fine up until very recently (like a couple of days ago, I think). Changing the Android instance to use imap.zoho.com does not fix the problem.

I have a couple more desktop instances I can check, but won't have access to them until the day after tomorrow.

gerryfrancis commented 4 years ago

According to Zoho, imap.zoho.eu is for "home" customers, and imappro.zoho.eu for business customers. Also the top level domain is not com, but eu. See https://www.zoho.com/mail/help/imap-access.html#alink1 .

quartznets commented 4 years ago

According to Zoho, imap.zoho.eu is for "home" customers, and imappro.zoho.eu for business customers.

imappro is also for custom domains. Custom domains users are considered 'business customers' in Zoho.

Also the top level domain is not com, but eu. See https://www.zoho.com/mail/help/imap-access.html#alink1 .

It seems we have completely different information on the same link. For me it's clearly com and it's working perfectly using any IMAP client (except Deltachat)

Screenshot_2020-08-16_00-43-06

gerryfrancis commented 4 years ago

Quite interesting, I get this on my screen:

grafik

Hocuri commented 4 years ago

The error that was posted are characters as ASCII numbers. All numbers replaced by characters:

src/scheduler.rs:147: IMAP IDLE protocol failed to init/complete: io: Error(([* FLAGS (\Answered \Flagged \Deleted \Seen \Draft \*)
* OK [PERMANENTFLAGS (\* \Answered \Flagged \Deleted \Seen \Draft)] Permanent flags
A0003 OK [READ-WRITE] SELECT completed], Alt)) during parsing of [* FLAGS (\Answered \Flagged \Deleted \Seen \Draft \*)
* OK [PERMANENTFLAGS (\* \Answered \Flagged \Deleted \Seen \Draft)] Permanent flags
A0003 OK [READ-WRITE] SELECT completed]

gerryfrancis commented 4 years ago

@Hocuri The first A0003 statement includes , Alt, while the second does not.

link2xt commented 4 years ago

Made a workaround: https://github.com/djc/tokio-imap/pull/91

link2xt commented 4 years ago

The problem is that FLAGS response is not allowed to contain \* according to https://tools.ietf.org/html/rfc3501

Could someone with a Zoho account send a request to fix it? You can point to https://github.com/djc/tokio-imap/pull/91/files

cc @quartznets @EclecticSoftwareEngineer

EclecticSoftwareEngineer commented 4 years ago

I'm happy to do that tomorrow. Will confirm when done.

Thanks for your work on this!

David

link2xt commented 4 years ago

Besides djc/tokio-imap#91 I wonder if we can do something to ignore any unknown/unparseable responses and skip to next response line. Maybe in async-imap. cc @dignifiedquire

EclecticSoftwareEngineer commented 4 years ago

I have reported this issue to Zoho, and am awaiting a response.

David Trudgett

EclecticSoftwareEngineer commented 4 years ago

I received the following reply from Zoho tech support:

Hello David,

Greetings from the Zoho mail team.

Sorry for the inadvertent delay in responding.

On checking the issue that you had reported, we found that '*' can be included in flag response according to RFC standard. And, it is mentioned in the RFC you have shared with us, https://tools.ietf.org/html/rfc3501, "The PERMANENTFLAGS list can also include the special flag *, which indicates that it is possible to create new keywords by attempting to store those flags in the mailbox." We will not be able to remove the character from our Flags response as of now.

Please check the attached file for your reference.

Do write back to us for further queries or clarifications. We will be happy to help you.

Regards, Kowsalya

link2xt commented 4 years ago

But the problem is that in this case \* appears not in PERMANENTFLAGS, but in FLAGS:

* FLAGS (\Answered \Flagged \Deleted \Seen \Draft \*)

EclecticSoftwareEngineer commented 4 years ago

Yes, that makes sense to me. There would not even be any meaning to having * in FLAGS, in that case.

I will reply to them with this clarification.

Thanks, David

EclecticSoftwareEngineer commented 4 years ago

I have sent the clarification and am now awaiting a response.

EclecticSoftwareEngineer commented 4 years ago

On general principles, I would comment that for optimum reliability and interoperability, one should be lenient in parsing incoming data, and strict in formatting outgoing data. We can't control the rest of the Internet, but we can control ourselves. In this case, it would seem to make sense to ignore and possibly log unexpected, invalid or unknown flags, rather than to fail.

link2xt commented 4 years ago

On general principles, I would comment that for optimum reliability and interoperability, one should be lenient in parsing incoming data, and strict in formatting outgoing data. We can't control the rest of the Internet, but we can control ourselves. In this case, it would seem to make sense to ignore and possibly log unexpected, invalid or unknown flags, rather than to fail.

Sure, this will be fixed when a new version of imap-proto is released and DC updates to it.

EclecticSoftwareEngineer commented 4 years ago

FWIW, I have received a response from Zoho saying that the fix for the issue should come through in a "month or two'.

gerryfrancis commented 4 years ago

@EclecticSoftwareEngineer Thanks for letting us know. :+1:

hpk42 commented 4 years ago

With a new async-email release from @dignifiedquire we can fix this issue on our side, i think.

deltachat / deltachat-core-rust

Not receiving emails from Zoho IMAP #1834