No extra headers are present in case of Yahoo and Aol

[beczesz]

Summary In the past couple of weeks we started to observe that using Mailcore2 we don't get the extra headers anymore for any of the messages on Yahoo and Aol.

This is working fine using MailKit (Windows version). It happens on all phones all accounts so I think others encountered with this as well.

Platform(s)

<Android><iOS, macOS>

Happens on Mail Server

<Aol>
<Yahoo>

[beczesz]

Note: on Android the connection logger is not working, so I can't attach connection logs.

[beczesz]

iOS connection logs

 8 UID FETCH 40026 (UID FLAGS RFC822.SIZE BODYSTRUCTURE INTERNALDATE BODY.PEEK[HEADER.FIELDS (Date Subject From Sender Reply-To To Cc Message-ID References In-Reply-To X-Myapp-Fid X-Myapp-Token)])

 * 2 FETCH (FLAGS ($NotJunk) UID 40026 INTERNALDATE "29-Jun-2022 16:41:14 +0000" RFC822.SIZE 29980 BODYSTRUCTURE ((("text" "plain" ("charset" "utf-8") NIL NIL "7bit" 1181 14 NIL ("inline" NIL) NIL NIL)("text" "html" ("charset" "utf-8") NIL NIL "quoted-printable" 17351 504 NIL ("inline" NIL) NIL NIL) "alternative" ("boundary" "62bc80a0_507ed7ab_c562") NIL)("application" "octet-stream" NIL NIL NIL "base64" 4878 NIL ("attachment" ("filename" "*****")) NIL NIL) "mixed" ("boundary" "62bc80a0_2eb141f2_c562") NIL) BODY[HEADER.FIELDS (DATE SUBJECT FROM SENDER REPLY-TO TO CC MESSAGE-ID REFERENCES IN-REPLY-TO X-MYAPP-FID X-MYAPP-TOKEN)] {234}
To: ****@yahoo.com
From: "****" <*****@gmail.com>
Date: Wed, 29 Jun 2022 16:41:04 +0000 (UTC)
Message-ID: <165******@myapp.com>
Subject: Sample subject

)

 8 OK UID FETCH completed

[beczesz]

@dinhvh Is there any potential workaround for this issue? This is pretty critical because a lot of functionality is dependent on the extra headers in the email headers.

Thank you in advance

[dinhvh]

Have you tried fetching all the headers instead?

[beczesz]

Thank you for your response. This is how I fetch the headers on Android

IMAPFetchMessagesOperation operation = session.fetchMessagesByUIDOperation(mFolderName, mRequestKind, nextBatch); operation.setExtraHeaders(getExtraHeaderKeyList()); operation.start(new OperationCallback() { [...]

The setExtraHeaderKeyList() according to the documentation must be set http://libmailcore.com/api/java/com/libmailcore/IMAPFetchMessagesOperation.html, but even if I set it to null, or I set it to contain only a single item then it returns none of the extra headers.

Is there another way to fetch all the headers?

[beczesz]

@dinhvh I will try to see where the problem is coming from. Could you give me some hints on where should I start it in the code?

[beczesz]

Have you tried fetching all the headers instead?

Hi @dinhvh ,

I think partly I figured out the reason why in case of Yahoo and Aol the extra headers are not downloaded. It seems that the IMAP fetch request gets all capitalized.

So on Gmail this is the IMAP request

3 FETCH (UID 33 MODSEQ (114035) INTERNALDATE "27-Jul-2022 14:31:35 +0000" FLAGS (NonJunk) ENVELOPE (   [...]  
BODY[**HEADER.FIELDS (References X-Myapp-ID)** ]

On Yahoo it will be

3 FETCH (UID 33 MODSEQ (114035) INTERNALDATE "27-Jul-2022 14:31:35 +0000" FLAGS (NonJunk) ENVELOPE (   [...]  
BODY[**HEADER.FIELDS (REFERENCES X-MYAPP-ID)** ]

Because of this capitalization Yahoo won't return any extra headers. I think the bug should be properly resolved in libetpan, but as a workaround we could use the IMAPMessagesRequestKindFullHeaders.

Unfortunately with the current Mailcore2 version is is still not working because this flag is not handled -> The request will still contain BODY[**HEADER.FIELDS (REFERENCES X-MYAPP-ID)**

I have a fix with this workaround and submit a PR soon.

@dinhvh Could you please give me a hint where should I look for this capitalization issue in libetpan? Thank you!