Header Parsing Problem?

[dkalliv]

Hi mobizt, I'm not sure if I'm missing something, but when receiving messages via IMAP, I can't get any properties populated apart from msg.msgNo and msg.UID. I tried using an ESP32 and ESP8266 (Wemos D1 Mini) with the Read_Single_Email example and library Version 1.0.8. Interestingly debug (333) outputs all the properties (see console output below). With your old library (ESP32-Mail-Client) I am able to access all the properties, but I would like to use the new library. Any help would be greatly appreciated.

Kind regards

Dkalliv

Connecting to IMAP server...

C: ESP Mail Client v1.0.8 C: connect to IMAP server C: host > imap.swizzonic-mail.ch C: port > 993 C: starting socket C: connecting to Server C: seeding the random number generator C: setting up the SSL/TLS structure C: setting hostname for TLS session C: performing the SSL handshaking C: verifying peer X.509 certificate C: certificate verified IMAP server connected C: server connected

Get the capability...

C: get the capability $ CAPABILITY

• OK swizzonic-mail.ch CommuniGate Pro IMAP Server 6.2.15 ready
• CAPABILITY IMAP4 IMAP4REV1 ACL NAMESPACE UIDPLUS IDLE LITERAL+ QUOTA ID MULTIAPPEND LISTEXT CHILDREN BINARY ESEARCH LOGIN-REFERRALS UNSELECT SASL-IR LIST-EXTENDED SPECIAL-USE CREATE-SPECIAL-USE XLIST AUTH=LOGIN AUTH=PLAIN $ OK completed

Logging in...

C: send imap command, LOGIN $ LOGIN 3do@xxxxxxxxxx.xx ***** $ OK completed

Reading the list of mailboxes...

C: send imap command, LIST $ LIST "" *

• LIST (\HasNoChildren \Marked) "/" INBOX $ OK completed

Available folders: INBOX

Selecting the INBOX folder...

C: open the mailbox folder $ EXAMINE "INBOX"

• FLAGS (\Answered \Flagged \Deleted \Seen \Draft $MDNSent $Hidden $Media $Forwarded Junk $Label1 $Label2 $Label3)
• OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft $MDNSent $Hidden $Media $Forwarded Junk $Label1 $Label2 $Label3)] limited
• 5 EXISTS
• 5 RECENT
• OK [UIDNEXT 6] predicted next UID
• OK [UIDVALIDITY 402276210] UIDs valid
• OK [UNSEEN 1] message 1 is first unseen $ OK [READ-ONLY] EXAMINE completed

Info of the selected folder Total Messages: 5 Predicted next UID: 6 Flags: \Answered, \Flagged, \Deleted, \Seen, \Draft, $MDNSent, $Hidden, $Media, $Forwarded, Junk, $Label1, $Label2, $Label3

Fetching message 1, UID: 5

C: send imap command, FETCH C: fetch message header $ UID FETCH 5 BODY.PEEK[HEADER.FIELDS (SUBJECT FROM TO DATE CC Message-ID Accept-Language content-type Content-transfer-encoding Content-Language)]

• 5 FETCH (BODY[HEADER.FIELDS ("SUBJECT" "FROM" "TO" "DATE" "CC" "Message-ID" "Accept-Language" "content-type" "Content-transfer-encoding" "Content-Language")] {282} Subject: test 5 From: 2do@xxxxxxxxxx.xx To: "3do@xxxxxxxxxx.xx" 3do@xxxxxxxxxx.xx Date: Sun, 27 Dec 2020 22:42:28 +0100 Message-ID: ximss-161881787@switchplus-mail.ch Content-Type: text/plain; charset="utf-8"; format="flowed" Content-Transfer-Encoding: 8bit UID 5) $ OK completed C: fetch body part header, 1 $ UID FETCH 5 BODY.PEEK[header.fields (content-type Content-transfer-encoding)]
• 5 FETCH (BODY[header.fields ("content-type" "Content-transfer-encoding")] {95} Content-Type: text/plain; charset="utf-8"; format="flowed" Content-Transfer-Encoding: 8bit UID 5) $ OK completed

Downloading messages...

C: download plain TEXT message $ UID FETCH 5 BODY.PEEK[TEXT] download /email_data/5/msg.txt, 100% $ OK completed

Saving message header to file...

C: Free Heap: 217404

Logging out...

C: send imap command, LOGOUT $ LOGOUT

• BYE CommuniGate Pro IMAP Server closing connection $ OK completed ! E: SSL - The peer notified us that the connection is going to be closed C: cleaning SSL connection

Message fetch cmpleted

C: message fetch completed

################################ Messsage Number: 1 Messsage UID: 5 Messsage ID: Accept Language: Content Language: From: From Charset: To: To Charset: CC: CC Charset: Date: Subject: Subject Charset: Text Message: tst Text Message Charset: utf-8 Text Message Transfer Encoding: 8bit HTML Message: HTML Message Charset: HTML Message Transfer Encoding:

Free Heap: 260020

[mobizt]

Can you view the message source? You can view the raw data of message from web browser in PC. Please remove the private data from message source and post it here. I want to check what's the problem in that message.

[dkalliv]

Below I've attached the message source of the particular message, however no messages retrieved from this provider seem to work...

Return-Path: 2xx@xxxxxxxxxxxx.xx Received: from [212.51.135.9] (account 2xx@xxxxxxxxxxxx.xx) by xxxxxxxxxxxx.xx (CommuniGate Pro XIMSS 6.2.15) with HTTPU id 161881748 for 3xx@xxxxxxxxxxxx.xx; Sun, 27 Dec 2020 22:42:28 +0100 X-Mailer: CommuniGate Pronto! HTML5 6.2.4612 Subject: test 5 MIME-Version: 1.0 From: 2xx@xxxxxxxxxxxx.xx To: "3xx@xxxxxxxxxxxx.xx" 3xx@xxxxxxxxxxxx.xx Date: Sun, 27 Dec 2020 22:42:28 +0100 Message-ID: ximss-161881787@switchplus-mail.ch Content-Type: text/plain; charset="utf-8"; format="flowed" Content-Transfer-Encoding: 8bit

tst

[mobizt]

It should be completed. Is it completed?

[dkalliv]

It is complete, it's rather short, since it is sent from another account on the same mailserver. Below is an example sent from a different mailserver (same problem). I extracted the message source with Thunderbird, as I don't know how to do it with the browser...

Return-Path: xxxx@xxxxxxxxxx.xx Received: from malta.iway.ch ([212.25.24.40] verified) by switchplus-mail.ch (CommuniGate Pro SMTP 6.2.15) with ESMTP id 161943087 for xxx@xxxxxxxxxxxxx.xx; Mon, 28 Dec 2020 13:10:11 +0100 Received: from localhost (localhost [127.0.0.1]) by localhost (Postfix) with ESMTP id 9A1D720017B for xxx@xxxxxxxxxxxxx.xx; Mon, 28 Dec 2020 13:10:11 +0100 (CET) X-Virus-Scanned: iway amavisd-new at malta.iway.ch X-Spam-Flag: NO X-Spam-Score: 0.001 X-Spam-Level: X-Spam-Status: No, score=0.001 tagged_above=-999 required=5 tests=[HTML_MESSAGE=0.001, RCVD_IN_DNSWL_BLOCKED=0.001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001] autolearn=disabled X-Spam-Report:

• 0.0 RCVD_IN_DNSWL_BLOCKED RBL: ADMINISTRATOR NOTICE: The query to DNSWL
• was blocked. See
• http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for
• more information.
• [195.186.120.133 listed in list.dnswl.org]
• -0.0 SPF_PASS SPF: sender matches SPF record
• 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record
• 0.0 HTML_MESSAGE BODY: HTML included in message
• -0.0 RCVD_IN_MSPIKE_H2 RBL: Average reputation (+2)
• [195.186.120.133 listed in wl.mailspike.net] Received: from malta.iway.ch ([127.0.0.1]) by localhost (malta.iway.ch [127.0.0.1]) (amavisd-new, port 10031) with ESMTP id DtDjE6IXsj0E for xxx@xxxxxxxxxxxxx.xx; Mon, 28 Dec 2020 13:10:06 +0100 (CET) Received-SPF: pass (bluewin.ch ... _spf.bluewin.ch: Sender is authorized to use 'xxxx@xxxxxxxxxx.xx' in 'mfrom' identity (mechanism 'include:_netblocks.bluewin.ch' matched)) receiver=malta.iway.ch; identity=mailfrom; envelope-from="xxxx@xxxxxxxxxx.xx"; client-ip=195.186.120.133 X-Greylist: client address auto-whitelisted by SQLgrey-1.8.0 Received: from vimdzmsp-sfwd05.bluewin.ch (vimdzmsp-sfwd05.bluewin.ch [195.186.120.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by malta.iway.ch (Postfix) with ESMTPS id 1C1922001DC for xxx@xxxxxxxxxxxxx.xx; Mon, 28 Dec 2020 13:10:05 +0100 (CET) Received: from X250 ([212.51.135.9]) by vimdzmsp-sfwd05.bluewin.ch Swisscom AG with ESMTPA id trLhkVvLpGCW8trLhkoe5T; Mon, 28 Dec 2020 13:10:05 +0100 X-Bluewin-Spam-Analysis: v=2.4 cv=IOzHtijG c=1 sm=1 tr=0 ts=5fe9cb1d a=JIbG404j7Vy8dhpSN5W1tg==:117 a=JIbG404j7Vy8dhpSN5W1tg==:17 a=DAwyPP_o2Byb1YXLmDAA:9 a=tENOcLu-v82Jmo2qmpMA:9 a=CjuIK1q_8ugA:10 a=yMhMjlubAAAA:8 a=SSmOFEACAAAA:8 a=R3l9YQUt1bmqkSa4GCUA:9 a=gKO2Hq4RSVkA:10 a=UiCQ7L4-1S4A:10 a=hTZeC7Yk6K0A:10 a=frz4AuCg-hUA:10 X-Bluewin-Spam-Score: 0.00 X-FXIT-IP: IPv4[212.51.135.9] Epoch[1609157405] X-Bluewin-AuthAs: xxxx@xxxxxxxxxx.xx From: "xxxxxxxxxxxxxxxxxxxxxxxxx" xxxx@xxxxxxxxxx.xx To: xxx@xxxxxxxxxxxxx.xx Subject: New Test Date: Mon, 28 Dec 2020 13:10:05 +0100 Message-ID: 000001d6dd12$60a38f90$21eaaeb0$@bluewin.ch MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_NextPart_000_0001_01D6DD1A.C267F790" X-Mailer: Microsoft Outlook 16.0 Thread-Index: AdbdEktUQskUX4hrQxSWq19l6c0wyg== Content-Language: de-ch X-CMAE-Envelope: MS4xfBFUR4Npu+JoVy2DvQ+Za21KIlXM125cai5mEJTy0nlNSY6EwbZ4p/JJWQ0AZMiKsXHu2nuPWVMfaZT96b7PqnVOaJp/3MQV6XWTvDV6NVM0nWzpih8q wv/BBYUE3+vAIBD5aSc+5o4eiZNDeqzmPkvaEk2+rgWCep8Xdf5XntfZ

This is a multipart message in MIME format.

------=_NextPart_000_0001_01D6DD1A.C267F790 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit

tst (lazy form of test)

------=_NextPart_000_0001_01D6DD1A.C267F790 Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

<html xmlns:v=3D"urn:schemas-microsoft-com:vml" = xmlns:o=3D"urn:schemas-microsoft-com:office:office" = xmlns:w=3D"urn:schemas-microsoft-com:office:word" = xmlns:m=3D"http://schemas.microsoft.com/office/2004/12/omml" = xmlns=3D"http://www.w3.org/TR/REC-html40"><META = HTTP-EQUIV=3D"Content-Type" CONTENT=3D"text/html; = charset=3Dus-ascii"><meta name=3DGenerator content=3D"Microsoft Word 15 = (filtered medium)"><body lang=3DDE-CH = link=3D"#0563C1" vlink=3D"#954F72" style=3D'word-wrap:break-word'>

tst (lazy form of = test)</o:p>

 </o:p>

<p = class=3DMsoNormal> </o:p>

------=_NextPart_000_0001_01D6DD1A.C267F790--

[mobizt]

Is this the source from the same message UID 5 that you test? I see some different from the debug. From the debug, it is only plain text (single part body) but the message source has multipart body included html and plain text.

[dkalliv]

No, this is a different message. The source for message UID 5 from the log file is the short one I've sent initially. If you want I can also send you the log file for the multipart message. The behavior is however the same...

[mobizt]

I don't want other message, I need only UID 5 full message source as you mentioned on the first post. It should complete otherwise your imap server is not RFC compliance even the message can read in the old library.

[dkalliv]

Well the short message source (see my previous message) is all I get from Thunderbird for message UID 5. Maybe I'm not extracting the source the right way. How would I extract the source using a web browser? This is not an IMAP server I run myself, but one operated by a commercial ISP. If there is a problem with RFC compliance, what is missing?

[mobizt]

It's a IMAP server which run on your domain or host service provider.

I try to register for this service too but it's not allow for international user (only swiss).

The message source should not in the short text like that. The data you post for UID5 called message envelope which missing some information of server.

Please test for me again by edit this line

from

if (imap->_imap_cmd != esp_mail_imap_cmd_search && imap->_imap_cmd != esp_mail_imap_cmd_fetch_body_text && imap->_imap_cmd != esp_mail_imap_cmd_fetch_body_attachment && imap->_imap_cmd != esp_mail_imap_cmd_fetch_body_inline)
              esp_mail_debug((const char *)response);

to

  esp_mail_debug((const char *)response);

and post the debug message.

Warning, after you edit and send me a debug message, please undo the change because it will print huge data of message body and can cause your IDE or serial monitor to freeze.

[dkalliv]

Hmm, not sure if I'm missing something here. This change didn't do much, (basically I just had to comment line 5360 in ESP_Mail_Client.cpp). Below is the debug message. If it helps, I can create a test account for you (I just need a way to send you the credentials).

Connecting to IMAP server...

C: ESP Mail Client v1.0.8 C: connect to IMAP server C: host > imap.swizzonic-mail.ch C: port > 993 C: starting socket C: connecting to Server C: seeding the random number generator C: setting up the SSL/TLS structure C: setting hostname for TLS session C: performing the SSL handshaking C: verifying peer X.509 certificate C: certificate verified IMAP server connected C: server connected

Get the capability...

C: get the capability $ CAPABILITY

• OK swizzonic-mail.ch CommuniGate Pro IMAP Server 6.2.15 ready
• CAPABILITY IMAP4 IMAP4REV1 ACL NAMESPACE UIDPLUS IDLE LITERAL+ QUOTA ID MULTIAPPEND LISTEXT CHILDREN BINARY ESEARCH LOGIN-REFERRALS UNSELECT SASL-IR LIST-EXTENDED SPECIAL-USE CREATE-SPECIAL-USE XLIST AUTH=LOGIN AUTH=PLAIN $ OK completed

Logging in...

C: send imap command, LOGIN $ LOGIN 3xx@xxxxxxxxxxxxx.xx xxxxxxxxxx $ OK completed

Reading the list of mailboxes...

C: send imap command, LIST $ LIST "" *

• LIST (\HasNoChildren \UnMarked) "/" INBOX
• LIST (\HasNoChildren \UnMarked) "/" Trash $ OK completed

Available folders: INBOX, Trash

Selecting the INBOX folder...

C: open the mailbox folder $ EXAMINE "INBOX"

• FLAGS (\Answered \Flagged \Deleted \Seen \Draft $MDNSent $Hidden $Media $Forwarded Junk $Label1 $Label2 $Label3)
• OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft $MDNSent $Hidden $Media $Forwarded Junk $Label1 $Label2 $Label3)] limited
• 7 EXISTS
• 0 RECENT
• OK [UIDNEXT 8] predicted next UID
• OK [UIDVALIDITY 402276210] UIDs valid
• OK [UNSEEN 1] message 1 is first unseen $ OK [READ-ONLY] EXAMINE completed

Info of the selected folder Total Messages: 7 Predicted next UID: 8 Flags: \Answered, \Flagged, \Deleted, \Seen, \Draft, $MDNSent, $Hidden, $Media, $Forwarded, Junk, $Label1, $Label2, $Label3

Fetching message 1, UID: 5

C: send imap command, FETCH C: fetch message header $ UID FETCH 5 BODY.PEEK[HEADER.FIELDS (SUBJECT FROM TO DATE CC Message-ID Accept-Language content-type Content-transfer-encoding Content-Language)]

• 5 FETCH (BODY[HEADER.FIELDS ("SUBJECT" "FROM" "TO" "DATE" "CC" "Message-ID" "Accept-Language" "content-type" "Content-transfer-encoding" "Content-Language")] {282} Subject: test 5 From: 2xx@xxxxxxxxxxxxx.xx To: "3xx@xxxxxxxxxxxxx.xx" 3xx@xxxxxxxxxxxxx.xx Date: Sun, 27 Dec 2020 22:42:28 +0100 Message-ID: ximss-161881787@switchplus-mail.ch Content-Type: text/plain; charset="utf-8"; format="flowed" Content-Transfer-Encoding: 8bit UID 5) $ OK completed C: fetch body part header, 1 $ UID FETCH 5 BODY.PEEK[header.fields (content-type Content-transfer-encoding)]
• 5 FETCH (BODY[header.fields ("content-type" "Content-transfer-encoding")] {95} Content-Type: text/plain; charset="utf-8"; format="flowed" Content-Transfer-Encoding: 8bit UID 5) $ OK completed

Downloading messages...

C: download plain TEXT message $ UID FETCH 5 BODY.PEEK[TEXT]

• 5 FETCH (BODY[TEXT] {5} tst UID 5) download /email_data/5/msg.txt, 100% $ OK completed $ OK completed

Saving message header to file...

C: Free Heap: 217268

Logging out...

C: send imap command, LOGOUT $ LOGOUT

• BYE CommuniGate Pro IMAP Server closing connection $ OK completed ! E: SSL - The peer notified us that the connection is going to be closed C: cleaning SSL connection

Message fetch cmpleted

C: message fetch completed

################################ Messsage Number: 1 Messsage UID: 5 Messsage ID: Accept Language: Content Language: From: From Charset: To: To Charset: CC: CC Charset: Date: Subject: Subject Charset: Text Message: tst Text Message Charset: utf-8 Text Message Transfer Encoding: 8bit HTML Message: HTML Message Charset: HTML Message Transfer Encoding:

Free Heap: 259936

[mobizt]

Thank you, I will check and update.

[dkalliv]

The test credentials are in your inbox...

[mobizt]

In your code, are you comment these line out?

config.enable.html = true;
config.enable.text = true;

If yes, the message body and header will not store in the session data unless save to file.

Then you can't see any fetch result.

The header parsed successfully then you'll se

Text Message Charset: utf-8
Text Message Transfer Encoding: 8bit

But html or text enable config is false, then no content and others header store in the session data.

[dkalliv]

I tested your library with the Read_Single_Email example, so both settings are enabled (true). When testing with other providers (Gmail for example) everything works as expected. With that particular provider however, properties like To, From, Subject, ... are not populated, while the body is there. Chances are, there are other providers out there, having the same problem. Fixing this might increase compatibility...

[mobizt]

I test with your test account and everything is ok when sending from other email to or send from and to that mailbox itself.

Can you send the message to the test account the same way you send message UID 5?

[mobizt]

The non standard may be at the smtp server that send email UID 5.

On client side, it's depending on how IMAP server that parses the message UID 5 based on the command and the information that required for identification are missing from message UID 5.

[dkalliv]

Just sent you a message through the same way as email UID 5 to your test account (UID 10). I also sent you two messages from other providers/SMTP gateways (bluewin UID 11 and Gmail UID 12). All messages show the same problem...

[mobizt]

The issue is at the reponse from this imap server when fetching the header fields.

When client sent this in a part of fetch command

BODY.PEEK[HEADER.FIELDS (SUBJECT FROM TO DATE CC Message-ID Accept-Language content-type Content-transfer-encoding Content-Language)]

Expected response (most tested imap servers)

BODY[HEADER.FIELDS (SUBJECT FROM TO DATE CC Message-ID Accept-Language content-type Content-transfer-encoding Content-Language)]

This imap server response

BODY[HEADER.FIELDS ("SUBJECT" "FROM" "TO" "DATE" "CC" "Message-ID" "Accept-Language" "content-type" "Content-transfer-encoding" "Content-Language")]

The different is the header fields returned are inside the quotes e.g. "SUBJECT" instead of SUBJECT.

This issue is fixed now, please update the library

[dkalliv]

This did the trick! Thank you, your library now also works with this IMAP server.