(Hi, it's me again)
I have an email in an IMAP inbox that apparently has a trailing space in the Subject. When I call Fetch, the full subject is returned including the trailing space. Later, when I call GetMessage, the Subject has been trimmed to remove the trailing space. This is causing problems for me because I am using the subject as part of a hash, so even a slight difference makes the hashes not match. I suspect this might be related to the encoding.
TL;DR: I expect the Subject, Date, Sender, and Message Id to match when returned from Fetch and GetMessage. I am probably making an incorrect assumption somewhere, possibly with encoding. Let me know if you have suggestions.
If you want the backstory, I'm trying to support both IMAP and POP3, saving messages in a local database and synchronizing periodically. As I understand it, not all POP3 servers support UIDLs, so you have to do some kind of logic to check if emails are the same (and not redownload them). This led me to implement an identification system where the base level of comparison is a hash of the date, subject, actual sender, and message Id. Even though this is an IMAP case, I can't necessarily assume the messages in my database have an IMAP unique id (e.g. it could be a locally composed message that I haven't uploaded yet). Thus, if I want to compare M local IMAP unique Ids to N server IMAP unique Ids, this would be O(M N), but if I use the hashcode, it's O (N log(M))...I think...anyway...I think you get the idea.
The subject of the message in question is "Hey Tobii, explore your new inbox". The log is below:
Connected to imaps://imap.mail.yahoo.com:993/
S: * OK [CAPABILITY IMAP4rev1 ID MOVE NAMESPACE XYMHIGHESTMODSEQ UIDPLUS LITERAL+ CHILDREN SASL-IR AUTH=PLAIN AUTH=XYMCOOKIEB64 AUTH=XOAUTH2 AUTH=OAUTHBEARER] IMAP4rev1 Hello
C: A00000000 AUTHENTICATE PLAIN **
S: A00000000 OK AUTHENTICATE completed
C: A00000001 CAPABILITY
S: CAPABILITY IMAP4rev1 ID MOVE NAMESPACE XYMHIGHESTMODSEQ UIDPLUS LITERAL+ CHILDREN
S: A00000001 OK CAPABILITY completed
C: A00000002 NAMESPACE
S: NAMESPACE (("" "/")) NIL NIL
S: A00000002 OK NAMESPACE completed
C: A00000003 LIST "" "INBOX"
S: LIST (\HasNoChildren) "/" "Inbox"
S: A00000003 OK LIST completed
C: A00000004 LIST "" ""
S: LIST (\Junk \HasNoChildren) "/" "Bulk Mail"
S: LIST (\Archive \HasNoChildren) "/" "Archive"
S: LIST (\Drafts \HasNoChildren) "/" "Draft"
S: LIST (\HasNoChildren) "/" "Inbox"
S: LIST (\Sent \HasNoChildren) "/" "Sent"
S: LIST (\Trash \HasNoChildren) "/" "Trash"
S: A00000004 OK LIST completed
C: A00000005 EXAMINE Draft
S: 0 EXISTS
S: 0 RECENT
S: OK [UIDVALIDITY 1518176805] UIDs valid
S: OK [UIDNEXT 1] Predicted next UID
S: FLAGS (\Answered \Deleted \Draft \Flagged \Seen $Forwarded $Junk $NotJunk)
S: OK [PERMANENTFLAGS ()] No permanent flags permitted
S: OK [HIGHESTMODSEQ 1]
S: A00000005 OK [READ-ONLY] EXAMINE completed; now in selected state
C: A00000006 EXAMINE Inbox
S: 4 EXISTS
S: 0 RECENT
S: OK [UIDVALIDITY 1518176805] UIDs valid
S: OK [UIDNEXT 6] Predicted next UID
S: FLAGS (\Answered \Deleted \Draft \Flagged \Seen $Forwarded $Junk $NotJunk)
S: OK [PERMANENTFLAGS ()] No permanent flags permitted
S: OK [HIGHESTMODSEQ 12]
S: A00000006 OK [READ-ONLY] EXAMINE completed; now in selected state
C: A00000007 FETCH 1: (UID FLAGS INTERNALDATE ENVELOPE)
S: 1 FETCH (FLAGS (\Seen $NotJunk) UID 1 INTERNALDATE "09-Feb-2018 11:46:48 +0000" ENVELOPE (NIL "Welcome to Yahoo!" (("Yahoo Mail" NIL "mail" "product.communications.yahoo.com")) (("Yahoo Mail" NIL "mail" "product.communications.yahoo.com")) ((NIL NIL "replies" "communications.yahoo.com")) ((NIL NIL "tobiidynavoxappdev" "yahoo.com")) NIL NIL NIL "tobiidynavoxappdev@yahoo.com.02092018114646.yahoo.com"))
S: 2 FETCH (FLAGS (\Seen $NotJunk) UID 2 INTERNALDATE "09-Feb-2018 11:54:04 +0000" ENVELOPE ("Fri, 09 Feb 2018 11:54:04 +0000" "=?UTF-8?Q?Hey=20Tobii=2C=20explore=20your=20new=20inbox=20?=" (("=?UTF-8?Q?Yahoo=20Mail?=" NIL "Yahoo" "communications.yahoo.com")) (("=?UTF-8?Q?Yahoo=20Mail?=" NIL "Yahoo" "communications.yahoo.com")) ((NIL NIL "replies" "communications.yahoo.com")) ((NIL NIL "tobiidynavoxappdev" "yahoo.com")) NIL NIL NIL "218103894.443316@communications.yahoo.com"))
S: 3 FETCH (FLAGS (\Seen $NotJunk) UID 3 INTERNALDATE "09-Feb-2018 12:10:08 +0000" ENVELOPE ("Fri, 09 Feb 2018 11:55:06 +0000" "=?UTF-8?Q?Check=20in=20anywhere=20with=20the=20app?=" (("=?UTF-8?Q?Yahoo=20Mail?=" NIL "Yahoo" "communications.yahoo.com")) (("=?UTF-8?Q?Yahoo=20Mail?=" NIL "Yahoo" "communications.yahoo.com")) ((NIL NIL "replies" "communications.yahoo.com")) ((NIL NIL "tobiidynavoxappdev" "yahoo.com")) NIL NIL NIL "218103895.238261@communications.yahoo.com"))
S: 4 FETCH (FLAGS (\Seen $NotJunk) UID 5 INTERNALDATE "09-Feb-2018 12:47:09 +0000" ENVELOPE (NIL "Security settings changed on your Yahoo account" (("Yahoo" NIL "no-reply" "cc.yahoo-inc.com")) (("Yahoo" NIL "no-reply" "cc.yahoo-inc.com")) (("Yahoo" NIL "no-reply" "cc.yahoo-inc.com")) ((NIL NIL "tobiidynavoxappdev" "yahoo.com")) NIL NIL NIL NIL))
S: A00000007 OK FETCH completed
C: A00000008 EXAMINE Sent
S: 0 EXISTS
S: 0 RECENT
S: OK [UIDVALIDITY 1518176805] UIDs valid
S: OK [UIDNEXT 1] Predicted next UID
S: FLAGS (\Answered \Deleted \Draft \Flagged \Seen $Forwarded $Junk $NotJunk)
S: OK [PERMANENTFLAGS ()] No permanent flags permitted
S: OK [HIGHESTMODSEQ 1]
S: A00000008 OK [READ-ONLY] EXAMINE completed; now in selected state
C: A00000009 EXAMINE Trash
S: 0 EXISTS
S: 0 RECENT
S: OK [UIDVALIDITY 1518176805] UIDs valid
S: OK [UIDNEXT 2] Predicted next UID
S: FLAGS (\Answered \Deleted \Draft \Flagged \Seen $Forwarded $Junk $NotJunk)
S: OK [PERMANENTFLAGS ()] No permanent flags permitted
S: OK [HIGHESTMODSEQ 4]
S: A00000009 OK [READ-ONLY] EXAMINE completed; now in selected state
C: A00000010 EXAMINE Inbox
S: 4 EXISTS
S: 0 RECENT
S: OK [UIDVALIDITY 1518176805] UIDs valid
S: OK [UIDNEXT 6] Predicted next UID
S: FLAGS (\Answered \Deleted \Draft \Flagged \Seen $Forwarded $Junk $NotJunk)
S: OK [PERMANENTFLAGS ()] No permanent flags permitted
S: OK [HIGHESTMODSEQ 12]
S: A00000010 OK [READ-ONLY] EXAMINE completed; now in selected state
C: A00000011 UID FETCH 1 (BODY.PEEK[])
S: 1 FETCH (UID 1 BODY[] {26308}
S: X-Apparently-To: tobiidynavoxappdev@yahoo.com; Fri, 09 Feb 2018 11:46:48 +0000
S: Return-Path: mail@product.communications.yahoo.com
S: Received-SPF: fail (domain of product.communications.yahoo.com does not designate 98.137.67.80 as permitted sender)
S: X-YMailISG: YOwtlwYWLDttiEL5EYoG1lg2GYrKrua2o3SvgJbdXGoIxWEQ
S: Tn6YIgl2M6hTRPlfW.BkXCbCQTcuiTDEWHPkG1grVdYHTtGTR4LuG9QILz06
S: puQ09.YAOpSxUWfponbudeP3MCHKVQnlo0Qn1jZ5amGnqMfaUQcY1oAM.agT
S: 3u6vLOEQ_TvQ9ihiwKx298tq_dsS8pPe7Y0lvpvLaSLht0BJsGFnUSxKszAF
S: LHlOnwSxYnwygqKXX6tg7wdEKt2Osx2zglhgX9HjTHRULwK5iov1nGrv.Rru
S: T.67B7UwxWRmp7RDT3LgPXIkkXjdSV0_fH7ZNFkUsJonBBznO0VaiRLynUkN
S: EvULBc8ovlpQoJysMo_vljeIbcmx9vi8X9zGkWCJb9I0d9rMl3YML1OZQkLV
S: WEPZP9QixrCeakaPzFjvrC5h8U_3G77J3BzN1zzlbslDXNgD7tg9FkIWpJO0
S: zzjerbE1daNuIYikuE4n.RXfMlgmpWzgGOOADn24_dByD1yCtn6gYVmtX0Rp
S: fC4LXMGI1.ZQHrsYErFY2K_bBwbQ4RoGBPXq5A3GJOe0hqW_aiJJ6AoT.CFA
S: egCP81rKNIgtgapa43GAgEUElNdEoN7IhzMrIfDpdfkPXkDJrmXyMcmmRwWT
S: ax5ZfJwZgf10gvvEON7SItyYPxNcOY1_VOE4hTy81iAv.aWFC2YX8X33eyDt
S: EmCztf9Aoti4JDIg5C40x5GPibiIe1OriXlX396Y.YNBLbmyU7svlcBgRVHa
S: 3K3fVQZaeABkinQljod8VejXjkm3Syv13NKN4Rjoe50Y2RKAL4EWSul32oTR
S: dAv6vT7.KdH6GwGR2OqaKoSi6kE7TieADAI7IRh8UK3tWnI267f2xMGYnVt
S: KlzrdbJr5RqrUrucEiS3v7Ytn8Uk3FJwbK6TnfX.mLpT0uZ1i3IKB2mX4DK
S: 7vfmVEUTP4aBvfrshLNFKg0aovBsbVs9QGWaJzWHaErtBlvX_wDvnKUh9dTU
S: wM.YduDCFEFJVtXwLJ9f.sGfgj6z2i1tgSFcRWbfzYIZXJC.6owgwJLBRYZM
S: om5GP3EoIhrKzCeXerZ5xkIx0AfaVwmCJ5YueanOdRhxYACFVpS2Q5DpxRbA
S: EW6MAmxRDVz69AebdotC.MIeOKWpyPW2oTy7VkhgLuOWhvymPoojyvhJtSD5
S: JCPumCy.vp1lZ_S1Ii7rKnuO6styfnaXr7PQ6_CwMKfgDyXO5wZmxHkouBSf
S: jp3uFTGcqrudENrd_QODRCoUz9NpQgcVc1PK3S1tDP3c.cU9B4CTy4_diwro
S: uNLJLFxYj2rr7MNncV.7X1cyAAUdQdwkxHl2XoYh2S2gqBR_dyT1OCGBqi2u
S: 8kG1h.JN5_PkUudwiLiT.Uvvs3s.3.6GSfrsSXPVsrnMvw_dfgp9AI8mwYfW
S: e2h41F6Z2C8AJwuqUk4BkYouIrEcCemc461cEnxMeWGgKBwoMoNF4Xr9KTH0
S: cTtGyF1KQEQq2Q--
S: X-Originating-IP: [98.137.67.80]
S: Authentication-Results: mta4313.mail.gq1.yahoo.com from=product.communications.yahoo.com; domainkeys=neutral (no sig); from=product.communications.yahoo.com; dkim=pass (ok)
S: Received: from 127.0.0.1 (EHLO sonic329-17.consmr.mail.gq1.yahoo.com) (98.137.67.80)
S: by mta4313.mail.gq1.yahoo.com with SMTPS; Fri, 09 Feb 2018 11:46:48 +0000
S: DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=product.communications.yahoo.com; s=201402-std-mrk-prd; t=1518176807; bh=+2dcRassy8Lwl0dBtg9lHe/SlbtC9aUkMHZ9W4js4FE=; h=From:Reply-To:To:Subject:From:Subject; b=jSqwTeeOcUbk9RvWKVJSyKWuD5m3NQGkRInF+kM6hvv1AXRyfu2H5z/twkoQSXpERcRhlGTo/r5F35Sgfg+DCBTNzG1mYtRkvY2sLbbkoruNV7kznyCD7LvP/UBZA9k2jy22Q2PKc13zKGQvHw9m9+4pKUhuAkCOS0L9Mr2b1q0QS0+OWb2spPTOjvjDMvsaMrimu7tPWtZDm0cWES7L3QBF2SSaGFfk14iluIwCZMQEIzMpOOuYmVqrhXGj/dzfV29QZRT366ceK92pwSn4VM8VLYkVFDLQ+gtxg59GYUyE86BPbuEKpUzMHLRKqNtnLBDbfScMktSOH/dl8JLJvg==
S: Received: from sonic.gate.mail.ne1.yahoo.com by sonic329.consmr.mail.gq1.yahoo.com with HTTP; Fri, 9 Feb 2018 11:46:47 +0000
S: From: "Yahoo Mail" mail@product.communications.yahoo.com
S: Reply-To: replies@communications.yahoo.com
S: To: tobiidynavoxappdev@yahoo.com
S: Message-ID: tobiidynavoxappdev@yahoo.com.02092018114646.yahoo.com
S: Subject: Welcome to Yahoo!
S: X-Yahoo-Newman-Property: ydirect
S: Content-Type: text/html
S: Content-Length: 22653
S:
S: <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
S:
S:
S:
S:
S:
S:
S: Yahoo!
S:
S:
S:
S:
S:
S:
S:
S:
S:
S:
S:
S:
S:
S:
S:
S:
S:
S:
S:
S:
S:
S:
S:
S:
S:
S:
S:
S:
S:
S:
S:
S:
S:
S:
S:
S:
S:
S:
S:
S:
S:
S:
S:
S:
Welcome to Yahoo,
S:
S:
S:
Tobii
S:
S:
S:
S:
S:
S:
S:
S:
S:
S: Congrats! Your Yahoo username tobiidynavoxappdev@yahoo.com
S: is now active. Take a look at all the exciting
S: destinations you have ahead.
S:
S:
S:
S:
S:
S:
S:
S:
S:
S:
S:
S:
S:
S:
S:
S:
S:
S:
S:
S:
S:
S:
S:
S:
S:
S:
S:
S:
S: Discover the latest
S: and greatest
S:
S:
S:
S:
S:
S:
S: Need an organized inbox? You got it.
S: News from Wall Street? No problem.
S: Fantasy Football smack talk? It's all here.
S:
S:
<td class=3D"txt3" align=3D"right"=
S: valign=3D"top" style=3D"font-family:'HelveticaNeueMed', Helvetica, sans-=
S: serif; font-size:17px; color:#26282a; line-height:20px; padding-top: 12px=
S: ; padding-bottom: 22px; font-weight:normal;">Time to set up your inbox yo=
S: ur way. =0A Add a splash of color with adju=
S: stable themes. =0A
<td class=3D"txt5" align=3D"left" valign=3D"top" style=3D"font-f=
S: amily:'HelveticaNeueMed', Helvetica, sans-serif; font-size:17px; color:#2=
S: 6282a; line-height:20px; padding-top: 13px; padding-bottom: 22px; font-we=
S: ight:normal;">Yahoo Mail makes it easy to get organized. =0A =
S: Take a look around. We think you=E2=80=99ll like what y=
S: ou see.=0A
You just allowed apps that may use a less secure sign-in method to access your Yahoo account. This change allows email applications that do not meet Yahoo’s recommended security standards to access your account and may leave your account more vulnerable to security risks.
S:
S:
S:
S:
S:
S:
S:
S:
S:
To better secure your Yahoo account, we strongly recommend that you:
S:
S:
S:
S:
S:
S:
S:
S:
S:
1.Turn off "Allow apps that use less secure sign in" here:
(Hi, it's me again) I have an email in an IMAP inbox that apparently has a trailing space in the Subject. When I call Fetch, the full subject is returned including the trailing space. Later, when I call GetMessage, the Subject has been trimmed to remove the trailing space. This is causing problems for me because I am using the subject as part of a hash, so even a slight difference makes the hashes not match. I suspect this might be related to the encoding.
TL;DR: I expect the Subject, Date, Sender, and Message Id to match when returned from Fetch and GetMessage. I am probably making an incorrect assumption somewhere, possibly with encoding. Let me know if you have suggestions.
If you want the backstory, I'm trying to support both IMAP and POP3, saving messages in a local database and synchronizing periodically. As I understand it, not all POP3 servers support UIDLs, so you have to do some kind of logic to check if emails are the same (and not redownload them). This led me to implement an identification system where the base level of comparison is a hash of the date, subject, actual sender, and message Id. Even though this is an IMAP case, I can't necessarily assume the messages in my database have an IMAP unique id (e.g. it could be a locally composed message that I haven't uploaded yet). Thus, if I want to compare M local IMAP unique Ids to N server IMAP unique Ids, this would be O(M N), but if I use the hashcode, it's O (N log(M))...I think...anyway...I think you get the idea.
The subject of the message in question is "Hey Tobii, explore your new inbox". The log is below: