search

nicm / fdm

fdm source code
270 stars 50 forks source link

Cannot display Unicode content #132

Closed pepa65 closed 1 year ago

pepa65 commented 1 year ago

When fdm communicates with Courier, the latter gets the impression that fdm is not implementing RFC6856 correctly when trying to send certain messages, which results in Courier wrapping the email with this message:

This E-mail message was determined to be Unicode-formatted
but your E-mail reader does not support Unicode E-mail.

Please use an E-mail reader that supports POP3 with UTF-8
(see https://tools.ietf.org/html/rfc6856.html).

This can also happen when the sender's E-mail program does not
correctly format the sent message.

The original message is included as a separate attachment
so that it can be downloaded manually.

I can solve this by switching back to Dovecot (which I've always used), but Courier was super easy to set up.

Is this an issue that would be easy or hard to fix?

nicm commented 1 year ago

Who knows? What message? What does the not correct format look like?

pepa65 commented 1 year ago

I can send you an example, I'd rather not paste it here. I thought you would have heard of it before, I think it's a 4 year old issue with Courier not being lenient and everybody switching away to Dovecot (notably all Plesk users). The subject of all these mails is "Cannot display Unicode content".

I'm hoping it's just an issue of giving a certain response in the interchange...

This is an example thread on Courier's listserve: https://www.mail-archive.com/courier-imap@lists.sourceforge.net/msg04747.html

(It happens for IMAP and POP3.)

pepa65 commented 1 year ago

This is the raw content of the email after Courier wrapped it because of the way fdm asked for it:

Received: by localhost (fdm 2.2, account "Mails") 
    with pop3 (version=TLSv1/SSLv3 ECDHE-RSA-AES256-GCM-SHA384 256 bits)
    Mon, 04 Sep 2023 04:54:03 +0700
From: Mail Delivery Subsystem <postmaster>
Subject: Cannot display Unicode content
Mime-Version: 1.0
Content-Type: multipart/mixed; boundary="=_322920-1693778043"

--=_322920-1693778043
Content-Type: text/plain

This E-mail message was determined to be Unicode-formatted
but your E-mail reader does not support Unicode E-mail.

Please use an E-mail reader that supports POP3 with UTF-8
(see https://tools.ietf.org/html/rfc6856.html).

This can also happen when the sender's E-mail program does not
correctly format the sent message.

The original message is included as a separate attachment
so that it can be downloaded manually.

--=_322920-1693778043
Content-Type: message/global

Return-Path: <return@sh.barchettigroup.net>
X-Original-To: peterkelly@passchier.net
Delivered-To: mailsink@mails.zwer.men
Received: from sh.barchettigroup.net (sh.barchettigroup.net [185.126.115.144])
    by mails.zwer.men (Postfix) with ESMTP id 7A6E620256
    for <peterkelly@passchier.net>; Mon,  4 Sep 2023 04:53:44 +0700 (+07)
MIME-Version: 1.0
Message-Id: <syfBekQ.60423.264.agk@peterkelly.edu>
From: AARP® <info@sh.barchettigroup.net>
Subject: You_have_won an #59363812
Reply-To: reply_mlto@sh.barchettigroup.net
To: peterkelly@passchier.net
Content-Type: multipart/alternative; boundary="----=_Part_608273_718083848.1689092679035"
Date: Sun, 03 Sep 2023 21:53:20 +0000

------=_Part_608273_718083848.1689092679035
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

------=_Part_608273_718083848.1689092679035
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<!DOCTYPE HTML>
<html lang=3D"en-US"> <head>  <meta http-equiv=3D"Content-Type" content=3D"=
text/html; charset=3Dutf-8"> <meta name=3D"viewport" content=3D"width=3Ddev=
ice-width, initial-scale=3D1, maximum-scale=3D1">   <style type=3D"text/css=
" data-premailer=3D"ignore">=20
@media only screen and (max-width:480px) { .footer-main-width { width: 100%=
 !important; }  .footer-mobile-hidden { display: none !important; }  .foote=
r-mobile-hidden { display: none !important; }  .footer-column { display: bl=
ock !important; }  .footer-mobile-stack { display: block !important; }  .fo=
oter-mobile-stack-padding { padding-top: 3px; } }=20
/* IE: correctly scale images with w/h attbs */ img { -ms-interpolation-mod=
e: bicubic; }=20
layout { min-width: 100%; }=20
table { table-layout: fixed; } .shell_outer-row { table-layout: auto; }=20
/* Gmail/Web viewport fix */ u + .body .shell_outer-row { width: 620px; }=
=20
@media screen {  @font-face { font-family: 'Libre Baskerville'; font-style:=
 normal; font-weight: 400; src: local('Libre Baskerville'), local('LibreBas=
kerville-Regular'), url(https://fonts.gstatic.com/s/librebaskerville/v5/kmK=
nZrc3Hgbbcjq75U4uslyuy4kn0qNZaxMaC82U.woff2) format('woff2'); unicode-range=
: U+0000-00FF, U+0131, U+0152-0153, U+02C6, U+02DA, U+02DC, U+2000-206F, U+=
2074, U+20AC, U+2212, U+2215; }  }  /* LIST AND p STYLE OVERRIDES */ .text =
text_content-cell p { margin: 0; padding: 0; margin-bottom: 0; } .text .te=
xt_content-cell ul, .text .text_content-cell ol { padding: 0; margin: 0 0 0=
 40px; } .text .text_content-cell li { padding: 0; margin: 0; /* line-heigh=
t: 1.2; Remove after testing */ } /* Text Link Style Reset */ a { text-deco=
ration: underline; } /* iOS: Autolink styles inherited */ a=
 { text-decoration: underline !important; font-size: inherit !impor=
tant; font-family: inherit !important; font-weight: inherit !important; lin=
e-height: inherit !important; color: inherit !important; } /* FF/Chrome: Sm=
ooth font rendering */ .text .text_content-cell { -webkit-font-smoothing: a=
ntialiased; -moz-osx-font-smoothing: grayscale; }=20
</style> <!--[if gte mso 9]> <style id=3D"ol-styles">=20
/* OUTLOOK-SPECIFIC STYLES */ li { text-indent: -1em; padding: 0; margin: 0=
; /* line-height: 1.2; Remove after testing */ } ul, ol { padding: 0; margi=
n: 0 0 0 40px; } p { margin: 0; padding: 0; margin-bottom: 0; }=20
</style> <![endif]-->  <style>@media only screen and (max-width:480px) {
button_content-cell {
padding-top: 10px !important; padding-right: 20px !important; padding-botto=
m: 10px !important; padding-left: 20px !important;
}
button_border-row .button_content-cell {
padding-top: 10px !important; padding-right: 20px !important; padding-botto=
m: 10px !important; padding-left: 20px !important;
}
column .content-padding-horizontal {
padding-left: 20px !important; padding-right: 20px !important;
}
layout .column .content-padding-horizontal .content-padding-horizontal {
padding-left: 0px !important; padding-right: 0px !important;
}
layout .column .content-padding-horizontal .block-wrapper_border-row .cont=
ent-padding-horizontal {
padding-left: 20px !important; padding-right: 20px !important;
}
dataTable {
overflow: auto !important;
}
dataTable .dataTable_content {
width: auto !important;
}
image--mobile-scale .image_container img {
width: auto !important;
}
image--mobile-center .image_container img {
margin-left: auto !important; margin-right: auto !important;
}
layout-margin .layout-margin_cell {
padding: 0px 20px !important;
}
layout-margin--uniform .layout-margin_cell {
padding: 20px 20px !important;
}
scale {
width: 100% !important;
}
stack {
display: block !important; box-sizing: border-box;
}
hide {
display: none !important;
}
u + .body .shell_outer-row {
width: 100% !important;
}
socialFollow_container {
text-align: center !important;
}
text .text_content-cell {
font-size: 16px !important;
}
text .text_content-cell h1 {
font-size: 24px !important;
}
text .text_content-cell h2 {
font-size: 20px !important;
}
text .text_content-cell h3 {
font-size: 20px !important;
}
text--sectionHeading .text_content-cell {
font-size: 24px !important;
}
text--heading .text_content-cell {
font-size: 24px !important;
}
text--dataTable .text_content-cell .dataTable .dataTable_content-cell {
font-size: 14px !important;
}
text--dataTable .text_content-cell .dataTable th.dataTable_content-cell {
font-size: px !important;
}
}
</style>
</head> <body class=3D"body template template--en-US" data-template-version=
=3D"1.27.0" data-canonical-name=3D"CPE-PT17183" lang=3D"en-US" align=3D"cen=
ter" style=3D"-ms-text-size-adjust: 100%; -webkit-text-size-adjust: 100%; m=
in-width: 100%; width: 100%; margin: 0px; padding: 0px;"> <div id=3D"prehea=
der" style=3D"color: transparent; display: none; font-size: 1px; line-heigh=
t: 1px; max-height: 0px; max-width: 0px; opacity: 0; overflow: hidden;"><sp=
an data-entity-ref=3D"preheader">Learn more about autism and those affected=
 by it.</span></div> <div id=3D"tracking-image" style=3D"color: transparent=
; display: none; font-size: 1px; line-height: 1px; max-height: 0px; max-wid=
th: 0px; opacity: 0; overflow: hidden;"><img src=3D"https://r20.rs6.net/on.=
jsp?ca=3D5bc5813d-22bd-4ef3-979b-1615098e3d96&a=3D1140529738186&c=3D51de4f8=
2-4749-11ee-8f6d-fa163e25e9d8&ch=3D51df3d48-4749-11ee-8f6d-fa163e25e9d8" / =
alt=3D""></div> <div class=3D"shell" lang=3D"en-US" style=3D"background-col=
or: #ffffff;">  <table class=3D"shell_panel-row" width=3D"100%" border=3D"0=
" cellpadding=3D"0" cellspacing=3D"0" style=3D"background-color: #ffffff;" =
bgcolor=3D"#ffffff"> <tr class=3D""> <td class=3D"shell_panel-cell" style=
=3D"" align=3D"center" valign=3D"top"> <table class=3D"shell_width-row scal=
e" style=3D"width: 620px;" align=3D"center" border=3D"0" cellpadding=3D"0" =
cellspacing=3D"0"> <tr> <td class=3D"shell_width-cell" style=3D"padding: 15=
px 10px;" align=3D"center" valign=3D"top"> <table class=3D"shell_content-ro=
w" width=3D"100%" align=3D"center" border=3D"0" cellpadding=3D"0" cellspaci=
ng=3D"0"> <tr> <td class=3D"shell_content-cell" style=3D"background-color: =
#FFFFFF; padding: 0; border: 0px solid #0052a0;" align=3D"center" valign=3D=
"top" bgcolor=3D"#FFFFFF"> <table class=3D"layout layout--1-column" style=
=3D"background-color: #000000; table-layout: fixed;" width=3D"100%" border=
=3D"0" cellpadding=3D"0" cellspacing=3D"0" bgcolor=3D"#000000"> <tr> <td cl=
ass=3D"column column--1 scale stack" style=3D"width: 100%;" align=3D"center=
" valign=3D"top">
<table class=3D"text text--padding-vertical" width=3D"100%" border=3D"0" ce=
llpadding=3D"0" cellspacing=3D"0" style=3D"table-layout: fixed;"> <tr> <td =
class=3D"text_content-cell content-padding-horizontal" style=3D"text-align:=
 left; font-family: Arial,Verdana,Helvetica,sans-serif; color: #000000; fon=
t-size: 16px; line-height: 1.2; display: block; word-wrap: break-word; padd=
ing: 10px 20px;" align=3D"left" valign=3D"top">
<p style=3D"text-align: center; margin: 0;" align=3D"center"><a href=3D"htt=
ps://r20.rs6.net/tn.jsp?f=3D001-urlE9Dxb73e2l_4RBUJr4Uw_3VnXiZzbhzc05IV04sn=
egB6rz85ljoGnbvZw05OMUjsxpNTj3qv4dNY5Rg0dVojRmvIdVt0mTPV8ulZnJrPyktwy3Hnd0C=
ZOTbVOnY_6qknEIJ7FHZn0jDf9jc0WShaOofKm_syOBfHFqR5h2ee7yxp5U3BlXI9S63aYLMGXp=
CcXQplStY=3D&c=3DlX1Bq00DjO0Dw_gKwbOZE50P-PL09mAvawPkNBvhXS4flhSmdyRmUw=3D=
=3D&ch=3DIS1xJ3xpySCi0XZmfBbq_0CTRD8aqF1Sp2UP2W_epfDl2dni0XMEfQ=3D=3D" targ=
et=3D"_blank" style=3D'font-family: "Libre Baskerville", serif; font-weight=
: bold; background-color: rgb(0, 0, 0); color: rgb(252, 93, 5); font-size: =
17px; text-decoration: none;'>Confirming your AARP reward  &gt;&gt;</a></p>
<p style=3D"margin: 0;">&gt;</p>
</td> </tr> </table> <table class=3D"image image--padding-vertical image--m=
obile-scale image--mobile-center" width=3D"100%" border=3D"0" cellpadding=
=3D"0" cellspacing=3D"0"> <tr> <td class=3D"image_container content-padding=
-horizontal" align=3D"center" valign=3D"top" style=3D"padding: 10px 20px;">=
 <a href=3D"https://r20.rs6.net/tn.jsp?f=3D001-urlE9Dxb73e2l_4RBUJr4Uw_3VnX=
iZzbhzc05IV04snegB6rz85ljoGnbvZw05OMUjsxpNTj3qv4dNY5Rg0dVojRmvIdVt0mTPV8ulZ=
nJrPyktwy3Hnd0CZOTbVOnY_6qknEIJ7FHZn0jDf9jc0WShaOofKm_syOBfHFqR5h2ee7yxp5U3=
BlXI9S63aYLMGXpCcXQplStY=3D&c=3DlX1Bq00DjO0Dw_gKwbOZE50P-PL09mAvawPkNBvhXS4=
flhSmdyRmUw=3D=3D&ch=3DIS1xJ3xpySCi0XZmfBbq_0CTRD8aqF1Sp2UP2W_epfDl2dni0XME=
fQ=3D=3D" data-trackable=3D"true"><img data-image-content class=3D"image_co=
ntent" width=3D"549" src=3D"https://files.constantcontact.com/ac1bedc8901/1=
8145bb9-f9eb-4bae-8cb1-91a3a4608244.png?rdr=3Dtrue" alt=3D"Autism Awareness=
 Month" style=3D"display: block; height: auto; max-width: 100%;"></a> </td>=
 </tr> </table> </td> </tr> </table> <table class=3D"layout layout--1-colum=
n" style=3D"background-color: #000000; table-layout: fixed;" width=3D"100%"=
 border=3D"0" cellpadding=3D"0" cellspacing=3D"0" bgcolor=3D"#000000"> <tr>=
 <td class=3D"column column--1 scale stack" style=3D"width: 100%;" align=3D=
"center" valign=3D"top"> <table class=3D"button button--padding-vertical" w=
idth=3D"100%" border=3D"0" cellpadding=3D"0" cellspacing=3D"0" style=3D"tab=
le-layout: fixed;"> <tr> <td class=3D"button_container content-padding-hori=
zontal" align=3D"center" style=3D"padding: 10px 20px;">    <table class=3D"=
button_content-row" style=3D"background-color: #fc5d05; width: inherit; bor=
der-radius: 10px; border-spacing: 0; border: none;" border=3D"0" cellpaddin=
g=3D"0" cellspacing=3D"0" bgcolor=3D"#fc5d05"> <tr> <td class=3D"button_con=
tent-cell" style=3D"padding: 12px 25px;" align=3D"center"> <a class=3D"butt=
on_link" href=3D"https://r20.rs6.net/tn.jsp?f=3D001-urlE9Dxb73e2l_4RBUJr4Uw=
_3VnXiZzbhzc05IV04snegB6rz85ljoGnbvZw05OMUjsxpNTj3qv4dNY5Rg0dVojRmvIdVt0mTP=
V8ulZnJrPyktwy3Hnd0CZOTbVOnY_6qknEIJ7FHZn0jDf9jc0WShaOofKm_syOBfHFqR5h2ee7y=
xp5U3BlXI9S63aYLMGXpCcXQplStY=3D&c=3DlX1Bq00DjO0Dw_gKwbOZE50P-PL09mAvawPkNB=
vhXS4flhSmdyRmUw=3D=3D&ch=3DIS1xJ3xpySCi0XZmfBbq_0CTRD8aqF1Sp2UP2W_epfDl2dn=
i0XMEfQ=3D=3D" data-trackable=3D"true" style=3D"font-weight: bold; color: #=
FFFFFF; font-family: Arial,Verdana,Helvetica,sans-serif; font-size: 16px; w=
ord-wrap: break-word; text-decoration: none;">GET STARTED, It's FREE</a> </=
td> </tr> </table>    </td> </tr> </table>   </td> </tr> </table> <table cl=
ass=3D"layout layout--1-column" style=3D"background-color: #000000; table-l=
ayout: fixed;" width=3D"100%" border=3D"0" cellpadding=3D"0" cellspacing=3D=
"0" bgcolor=3D"#000000"> <tr> <td class=3D"column column--1 scale stack" st=
yle=3D"width: 100%;" align=3D"center" valign=3D"top"><div class=3D"spacer" =
style=3D"line-height: 200px; height: 200px;">&#x200a;</div></td> </tr> </ta=
ble>  </td> </tr> </table> </td> </tr> </table> </td> </tr> <tr> <td class=
=3D"shell_panel-cell shell_panel-cell--systemFooter" style=3D"" align=3D"ce=
nter" valign=3D"top"> <table class=3D"shell_width-row scale" style=3D"width=
: 100%;" align=3D"center" border=3D"0" cellpadding=3D"0" cellspacing=3D"0">=
 <tr> <td class=3D"shell_width-cell" style=3D"padding: 0px;" align=3D"cente=
r" valign=3D"top"> <table class=3D"shell_content-row" width=3D"100%" align=
=3D"center" border=3D"0" cellpadding=3D"0" cellspacing=3D"0"> <tr> <td clas=
s=3D"shell_content-cell" style=3D"background-color: #FFFFFF; padding: 0; bo=
rder: 0 solid #0052a0;" align=3D"center" valign=3D"top" bgcolor=3D"#FFFFFF"=
> <table class=3D"layout layout--1-column" style=3D"table-layout: fixed;" w=
idth=3D"100%" border=3D"0" cellpadding=3D"0" cellspacing=3D"0"> <tr> <td cl=
ass=3D"column column--1 scale stack" style=3D"width: 100%;" align=3D"center=
" valign=3D"top"> <table class=3D"footer" width=3D"100%" border=3D"0" cellp=
adding=3D"0" cellspacing=3D"0" style=3D"font-family: Verdana,Geneva,sans-se=
rif; color: #5d5d5d; font-size: 12px;"> <tr> <td class=3D"footer_container"=
 align=3D"center"> <table class=3D"footer-container" width=3D"100%" cellpad=
ding=3D"0" cellspacing=3D"0" border=3D"0" style=3D"background-color: #fffff=
f; margin-left: auto; margin-right: auto; table-layout: auto !important;" b=
gcolor=3D"#ffffff">
<tr>
<td width=3D"100%" align=3D"center" valign=3D"top" style=3D"width: 100%;">
<div class=3D"footer-max-main-width" align=3D"center" style=3D"margin-left:=
 auto; margin-right: auto; max-width: 100%;">
<table width=3D"100%" cellpadding=3D"0" cellspacing=3D"0" border=3D"0">
<tr>
<td class=3D"footer-layout" align=3D"center" valign=3D"top" style=3D"paddin=
g: 16px 0px;">
<table class=3D"footer-main-width" style=3D"width: 580px;" border=3D"0" cel=
lpadding=3D"0" cellspacing=3D"0">
<tr>
<td class=3D"footer-text" align=3D"center" valign=3D"top" style=3D"color: #=
5d5d5d; font-family: Verdana,Geneva,sans-serif; font-size: 12px; padding: 4=
px 0px;">

</div>
</td>
</tr>
</table>
</td>
</tr>
</table>
</div>
</td>
</tr>
</table> </td> </tr> </table>   </td> </tr> </table>  </td> </tr> </table> =
</td> </tr> </table> </td> </tr>  </table> </div>  </body> </html>

op/4795_md/1/898/2685/88/37176

------=_Part_608273_718083848.1689092679035--

--=_322920-1693778043--

Thunderbird is not affected, so the mails where this happens to get shown 'normally' on IMAP, but when I POP them into my local Dovecot with fdm, they get like this.

nicm commented 1 year ago

fdm does not support RFC6856 but there is nothing to it except telling the server to use UTF-8. Doesn't Courier have an option to turn this off and just send the mail as it is?

pepa65 commented 1 year ago

It would be the best for everyone if Courier did offer some option to not do this, but as far as I know they don't... This has severely impacted their install base over the past few years, which is tragic because it is still actively maintained and tries to stay ahead of the curve.

I was hoping a simple 'lie' on behalf of fdm would be sufficient, but I understand if you don't want to go there.

nicm commented 1 year ago

Yes, it seems a bit stupid because fdm can't be the only POP3 client which was written before RFC6856 and many of them would be 8-bit clean already whether or not they say so. So having an option would be sensible.

RFC6856 literally has nothing more than the client sending a "UTF8" command to tell the server it might send UTF-8, there is nothing complicated so it would be fairly easy to add.

pepa65 commented 1 year ago

It would be fantastic if you could add it! Over the years I've build a lot of infrastructure around fdm, so I would love to put this issue to rest and keep using it.

nicm commented 1 year ago

Does this work? x.diff.txt

fdm doesn't ask for the POP3 capabilities extension to know if it can send UTF8 but I don't think we need to - there seems no reason to send CAPA which may or may not exist just to see if we can send UTF8 which may or may not exist.

pepa65 commented 1 year ago

That totally did it!! Thank you for all your work, very much appreciated!

pepa65 commented 1 year ago

(For the record, I tried getmail5 and golang's POP3 which both had the same problem. Not sure if I somehow overlooked one of their options, but fdm now has it baked in!! If issued a UTF8 command before RETR, all goes well too with golang's library.)

nicm commented 1 year ago

Great, I have applied this now and will look again if for any reason it breaks some other server. Thanks!