search

markov2 / perl5-MailTools

Various ancient e-mail related modules
http://perl.overmeer.net/CPAN
1 stars 2 forks source link

UTF-8 message text #2

Closed tester0077 closed 6 years ago

tester0077 commented 6 years ago

Recently I received a POP3 message which appears to have been generated by Perl5 Mail::Internet v2.04

My familiarity with all of the relevant standards is very rudimentary, but ... 1) The main issue I ran into was that the message text contained raw and unencoded UTF-9 sequences such as c3 89. It is my understanding that all content must be encoded to avoid using bit 8 of the byte. Clearly this is not the case.

2) the message header did not contain any indication of the MIME content or character set specification. Being as unfamiliar with the standards as I am, there might be a default for those, but I have not found any references for this. By default, even Thunderbird 52.9.1 cannot display either the message nor the source properly without forcing the display to use UTF-8

markov2 commented 6 years ago

Hi Tester,

Mail::Internet saw its first release somewhere in 1995... it is ancient code, only maintained because many books show examples, and old code is still in use.

Mail::Internet does not help people much with correctly creating emails. So, it can be used to generate any email, as long as you know what you are doing. Probably the sender made a mistake... contact him, not me. You could use Mail::Box to generate mails with a much smaller chance on violating the rules.

To respond to your points:

ad 1) emails can contain raw utf8 when the Content-Type is like "text/plain; charset=UTF-8" No problem, even when Content-Transfer-EncodingL 7bit. Read about the encodings in https://docs.microsoft.com/en-us/previous-versions/office/developer/exchange-server-2010/aa563317(v%3dexchg.140)... testing the body is simply ignored.

ad 2) The standars are many RFCs. Start reading at https://www.rfc-editor.org/info/rfc5322 Read more about the Content-Type header on page 9 (section 5) of https://www.rfc-editor.org/rfc/rfc2045.txt Section 5.2 tells me that "Content-type: text/plain; charset=us-ascii" is the default. So when you see UTF8, then the mail reader will not like it.

Mail::Internet itself expects the programmer to set the right Content-Type... but missing makes it still a correct message (according to the RFC). The programmer, however, produces a body which does not match the charset. It's not an issue for MailTools. Agree?

tester0077 commented 6 years ago

FWIW, my interest is only in being able to decode and, if necessary, to display the message content as part of my POP3 mail screener, so please forgive me if I don't seem to really understand all of the issues involved behind the scene. I am merely trying to provide some feedback when I found the library 'in the wild' Also, I quite understand your point about Mail::Internet being somewhat dated :-)

OTOH, I have also contacted the sender - their version is several versions behind the latest release on this site, but I suppose that is the inherent danger in using any library. I have no idea as to just how much the sender relies on the library, nor have I really investigated the library at all. In any case, updating the library would not help the sender. Still, I see some rather recent changes to the library and possibly some of these issues can be fixed for any new users. If the library code was used to actually send the message, then, ( IMO, ;-) ) it ought to at least provide a minimal set of defaults. Again, I am in no position to judge and cam only state what I found in message claiming to have been prepared/sent by this library. I very much appreciate the references, though, again, I am not sure at all which issues I found are due to the use of the library. As for the body not matching the (unstated) default type, I have to agree. But, if the library was also used to actually post the message, then it ought to at least warn the user about his use of bad content.

Although I don't know what version of any of the standards the library claims or is/was intended to adhere to, I would still argue, that the issue concerning "no high bit in any character" - or 7-Bit ASCII goes back far enough, that if the library does provide the facility to actually post a message, then it ought to make sure this condition is observed.

tester0077 commented 6 years ago

After doing a bit more research, I did find this MIME lint checker for e-mail message compliance to RFC 822 etc - see: https://tools.ietf.org/tools/msglint/ After pasting the received message, I got pages ( actually about almost 500 lines) of mostly ERRORs of all sorts, with most of them being of the sort: WARNING: line 34 too long (122 chars); text/plain shouldn't need folding (RFC 2046-4.1.1) ERROR: Content-Transfer-Encoding doesn't permit 8-bit characters line 36 ERROR: character set doesn't permit 8-bit characters (0303, 0xc3) line 36 ERROR: character set doesn't permit 8-bit characters (0203, 0x83) line 36 ERROR: character set doesn't permit 8-bit characters (0342, 0xe2) line 36 I don't know if you want to follow up on this and if you do, I can provide more details,

markov2 commented 6 years ago

As I explained to you, Mail::Internet is poor, and does not check whether the content headers and message body match. Mistakes are left to the user of this library.

It's not a bug of MailTools, and I therefore close this issue.