MIME renderer: wrong header line break with long subject?

[1409fbaa-f956-4d99-a567-589cf071a381]

BPO	1645148
Nosy	@warsaw, @cjw296

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields: ```python assignee = 'https://github.com/warsaw' closed_at = created_at = labels = ['library'] title = 'MIME renderer: wrong header line break with long subject?' updated_at = user = 'https://bugs.python.org/kxroberto' ``` bugs.python.org fields: ```python activity = actor = 'barry' assignee = 'barry' closed = True closed_date = closer = 'barry' components = ['Library (Lib)'] creation = creator = 'kxroberto' dependencies = [] files = [] hgrepos = [] issue_num = 1645148 keywords = [] message_count = 10.0 messages = ['31096', '31097', '31098', '31099', '31100', '31101', '31102', '66699', '66700', '84604'] nosy_count = 6.0 nosy_names = ['barry', 'ggenellina', 'kxroberto', 'aalbrecht', 'cjw296', 'kayne'] pr_nums = [] priority = 'high' resolution = 'duplicate' stage = None status = 'closed' superseder = None type = None url = 'https://bugs.python.org/issue1645148' versions = [] ```

[1409fbaa-f956-4d99-a567-589cf071a381]

>>> from email.MIMEText import MIMEText
>>> o=MIMEText('hello')
>>> o['Subject']='1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 1 2 3 4
5 6 7 8 9 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 '
>>> o.as_string()
'Content-Type: text/plain; charset="us-ascii"\nMIME-Version: 1.0\nContent-Transf
er-Encoding: 7bit\nSubject: 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8
9 1 2 3 4 5 6 7 8\n\t9 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 \n\
nhello'
>>>

The '6 7 8\n\t9 1 2 3' clashes together to 6 7 89 1 2 3 without space between 89 in usual mail readers. Is this an error and should be :

'6 7 8 \n\t9 1 2 3'

?

as there is also the space preserved in '6 7 8 9 \n\nhello'

[warsaw]

Tokio, I'm not so sure about adding that extra space. I tested all the mail readers at my disposal on Linux and OSX. I don't have any Windows machines available so I couldn't test Outlook or OE. (If you have them, please indicate what they do in a comment here!) Here's what I found:

OSX: Mail.app 2.1.1 -- the extra space produces an extra space between the 8 and 9 both in the message summary and in the Subject header in the preview pane. The no extra space message looks fine. Thunderbird 1.5.0.10 -- The extra space message produces an extra space in the message summary, while the no extra space message looks fine here. But the weird thing is that in both the extra space and non-extra space cases, the Subject header in the preview pane both have extra spaces. It looks to me like the leading tab is being inserted into the Subject header view. Entourage 11.3.3 -- The extra space shows up in both the summary and preview panes but only in the message with the extra space

Linux Thunderbird 1.5.0.10 -- both messages get a tab between the 8 and 9 in both the summary and preview pane. This is for both the extra space message and the no-extra space message; afaict, there's no difference between the two and this is definitely a difference between the Linux version and the OSX version of the same mail reader. Evolution 2.9.92 -- the extra space message produces an extra space between the 8 and 9 in both the summary and preview panes. The no-extra space message looks fine. Sylpheed Claws 2.6.0 -- the extra space message produces an extra space between the 8 and 9 in both the summary and preview panes. The no-extra space message looks file.

So anyway, afaict, the extra space is not appropriate for any of the non-Windows mail readers.

[warsaw]

Whoops, this wasn't supposed to be a response to Tokio, but instead the OP.

[1409fbaa-f956-4d99-a567-589cf071a381]

What would be the RFC definition for a correct auto-line break in a (subject) mail header line? Wouldn't it be more simple to not do any auto-line break for the subject? or is there a requirement for the line break in the RFC. Don't think any reader program would fail because of >79 char subject header lines.

[1fd7a44c-f7f2-43ed-9c9f-bafa512b8598]

Quoting RFC2822 section 2.2.3 \ftp://ftp.rfc-editor.org/in-notes/rfc2822.txt\:

"""The general rule is that wherever this standard allows for folding white space (not simply WSP characters), a CRLF may be inserted before any WSP. For example, the header field:

       Subject: This is a test

can be represented as:

       Subject: This
        is a test

[...]Unfolding is accomplished by simply removing any CRLF that is immediately followed by WSP."""

That is, folding is done by inserting ONLY the sequence CRLF before any folding whitespace. When the header is interpreted, the whitespace is NOT removed, only the CRLF. The posted Subject header should become "Subject: 1 2 3...7 8\n\r 9 1 2...'

I think this is a bug in the email.header.Header class; its __init__ says, about the continuation_ws argument: "[it] must be RFC 2822 compliant folding whitespace (usually either a space or a hard tab) which will be prepended to continuation lines.". Folding does not involve *prepending any whitespace, just inserting CRLF right before *existing whitespace.

Note that this is wrong even for the old RFC 822 (with slightly different rules for line folding.)

[1409fbaa-f956-4d99-a567-589cf071a381]

the bug appears to be quite frequent now (once one knows it :-) ). Possibly for the same reason one sees this kind of subject alteration by one space often on the usenet.

[9ab3d873-f991-4ebe-bba0-d236aa680c14]

The problem occurs, when you are using strings, not when you use the Header class directly. The reason for this behaviour is the way the Generator creates Header instances from strings: It initializes the class with continuation_ws set to '\t' instead of the default ' ' (email/generator.py in method _write_headers).

>>> from email.MIMEText import MIMEText
>>> msg = MIMEText("Just a test.")
>>> msg["Subject"] = "foo "*22
>>> print msg.as_string()
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Subject: foo foo foo foo foo foo foo foo foo foo foo foo foo foo foo foo foo
        foo foo foo foo foo 

Just a test.

>>

Tab as the whitespace character is no problem for most email clients. But i.e. Outlook removes it.

When using the Header class you get the following

>>> from email.MIMEText import MIMEText
>>> from email.header import Header
>>> msg = MIMEText("Just a test.")
>>> msg["Subject"] = Header("foo "*22)
>>> print msg.as_string()
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Subject: foo foo foo foo foo foo foo foo foo foo foo foo foo foo foo foo foo foo foo
 foo foo foo 

Just a test.

>>

The last message shows up in Outlook correctly.

[9f149e3d-bcfa-4679-865b-a72ded675531]

Just struck this myself, found Andi's solution to work.

Constructing the header using email.header stops it from breaking the line awkwardly (vs. just storing a string).

Suggest the documentation example page be updated to use header() in place of straight strings.

(I can only guess as to they why of this, it seems not to hang the indent more than a space when using header).

[9f149e3d-bcfa-4679-865b-a72ded675531]

Argg, yes, as Andi explained it's the tab (not sure how I missed that on first reading).

[cjw296]

This bug is a combination of [bpo-1974] and [bpo-5612].

[bpo-1974] is being worked on.

[bpo-5612] is fixed in Python 3, but will need more work to be fixed in Python 2, *if* anyone cares about it...

Please followup on one of these issues...