python / cpython

The Python programming language
https://www.python.org
Other
63.42k stars 30.37k forks source link

email.Generator does not separate headers with "\r\n" #42554

Closed df3f657e-c66f-4df6-bced-d446c75558df closed 14 years ago

df3f657e-c66f-4df6-bced-d446c75558df commented 19 years ago
BPO 1349106
Nosy @warsaw, @bitdancer
Files
  • email_linesep.patch
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields: ```python assignee = 'https://github.com/bitdancer' closed_at = created_at = labels = ['easy', 'type-feature', 'library'] title = 'email.Generator does not separate headers with "\\r\\n"' updated_at = user = 'https://bugs.python.org/manlioperillo' ``` bugs.python.org fields: ```python activity = actor = 'r.david.murray' assignee = 'r.david.murray' closed = True closed_date = closer = 'r.david.murray' components = ['Library (Lib)'] creation = creator = 'manlioperillo' dependencies = [] files = ['19319'] hgrepos = [] issue_num = 1349106 keywords = ['patch', 'easy'] message_count = 16.0 messages = ['54654', '54655', '54656', '54657', '54658', '54659', '54660', '54661', '54662', '97592', '108373', '119183', '119200', '119277', '119278', '119478'] nosy_count = 6.0 nosy_names = ['barry', 't-v', 'manlioperillo', 'r.david.murray', 'l0nwlf', 'Malcolm.Box'] pr_nums = [] priority = 'normal' resolution = 'accepted' stage = 'resolved' status = 'closed' superseder = None type = 'enhancement' url = 'https://bugs.python.org/issue1349106' versions = ['Python 3.2'] ```

    df3f657e-c66f-4df6-bced-d446c75558df commented 19 years ago

    Regards.

    The email.Generator module does not separates headers with "\r\n".

    Manlio Perillo

    warsaw commented 18 years ago

    Logged In: YES user_id=12800

    Correct; this is by design. If you're worried about protocols such as RFC 2821 requiring \r\n line endings, don't. The smtplib module automatically ensures proper line endings for the on-the-wire communication.

    df3f657e-c66f-4df6-bced-d446c75558df commented 18 years ago

    Logged In: YES user_id=1054957

    Ok, thanks. But what if I don't use the smtplib module?

    I discovered the bug because I have written a small NNTP server with twisted, using email module for parsing...

    warsaw commented 18 years ago

    Logged In: YES user_id=12800

    The module that speaks the wire protocol should do the conversion. IMO, there's no other way to guarantee that you're RFC compliant. You could be getting your data from the email package, but you could be getting it from anywhere else, and /that/ source may not be RFC line ended either. Since you can't change every possible source of data for NNTP or SMTP, your network interface must guarantee conformance.

    df3f657e-c66f-4df6-bced-d446c75558df commented 18 years ago

    Logged In: YES user_id=1054957

    I do not agree here (but I'm not an expert).

    First - the documentation says: """The email package attempts to be as RFC-compliant as possible, supporting in addition to RFC 2822, such MIME-related RFCs as RFC 2045, RFC 2046, RFC 2047, and RFC 2231. """

    But, as I can see, the generated email does not conform to RFC 2822.

    Second - I use email package as a "filter". read raw email text, do some processing, generate raw email text.

    Really, I don't understand why generated headers don't are separed by '\r\n' and one must rely on an external tool for the right conversion.

    Thanks.

    warsaw commented 18 years ago

    Logged In: YES user_id=12800

    I hear what you're saying, but so far, it has been more convenient for developers when the generator outputs native line endings. I can see a case for a flag or other switch on the Generator instance to force RFC 2822 line endings.

    I would suggest joining the email-sig and posting a request there so the issue can be discussed as an RFE.

    df3f657e-c66f-4df6-bced-d446c75558df commented 18 years ago

    Logged In: YES user_id=1054957

    But the generator does not output in native line endings!

    On Windows:
    >>> from email.Message import Message
    >>> msg = Message()
    >>> msg["From"] = "me"
    >>> msg["To"] = "you"
    >>> print repr(msg.as_string())
    'From: me\nTo: you\n\n'
    d79c495d-f085-4fd9-ab6f-4faab65c23c2 commented 17 years ago

    Hi,

    could you please reconsider closing this bug and consider fixing it or at least providing an option for standard behaviour? Leaving aside the question of performance impact of postprocessing in longer mails (for those, email may not a be good optionin the first place), the module as is renders the email.Generator mostly useless for multipart messages with binary data that needs to be standards compliant, e.g. Multipart-Messages containing images, possibly signed or uploading (with httplib) multipart/form-data.

    Thank you for your consideration.

    Kind regards

    Thomas

    warsaw commented 17 years ago

    I am reopening this as a feature request. I still think it's better for protocols that require these line endings to ensure that their data is standards compliant, but I can see that there may be other use cases where you'd want to generate protocol required line endings. I'm not totally convinced, but it's worth opening the issue for now and discussing this on the email-sig.

    bitdancer commented 14 years ago

    Consider also bpo-975330.

    6ebe095a-694e-4982-ab5a-3ac7a21705dd commented 14 years ago

    Echoing the comment of Thomas Viehmann, the current behaviour makes it impossible to use this library to generate a correct multipart/mixed message with 7_or_8_bit encoding on Unix.

    The MIME standard specifies that headers are to be CRLF terminated - this is independent of any lower-level transport encoding.

    Binary bodies are simply octet sequences - so may contain \n or \r characters.

    The correct MIME encoding would have the headers terminated with CRLF, and the bodies as raw byte sequences. However on Unix, since the headers are generated with \n not CRLF, you can't even post-process this (e.g. message.as_string().replace('\n', '\r\n') as that will screw up the binary body data.

    The current behaviour is basically wrong and should be fixed.

    bitdancer commented 14 years ago

    Malcolm: a Content-Transfer-Encoding of 8bit may only contain \r and \n characters as part of the line ending sequence. 8bit is *not* binary; to use a CTE of binary the SMTP server must support BINARYMIME, which I don't think is all that common yet. At any rate, my up-to-date postfix server doesn't support it.

    And in any case, the email package doesn't support the binary CTE, so it's kind of irrelevant anyway :)

    All that said, however, it certainly seems useful to be able to generate crlf terminated messages as an option. And it turns out that that capability is needed in order to be able to generate binary messages in Python3 while still remaining backward compatible with the behavior of Python2 when parsing binary messages (see bpo-10134). (Actually I think the scenario I'm finding problematic in Python3 is also problematic in Python2, it's just that there are tricks you can use to work around it in Python2 that aren't allowed in Python3 because of the byte/string separation.)

    So, attached find a patch implementing a linesep option in Generator.flatten and Header.encode.

    Note that this patch also fleshes out the currently abbreviated documentation for BytesGenerator.

    6ebe095a-694e-4982-ab5a-3ac7a21705dd commented 14 years ago

    David: Great to see a patch for this.

    You're right of course, 8bit isn't binary - I meant binary. The main place this shows up is when you're using MIME not in email (e.g. on the web), where binary transport is entirely possible.

    This fix should mean that the MIME libraries are much more usable in non-email environments. Thanks.

    bitdancer commented 14 years ago

    Updated patch that adds a missing test for BytesGenerator.flatten and fixes the bugs in it. Also added versionchanged tags to the docs for the linesep argument.

    I think this is ready to go in.

    bitdancer commented 14 years ago

    Removed some debugging cruft from the latest patch.

    bitdancer commented 14 years ago

    Committed in r85811.