AttributeError when parsing multipart email with invalid non-decodable Content-Transfer-Encoding

[ce351957-9491-4262-b6c6-0a2cb13dc12f]

BPO	30835
Nosy	@warsaw, @bitdancer, @ajdlinux, @miss-islington, @websurfer5
PRs	python/cpython#2544 python/cpython#13598 python/cpython#13820 python/cpython#13821
Files	testprog.py: Example program that demonstrates the error

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields: ```python assignee = None closed_at = None created_at = labels = ['3.7', '3.8', 'type-bug', 'library', 'expert-email'] title = 'AttributeError when parsing multipart email with invalid non-decodable Content-Transfer-Encoding' updated_at = user = 'https://github.com/ajdlinux' ``` bugs.python.org fields: ```python activity = actor = 'miss-islington' assignee = 'none' closed = False closed_date = None closer = None components = ['Library (Lib)', 'email'] creation = creator = 'Andrew Donnellan' dependencies = [] files = ['46989'] hgrepos = [] issue_num = 30835 keywords = ['patch'] message_count = 4.0 messages = ['297584', '297771', '344620', '344698'] nosy_count = 5.0 nosy_names = ['barry', 'r.david.murray', 'Andrew Donnellan', 'miss-islington', 'Jeffrey.Kintscher'] pr_nums = ['2544', '13598', '13820', '13821'] priority = 'normal' resolution = None stage = 'patch review' status = 'open' superseder = None type = 'behavior' url = 'https://bugs.python.org/issue30835' versions = ['Python 3.6', 'Python 3.7', 'Python 3.8'] ```

[ce351957-9491-4262-b6c6-0a2cb13dc12f]

Parsing an email containing a multipart Content-Type, along with a Content-Transfer-Encoding containing an invalid (non-ASCII-decodable) byte will fail.

email.feedparser.FeedParser._parsegen() calls "self._cur.get('content-transfer-encoding', '8bit')" to get the header.

It then tries to check whether the C-T-E is in the allowable set of ('7bit', '8bit', 'binary'), and to do so case-insensitively, it tries to convert the header to lowercase. However, because there's an invalid character in there, it's dealing with a Header object rather than a str. Hence it throws an AttributeError.

Correct behaviour would be to convert the Header to a str, see that it's not valid, and continue on to handle the defect as usual.

Thanks to Daniel Axtens for finding this bug as he was running the AFL fuzzer on the email parsing code in Patchwork (https://github.com/getpatchwork/patchwork).

Pull request incoming.

[bitdancer]

There's a deeper problem here involving how Header is used in compat32 that I've been aware of for a while but haven't had time to try to think through a fix for (there may not be one, given the history of the compat32 code). In the meantime, the proposed fix is reasonable. (It isn't needed for the new policies, but it doesn't hurt.)

[warsaw]

New changeset aa79707262f893428665ef45b5e879129abca4aa by Barry Warsaw (Abhilash Raj) in branch 'master': bpo-30835: email: Fix AttributeError when parsing invalid CTE (GH-13598) https://github.com/python/cpython/commit/aa79707262f893428665ef45b5e879129abca4aa

[miss-islington]

New changeset f62a372928fbf6a2ba722f12f069b75ca6ad16fb by Miss Islington (bot) in branch '3.7': bpo-30835: email: Fix AttributeError when parsing invalid CTE (GH-13598) https://github.com/python/cpython/commit/f62a372928fbf6a2ba722f12f069b75ca6ad16fb

[furkanonder]

@warsaw The issue seems to be solved. Can we close it?