42624147-0ecb-4866-950c-a39f74db61aa
commented
10 years ago Bugzilla sends e-mail in a format where =?UTF-8 is not preceded by whitespace. This makes email.headerregistry.UnstructuredHeader (and email._header_value_parser on the background) not recognise the structure.
>>> import email.headerregistry, pprint
>>> x = {}; email.headerregistry.UnstructuredHeader.parse('[Bug 64155]\tNew:=?UTF-8?Q?=20non=2Dascii=20bug=20t=C3=A9st?=;\trussian text:=?UTF-8?Q?=20=D0=90=D0=91=D0=92=D0=93=D2=90=D0=94', x); pprint.pprint(x)
{'decoded': '[Bug 64155]\tNew:=?UTF-8?Q?=20non=2Dascii=20bug=20t=C3=A9st?=;\t'
'russian text:=?UTF-8?Q?=20=D0=90=D0=91=D0=92=D0=93=D2=90=D0=94',
'parse_tree': UnstructuredTokenList([ValueTerminal('[Bug'), WhiteSpaceTerminal(' '), ValueTerminal('64155]'), WhiteSpaceTerminal('\t'), ValueTerminal('New:=?UTF-8?Q?=20non=2Dascii=20bug=20t=C3=A9st?=;'), WhiteSpaceTerminal('\t'), ValueTerminal('russian'), WhiteSpaceTerminal(' '), ValueTerminal('text:=?UTF-8?Q?=20=D0=90=D0=91=D0=92=D0=93=D2=90=D0=94')])}versus
>>> x = {}; email.headerregistry.UnstructuredHeader.parse('[Bug 64155]\tNew: =?UTF-8?Q?=20non=2Dascii=20bug=20t=C3=A9st?=;\trussian text: =?UTF-8?Q?=20=D0=90=D0=91=D0=92=D0=93=D2=90=D0=94', x); pprint.pprint(x)
{'decoded': '[Bug 64155]\tNew: non-ascii bug tést;\trussian text: АБВГҐД',
'parse_tree': UnstructuredTokenList([ValueTerminal('[Bug'), WhiteSpaceTerminal(' '), ValueTerminal('64155]'), WhiteSpaceTerminal('\t'), ValueTerminal('New:'), WhiteSpaceTerminal(' '), EncodedWord([WhiteSpaceTerminal(' '), ValueTerminal('non-ascii'), WhiteSpaceTerminal(' '), ValueTerminal('bug'), WhiteSpaceTerminal(' '), ValueTerminal('tést')]), ValueTerminal(';'), WhiteSpaceTerminal('\t'), ValueTerminal('russian'), WhiteSpaceTerminal(' '), ValueTerminal('text:'), WhiteSpaceTerminal(' '), EncodedWord([WhiteSpaceTerminal(' '), ValueTerminal('АБВГҐД')])])}I have attached the raw e-mail as attachment.
Judging by the code, this is supposed to work (while raising a Defect -- "missing whitespace before encoded word"), but the code splits by whitespace:
tok, *remainder = _wsp_splitter(value, 1)
which swallows the encoded section in one go. In a second attachment, I added a patch which 1) adds a test case for this and 2) implements a solution, but the solution is unfortunately not in the style of the rest of the module.
In the meanwhile, I've chosen a monkey-patching approach to work around the issue:
import email._header_value_parser, email.headerregistry
def get_unstructured(value):
value = value.replace("=?UTF-8?Q?=20", " =?UTF-8?Q?")
return email._header_value_parser.get_unstructured(value)
email.headerregistry.UnstructuredHeader.value_parser = staticmethod(get_unstructured)
83d2e70e-e599-4a04-b820-3814bbdb9bef
maxking
bitdancer
warsaw
miss-islington
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields: ```python assignee = None closed_at =
created_at =
labels = ['3.8', 'type-bug', '3.7', 'expert-email']
title = 'email._header_value_parser does not recognise in-line encoding changes'
updated_at =
user = 'https://bugs.python.org/valhallasw'
```
bugs.python.org fields:
```python
activity =
actor = 'miss-islington'
assignee = 'none'
closed = True
closed_date =
closer = 'maxking'
components = ['email']
creation =
creator = 'valhallasw'
dependencies = []
files = ['34984', '34985']
hgrepos = []
issue_num = 21315
keywords = ['patch']
message_count = 12.0
messages = ['216908', '238956', '342739', '342745', '342750', '342755', '342756', '343070', '343271', '344750', '344841', '351091']
nosy_count = 5.0
nosy_names = ['barry', 'r.david.murray', 'valhallasw', 'maxking', 'miss-islington']
pr_nums = ['13425', '13846', '15655']
priority = 'normal'
resolution = None
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue21315'
versions = ['Python 3.7', 'Python 3.8']
```