python / cpython

The Python programming language
https://www.python.org
Other
63.08k stars 30.21k forks source link

Wrong behavior for '\xff\n'.decode('gb2312', 'ignore') #56225

Closed 7b5b780f-2066-4b32-bf99-86bf942974e3 closed 13 years ago

7b5b780f-2066-4b32-bf99-86bf942974e3 commented 13 years ago
BPO 12016
Nosy @malemburg, @terryjreedy, @hyeshik, @vstinner, @ezio-melotti
Dependencies
  • bpo-12057: HZ codec has no test
  • Files
  • cjk_decode.patch
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields: ```python assignee = None closed_at = created_at = labels = ['type-bug', 'expert-unicode'] title = "Wrong behavior for '\\xff\\n'.decode('gb2312', 'ignore')" updated_at = user = 'https://bugs.python.org/cdqzzy' ``` bugs.python.org fields: ```python activity = actor = 'vstinner' assignee = 'none' closed = True closed_date = closer = 'vstinner' components = ['Unicode'] creation = creator = 'cdqzzy' dependencies = ['12057'] files = ['22241'] hgrepos = [] issue_num = 12016 keywords = ['patch'] message_count = 14.0 messages = ['135268', '135410', '135419', '135422', '135423', '135447', '135767', '135769', '137340', '137591', '137595', '137602', '140004', '140006'] nosy_count = 7.0 nosy_names = ['lemburg', 'terry.reedy', 'hyeshik.chang', 'vstinner', 'ezio.melotti', 'python-dev', 'cdqzzy'] pr_nums = [] priority = 'normal' resolution = 'fixed' stage = 'test needed' status = 'closed' superseder = None type = 'behavior' url = 'https://bugs.python.org/issue12016' versions = ['Python 3.3'] ```

    7b5b780f-2066-4b32-bf99-86bf942974e3 commented 13 years ago

    let s='\xff\n' The expected result of s.decode('gb2312', 'ignore') is u"\n", while in 2.6.6 it is u"". s can be replaced with chr(m) + chr(n) , where m is in range of 128\~255, and n in 0\~127. In the above cases, try decoding from chr(n) will never interfere with later parts in the string if there is any, since chr(n) do not start a multibyte sequence.

    terryjreedy commented 13 years ago

    u'' in 2.7.1 also, on winxp

    vstinner commented 13 years ago

    So the correct result for b'\xff\n'.decode('gb2312', 'replace') is u'?\n'?

    7b5b780f-2066-4b32-bf99-86bf942974e3 commented 13 years ago

    So the correct result for b'\xff\n'.decode('gb2312', 'replace') is u'?\n'?

    I think it should be so. This behavior does not leave out possible information, has no side-effect on later decodings, and should the '\n' indeed be redundant, an output of u'?\n' would unlikely cause confusions.

    Though, I have no knowledge on this subject code-wise. If a change of the behavior will have an impact on performance, maybe the change should not come in.

    vstinner commented 13 years ago

    _codecs_cn implements different multibyte encodings: gb2312, gbkext, gbcommon, gb18030ext, gbk, gb18030.

    And there are other Asian multibyte encodings: big5 family, ISO 2202 family, JIS family, korean encodings (KSX1001, EUC_KR, CP949, ...), Big5, CP950, ...

    All of them ignore the all bytes if one byte of a multibyte sequence is invalid (lile 0xFF 0x0A: replaced by ? instead of ?\n using replace error handler).

    I don't think that you can/should patch only one encoding: we should use the same rule for all encodings.

    By the way, do you have any document explaining which result is the good one (? or ?\n)? For UTF-8, we have well defined standards explaining exactly what to do with invalid byte sequences => see issue bpo-8271. It is easy to fix the decoders, but I would like to be sure that your proposed change is the right way to decode these encodings.

    Change the multibyte encodings can also concern the security. Read for example the following section "Check byte strings before decoding them to character strings" of my book: http://www.haypocalc.com/tmp/unicode-2011-03-25/html/issues.html#check-byte-strings-before-decoding-them-to-character-strings (https://github.com/haypo/unicode_book/wiki)

    7b5b780f-2066-4b32-bf99-86bf942974e3 commented 13 years ago

    I do not have documents on this subject. Though, I found that GNU iconv(1) behaves the same as my proposed behavior. My reading of the source code suggests that iconv(1) treat all encodings equally, which I think should also be true for python.

    As of security concerns, I do not think the change in decoding function itself would introduce any security vulnerabilities. If a security issue arises because of the proposed change, there must be improper code out side of python, which is out of python's control. That said, the proposed change is unlikely to introduce new security vulnerability, as all it does in effect is retaining a few ascii characters in the string to the output as opposed to removing. In the issue of wordpress, if we suppose that wordpress was written in python, and that the attacker was using gb2312 encoded strings instead of gbk, then my proposed change would by chance fix the issue, as the backslash would be retained when we decode the string.

    vstinner commented 13 years ago

    I asked if the change is correct on iconv mail list. Here is a copy of an answer.

    De: Bruno Haible À: [iconv mailing list] Cc: Victor Stinner Sujet: Re: [bug-gnu-libiconv] Invalid byte sequences and multiybyte encodings Date: Tue, 10 May 2011 14:52:09 +0200

    Hi,

    Someone opened an issue in Python bug tracker asking to change how invalid multibyte sequences are handled. http://bugs.python.org/issue12016

    For UTF-8 the recommended way of handling malformed input is written down in \http://www.cl.cam.ac.uk/~mgk25/ucs/examples/UTF-8-test.txt\. But the principle applies to any encoding with a variable number of bytes per character: When an invalid or malformed byte sequence is found, the smallest such byte sequence is transformed to U+FFFD (replacement character).

    In particular, normally, if the first byte that is considered "wrong" or "invalid" is a valid starter byte, the malformed byte sequence should be considered to end before that byte. If it is not a valid starter byte, then use your judgement.

    For an example implementation, see \http://git.savannah.gnu.org/gitweb/?p=gnulib.git;a=blob;f=lib/unistr/u8-mbtouc.c;hb=HEAD\ Here the return value is the number of bytes consumed. Look carefully when it is 1, 2, 3, or 4.

    b'\xffabc'.decode('gb2312', 'replace') gives "�bc". The 'a' character is seen as part of a multibyte character of 2 bytes. Because {0xFF, 0x61} is invalid in GB2312, the two bytes are replaced by U+FFFD.

    Is it the "right" way to to do?

    It is better to replace only the 0xFF byte with U+FFFD, because 0x61 is a valid first byte (even a complete character).

    UTF-8 decoder changed recently to ignore a single byte and restart the decoder, so '\xF1\x80\x41\x42\x43' is now decoded "�ABC" instead "�C". Should we do the same for all encodings?

    Generally, yes.

    Or at least for asian encodings (gb2312, gbk, gb18030, big5 family, ISO 2202 family, JIS family, EUC_KR, CP949, Big5, CP950, ...)?

    For stateful encodings of the ISO 2202 family, you may want to ignore/replace a complete escape sequence, where the syntax of escape sequences is defined through general rules.

    Bruno -- In memoriam Siegfried Rädel \http://en.wikipedia.org/wiki/Siegfried_Rädel\

    vstinner commented 13 years ago

    Oh, the HZ codec has no test! And what is this horrible BLOB, Lib/test/cjkencodings_test.py?

    vstinner commented 13 years ago

    I can now work confidently on this issue. I will try to patch all CJK decoders to only replace 1 invalid byte by U+FFFD (and not 2, 3 or 4 bytes) and try to write a test for each case (each byte sequence generating a different error).

    1762cc99-3127-4a62-9baf-30c3d0f51ef7 commented 13 years ago

    New changeset 3610841f7357 by Victor Stinner in branch '3.2': Issue bpo-12016: Reindent decoders of HK and JP codecs http://hg.python.org/cpython/rev/3610841f7357

    New changeset aa07c1237f4e by Victor Stinner in branch 'default': (Merge 3.2) Issue bpo-12016: Reindent decoders of HK and JP codecs http://hg.python.org/cpython/rev/aa07c1237f4e

    New changeset 685351d65592 by Victor Stinner in branch '2.7': Issue bpo-12016: Reindent decoders of HK and JP codecs http://hg.python.org/cpython/rev/685351d65592

    1762cc99-3127-4a62-9baf-30c3d0f51ef7 commented 13 years ago

    New changeset 8572bf1b56ec by Victor Stinner in branch '3.2': Issue bpo-12016: Add test_errorhandle() to TestBase_Mapping of http://hg.python.org/cpython/rev/8572bf1b56ec

    New changeset c3dc94d53ef8 by Victor Stinner in branch 'default': (Merge 3.2) Issue bpo-12016: Add test_errorhandle() to TestBase_Mapping of http://hg.python.org/cpython/rev/c3dc94d53ef8

    New changeset 53912b58eee6 by Victor Stinner in branch '2.7': Issue bpo-12016: Add test_errorhandle() to TestBase_Mapping of http://hg.python.org/cpython/rev/53912b58eee6

    vstinner commented 13 years ago

    cjk_decode.patch:

    Because I consider this issue as a bug, I would like to apply this patch to 2.7, 3.2 and 3.3.

    1762cc99-3127-4a62-9baf-30c3d0f51ef7 commented 13 years ago

    New changeset 16cbd84de848 by Victor Stinner in branch 'default': Issue bpo-12016: Multibyte CJK decoders now resynchronize faster http://hg.python.org/cpython/rev/16cbd84de848

    vstinner commented 13 years ago

    Because I consider this issue as a bug, I would like to apply this patch to 2.7, 3.2 and 3.3.

    It is maybe a bug but it is also an important change on Python behaviour, so finally I prefer to only change (fix) Python 3.3.

    Thanks for reporting the bug zy (cdqzzy). Tell me if it now behaves as you expected.

    I'm closing this issue because the initial issue is now fixed.