python / cpython

The Python programming language
https://www.python.org
Other
63.73k stars 30.53k forks source link

IDNA2008 encoding is missing #61507

Open 52c31243-b163-4522-9315-f114eecf8874 opened 11 years ago

52c31243-b163-4522-9315-f114eecf8874 commented 11 years ago
BPO 17305
Nosy @loewis, @gpshead, @tiran, @bitdancer, @njsmith, @asvetlov, @ambv, @socketpair, @berkerpeksag, @Lukasa, @wumpus, @miss-islington, @epicfaace, @akulakov, @case
PRs
  • python/cpython#25208
  • python/cpython#25210
  • python/cpython#25211
  • Files
  • idna_translate.py
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields: ```python assignee = None closed_at = None created_at = labels = ['expert-SSL', 'type-feature', 'library', '3.10'] title = 'IDNA2008 encoding is missing' updated_at = user = 'https://bugs.python.org/marten' ``` bugs.python.org fields: ```python activity = actor = 'case' assignee = 'none' closed = False closed_date = None closer = None components = ['Library (Lib)', 'SSL'] creation = creator = 'marten' dependencies = [] files = ['29256'] hgrepos = [] issue_num = 17305 keywords = ['patch'] message_count = 35.0 messages = ['183104', '183106', '183144', '183147', '183149', '183159', '183160', '183199', '183202', '205009', '205034', '217092', '217218', '278493', '279904', '310923', '310924', '310943', '310988', '311007', '311009', '311013', '349642', '349643', '349840', '349855', '349886', '370635', '389932', '390246', '390288', '390291', '390296', '391980', '396587'] nosy_count = 20.0 nosy_names = ['loewis', 'gregory.p.smith', 'christian.heimes', 'r.david.murray', 'njs', 'asvetlov', 'lukasz.langa', 'socketpair', 'underrun', 'berker.peksag', 'era', 'marten', 'Lukasa', 'wumpus', 'SamWhited', 'Socob', 'miss-islington', 'epicfaace', 'andrei.avk', 'case'] pr_nums = ['25208', '25210', '25211'] priority = 'high' resolution = None stage = 'patch review' status = 'open' superseder = None type = 'enhancement' url = 'https://bugs.python.org/issue17305' versions = ['Python 3.10'] ```

    52c31243-b163-4522-9315-f114eecf8874 commented 11 years ago

    Since Python 2.3 the idna encoding is available for Internationalized Domain Names. But the current encoding doesn't work according to the latest version of the spec.

    There is a new IDNA2008 specification (RFCs 5890-5894). Although I'm not very deep into all the changes, I know that at least the nameprep has changed. For example, the German sharp S ('ß') isn't replaced by 'ss' any longer.

    The attached file shows the difference between the expected translation and the actual translation.

    bitdancer commented 11 years ago

    How are they handling interoperability?

    52c31243-b163-4522-9315-f114eecf8874 commented 11 years ago

    At least from the GNU people, two separate projects exists for this matter:

    libidn, the original IDNA translation (http://www.gnu.org/software/libidn/) libidn2, the IDNA2008 translation (http://www.gnu.org/software/libidn/libidn2/manual/libidn2.html)

    Btw.: Does Python provide a way to decode the ASCII-representation back to UTF-8?

    >>> name.encode('idna')
    'xn--mller-kva.com'
    
    >>> name.encode('idna').decode('utf-8')
    u'xn--mller-kva.com'

    Otherwise I'd look for Python bindings of libidn2 or idnkit-2.

    52c31243-b163-4522-9315-f114eecf8874 commented 11 years ago

    For the embedded Python examples, please prepend the following lines:

    from __future__ import unicode_literals
    name='müller.com'

    So regarding interoperability: Usually you only use one implementation in your code and hopefully the latest release, but in case someone needs to old one, maybe there should be a separate encodings.idna2008 class.

    bitdancer commented 11 years ago

    Does this mean the differences are only in the canonicalization of unicode values? IDNA is a wire protocol, which means that an application can't know if it is being asked to decode an idna1 or idna2 string unless there's something in the protocol that tells it. But if the differences are only on the encoding side, and an idna1 decoder will "do the right thing" with the idna2 string, then that would be interoperable. I'll have to read the standard, but I don't have time right now :)

    idna is a codec:

    >>> b'xn--mller-kva.com'.decode('idna')
    'müller.com'

    (that's python3, it'll be a unicode string in python2, obviously).

    52c31243-b163-4522-9315-f114eecf8874 commented 11 years ago

    IDNA2008 should be backwards compatible. I can try to explain it in a practical example:

    DENIC was the first registry that actually used IDNA2008 - at a time, where not even libidn2 officially included the changes required for it. This was mainly due to the point, that the German Latin Small Letter Sharp S ('ß') was treated differently to other German Umlauts ('ä', 'ö', 'ü') in the original IDNA spec: It was not punycoded, because the nameprep already replaced it by 'ss'. Replacing 'ß' with 'ss' is in general correct in German (e.g. if your keyboard doesn't allow to enter 'ß'), but then 'ä' would have to be replaced by 'ae', 'ö' by 'oe' and 'ü' by 'ue' as well.

    Punycoding 'ä', 'ö', 'ü', but not 'ß' was inconsistent and it wouldn't allow to register a domain name like straße.de, because it was translated to strasse.de. Therefor DENIC supported IDNA2008 very early to allow the registration of domain names containing 'ß'.

    The only thing I'm aware of in this situation is, that previously straße.de was translated to strasse.de, while with IDNA2008 it's being translated to xn--strae-oqa.de. So people that have hardcoded a URL containing 'ß' and who are expecting it to be translated to 'ss' would fail, because with IDNA2008 it would be translated to a different ASCII-hostname. But those people could just change 'ß' to 'ss' in their code and everything would work again.

    On the contrary, people that have registered a domain name containing 'ß' in the meantime couldn't access it right now by specifying the IDN version, because it would be translated to the wrong domain name with the current Python IDNA encoding. So the current IDNA-Encoding should be upgraded to IDNA2008.

    bitdancer commented 11 years ago

    That doesn't sound like interoperability to me, that sounds like backward incompatibility :(. I hope you are right that it only affects people with hardcoded domain names, but that is still an issue.

    In any case, since this is a new feature it can only go into Python3.4, however we decide to do it.

    52c31243-b163-4522-9315-f114eecf8874 commented 11 years ago

    I found an interesting link about this issue:

    http://www.unicode.org/faq/idn.html

    I also checked a domain name of a client that ends with 'straße.de': IE, Firefox and Chrome still use IDNA2003, Opera already does IDNA2008.

    In IDNA2008 a lot of characters aren't allowed any longer (like symbols or strike-through letters). But I think this doesn't have any practical relevance, because even while IDNA2003 formally allowed these characters, domain name registries disallowed to register internationalized domain names containing any of these characters.

    Most registries restricted the allowed characters very strong, e.g. in the .de zone you cannot use Japanese characters, only those in use within the German language. Some other registries expect you to submit a language property during the domain registration and then only special characters within that language are allowed in the domain name. Also, most registries don't allow to register a domain name that mixes different languages.

    So IDNA2008 is the future and hopefully shouldn't break a lot. I don't know of any real life use of the IDNA encoding other than DNS / URLs. I don't know how many existing modules in PyPI working with URLs already make use of the current encodings.idna class but I guess it would cause more work if they all would have to change their code to use name.encode('idna2008') or work with an outdated encoding in the end if unchanged than just silentely switching to IDNA2008 for encodings.idna and add encodings.idna2003 for those who really need the old one for some reason. Reminds me a bit on the range() / xrange() thing. Now the special new xrange() is the default and called just range() again. I guess in some years we'll look back on the IDNA2003/2008 transition the same way.

    bitdancer commented 11 years ago

    Ah, excellent, that document looks like exactly what I was looking for.

    Now, when someone is going to get around to working on this, I don't know.

    (Note that the xrange/range change was made at the Python2/Python3 boundary, where we broke backward compatibility. I doubt that we are ever going to do that kind of transition again, but we do have ways to phase in changes in the default behavior over time.)

    4e71a086-d468-4333-b7b0-fbd5381908f7 commented 11 years ago

    At least the following existing domain names are rejected by the current implementation, apparently because they are not IDNA2003-compatible.

    XN----NNC9BXA1KSA.COM XN--14-CUD4D3A.COM XN--YGB4AR5HPA.COM XN---14-00E9E9A.COM XN--MGB2DAM4BK.COM XN--6-ZHCPPA1B7A.COM XN--3-YMCCH8IVAY.COM XN--3-YMCLXLE2A3F.COM XN--4-ZHCJXA0E.COM XN--014-QQEUW.COM XN--118-Y2EK60DC2ZB.COM

    As a workaround, in the code where I needed to process these, I used a fallback to string[4:].decode('punycode'); this was in a code path where I had already lowercased the string and established that string[0:4] == 'xn--'.

    As a partial remedy, supporting a relaxed interpretation of the spec somehow would be useful; see also (tangentially) issue bpo-12263.

    52c31243-b163-4522-9315-f114eecf8874 commented 11 years ago

    There's nice library called idna on PyPI doing idna2008: https://pypi.python.org/pypi/idna/0.1

    I'd however prefer this standard encoding to be part of standard python.

    e3bb7a8a-6d23-40c5-b27f-44cffc49d48a commented 10 years ago

    It is worth noting that the do exist some domains that have been registered in the past that work with IDNA2003 but not IDNA2008.

    There definitely needs to be IDNA2008 support, for my use case I need to attempt IDNA2008 and then fall back to IDNA2003.

    When support for IDNA2008 is added, please retain support for IDNA2003.

    I would say that ideally there would be a codec that could handle both - attempt to use IDNA2008 and on error fallback to idna2003. I realize this isn't "official" but it would certainly be useful.

    61337411-43fc-4a9c-b8d5-4060aede66d0 commented 10 years ago

    I would propose this approach:

    1. Python should implement both IDNA2008 and UTS#46, and keep IDNA2003
    2. "idna" should become an alias for "idna2003".
    3. The socket module and all other place that use the "idna" encoding should use "uts46" instead.
    4. Pre-existing implementations of IDNA 2008 should be used as inspirations at best; Python will need a new implementation from scratch, one that puts all relevant tables into the unicodedata module if they aren't there already. This is in particular where the idna 0.1 library fails. The implementation should refer to the relevant parts of the specification, to be easily reviewable for correctness.

    Contributions are welcome.

    tiran commented 8 years ago

    I'm considering lack of IDNA 2008 a security issue for applications that perform DNS lookups and X.509 cert validation. Applications may end up connecting to the wrong machine and even validate the cert correctly.

    Wrong:

    >>> import socket
    >>> u'straße.de'.encode('idna')
    'strasse.de'
    >>> socket.gethostbyname(u'straße.de'.encode('idna'))
    '72.52.4.119'
    
    Correct:
    >>> import idna
    >>> idna.encode(u'straße.de')
    'xn--strae-oqa.de'
    >>> socket.gethostbyname(idna.encode(u'straße.de'))
    '81.169.145.78'
    tiran commented 8 years ago

    I reported the issue for curl, CVE-2016-8625 https://curl.haxx.se/docs/adv_20161102K.html

    874c7ef8-1afe-4082-ade7-d41534a84280 commented 6 years ago

    I am avoiding Python's built-in libraries as much as possible in my aiohttp-based crawler because of this issue, but I cannot open a connection to https://xn--ho-hia.de because there is an 'IDNA does not round-trip' raise in the python 3.6 library ssl.py code.

    Happy to provide a code sample. I guess the 500-line async crawler in Guido's book was never used on German websites.

    njsmith commented 6 years ago

    Greg: That's bpo-28414. There's currently no motion towards builtin IDNA 2008 support (this bug), but I *think* in 3.7 the ssl module will be able to handle pre-encoded A-labels like that. I'm a little confused about the exact status right now but there's been lots of dicussion about that specific issue and I think Christian is planning to get one of the relevant PRs merged ASAP.

    tiran commented 6 years ago

    A fix will land in 3.7 and maybe get backported to 3.6. Stay tuned!

    tiran commented 6 years ago

    bpo-31399 has fixed hostname matching for IDNA 2003 compatible domain names. IDNA 2008 domain names with German ß are still broken, for example:

    UnicodeError: ('IDNA does not round-trip', b'xn--knigsgchen-b4a3dun', b'xn--knigsgsschen-lcb0w')

    bitdancer commented 6 years ago

    What we need for this issue is someone volunteering to writing the code. Given how long it has already been, I don't think anyone already on the core team is going to pick it up.

    tiran commented 6 years ago

    I lack the expertise and time to implement IDNA 2008 with UTS46 codec. I considered GNU libidn2, but the library requires two more helper libraries and LGPLv3 might be an issue for us.

    njsmith commented 6 years ago

    The "obvious" solution would be to move the "idna" module into the stdlib, but someone would still have to work that out, and it's clearly not happening for 3.7.

    ccbe4879-e54b-43e7-9c63-280a74d52875 commented 5 years ago

    Why would chrome still be using IDNA 2003 to link http://straße.de to http://strasse.de?

    tiran commented 5 years ago

    You have to ask the Chrome team.

    ccbe4879-e54b-43e7-9c63-280a74d52875 commented 5 years ago

    So is the consensus that the best way to do this is to move the "idna" library to stdlib, or implement it from scratch?

    asvetlov commented 5 years ago

    There is no consensus yet, IMHO. There is a lack of resources for the issue.

    tiran commented 5 years ago

    There is no consensus yet. Besides https://pypi.org/project/idna/ we could also consider to wrap libidn2 and ship it. Both PyPI idna and libidn2 have potential licensing issues. I don't like the idea to reinvent the wheel and implement our own idna2008 codec. It's not a trivial task.

    Once Python has a working idna2008 encoder, we need to address integration into socket, ssl, http, and asyncio module.

    tiran commented 4 years ago

    BPO bpo-40845 is another case of IDNA 2003 / 2008 bug.

    e3bb7a8a-6d23-40c5-b27f-44cffc49d48a commented 3 years ago

    why the downgrade from security to enhancement and critical to high?

    this is a significant issue that can impact everything from phishing to TLS certificate domain validation and SNI.

    tiran commented 3 years ago

    The issue has been waiting for contributions for 8 years now. So far nobody has shown an interested to address the problem and contribute an IDNA 2008 codec to Python's standard library.

    gpshead commented 3 years ago

    My PR merely adds a note to the docs linking to idna on pypi. Don't get excited, it doesn't implement anything. :P

    re "Once Python has a working idna2008 encoder, we need to address integration into socket, ssl, http, and asyncio module."

    ... doing that _could_ be the same can of worms the browsers all had to go through? We'd need to decide which behavior we wanted; pure? or matching what browsers do? I suspect that is equivalent to the pypi idna https://github.com/kjd/idna 's uts46=True + transitional=True mode [*] but anyone doing this work would need to figure that out for sure if we wanted to default to behaving like browsers with the transitional compatibility mode.

    That there is a need for a couple options on top of idna2008 as an encoding suggests it may not be a great fit for the Python codecs encodings system as those use a single string name. We'd need to permute the useful possible combos of flag behavior in the names. idna2003, idna2008, idna2008uts46, idna2008uts46transitional, and other combos of those if alternate combinations are deemed relevant.

    I worry that a browser-transitional-behavior-matching situation may change over time as TLDs decide when to change their policies. Is that an irrational fear? Browsers are well equipped to deal with this as they've got frequent updates. A PyPI package could as well.

    [*] Browser history:

    fwiw people wondering _why_ browsers like Chrome and Firefox don't "just blindly use idna2008 for everything" should go read the backwards compatibility transitional rationale and security concerns in https://bugs.chromium.org/p/chromium/issues/detail?id=61328 and https://bugzilla.mozilla.org/show_bug.cgi?id=479520

    (caution: be ready to filter out the random internet whiners from those threads)

    miss-islington commented 3 years ago

    New changeset 1d023e374cf96d143b065242131ddc9b889f9a1e by Gregory P. Smith in branch 'master': bpo-17305: Link to the third-party idna package. (GH-25208) https://github.com/python/cpython/commit/1d023e374cf96d143b065242131ddc9b889f9a1e

    miss-islington commented 3 years ago

    New changeset c7ccb0ff61e443633d0c54cb18b5633a8e95b30c by Miss Islington (bot) in branch '3.9': bpo-17305: Link to the third-party idna package. (GH-25208) https://github.com/python/cpython/commit/c7ccb0ff61e443633d0c54cb18b5633a8e95b30c

    ambv commented 3 years ago

    New changeset 2760a6711b0f510afbd09b19949bee786e098af9 by Miss Islington (bot) in branch '3.8': bpo-17305: Link to the third-party idna package. (GH-25208) (bpo-25211) https://github.com/python/cpython/commit/2760a6711b0f510afbd09b19949bee786e098af9

    akulakov commented 3 years ago

    Maybe deprecate idna so that users are strongly prompted to consider the pypi idna?

    JannesAlthoff commented 2 years ago

    I think IDNA2008 with TR46 without Transitional should be used, because Browser use this, and it should replace the current idna codec, the old codec's and IDNA2008 with other options could be provided under different codec names.

    This should be done because the current behavior is a high severity vulnerability. And replacing idna2003 with TR46 without Transitional matches most Browsers behavior.

    JannesAlthoff commented 2 years ago

    A not so good temporary fix could be not removing the ezset, endsigma, ZWJ and ZWNJ int the stringprep function.

    This should be easy:

    newlabel.append(stringprep.map_table_b2(c))

    Would be replaced by

    newlabel.append(stringprep.map_table_b2(c) if c!="ß" and c!="ς" else c)