search

psf / requests

A simple, yet elegant, HTTP library.
https://requests.readthedocs.io/en/latest/
Apache License 2.0
52.16k stars 9.33k forks source link

Relax IDNA2008 requirement? #5845

Closed snarfed closed 3 years ago

snarfed commented 3 years ago

Hi all! First, thank you for all your hard work developing and maintaining requests. I know firsthand how thankless and time consuming managing an open source project can be, and I can only imagine how much more so on a project this popular. We appreciate it!

I originally raised this as https://github.com/psf/requests/issues/3687#issuecomment-835688746, after that issue was closed. I suspect no one noticed, so I'm raising as a new issue.

requests currently uses the idna library to check input URLs for IDNA2008 compliance, and rejects URLs that don't comply. This breaks non-compliant URLs with emoji characters, like http://☃.net/, which you all said was intentional in https://github.com/psf/requests/issues/3687#issuecomment-260757981 (also see https://github.com/psf/requests/issues/3683#issuecomment-261240279), since those domains' time is arguably limited, ie they're effectively "dead domains walking." Understood.

However, not all TLDs require IDNA2008 compliance. Unlike gTLDs, ccTLDs generally get to choose their own domain policies - background from Wikipedia, ICANN, a GoDaddy representative - and a handful of them have stuck with IDNA2003, UTS#46, or related variants. (Not to mention older proprietary schemes like ThaiURL 😁.) For example, .ws, .la, .ai, .to, and .fm evidently explicitly allow emoji.

Similarly, afaik domain owners can do whatever they want with their own subdomains. So thanks to Punycode, third level (and beyond) hostnames like https://🌏➡➡❤🔒.ayeshious.com and https://🔒🔒🔒.scotthelme.co.uk seem to not be at risk of breaking due to gTLD registries enforcing IDNA2008 on pay-level domain registrations.

Any chance you all could relax the IDNA2008 requirement so that you support both of those kinds of domains?

Right now, I'm working around this with code like this, using the domain2idna library, to support at least IDNA2003 in addition to IDNA2008. It'd be nice not to have to.

try:
  resp = requests.get(url, ...)
except requests.exceptions.InvalidURL:
  punycode = domain2idna(url)
  if punycode != url:
    # the domain is valid idna2003 but not idna2008. encode and try again.
    resp = requests.get(punycode, ...)

Thanks again for listening, and for maintaining requests!

sethmlarson commented 3 years ago

This is not likely to land as there are additional security requirements to be mindful of when using IDNA2003, hence why it's preferable to use IDNA2008. My recommendation in this case is to do the normalization yourself and pass Requests an ASCII-only host.

snarfed commented 3 years ago

Fair enough, understood. Thanks for the response.