search

urllib3 / urllib3

urllib3 is a user-friendly HTTP client library for Python
https://urllib3.readthedocs.io
MIT License
3.74k stars 1.13k forks source link

InvalidCodepoint exception with wildcard certificate name ? [Wildcard/CN=*.] #981

Closed spazm closed 7 years ago

spazm commented 7 years ago

Error

elasticsearch.exceptions.ConnectionError: ConnectionError(Codepoint U+002A at position 1 of u'*' not allowed) caused by: InvalidCodepoint(Codepoint U+002A at position 1 of u'*' not allowed)

Overview

I'm seeing an InvalidCodepoint exception connecting to my elasticsearch server via ssl after upgrading to urllib3==1.17. This worked correctly with urllib3=1.16.

What can I do to help debug this issue? Can I turn up logging? Is this a problem with how elasticsearch is calling urllib3? I've simplified with a direct urllib3 example.

Lemma 1.

In 1.17 urllib3 switched to using idna.

Lemma 2

The host has a wildcard certificate valid for *.prod.example.com

Hypothesis

I assume the wildcard certificate is somehow related to the u'*' in the invalid code point error provided from idna.

Reproduction Steps

Failing elasticsearch script

urllib3_test.py:

#!/usr/bin/env python

import certifi
from elasticsearch import Elasticsearch
import urllib3.contrib.pyopenssl

urllib3.contrib.pyopenssl.inject_into_urllib3()

host = u"rdb-test-andrew-01.prod.example.com"
es_client=Elasticsearch(host,
                        use_ssl=True,
                        verify_certs=True,
                        ca_certs=certifi.where())
es_client.info()

Recreate environment

virtualenv env-test && . env-test/bin/activate
pip install elasticsearch==2.4.0 urllib3==1.17 certifi==2016.8.31 
pip install urllib3[secure]

$ ./urllib3_test.py
Traceback (most recent call last):
  File "./urllib3_test.py", line 14, in <module>
    es_client.info()
  File "/mnt/my-candidates/env-test/local/lib/python2.7/site-packages/elasticsearch/client/utils.py", line 69, in _wrapped
    return func(*args, params=params, **kwargs)
  File "/mnt/my-candidates/env-test/local/lib/python2.7/site-packages/elasticsearch/client/__init__.py", line 220, in info
    return self.transport.perform_request('GET', '/', params=params)
  File "/mnt/my-candidates/env-test/local/lib/python2.7/site-packages/elasticsearch/transport.py", line 327, in perform_request
    status, headers, data = connection.perform_request(method, url, params, body, ignore=ignore, timeout=timeout)
  File "/mnt/my-candidates/env-test/local/lib/python2.7/site-packages/elasticsearch/connection/http_urllib3.py", line 105, in perform_request
    raise ConnectionError('N/A', str(e), e)
elasticsearch.exceptions.ConnectionError: ConnectionError(Codepoint U+002A at position 1 of u'*' not allowed) caused by: InvalidCodepoint(Codepoint U+002A at position 1 of u'*' not allowed)

Installed versions:

pip freeze

argparse==1.2.1
certifi==2016.08.31
cffi==1.8.3
cryptography==1.5
distribute==0.6.24
elasticsearch==2.4.0
enum34==1.1.6
idna==2.1
ipaddress==1.0.17
pyOpenSSL==16.1.0
pyasn1==0.1.9
pycparser==2.14
six==1.10.0
urllib3==1.17
wsgiref==0.1.2

Working example in urllib3==1.16

% pip install urllib3==1.16
% pip install pyopenssl ndg-httpsclient pyasn1

% ./urllib3_test.py
# success! no error

% pip freeze
argparse==1.2.1
certifi==2016.08.31
cffi==1.8.3
cryptography==1.5
distribute==0.6.24
elasticsearch==2.4.0
enum34==1.1.6
idna==2.1
ipaddress==1.0.17
ndg-httpsclient==0.4.2
pyOpenSSL==16.1.0
pyasn1==0.1.9
pycparser==2.14
six==1.10.0
urllib3==1.16
wsgiref==0.1.2

Simplified test

#!/usr/bin/env python

import certifi
import urllib3
import urllib3.contrib.pyopenssl

urllib3.contrib.pyopenssl.inject_into_urllib3()

url = u"https://rdb-test-andrew-01.prod.example.com:9200"
connection_pool = urllib3.PoolManager(
        cert_reqs='CERT_REQUIRED',
        ca_certs=certifi.where())
resp = connection_pool.request('GET', url)
print resp.data
  File "./urllib3_test_direct.py", line 13, in <module>
    resp = connection_pool.request('GET', url)
  File "/mnt/my-candidates/env-test/local/lib/python2.7/site-packages/urllib3/request.py", line 66, in request
    **urlopen_kw)
  File "/mnt/my-candidates/env-test/local/lib/python2.7/site-packages/urllib3/request.py", line 87, in request_encode_url
    return self.urlopen(method, url, **extra_kw)
  File "/mnt/my-candidates/env-test/local/lib/python2.7/site-packages/urllib3/poolmanager.py", line 244, in urlopen
    response = conn.urlopen(method, u.request_uri, **kw)
  File "/mnt/my-candidates/env-test/local/lib/python2.7/site-packages/urllib3/connectionpool.py", line 594, in urlopen
    chunked=chunked)
  File "/mnt/my-candidates/env-test/local/lib/python2.7/site-packages/urllib3/connectionpool.py", line 350, in _make_request
    self._validate_conn(conn)
  File "/mnt/my-candidates/env-test/local/lib/python2.7/site-packages/urllib3/connectionpool.py", line 833, in _validate_conn
    conn.connect()
  File "/mnt/my-candidates/env-test/local/lib/python2.7/site-packages/urllib3/connection.py", line 324, in connect
    cert = self.sock.getpeercert()
  File "/mnt/my-candidates/env-test/local/lib/python2.7/site-packages/urllib3/contrib/pyopenssl.py", line 312, in getpeercert
    'subjectAltName': get_subj_alt_name(x509)
  File "/mnt/my-candidates/env-test/local/lib/python2.7/site-packages/urllib3/contrib/pyopenssl.py", line 185, in get_subj_alt_name
    for name in ext.get_values_for_type(x509.DNSName)
  File "/mnt/my-candidates/env-test/local/lib/python2.7/site-packages/urllib3/contrib/pyopenssl.py", line 141, in _dnsname_to_stdlib
    name = idna.encode(name)
  File "/mnt/my-candidates/env-test/local/lib/python2.7/site-packages/idna/core.py", line 355, in encode
    result.append(alabel(label))
  File "/mnt/my-candidates/env-test/local/lib/python2.7/site-packages/idna/core.py", line 276, in alabel
    check_label(label)
  File "/mnt/my-candidates/env-test/local/lib/python2.7/site-packages/idna/core.py", line 253, in check_label
    raise InvalidCodepoint('Codepoint {0} at position {1} of {2} not allowed'.format(_unot(cp_value), pos+1, repr(label)))
idna.core.InvalidCodepoint: Codepoint U+002A at position 1 of u'*' not allowed

Certificate Info


---
Certificate chain
 0 s:/OU=Domain Control Validated/OU=PositiveSSL Wildcard/CN=*.prod.ziprecruiter.com
   i:/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO RSA Domain Validation Secure Server CA
 1 s:/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO RSA Domain Validation Secure Server CA
   i:/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO RSA Certification Authority
 2 s:/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO RSA Certification Authority
   i:/C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root
subject=/OU=Domain Control Validated/OU=PositiveSSL Wildcard/CN=*.prod.example.com
issuer=/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO RSA Domain Validation Secure Server CA

ObNote

replaced our real commercial domain name with example.com.

nateprewitt commented 7 years ago

Hey @spazm, I believe this may be related to #979 which has been addressed in master with #980. If you have a moment could you run your codebase with the current master branch and see if this solves the issue?

spazm commented 7 years ago

Checked out master (3de23d0edef737a8b72d33140a3ad44af19a565b) and installed. Fixed! Thanks for the quick response, @nateprewitt.

LGTM. 👍

Failing with 1.17

pip install urllib3==1.17 && ./urllib3_test_direct.py
Downloading/unpacking urllib3==1.17
  Downloading urllib3-1.17.tar.gz (181Kb): 181Kb downloaded
  Running setup.py egg_info for package urllib3

    warning: no previously-included files matching '*' found under directory 'docs/_build'
Installing collected packages: urllib3
  Found existing installation: urllib3 dev
    Uninstalling urllib3:
      Successfully uninstalled urllib3
  Running setup.py install for urllib3

    warning: no previously-included files matching '*' found under directory 'docs/_build'
Successfully installed urllib3
Cleaning up...
Traceback (most recent call last):
  File "./urllib3_test_direct.py", line 13, in <module>
    resp = connection_pool.request('GET', url)
  File "/mnt/my-candidates/env-test/local/lib/python2.7/site-packages/urllib3/request.py", line 66, in request
    **urlopen_kw)
  File "/mnt/my-candidates/env-test/local/lib/python2.7/site-packages/urllib3/request.py", line 87, in request_encode_url
    return self.urlopen(method, url, **extra_kw)
  File "/mnt/my-candidates/env-test/local/lib/python2.7/site-packages/urllib3/poolmanager.py", line 244, in urlopen
    response = conn.urlopen(method, u.request_uri, **kw)
  File "/mnt/my-candidates/env-test/local/lib/python2.7/site-packages/urllib3/connectionpool.py", line 594, in urlopen
    chunked=chunked)
  File "/mnt/my-candidates/env-test/local/lib/python2.7/site-packages/urllib3/connectionpool.py", line 350, in _make_request
    self._validate_conn(conn)
  File "/mnt/my-candidates/env-test/local/lib/python2.7/site-packages/urllib3/connectionpool.py", line 833, in _validate_conn
    conn.connect()
  File "/mnt/my-candidates/env-test/local/lib/python2.7/site-packages/urllib3/connection.py", line 324, in connect
    cert = self.sock.getpeercert()
  File "/mnt/my-candidates/env-test/local/lib/python2.7/site-packages/urllib3/contrib/pyopenssl.py", line 312, in getpeercert
    'subjectAltName': get_subj_alt_name(x509)
  File "/mnt/my-candidates/env-test/local/lib/python2.7/site-packages/urllib3/contrib/pyopenssl.py", line 185, in get_subj_alt_name
    for name in ext.get_values_for_type(x509.DNSName)
  File "/mnt/my-candidates/env-test/local/lib/python2.7/site-packages/urllib3/contrib/pyopenssl.py", line 141, in _dnsname_to_stdlib
    name = idna.encode(name)
  File "/mnt/my-candidates/env-test/local/lib/python2.7/site-packages/idna/core.py", line 355, in encode
    result.append(alabel(label))
  File "/mnt/my-candidates/env-test/local/lib/python2.7/site-packages/idna/core.py", line 276, in alabel
    check_label(label)
  File "/mnt/my-candidates/env-test/local/lib/python2.7/site-packages/idna/core.py", line 253, in check_label
    raise InvalidCodepoint('Codepoint {0} at position {1} of {2} not allowed'.format(_unot(cp_value), pos+1, repr(label)))
idna.core.InvalidCodepoint: Codepoint U+002A at position 1 of u'*' not allowed

Working with HEAD (3de23d0)

$ pip install ./urllib3/ && ./urllib3_test_direct.py
Unpacking ./urllib3
  Running setup.py egg_info for package from file:///mnt/my-candidates/urllib3

    warning: no previously-included files matching '*' found under directory 'docs/_build'
Installing collected packages: urllib3
  Found existing installation: urllib3 1.17
    Uninstalling urllib3:
      Successfully uninstalled urllib3
  Running setup.py install for urllib3

    warning: no previously-included files matching '*' found under directory 'docs/_build'
Successfully installed urllib3
Cleaning up...
{
  "name" : "rdb-test-andrew-01.prod",
  "cluster_name" : "rdb-test-andrew.prod",
  "version" : {
    "number" : "2.4.0",
    "build_hash" : "ce9f0c7394dee074091dd1bc4e9469251181fc55",
    "build_timestamp" : "2016-08-29T09:14:17Z",
    "build_snapshot" : false,
    "lucene_version" : "5.5.2"
  },
  "tagline" : "You Know, for Search"
}

urllib3_test_direct.py

(same as previous comment)

#!/usr/bin/env python

import certifi
import urllib3
import urllib3.contrib.pyopenssl

urllib3.contrib.pyopenssl.inject_into_urllib3()

url = u"https://rdb-test-andrew-01.prod.example.com:9200"
connection_pool = urllib3.PoolManager(
        cert_reqs='CERT_REQUIRED',
        ca_certs=certifi.where())
resp = connection_pool.request('GET', url)
print resp.data
nateprewitt commented 7 years ago

Great, glad it helped @spazm.

derekbekoe commented 7 years ago

Wondering if an update to the urllib3 package can be published with this fix? cc @haocs Thanks

Lukasa commented 7 years ago

It can do. It's on my schedule, but won't happen before next week.

derekbekoe commented 7 years ago

Okay thanks for the reply.