urllib3 is not part of requests anymore

[rafaelpirolla]

Describe the bug pip install requests will install latest version of the lib this doesn't work on latest version of the requests lib: requests.packages.urllib3.disable_warnings(InsecureRequestWarning)

This logs fills up the FS in days.

To Reproduce Steps to reproduce the behavior:

1. Use any ADC with self signed cert
2. Latest as today
3. Any
4. Log: /usr/lib/python2.7/site-packages/urllib3/connectionpool.py:986: InsecureRequestWarning: Unverified HTTPS request is being made to host

Expected behavior No log.

Probable solution Something in the lines of: Lines to remove: -from requests.packages.urllib3.exceptions import InsecureRequestWarning -requests.packages.urllib3.disable_warnings(InsecureRequestWarning) -requests.packages.urllib3.disable_warnings(SubjectAltNameWarning)

Lines to add in the proper place: import urllib3 urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning) urllib3.disable_warnings(urllib3.exceptions.SubjectAltNameWarning)

[aroraharsh23]

Thanks for updating. Will verify the same and update in script.

[aroraharsh23]

@rafaelpirolla Can you specify the requests version. I just installed requests latest : requests-2.23.0 and it did install urllib3-1.25.9

########## python2.7 -m pip install requests Collecting requests Using cached https://files.pythonhosted.org/packages/1a/70/1935c770cb3be6e3a8b78ced23d7e0f3b187f5cbfab4749523ed65d7c9b1/requests-2.23.0-py2.py3-none-any.whl Requirement already satisfied (use --upgrade to upgrade): chardet<4,>=3.0.2 in /usr/local/lib/python2.7/dist-packages (from requests) Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 (from requests) Using cached https://files.pythonhosted.org/packages/e1/e5/df302e8017440f111c11cc41a6b432838672f5a70aa29227bf58149dc72f/urllib3-1.25.9-py2.py3-none-any.whl Requirement already satisfied (use --upgrade to upgrade): idna<3,>=2.5 in /usr/local/lib/python2.7/dist-packages (from requests) Requirement already satisfied (use --upgrade to upgrade): certifi>=2017.4.17 in /usr/local/lib/python2.7/dist-packages (from requests) Installing collected packages: urllib3, requests Successfully installed requests-2.23.0 urllib3-1.25.9 ################

[rafaelpirolla]

That's the issue. For this version these lines doesn't work:

from requests.packages.urllib3.exceptions import InsecureRequestWarning

requests.packages.urllib3.disable_warnings(InsecureRequestWarning)
requests.packages.urllib3.disable_warnings(SubjectAltNameWarning)

They run without errors but they don't supres the errors thrown if you access an https url with untrusted certificate.

[aroraharsh23]

Ok, will update then

[rafaelpirolla]

Tricky... I actually could not replicate: docker run -it --rm --name came-31 python:2-alpine sh

#pip install requests
Successfully installed certifi-2020.4.5.1 chardet-3.0.4 idna-2.9 requests-2.23.0 urllib3-1.25.9

apk add openssl

openssl \
  req \
  -newkey rsa:2048 -nodes \
  -keyout privkey.pem \
  -x509 -days 36500 -out certificate.pem \
  -subj "/C=XX/ST=NRW/L=Earth/O=CompanyName/OU=IT/CN=localhost/emailAddress=admin@localhost"

apk add vim

server.py:

#!/usr/local/bin/python
# Sets up an HTTPS server that serves directory contents
import sys
import ssl

# Settings
listen_target = ('localhost', 9999)  # https://localhost:9999/
certificate_file = './certificate.pem'
private_key_file = './privkey.pem'

# Python 3 version
if sys.version_info[0] == 3:
    import http.server
    httpd = http.server.HTTPServer(listen_target, http.server.SimpleHTTPRequestHandler)
# Python 2 version
elif sys.version_info[0] == 2:
    import BaseHTTPServer, SimpleHTTPServer
    httpd = BaseHTTPServer.HTTPServer(listen_target, SimpleHTTPServer.SimpleHTTPRequestHandler)

# Wrap the socket with SSL
httpd.socket = ssl.wrap_socket(httpd.socket,
               certfile=certificate_file, keyfile=private_key_file, server_side=True)

# Start listening
httpd.serve_forever()

chmod a+x server.py ./server.py &

get_simple.py:

import requests
response = requests.get("https://localhost:9999",verify=False)

python get_simple.py /usr/local/lib/python2.7/site-packages/urllib3/connectionpool.py:986: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings

get_oldversion.py

import requests
from requests.packages.urllib3.exceptions import InsecureRequestWarning

requests.packages.urllib3.disable_warnings(InsecureRequestWarning)

response = requests.get("https://localhost:9999",verify=False)

python get_oldversion.py (no warning)

get_newversion.py:

import requests
import urllib3

urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)

response = requests.get("https://localhost:9999",verify=False)

python get_newversion.py (no warning)

[rafaelpirolla]

I need to set NS_VALIDATE_CERT environment variable to no to have https with verify=False...

[aroraharsh23]

By default, there is no cert validation, so if you don't need to set anything. if you want to validate then only you need to set NS_VALIDATE_CERT to True and provide certpath. I hope documentation is clear on this. If you don't want https then just set --secure='no'. Please note, by default it is 'https' with no cert validation.

[rafaelpirolla]

Got it. I still don't know why adding these lines: import urllib3 urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)

And removing these: from requests.packages.urllib3.exceptions import InsecureRequestWarning requests.packages.urllib3.disable_warnings(InsecureRequestWarning)

Makes the script start working in my test environment if the test I created in the docker environment works with both options (i.e. doesn't print warnings).

[aroraharsh23]

I tested on 2 different Setups, installing requests from scratch(urllib3 is clearly a part of this). No warnings anywhere. Maybe you can try uninstalling and installing requests again(if you haven't done that), or try in a separate system. It should be fine.

[rafaelpirolla]

I could reproduce on amazon linux 2 on amazon, not in the container though.

[rafaelpirolla]

Found the issue: there is some package redundancy when installing awscli through system package manager.

Solution: Removing from OS using yum and installing everything through pip made it work.

Thanks for all the support!

Anyhow, I believe some investigation on how to set up the underlying urllib3 after requests >= 2.16.0 would be good. This troubleshooting helped me and it seems both syntaxes are supported? https://github.com/psf/requests/issues/4096

[rafaelpirolla]

Anyhow I think you could close this one but put some TODO somewhere?

[aroraharsh23]

Sure, i would take this into consideration for our next release, which will be on python3, that will require revisiting all pip requirements.