search

CiscoDevNet / virl2-client

Client library for the Cisco VIRL 2 Network Simulation Platform
Apache License 2.0
56 stars 18 forks source link

ClientLibrary(ssl_verify=False) does not disable SSL cert verification #23

Closed dgethings closed 1 year ago

dgethings commented 1 year ago

Hi,

It looks like disabling SSL verification in the library does not work. Here's my setup:

❯ python --version
Python 3.10.8

poetry show virl2_client name : virl2-client
version : 2.4.1
description : VIRL2 Client Library

dependencies

from pathlib import Path

from virl2_client import ClientLibrary

client = ClientLibrary("https://x.x.x.x", "user", "password", ssl_verify=False)

# this lab name exists on CML system
lab = client.find_labs_by_title("My lab")[0]

I get the following output when I run the above script:

❯ python src/main.py      
SSL Verification disabled
Traceback (most recent call last):
  File "/Users/gethdav/dbConda-2022_05-py39/envs/virt-lab-man/lib/python3.10/site-packages/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
  File "/Users/gethdav/dbConda-2022_05-py39/envs/virt-lab-man/lib/python3.10/site-packages/urllib3/connectionpool.py", line 386, in _make_request
    self._validate_conn(conn)
  File "/Users/gethdav/dbConda-2022_05-py39/envs/virt-lab-man/lib/python3.10/site-packages/urllib3/connectionpool.py", line 1042, in _validate_conn
    conn.connect()
  File "/Users/gethdav/dbConda-2022_05-py39/envs/virt-lab-man/lib/python3.10/site-packages/urllib3/connection.py", line 414, in connect
    self.sock = ssl_wrap_socket(
  File "/Users/gethdav/dbConda-2022_05-py39/envs/virt-lab-man/lib/python3.10/site-packages/urllib3/util/ssl_.py", line 453, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(sock, context, tls_in_tls)
  File "/Users/gethdav/dbConda-2022_05-py39/envs/virt-lab-man/lib/python3.10/site-packages/urllib3/util/ssl_.py", line 495, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock)
  File "/Users/gethdav/dbConda-2022_05-py39/envs/virt-lab-man/lib/python3.10/ssl.py", line 513, in wrap_socket
    return self.sslsocket_class._create(
  File "/Users/gethdav/dbConda-2022_05-py39/envs/virt-lab-man/lib/python3.10/ssl.py", line 1071, in _create
    self.do_handshake()
  File "/Users/gethdav/dbConda-2022_05-py39/envs/virt-lab-man/lib/python3.10/ssl.py", line 1342, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:997)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/gethdav/dbConda-2022_05-py39/envs/virt-lab-man/lib/python3.10/site-packages/requests/adapters.py", line 489, in send
    resp = conn.urlopen(
  File "/Users/gethdav/dbConda-2022_05-py39/envs/virt-lab-man/lib/python3.10/site-packages/urllib3/connectionpool.py", line 787, in urlopen
    retries = retries.increment(
  File "/Users/gethdav/dbConda-2022_05-py39/envs/virt-lab-man/lib/python3.10/site-packages/urllib3/util/retry.py", line 592, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='x.x.x.x', port=443): Max retries exceeded with url: /api/v0/system_information (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:997)')))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/gethdav/git/naas/virt-lab-man/src/main.py", line 9, in <module>
    client = ClientLibrary("https://x.x.x.x", "user", "password", ssl_verify=False)
  File "/Users/gethdav/dbConda-2022_05-py39/envs/virt-lab-man/lib/python3.10/site-packages/virl2_client/virl2_client.py", line 254, in __init__
    self.check_controller_version()
  File "/Users/gethdav/dbConda-2022_05-py39/envs/virt-lab-man/lib/python3.10/site-packages/virl2_client/virl2_client.py", line 427, in check_controller_version
    controller_version or self.system_info().get("version", "").split("-")[0]
  File "/Users/gethdav/dbConda-2022_05-py39/envs/virt-lab-man/lib/python3.10/site-packages/virl2_client/virl2_client.py", line 411, in system_info
    response = self.session.get(urljoin(self._base_url, "system_information"))
  File "/Users/gethdav/dbConda-2022_05-py39/envs/virt-lab-man/lib/python3.10/site-packages/requests/sessions.py", line 600, in get
    return self.request("GET", url, **kwargs)
  File "/Users/gethdav/dbConda-2022_05-py39/envs/virt-lab-man/lib/python3.10/site-packages/requests/sessions.py", line 587, in request
    resp = self.send(prep, **send_kwargs)
  File "/Users/gethdav/dbConda-2022_05-py39/envs/virt-lab-man/lib/python3.10/site-packages/requests/sessions.py", line 701, in send
    r = adapter.send(request, **kwargs)
  File "/Users/gethdav/dbConda-2022_05-py39/envs/virt-lab-man/lib/python3.10/site-packages/requests/adapters.py", line 563, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='x.x.x.x', port=443): Max retries exceeded with url: /api/v0/system_information (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:997)')))

I had a look through the source code. I think the problem is that while the ssl_verify argument is "set" on the session object it is not actually used by the session. For example:

>>> import requests
>>> s = requests.Session()
>>> s.verify = False
>>> s.get("https://x.x.x.x")Traceback (most recent call last):
  File "/Users/gethdav/dbConda-2022_05-py39/envs/virt-lab-man/lib/python3.10/site-packages/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
  File "/Users/gethdav/dbConda-2022_05-py39/envs/virt-lab-man/lib/python3.10/site-packages/urllib3/connectionpool.py", line 386, in _make_request
    self._validate_conn(conn)
  File "/Users/gethdav/dbConda-2022_05-py39/envs/virt-lab-man/lib/python3.10/site-packages/urllib3/connectionpool.py", line 1042, in _validate_conn
    conn.connect()
  File "/Users/gethdav/dbConda-2022_05-py39/envs/virt-lab-man/lib/python3.10/site-packages/urllib3/connection.py", line 414, in connect
    self.sock = ssl_wrap_socket(
  File "/Users/gethdav/dbConda-2022_05-py39/envs/virt-lab-man/lib/python3.10/site-packages/urllib3/util/ssl_.py", line 453, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(sock, context, tls_in_tls)
  File "/Users/gethdav/dbConda-2022_05-py39/envs/virt-lab-man/lib/python3.10/site-packages/urllib3/util/ssl_.py", line 495, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock)
  File "/Users/gethdav/dbConda-2022_05-py39/envs/virt-lab-man/lib/python3.10/ssl.py", line 513, in wrap_socket
    return self.sslsocket_class._create(
  File "/Users/gethdav/dbConda-2022_05-py39/envs/virt-lab-man/lib/python3.10/ssl.py", line 1071, in _create
    self.do_handshake()
  File "/Users/gethdav/dbConda-2022_05-py39/envs/virt-lab-man/lib/python3.10/ssl.py", line 1342, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:997)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/gethdav/dbConda-2022_05-py39/envs/virt-lab-man/lib/python3.10/site-packages/requests/adapters.py", line 489, in send
    resp = conn.urlopen(
  File "/Users/gethdav/dbConda-2022_05-py39/envs/virt-lab-man/lib/python3.10/site-packages/urllib3/connectionpool.py", line 787, in urlopen
    retries = retries.increment(
  File "/Users/gethdav/dbConda-2022_05-py39/envs/virt-lab-man/lib/python3.10/site-packages/urllib3/util/retry.py", line 592, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='x.x.x.x', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:997)')))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/gethdav/dbConda-2022_05-py39/envs/virt-lab-man/lib/python3.10/site-packages/requests/sessions.py", line 600, in get
    return self.request("GET", url, **kwargs)
  File "/Users/gethdav/dbConda-2022_05-py39/envs/virt-lab-man/lib/python3.10/site-packages/requests/sessions.py", line 587, in request
    resp = self.send(prep, **send_kwargs)
  File "/Users/gethdav/dbConda-2022_05-py39/envs/virt-lab-man/lib/python3.10/site-packages/requests/sessions.py", line 701, in send
    r = adapter.send(request, **kwargs)
  File "/Users/gethdav/dbConda-2022_05-py39/envs/virt-lab-man/lib/python3.10/site-packages/requests/adapters.py", line 563, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='x.x.x.x', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:997)')))

But if I pass the verify option to the get() call, it works:

>>> s.get("https://x.x.x.x", verify=False)
/Users/gethdav/dbConda-2022_05-py39/envs/virt-lab-man/lib/python3.10/site-packages/urllib3/connectionpool.py:1045: InsecureRequestWarning: Unverified HTTPS request is being made to host 'x.x.x.x'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
  warnings.warn(
<Response [200]>

The URL referenced in the comments above where ssl_verify is saved (http://docs.python-requests.org/en/master/user/advanced/#ssl-cert-verification) is misleading. Looking at the requests.Session.get() the verify property does not appear to be used. Running the example I provided through the debugger appears to confirm this:

❯ python -m pdb src/main.py
> /Users/gethdav/git/naas/virt-lab-man/src/main.py(1)<module>()
-> import os
(Pdb) b /Users/gethdav/dbConda-2022_05-py39/envs/virt-lab-man/lib/python3.10/site-packages/requests/sessions.py:599
Breakpoint 1 at /Users/gethdav/dbConda-2022_05-py39/envs/virt-lab-man/lib/python3.10/site-packages/requests/sessions.py:599
(Pdb) c
SSL Verification disabled
> /Users/gethdav/dbConda-2022_05-py39/envs/virt-lab-man/lib/python3.10/site-packages/requests/sessions.py(599)get()
-> kwargs.setdefault("allow_redirects", True)
(Pdb) kwargs
{}
(Pdb) n
> /Users/gethdav/dbConda-2022_05-py39/envs/virt-lab-man/lib/python3.10/site-packages/requests/sessions.py(600)get()
-> return self.request("GET", url, **kwargs)
(Pdb) kwargs
{'allow_redirects': True}
(Pdb) self.request("GET", url, **kwargs)
*** requests.exceptions.SSLError: HTTPSConnectionPool(host='x.x.x.x', port=443): Max retries exceeded with url: /api/v0/system_information (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:997)')))
rschmied commented 1 year ago

thanks for reporting... we'll look into it as we're about to release version 2.5. Might have been resolved already as we're also moving away from requests and use httpx instead.

rschmied commented 1 year ago

can we close this now that 2.5 has been released?

tmikuska commented 1 year ago

I believe that this issue was caused by https://github.com/psf/requests/issues/3829.

It may be sufficient to unset REQUESTS_CA_BUNDLE in the environment.

As @rschmied said, we already switched to httpx and we are not going to backport this change -> closing.