search

NDAR / nda-tools

Python package for interacting with NDA web services. Used to validate, submit, and download data to and from NDA.
MIT License
47 stars 22 forks source link

SSLCertVerificationError #83

Closed jmschabdach closed 7 months ago

jmschabdach commented 8 months ago

Running NDATools Version 0.2.25 with Python version 3.9 and numpy version 1.22.14 on Mac OS Ventura 13.5.2. I've set up my keyring username and password for nda-tools. When I try to run a basic command to download a packaged dataset to my local computer, I get garbled strings starting with 

Retrying (Retry(total=9, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1129)'))'

and ending with the following error messages

Traceback (most recent call last):
  File "/Users/youngjm/opt/miniconda3/envs/ndatools/lib/python3.9/site-packages/urllib3/connectionpool.py", line 715, in urlopen
    httplib_response = self._make_request(
  File "/Users/youngjm/opt/miniconda3/envs/ndatools/lib/python3.9/site-packages/urllib3/connectionpool.py", line 404, in _make_request
    self._validate_conn(conn)
  File "/Users/youngjm/opt/miniconda3/envs/ndatools/lib/python3.9/site-packages/urllib3/connectionpool.py", line 1058, in _validate_conn
    conn.connect()
  File "/Users/youngjm/opt/miniconda3/envs/ndatools/lib/python3.9/site-packages/urllib3/connection.py", line 419, in connect
    self.sock = ssl_wrap_socket(
  File "/Users/youngjm/opt/miniconda3/envs/ndatools/lib/python3.9/site-packages/urllib3/util/ssl_.py", line 449, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(
  File "/Users/youngjm/opt/miniconda3/envs/ndatools/lib/python3.9/site-packages/urllib3/util/ssl_.py", line 493, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
  File "/Users/youngjm/opt/miniconda3/envs/ndatools/lib/python3.9/ssl.py", line 501, in wrap_socket
    return self.sslsocket_class._create(
  File "/Users/youngjm/opt/miniconda3/envs/ndatools/lib/python3.9/ssl.py", line 1074, in _create
    self.do_handshake()
  File "/Users/youngjm/opt/miniconda3/envs/ndatools/lib/python3.9/ssl.py", line 1343, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1129)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/youngjm/opt/miniconda3/envs/ndatools/lib/python3.9/site-packages/requests/adapters.py", line 486, in send
    resp = conn.urlopen(
  File "/Users/youngjm/opt/miniconda3/envs/ndatools/lib/python3.9/site-packages/urllib3/connectionpool.py", line 827, in urlopen
    return self.urlopen(
  File "/Users/youngjm/opt/miniconda3/envs/ndatools/lib/python3.9/site-packages/urllib3/connectionpool.py", line 827, in urlopen
    return self.urlopen(
  File "/Users/youngjm/opt/miniconda3/envs/ndatools/lib/python3.9/site-packages/urllib3/connectionpool.py", line 827, in urlopen
    return self.urlopen(
  [Previous line repeated 7 more times]
  File "/Users/youngjm/opt/miniconda3/envs/ndatools/lib/python3.9/site-packages/urllib3/connectionpool.py", line 799, in urlopen
    retries = retries.increment(
  File "/Users/youngjm/opt/miniconda3/envs/ndatools/lib/python3.9/site-packages/urllib3/util/retry.py", line 592, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='gpop.s3.amazonaws.com', port=443): Max retries exceeded with url: [security info removed] (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1129)')))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/youngjm/opt/miniconda3/envs/ndatools/lib/python3.9/site-packages/NDATools/Download.py", line 499, in download_from_s3link
    with s.get(s3_link, stream=True) as response:
  File "/Users/youngjm/opt/miniconda3/envs/ndatools/lib/python3.9/site-packages/requests/sessions.py", line 602, in get
    return self.request("GET", url, **kwargs)
  File "/Users/youngjm/opt/miniconda3/envs/ndatools/lib/python3.9/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
  File "/Users/youngjm/opt/miniconda3/envs/ndatools/lib/python3.9/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
  File "/Users/youngjm/opt/miniconda3/envs/ndatools/lib/python3.9/site-packages/requests/adapters.py", line 517, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='gpop.s3.amazonaws.com', port=443): Max retries exceeded with url: [security info removed] (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1129)')))

Traceback (most recent call last):
  File "/Users/youngjm/opt/miniconda3/envs/ndatools/bin/downloadcmd", line 8, in <module>
    sys.exit(main())
  File "/Users/youngjm/opt/miniconda3/envs/ndatools/lib/python3.9/site-packages/NDATools/clientscripts/downloadcmd.py", line 200, in main
    s3Download.start()
  File "/Users/youngjm/opt/miniconda3/envs/ndatools/lib/python3.9/site-packages/NDATools/Download.py", line 198, in start
    self.download_package_metadata_file()
  File "/Users/youngjm/opt/miniconda3/envs/ndatools/lib/python3.9/site-packages/NDATools/Download.py", line 889, in download_package_metadata_file
    with gzip.open(download_location, 'rb') as f_in:
  File "/Users/youngjm/opt/miniconda3/envs/ndatools/lib/python3.9/gzip.py", line 58, in open
    binary_file = GzipFile(filename, gz_mode, compresslevel)
  File "/Users/youngjm/opt/miniconda3/envs/ndatools/lib/python3.9/gzip.py", line 173, in __init__
    fileobj = self.myfileobj = builtins.open(filename, mode or 'rb')
FileNotFoundError: [Errno 2] No such file or directory: '/Users/youngjm/NDA/nda-tools/downloadcmd/packages/1221956/package_file_metadata.txt.gz'

My interpretation is that a problem with SSL certificate validation is preventing the package_file_metadata.txt.gz file from being downloaded, which causes the final error.

Since I'm trying to download to my local machine, do I need to set up any of the s3 configuration? Is there another way to resolve the SSL certificate error aside from setting verify=False as suggested in this post? Has anyone else run into this error?

gregmagdits commented 8 months ago

Since you are not downloading to a personal s3 bucket, you do not need to setup any special s3 configuration.

Users report this error when they are running the download within a network that is configured to reject self-signed certs by default. From the stack trace, it looks like your institutions network config is rejecting a request to amazon's s3 server.

The post you linked to would require a code change, so you could do that if you are comfortable enough with Python. Alternatively you could request that your organization's IT team help you fix the issue for you. They might be able to fix this for you by adding the offending cert-chain to the certifi module. Instructions are posted in one of the answers here - https://superuser.com/questions/1426277/ssl-sslcertverificationerror-ssl-certificate-verify-failed-certificate-verif

You could also run the download from a different computer, specifically one that is outside your institutions network. If you run the command on your home computer, it should work.

gregmagdits commented 7 months ago

Closing - no code changes in nda-tools will be made since this stems from the network configuration in which nda-tools is running.