SSL certificate error with python3.9

[tloubrieu-jpl]

🐛 Describe the bug

When requesting pds-gamma with python 3.9 there is an SSL certificate error

it works with python 3.7

See:

Traceback (most recent call last):
  File "/Users/loubrieu/tmp/test.py", line 6, in <module>
    bundles.bundle_by_lidvid('urn:nasa:pds:insight_documents::2.0')
  File "/Users/loubrieu/tmp/venv-api39/lib/python3.9/site-packages/pds/api_client/api/bundles_api.py", line 67, in bundle_by_lidvid
    return self.bundle_by_lidvid_with_http_info(lidvid, **kwargs)  # noqa: E501
  File "/Users/loubrieu/tmp/venv-api39/lib/python3.9/site-packages/pds/api_client/api/bundles_api.py", line 158, in bundle_by_lidvid_with_http_info
    return self.api_client.call_api(
  File "/Users/loubrieu/tmp/venv-api39/lib/python3.9/site-packages/pds/api_client/api_client.py", line 376, in call_api
    return self.__call_api(resource_path, method,
  File "/Users/loubrieu/tmp/venv-api39/lib/python3.9/site-packages/pds/api_client/api_client.py", line 186, in __call_api
    response_data = self.request(
  File "/Users/loubrieu/tmp/venv-api39/lib/python3.9/site-packages/pds/api_client/api_client.py", line 402, in request
    return self.rest_client.GET(url,
  File "/Users/loubrieu/tmp/venv-api39/lib/python3.9/site-packages/pds/api_client/rest.py", line 238, in GET
    return self.request("GET", url,
  File "/Users/loubrieu/tmp/venv-api39/lib/python3.9/site-packages/pds/api_client/rest.py", line 204, in request
    r = self.pool_manager.request(method, url,
  File "/Users/loubrieu/tmp/venv-api39/lib/python3.9/site-packages/urllib3/request.py", line 74, in request
    return self.request_encode_url(
  File "/Users/loubrieu/tmp/venv-api39/lib/python3.9/site-packages/urllib3/request.py", line 96, in request_encode_url
    return self.urlopen(method, url, **extra_kw)
  File "/Users/loubrieu/tmp/venv-api39/lib/python3.9/site-packages/urllib3/poolmanager.py", line 375, in urlopen
    response = conn.urlopen(method, u.request_uri, **kw)
  File "/Users/loubrieu/tmp/venv-api39/lib/python3.9/site-packages/urllib3/connectionpool.py", line 783, in urlopen
    return self.urlopen(
  File "/Users/loubrieu/tmp/venv-api39/lib/python3.9/site-packages/urllib3/connectionpool.py", line 783, in urlopen
    return self.urlopen(
  File "/Users/loubrieu/tmp/venv-api39/lib/python3.9/site-packages/urllib3/connectionpool.py", line 783, in urlopen
    return self.urlopen(
  File "/Users/loubrieu/tmp/venv-api39/lib/python3.9/site-packages/urllib3/connectionpool.py", line 755, in urlopen
    retries = retries.increment(
  File "/Users/loubrieu/tmp/venv-api39/lib/python3.9/site-packages/urllib3/util/retry.py", line 574, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='pds-gamma.jpl.nasa.gov', port=443): Max retries exceeded with url: /api/bundles/urn%3Anasa%3Apds%3Ainsight_documents%3A%3A2.0 (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1129)')))

📜 To Reproduce

Steps to reproduce the behavior:

1. python -m venv venv # using python 3.9
2. source venv
3. pip install pds.api-client
4. python test.py

test.py.zip

🕵️ Expected behavior

The code should return the requested bundle

📚 Version of Software Used

0.7.1

🩺 Test Data / Additional context

🏞Screenshots

🖥 System Info

• OS: [e.g. iOS]
• Browser [e.g. chrome, safari]
• Version [e.g. 22]

---

🦄 Related requirements

⚙️ Engineering Details

[tloubrieu-jpl]

The packages installed are:

% pip freeze
pds.api-client==0.7.1
python-dateutil==2.8.2
six==1.16.0
urllib3==1.26.7

[nutjob4life]

Hi @tloubrieu-jpl! Thanks for the thorough bug report.

I believe the sysadmins will need to address this by ensuring that Python 3.9 is built against a version of OpenSSL that supports the wildcard/intermediate certificates used by pds-gamma.

However, I'm unable to reproduce the error you're seeing. What python3.9 executable are you using?

Here's what I get:

Mon Oct 11 17:06:39 UTC 2021
$ hostname
pds-gamma.jpl.nasa.gov
$ /usr/local/python-3.9.5/bin/python3.9 -m venv venv
$ cd venv
$ bin/pip install --upgrade --quiet pip setuptools wheel build
$ bin/pip --quiet install pds.api-client
$ curl --silent --location --remote-name 'https://github.com/NASA-PDS/pds-api-client/files/7324095/test.py.zip'
$ unzip -qq test.py.zip
$ bin/python test.py
$ echo $?
0
$ echo \U+1F389
🎉

[tloubrieu-jpl]

Thanks @nutjob4life , @gxtchen originally reported this bug and I am not sure which python he is using. For myself I use python 3.9 on macos, I installed 2 weeks ago, but I don't remember how...

[nutjob4life]

@tloubrieu-jpl For a short-term workaround for @gxtchen, you could put this at the top of test.py:

import ssl
ssl._create_default_https_context = ssl._create_unverified_context

This will turn off all certificate validation until such time that the Python 3.9 installation can be fixed.

[tloubrieu-jpl]

An intermediate certificate is new (1-2 year old) which is required for certificates such as the one of JPL containing wildcard.

homebrew deployment provide the intermediate certificate.

[nutjob4life]

I think this should be marked as closed and wontfix since the issue is not in our software but in the Python interpreter running it.

[tloubrieu-jpl]

@gxtchen , you need to run pip install --upgrade certifi in your virtual environment. Let me know if that helps. That worked for me.

[nutjob4life]

We could add certifi as a dependency of this package. It shouldn't hurt anyone who's Python installations know how to use SSL, but it might help those who's Python installations don't.

[tloubrieu-jpl]

Yes that sounds good to me, I will add the dependency

[tloubrieu-jpl]

Oops, actually requirements.txt is generated by openapi-generator...

[tloubrieu-jpl]

I've added the instruction in the documentation, see commit e62a7e4