SSL error on python 3.9

[astrochun]

The CI actions recently failed with the following error. This was captured on py3.9. Note that I ran it locally on 3.10 and didn't have an issue:

Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.9.15/x64/lib/python3.9/site-packages/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
  File "/opt/hostedtoolcache/Python/3.9.15/x64/lib/python3.9/site-packages/urllib3/connectionpool.py", line 386, in _make_request
    self._validate_conn(conn)
  File "/opt/hostedtoolcache/Python/3.9.15/x64/lib/python3.9/site-packages/urllib3/connectionpool.py", line 1042, in _validate_conn
    conn.connect()
  File "/opt/hostedtoolcache/Python/3.9.15/x64/lib/python3.9/site-packages/urllib3/connection.py", line 414, in connect
    self.sock = ssl_wrap_socket(
  File "/opt/hostedtoolcache/Python/3.9.15/x64/lib/python3.9/site-packages/urllib3/util/ssl_.py", line 449, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(
  File "/opt/hostedtoolcache/Python/3.9.15/x64/lib/python3.9/site-packages/urllib3/util/ssl_.py", line 493, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
  File "/opt/hostedtoolcache/Python/3.9.15/x64/lib/python3.9/ssl.py", line 501, in wrap_socket
    return self.sslsocket_class._create(
  File "/opt/hostedtoolcache/Python/3.9.15/x64/lib/python3.9/ssl.py", line 1041, in _create
    self.do_handshake()
  File "/opt/hostedtoolcache/Python/3.9.15/x64/lib/python3.9/ssl.py", line 1310, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: UNSAFE_LEGACY_RENEGOTIATION_DISABLED] unsafe legacy renegotiation disabled (_ssl.c:1129)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.9.15/x64/lib/python3.9/site-packages/requests/adapters.py", line 489, in send
    resp = conn.urlopen(
  File "/opt/hostedtoolcache/Python/3.9.15/x64/lib/python3.9/site-packages/urllib3/connectionpool.py", line 787, in urlopen
    retries = retries.increment(
  File "/opt/hostedtoolcache/Python/3.9.15/x64/lib/python3.9/site-packages/urllib3/util/retry.py", line 592, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='www.osti.gov', port=443): Max retries exceeded with url: /dataexplorer/api/v1/records?site_ownership_code=PPPL&page=0 (Caused by SSLError(SSLError(1, '[SSL: UNSAFE_LEGACY_RENEGOTIATION_DISABLED] unsafe legacy renegotiation disabled (_ssl.c:1129)')))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.9.15/x64/bin/scraper", line 8, in <module>
    sys.exit(main())
  File "/home/runner/work/pdc-osti/pdc-osti/src/pdc_osti/scraper.py", line 404, in main
    s.run_pipeline()
  File "/home/runner/work/pdc-osti/pdc-osti/src/pdc_osti/scraper.py", line 354, in run_pipeline
    self.get_existing_datasets()
  File "/home/runner/work/pdc-osti/pdc-osti/src/pdc_osti/scraper.py", line 118, in get_existing_datasets
    r = requests.get(url)
  File "/opt/hostedtoolcache/Python/3.9.15/x64/lib/python3.9/site-packages/requests/api.py", line 73, in get
    return request("get", url, params=params, **kwargs)
  File "/opt/hostedtoolcache/Python/3.9.15/x64/lib/python3.9/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
  File "/opt/hostedtoolcache/Python/3.9.15/x64/lib/python3.9/site-packages/requests/sessions.py", line 587, in request
    resp = self.send(prep, **send_kwargs)
  File "/opt/hostedtoolcache/Python/3.9.15/x64/lib/python3.9/site-packages/requests/sessions.py", line 701, in send
    r = adapter.send(request, **kwargs)
  File "/opt/hostedtoolcache/Python/3.9.15/x64/lib/python3.9/site-packages/requests/adapters.py", line 563, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='www.osti.gov', port=443): Max retries exceeded with url: /dataexplorer/api/v1/records?site_ownership_code=PPPL&page=0 (Caused by SSLError(SSLError(1, '[SSL: UNSAFE_LEGACY_RENEGOTIATION_DISABLED] unsafe legacy renegotiation disabled (_ssl.c:1129)')))



[astrochun]

There are some work-around solutions mentioned here:

https://stackoverflow.com/questions/71603314/ssl-error-unsafe-legacy-renegotiation-disabled

There's an active PR to fix this in cpython but that doesn't seem merged in 3.10. I think the problem stems with the GitHub servers using OpenSSL 3.0, while locally I'm using 1.1.1

[astrochun]

The Jan 1st monthly CI run failed for the same reason: https://github.com/pulibrary/pdc-osti/actions/runs/3816546780/jobs/6492246828