search

CADWRDeltaModeling / dms_datastore

Data download and management tools for continuous data for Pandas. See documentation https://cadwrdeltamodeling.github.io/dms_datastore/
https://cadwrdeltamodeling.github.io/dms_datastore/
MIT License
1 stars 0 forks source link

populate repo needs to be robust to failures #2

Closed dwr-psandhu closed 11 months ago

dwr-psandhu commented 11 months ago

Description

Ran populate_repo and after a number of successful downloads it crashed with a message as below That resulted in the downloads stopping. It would be better to catch and log the exception and continue and then deal with failures later.

What I Did

dwr_des generated an exception: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1091)> with trace:
Traceback (most recent call last):
  File "/home/psandhu/miniconda3/envs/dms/lib/python3.7/urllib/request.py", line 1350, in do_open
    encode_chunked=req.has_header('Transfer-encoding'))
  File "/home/psandhu/miniconda3/envs/dms/lib/python3.7/http/client.py", line 1262, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/home/psandhu/miniconda3/envs/dms/lib/python3.7/http/client.py", line 1308, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/home/psandhu/miniconda3/envs/dms/lib/python3.7/http/client.py", line 1257, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/home/psandhu/miniconda3/envs/dms/lib/python3.7/http/client.py", line 1028, in _send_output
    self.send(msg)
  File "/home/psandhu/miniconda3/envs/dms/lib/python3.7/http/client.py", line 968, in send
    self.connect()
  File "/home/psandhu/miniconda3/envs/dms/lib/python3.7/http/client.py", line 1432, in connect
    server_hostname=server_hostname)
  File "/home/psandhu/miniconda3/envs/dms/lib/python3.7/ssl.py", line 423, in wrap_socket
    session=session
  File "/home/psandhu/miniconda3/envs/dms/lib/python3.7/ssl.py", line 870, in _create
    self.do_handshake()
  File "/home/psandhu/miniconda3/envs/dms/lib/python3.7/ssl.py", line 1139, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1091)
dwr-psandhu commented 11 months ago

This failure is related to one of the websites not having a valid SSL certificate. The workaround would be to suspend SSL check for that url

dwr-psandhu commented 11 months ago

Fixed with latest push. SSL certification is bypassed for internal website