third-party SSL certificate?

[dbetchkal]

For those of us working behind stiff security measures is there a way to pass in the path to a third-party SSL certificate? I can install and import contextily fine, but as soon as I try to load tiles I inevitably get a SSLCertVerificationError.

Thank you!

[jorisvandenbossche]

I suppose this error is coming from requests, which we use to download the tiles ? (maybe https://requests.readthedocs.io/en/master/user/advanced/#ssl-cert-verification) I am not very familiar with this, but I suppose that requests has some interface to specify third-party SSL certificates? If that is the case, we can certainly try to provide an interface to pass through options for requests, I think.

[smHooper]

Yup, if you just provide an argument that gets passed to requests.get() or .post(), that should do it. I've only used the verify kwarg, but an alternative is cert. That would provide the most flexibility for those who either want to pass a third-party cert or cowboy it with verify=False.

[earwole1]

has there been any further work in this area? A coworker just ran into this issue. I can't replicate it on my end, unfortunately. (his network is locked down tighter than mine).

[dbetchkal]

has there been any further work in this area? A coworker just ran into this issue. I can't replicate it on my end, unfortunately. (his network is locked down tighter than mine).

I haven't tried to write something, but I've been meaning to... should be pretty straightforward.

[haley-sims]

Chiming in here as someone who has those aforementioned stiff security measures - it would be great to be able to pass the verify kwarg (or something similar) to add_basemap() (to be passed torequests.get()): https://github.com/geopandas/contextily/blob/main/contextily/tile.py#L395

Here's the full traceback from the SSL cert error:

---------------------------------------------------------------------------
SSLCertVerificationError                  Traceback (most recent call last)
File /srv/conda/envs/notebook/lib/python3.9/site-packages/urllib3/connectionpool.py:700, in HTTPConnectionPool.urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)
    699 if is_new_proxy_conn and http_tunnel_required:
--> 700     self._prepare_proxy(conn)
    702 # Make the request on the httplib connection object.

File /srv/conda/envs/notebook/lib/python3.9/site-packages/urllib3/connectionpool.py:996, in HTTPSConnectionPool._prepare_proxy(self, conn)
    994     conn.tls_in_tls_required = True
--> 996 conn.connect()

File /srv/conda/envs/notebook/lib/python3.9/site-packages/urllib3/connection.py:414, in HTTPSConnection.connect(self)
    412     context.load_default_certs()
--> 414 self.sock = ssl_wrap_socket(
    415     sock=conn,
    416     keyfile=self.key_file,
    417     certfile=self.cert_file,
    418     key_password=self.key_password,
    419     ca_certs=self.ca_certs,
    420     ca_cert_dir=self.ca_cert_dir,
    421     ca_cert_data=self.ca_cert_data,
    422     server_hostname=server_hostname,
    423     ssl_context=context,
    424     tls_in_tls=tls_in_tls,
    425 )
    427 # If we're using all defaults and the connection
    428 # is TLSv1 or TLSv1.1 we throw a DeprecationWarning
    429 # for the host.

File /srv/conda/envs/notebook/lib/python3.9/site-packages/urllib3/util/ssl_.py:449, in ssl_wrap_socket(sock, keyfile, certfile, cert_reqs, ca_certs, server_hostname, ssl_version, ciphers, ssl_context, ca_cert_dir, key_password, ca_cert_data, tls_in_tls)
    448 if send_sni:
--> 449     ssl_sock = _ssl_wrap_socket_impl(
    450         sock, context, tls_in_tls, server_hostname=server_hostname
    451     )
    452 else:

File /srv/conda/envs/notebook/lib/python3.9/site-packages/urllib3/util/ssl_.py:493, in _ssl_wrap_socket_impl(sock, ssl_context, tls_in_tls, server_hostname)
    492 if server_hostname:
--> 493     return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
    494 else:

File /srv/conda/envs/notebook/lib/python3.9/ssl.py:501, in SSLContext.wrap_socket(self, sock, server_side, do_handshake_on_connect, suppress_ragged_eofs, server_hostname, session)
    495 def wrap_socket(self, sock, server_side=False,
    496                 do_handshake_on_connect=True,
    497                 suppress_ragged_eofs=True,
    498                 server_hostname=None, session=None):
    499     # SSLSocket class handles server_hostname encoding before it calls
    500     # ctx._wrap_socket()
--> 501     return self.sslsocket_class._create(
    502         sock=sock,
    503         server_side=server_side,
    504         do_handshake_on_connect=do_handshake_on_connect,
    505         suppress_ragged_eofs=suppress_ragged_eofs,
    506         server_hostname=server_hostname,
    507         context=self,
    508         session=session
    509     )

File /srv/conda/envs/notebook/lib/python3.9/ssl.py:1041, in SSLSocket._create(cls, sock, server_side, do_handshake_on_connect, suppress_ragged_eofs, server_hostname, context, session)
   1040             raise ValueError("do_handshake_on_connect should not be specified for non-blocking sockets")
-> 1041         self.do_handshake()
   1042 except (OSError, ValueError):

File /srv/conda/envs/notebook/lib/python3.9/ssl.py:1310, in SSLSocket.do_handshake(self, block)
   1309         self.settimeout(None)
-> 1310     self._sslobj.do_handshake()
   1311 finally:

SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1129)

During handling of the above exception, another exception occurred:

MaxRetryError                             Traceback (most recent call last)
File /srv/conda/envs/notebook/lib/python3.9/site-packages/requests/adapters.py:489, in HTTPAdapter.send(self, request, stream, timeout, verify, cert, proxies)
    488 if not chunked:
--> 489     resp = conn.urlopen(
    490         method=request.method,
    491         url=url,
    492         body=request.body,
    493         headers=request.headers,
    494         redirect=False,
    495         assert_same_host=False,
    496         preload_content=False,
    497         decode_content=False,
    498         retries=self.max_retries,
    499         timeout=timeout,
    500     )
    502 # Send the request.
    503 else:

File /srv/conda/envs/notebook/lib/python3.9/site-packages/urllib3/connectionpool.py:787, in HTTPConnectionPool.urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)
    785     e = ProtocolError("Connection aborted.", e)
--> 787 retries = retries.increment(
    788     method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
    789 )
    790 retries.sleep()

File /srv/conda/envs/notebook/lib/python3.9/site-packages/urllib3/util/retry.py:592, in Retry.increment(self, method, url, response, error, _pool, _stacktrace)
    591 if new_retry.is_exhausted():
--> 592     raise MaxRetryError(_pool, url, error or ResponseError(cause))
    594 log.debug("Incremented Retry for (url='%s'): %r", url, new_retry)

MaxRetryError: HTTPSConnectionPool(host='stamen-tiles-a.a.ssl.fastly.net', port=443): Max retries exceeded with url: /terrain/3/0/1.png (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1129)')))

During handling of the above exception, another exception occurred:

SSLError                                  Traceback (most recent call last)
Cell In[27], line 2
      1 ax = all_endorheic_basins.plot(figsize=(20, 20))
----> 2 ctx.add_basemap(ax, crs="EPSG:4327")

File /srv/conda/envs/notebook/lib/python3.9/site-packages/contextily/plotting.py:121, in add_basemap(ax, zoom, source, interpolation, attribution, attribution_size, reset_extent, crs, resampling, **extra_imshow_args)
    117     left, right, bottom, top = _reproj_bb(
    118         left, right, bottom, top, crs, {"init": "epsg:3857"}
    119     )
    120 # Download image
--> 121 image, extent = bounds2img(
    122     left, bottom, right, top, zoom=zoom, source=source, ll=False
    123 )
    124 # Warping
    125 if crs is not None:

File /srv/conda/envs/notebook/lib/python3.9/site-packages/contextily/tile.py:222, in bounds2img(w, s, e, n, zoom, source, ll, wait, max_retries)
    220 x, y, z = t.x, t.y, t.z
    221 tile_url = provider.build_url(x=x, y=y, z=z)
--> 222 image = _fetch_tile(tile_url, wait, max_retries)
    223 tiles.append(t)
    224 arrays.append(image)

File /srv/conda/envs/notebook/lib/python3.9/site-packages/joblib/memory.py:594, in MemorizedFunc.__call__(self, *args, **kwargs)
    593 def __call__(self, *args, **kwargs):
--> 594     return self._cached_call(args, kwargs)[0]

File /srv/conda/envs/notebook/lib/python3.9/site-packages/joblib/memory.py:537, in MemorizedFunc._cached_call(self, args, kwargs, shelving)
    534         must_call = True
    536 if must_call:
--> 537     out, metadata = self.call(*args, **kwargs)
    538     if self.mmap_mode is not None:
    539         # Memmap the output at the first call to be consistent with
    540         # later calls
    541         if self._verbose:

File /srv/conda/envs/notebook/lib/python3.9/site-packages/joblib/memory.py:779, in MemorizedFunc.call(self, *args, **kwargs)
    777 if self._verbose > 0:
    778     print(format_call(self.func, args, kwargs))
--> 779 output = self.func(*args, **kwargs)
    780 self.store_backend.dump_item(
    781     [func_id, args_id], output, verbose=self._verbose)
    783 duration = time.time() - start_time

File /srv/conda/envs/notebook/lib/python3.9/site-packages/contextily/tile.py:252, in _fetch_tile(tile_url, wait, max_retries)
    250 @memory.cache
    251 def _fetch_tile(tile_url, wait, max_retries):
--> 252     request = _retryer(tile_url, wait, max_retries)
    253     with io.BytesIO(request.content) as image_stream:
    254         image = Image.open(image_stream).convert("RGBA")

File /srv/conda/envs/notebook/lib/python3.9/site-packages/contextily/tile.py:395, in _retryer(tile_url, wait, max_retries)
    375 """
    376 Retry a url many times in attempt to get a tile
    377 
   (...)
    392 request object containing the web response.
    393 """
    394 try:
--> 395     request = requests.get(tile_url, headers={"user-agent": USER_AGENT})
    396     request.raise_for_status()
    397 except requests.HTTPError:

File /srv/conda/envs/notebook/lib/python3.9/site-packages/requests/api.py:73, in get(url, params, **kwargs)
     62 def get(url, params=None, **kwargs):
     63     r"""Sends a GET request.
     64 
     65     :param url: URL for the new :class:`Request` object.
   (...)
     70     :rtype: requests.Response
     71     """
---> 73     return request("get", url, params=params, **kwargs)

File /srv/conda/envs/notebook/lib/python3.9/site-packages/requests/api.py:59, in request(method, url, **kwargs)
     55 # By using the 'with' statement we are sure the session is closed, thus we
     56 # avoid leaving sockets open which can trigger a ResourceWarning in some
     57 # cases, and look like a memory leak in others.
     58 with sessions.Session() as session:
---> 59     return session.request(method=method, url=url, **kwargs)

File /srv/conda/envs/notebook/lib/python3.9/site-packages/requests/sessions.py:587, in Session.request(self, method, url, params, data, headers, cookies, files, auth, timeout, allow_redirects, proxies, hooks, stream, verify, cert, json)
    582 send_kwargs = {
    583     "timeout": timeout,
    584     "allow_redirects": allow_redirects,
    585 }
    586 send_kwargs.update(settings)
--> 587 resp = self.send(prep, **send_kwargs)
    589 return resp

File /srv/conda/envs/notebook/lib/python3.9/site-packages/requests/sessions.py:701, in Session.send(self, request, **kwargs)
    698 start = preferred_clock()
    700 # Send the request
--> 701 r = adapter.send(request, **kwargs)
    703 # Total elapsed time of the request (approximately)
    704 elapsed = preferred_clock() - start

File /srv/conda/envs/notebook/lib/python3.9/site-packages/requests/adapters.py:563, in HTTPAdapter.send(self, request, stream, timeout, verify, cert, proxies)
    559         raise ProxyError(e, request=request)
    561     if isinstance(e.reason, _SSLError):
    562         # This branch is for urllib3 v1.22 and later.
--> 563         raise SSLError(e, request=request)
    565     raise ConnectionError(e, request=request)
    567 except ClosedPoolError as e:

SSLError: HTTPSConnectionPool(host='stamen-tiles-a.a.ssl.fastly.net', port=443): Max retries exceeded with url: /terrain/3/0/1.png (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1129)')))

[smHooper]

One potential workaround is to use os.environ['REQUESTS_CA_BUNDLE'] = your_cert_path somewhere in your own script before adding tiles with contextily. This sets the certificate to use globally while your script is running (you could also presumably set it for your OS, but then your script is less portable). I haven't been able to reproduce the contextily-specific requests issue for some reason, but it does resolve my problem when calling requests.get() directly.

[dbetchkal]

I can confirm that the above workaround using os.environ resolves my original issue.