certificate verify failed issue when using Get Namespaced Pod Exec

[yannpichon-su]

When using the Get Namespaced Pod Exec keywork on a k8s cluster using a custom CA, the following error occurs :

ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get issuer certificate (_ssl.c:1129)

Other keywords (Read Namespaced Pod Status, List Namespaced Pod By Pattern ...) are working as expected.

As a quick fix, I'm adding the following line in the _add_api method of the library :

def _add_api(self, reference, class_name):
    self.__dict__[reference] = class_name(self.api_client)
    if not self.cert_validation:
        self.__dict__[reference].api_client.rest_client.pool_manager.connection_pool_kw['cert_reqs'] = ssl.CERT_NONE
        self.__dict__[reference].api_client.configuration.verify_ssl = False

Am I missing something regarding the library configuration ?

Versions :

KubeLibrary: 0.8.0
Python: 3.9.13
Kubernetes: 1.24

KubeLibrary :

Library    KubeLibrary    kube_config=${KUBECONFIG_FILE}    cert_validation=False

KubeLibrary.Get Namespaced Pod Exec    
...    name=my-pod
...    namespace=${namespace}
...    argv_cmd=${command}

[m-wcislo]

Yeah your workaround is way to go, it does the current setting underneath so it is preferred, Please take a look at v0.8.1 at let me know if it works for you.

[fdaguin]

Thanks for the release @m-wcislo! However, removing the line https://github.com/devopsspiral/KubeLibrary/pull/124/files#diff-65edc7c7cbe7902bb5d18f0b4555bbf308e8cac89745ce248bec177fd9a9b334L279 introduced a regression.

For example, using the keyword List Namespaced Daemon Set leads to the following error:

WARN    Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get issuer certificate (_ssl.c:1129)'))': /k8s/clusters/***redacted***/apis/apps/v1/namespaces/demo/daemonsets?labelSelector=&watch=False
09:30:49.767
WARN    Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get issuer certificate (_ssl.c:1129)'))': /k8s/clusters/***redacted***/apis/apps/v1/namespaces/demo/daemonsets?labelSelector=&watch=False
09:30:49.850
WARN    Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get issuer certificate (_ssl.c:1129)'))': /k8s/clusters/***redacted***/apis/apps/v1/namespaces/demo/daemonsets?labelSelector=&watch=False
09:30:49.929
FAIL    MaxRetryError: HTTPSConnectionPool(host='***redacted***', port=443): Max retries exceeded with url: /k8s/clusters/***redacted***/apis/apps/v1/namespaces/demo/daemonsets?labelSelector=&watch=False (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get issuer certificate (_ssl.c:1129)')))
09:30:49.930
DEBUG   Traceback (most recent call last):
  File "***redacted***/.venv/lib/python3.9/site-packages/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
  File "***redacted***/.venv/lib/python3.9/site-packages/urllib3/connectionpool.py", line 386, in _make_request
    self._validate_conn(conn)
  File "***redacted***/.venv/lib/python3.9/site-packages/urllib3/connectionpool.py", line 1042, in _validate_conn
    conn.connect()
  File "***redacted***/.venv/lib/python3.9/site-packages/urllib3/connection.py", line 414, in connect
    self.sock = ssl_wrap_socket(
  File "***redacted***/.venv/lib/python3.9/site-packages/urllib3/util/ssl_.py", line 449, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(
  File "***redacted***/.venv/lib/python3.9/site-packages/urllib3/util/ssl_.py", line 493, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
  File "/home/linuxbrew/.linuxbrew/opt/python@3.9/lib/python3.9/ssl.py", line 501, in wrap_socket
    return self.sslsocket_class._create(
  File "/home/linuxbrew/.linuxbrew/opt/python@3.9/lib/python3.9/ssl.py", line 1041, in _create
    self.do_handshake()
  File "/home/linuxbrew/.linuxbrew/opt/python@3.9/lib/python3.9/ssl.py", line 1310, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get issuer certificate (_ssl.c:1129)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "***redacted***/.venv/lib/python3.9/site-packages/KubeLibrary/KubeLibrary.py", line 1378, in list_namespaced_daemon_set
    ret = self.appsv1.list_namespaced_daemon_set(namespace, watch=False, label_selector=label_selector)
  File "***redacted***/.venv/lib/python3.9/site-packages/kubernetes/client/api/apps_v1_api.py", line 3181, in list_namespaced_daemon_set
    return self.list_namespaced_daemon_set_with_http_info(namespace, **kwargs)  # noqa: E501
  File "***redacted***/.venv/lib/python3.9/site-packages/kubernetes/client/api/apps_v1_api.py", line 3296, in list_namespaced_daemon_set_with_http_info
    return self.api_client.call_api(
  File "***redacted***/.venv/lib/python3.9/site-packages/kubernetes/client/api_client.py", line 348, in call_api
    return self.__call_api(resource_path, method,
  File "***redacted***/.venv/lib/python3.9/site-packages/kubernetes/client/api_client.py", line 180, in __call_api
    response_data = self.request(
  File "***redacted***/.venv/lib/python3.9/site-packages/kubernetes/client/api_client.py", line 373, in request
    return self.rest_client.GET(url,
  File "***redacted***/.venv/lib/python3.9/site-packages/kubernetes/client/rest.py", line 241, in GET
    return self.request("GET", url,
  File "***redacted***/.venv/lib/python3.9/site-packages/kubernetes/client/rest.py", line 214, in request
    r = self.pool_manager.request(method, url,
  File "***redacted***/.venv/lib/python3.9/site-packages/urllib3/request.py", line 74, in request
    return self.request_encode_url(
  File "***redacted***/.venv/lib/python3.9/site-packages/urllib3/request.py", line 96, in request_encode_url
    return self.urlopen(method, url, **extra_kw)
  File "***redacted***/.venv/lib/python3.9/site-packages/urllib3/poolmanager.py", line 376, in urlopen
    response = conn.urlopen(method, u.request_uri, **kw)
  File "***redacted***/.venv/lib/python3.9/site-packages/urllib3/connectionpool.py", line 815, in urlopen
    return self.urlopen(
  File "***redacted***/.venv/lib/python3.9/site-packages/urllib3/connectionpool.py", line 815, in urlopen
    return self.urlopen(
  File "***redacted***/.venv/lib/python3.9/site-packages/urllib3/connectionpool.py", line 815, in urlopen
    return self.urlopen(
  File "***redacted***/.venv/lib/python3.9/site-packages/urllib3/connectionpool.py", line 787, in urlopen
    retries = retries.increment(
  File "***redacted***/.venv/lib/python3.9/site-packages/urllib3/util/retry.py", line 592, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='***redacted***', port=443): Max retries exceeded with url: /k8s/clusters/***redacted***/apis/apps/v1/namespaces/demo/daemonsets?labelSelector=&watch=False (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get issuer certificate (_ssl.c:1129)')))

Could it be reverted please?

[m-wcislo]

ok, reverted in 0.8.2, need to take closer look.

[fdaguin]

Thank you! From our tests, keeping both

    if not self.cert_validation:
        self.__dict__[reference].api_client.rest_client.pool_manager.connection_pool_kw['cert_reqs'] = ssl.CERT_NONE
        self.__dict__[reference].api_client.configuration.verify_ssl = False

was working as expected.

[m-wcislo]

Actually it is reverted in 0.8.3. yeah need to track down how the setting is propagated, latest setting should set both

[m-wcislo]

Ok, did some testing and cannot really reproduce it on my end, so have couple questions:

1. Did it worked in the past for you and is just broken recently? If yes what changes were made?
2. Your kubeconfig doesn't have :

   clusters:
   - cluster:
   certificate-authority-data: ?

   This is the easiest way to make your connection trust the API server cert, without need of use cert_validation=False

3. Are the failing (Get Namespaced Pod Exec) tests are executed somehow differently (different kubeconfig, settings etc.)than the ones that are passing (List Namespaced Daemon Set)?
4. Would it be possible to list packages versions you are using (pip freeze) at least for things in reuirements.txt