Pod service rigger Start failed with error '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get issuer certificate (_ssl.c:1131)')))

[haloxinyu]

Just follow the quick start steps to create redis-enterprise, but pod "services-rigger" always failed to start.

eli-master-60:~/eli # kubectl get all

NAME                                            READY   STATUS    RESTARTS       AGE
pod/my-rec-0                                    1/2     Running   0              21m
pod/my-rec-1                                    1/2     Running   3 (36s ago)    17m
pod/my-rec-services-rigger-7cdd4c5577-9pgtl     0/1     Error     9 (5m7s ago)   21m
pod/redis-enterprise-operator-66df8965f-wh7fg   2/2     Running   0              33m

NAME                  TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)             AGE
service/admission     ClusterIP   10.43.142.246   <none>        443/TCP             33m
service/my-rec        ClusterIP   10.43.199.102   <none>        9443/TCP,8001/TCP   21m
service/my-rec-prom   ClusterIP   None            <none>        8070/TCP            21m
service/my-rec-ui     ClusterIP   10.43.18.55     <none>        8443/TCP            21m

NAME                                        READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/my-rec-services-rigger      0/1     1            0           21m
deployment.apps/redis-enterprise-operator   1/1     1            1           33m

NAME                                                  DESIRED   CURRENT   READY   AGE
replicaset.apps/my-rec-services-rigger-7cdd4c5577     1         1         0       21m
replicaset.apps/redis-enterprise-operator-66df8965f   1         1         1       33m

NAME                      READY   AGE
statefulset.apps/my-rec   0/3     21m

ERROR Log:

2023-03-22 16:25:27,393 - services-rigger.config - INFO - Getting updated credentials
2023-03-22 16:25:27,394 - services-rigger.config - INFO - read username and password from kubernetes
--- Logging error ---
Traceback (most recent call last):
  File "/usr/lib64/python3.8/logging/__init__.py", line 1085, in emit
    msg = self.format(record)
  File "/usr/lib64/python3.8/logging/__init__.py", line 929, in format
    return fmt.format(record)
  File "/usr/lib64/python3.8/logging/__init__.py", line 668, in format
    record.message = record.getMessage()
  File "/usr/lib64/python3.8/logging/__init__.py", line 373, in getMessage
    msg = msg % self.args
TypeError: not all arguments converted during string formatting
Call stack:
  File "/usr/lib64/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib64/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/opt/redislabs/services-rigger/__main__.py", line 137, in <module>
    main()
  File "/opt/redislabs/services-rigger/__main__.py", line 42, in main
    config = Config()
  File "/opt/redislabs/services-rigger/config.py", line 82, in __init__
    self.configure()
  File "/opt/redislabs/services-rigger/config.py", line 222, in configure
    logger.info('services-rigger configured with:\nnamespace:%s\nredis-enterprise host:%s,\n'
Message: 'services-rigger configured with:\nnamespace:%s\nredis-enterprise host:%s,\nredis-enterprise username:%s,\nredis-enterprise port:%s,\nservice types:%s,\nowner ref:%s\nservice naming:%s,\nactive-active method:%s\ncrdb url suffix:%s\ningress annotations:%s\nistio gateway name:%s\nistio gateway port:%s\niteration sleep time:%s\n'
Arguments: ('redis', 'my-rec', 'demo@redislabs.com', 9443, ['cluster_ip', 'headless'], {'apiVersion': 'app.redislabs.com/v1alpha1', 'kind': 'RedisEnterpriseCluster', 'name': 'my-rec', 'uid': 'dc68a8f1-9f4e-4a12-9005-54cdd34369a7', 'controller': True, 'blockOwnerDeletion': True}, ['bdb_name'], None, None, None, {}, None, None, 0.5)
2023-03-22 16:25:27,403 - urllib3.connectionpool - WARNING - Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get issuer certificate (_ssl.c:1131)'))': /api/v1/namespaces/redis/services?labelSelector=app%3Dredis-enterprise-bdb%2Credis.io%2Fbdb
2023-03-22 16:25:27,405 - urllib3.connectionpool - WARNING - Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get issuer certificate (_ssl.c:1131)'))': /api/v1/namespaces/redis/services?labelSelector=app%3Dredis-enterprise-bdb%2Credis.io%2Fbdb
2023-03-22 16:25:27,407 - urllib3.connectionpool - WARNING - Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get issuer certificate (_ssl.c:1131)'))': /api/v1/namespaces/redis/services?labelSelector=app%3Dredis-enterprise-bdb%2Credis.io%2Fbdb
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 386, in _make_request
    self._validate_conn(conn)
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 1042, in _validate_conn
    conn.connect()
  File "/usr/local/lib/python3.8/site-packages/urllib3/connection.py", line 414, in connect
    self.sock = ssl_wrap_socket(
  File "/usr/local/lib/python3.8/site-packages/urllib3/util/ssl_.py", line 453, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(sock, context, tls_in_tls)
  File "/usr/local/lib/python3.8/site-packages/urllib3/util/ssl_.py", line 495, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock)
  File "/usr/lib64/python3.8/ssl.py", line 500, in wrap_socket
    return self.sslsocket_class._create(
  File "/usr/lib64/python3.8/ssl.py", line 1040, in _create
    self.do_handshake()
  File "/usr/lib64/python3.8/ssl.py", line 1309, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get issuer certificate (_ssl.c:1131)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib64/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib64/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/opt/redislabs/services-rigger/__main__.py", line 137, in <module>
    main()
  File "/opt/redislabs/services-rigger/__main__.py", line 51, in main
    services_handler.rigger_upgrade()
  File "/opt/redislabs/services-rigger/services.py", line 95, in rigger_upgrade
    old_bdbs = self.get_all_bdb_services(old_service_label)
  File "/opt/redislabs/services-rigger/services.py", line 278, in get_all_bdb_services
    services = self.k8s_v1_client.list_namespaced_service(self.config.namespace, label_selector=label_selector)
  File "/usr/local/lib/python3.8/site-packages/kubernetes/client/api/core_v1_api.py", line 16243, in list_namespaced_service
    return self.list_namespaced_service_with_http_info(namespace, **kwargs)  # noqa: E501
  File "/usr/local/lib/python3.8/site-packages/kubernetes/client/api/core_v1_api.py", line 16358, in list_namespaced_service_with_http_info
    return self.api_client.call_api(
  File "/usr/local/lib/python3.8/site-packages/kubernetes/client/api_client.py", line 348, in call_api
    return self.__call_api(resource_path, method,
  File "/usr/local/lib/python3.8/site-packages/kubernetes/client/api_client.py", line 180, in __call_api
    response_data = self.request(
  File "/usr/local/lib/python3.8/site-packages/kubernetes/client/api_client.py", line 373, in request
    return self.rest_client.GET(url,
  File "/usr/local/lib/python3.8/site-packages/kubernetes/client/rest.py", line 240, in GET
    return self.request("GET", url,
  File "/usr/local/lib/python3.8/site-packages/kubernetes/client/rest.py", line 213, in request
    r = self.pool_manager.request(method, url,
  File "/usr/local/lib/python3.8/site-packages/urllib3/request.py", line 74, in request
    return self.request_encode_url(
  File "/usr/local/lib/python3.8/site-packages/urllib3/request.py", line 96, in request_encode_url
    return self.urlopen(method, url, **extra_kw)
  File "/usr/local/lib/python3.8/site-packages/urllib3/poolmanager.py", line 376, in urlopen
    response = conn.urlopen(method, u.request_uri, **kw)
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 815, in urlopen
    return self.urlopen(
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 815, in urlopen
    return self.urlopen(
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 815, in urlopen
    return self.urlopen(
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 787, in urlopen
    retries = retries.increment(
  File "/usr/local/lib/python3.8/site-packages/urllib3/util/retry.py", line 592, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='10.43.0.1', port=443): Max retries exceeded with url: /api/v1/namespaces/redis/services?labelSelector=app%3Dredis-enterprise-bdb%2Credis.io%2Fbdb (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get issuer certificate (_ssl.c:1131)')))