cos-proxy fails on hook "downstream-logging-relation-changed" because of SSL: CERTIFICATE_VERIFY_FAILED

[jeffreychang911]

Bug Description

n SQA testrun ce8325a0-c0fe-46f8-af40-acd7b287c8de, cos-proxy fails to install in hook "downstream-logging-relation-changed".

To Reproduce

To reproduce, deploy cos and then the charmed kubernetes bundle, which includes cos. This issue seems to happen after Dec 13, after some cos update. This issue is not necessarily reproducible, we have seen this bundle deploy without this.

Environment

The environment is a juju maas controller hosting a charmed kubernetes deployment. This deployment is connected to cos, which is hosted on a microk8s, hosted on the same juju maas controller.

Relevant log output

2023-12-13 21:56:09 DEBUG juju.worker.uniter agent.go:22 [AGENT-STATUS] executing: running downstream-logging-relation-changed hook for cos-loki/0
2023-12-13 21:56:09 DEBUG juju.worker.uniter.runner runner.go:719 starting jujuc server  {unix @/var/lib/juju/agents/unit-cos-proxy-0/agent.socket <nil>}
2023-12-13 21:56:09 DEBUG unit.cos-proxy/0.juju-log server.go:325 downstream-logging:32: ops 2.8.0+8.g26c6e95 up and running.
2023-12-13 21:56:09 DEBUG unit.cos-proxy/0.juju-log server.go:325 downstream-logging:32: Emitting Juju event downstream_logging_relation_changed.
2023-12-13 21:56:09 DEBUG unit.cos-proxy/0.juju-log server.go:325 downstream-logging:32: Emitting custom event <VectorConfigChangedEvent via COSProxyCharm/VectorProvider[filebeat_downstream-logging]/on/config_changed[133]>.
2023-12-13 21:56:09 ERROR unit.cos-proxy/0.juju-log server.go:325 downstream-logging:32: Uncaught exception while in charm code:
Traceback (most recent call last):
  File "/usr/lib/python3.10/urllib/request.py", line 1348, in do_open
    h.request(req.get_method(), req.selector, req.data, headers,
  File "/usr/lib/python3.10/http/client.py", line 1283, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.10/http/client.py", line 1329, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.10/http/client.py", line 1278, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.10/http/client.py", line 1038, in _send_output
    self.send(msg)
  File "/usr/lib/python3.10/http/client.py", line 976, in send
    self.connect()
  File "/usr/lib/python3.10/http/client.py", line 1455, in connect
    self.sock = self._context.wrap_socket(self.sock,
  File "/usr/lib/python3.10/ssl.py", line 513, in wrap_socket
    return self.sslsocket_class._create(
  File "/usr/lib/python3.10/ssl.py", line 1100, in _create
    self.do_handshake()
  File "/usr/lib/python3.10/ssl.py", line 1371, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1007)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-cos-proxy-0/charm/./src/charm.py", line 517, in <module>
    main(COSProxyCharm)
  File "/var/lib/juju/agents/unit-cos-proxy-0/charm/venv/ops/main.py", line 436, in main
    _emit_charm_event(charm, dispatcher.event_name)
  File "/var/lib/juju/agents/unit-cos-proxy-0/charm/venv/ops/main.py", line 144, in _emit_charm_event
    event_to_emit.emit(*args, **kwargs)
  File "/var/lib/juju/agents/unit-cos-proxy-0/charm/venv/ops/framework.py", line 340, in emit
    framework._emit(event)
  File "/var/lib/juju/agents/unit-cos-proxy-0/charm/venv/ops/framework.py", line 842, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-cos-proxy-0/charm/venv/ops/framework.py", line 931, in _reemit
    custom_handler(event)
  File "/var/lib/juju/agents/unit-cos-proxy-0/charm/lib/charms/vector/v0/vector.py", line 227, in _on_log_relation_changed
    self.on.config_changed.emit(config=self.config)
  File "/var/lib/juju/agents/unit-cos-proxy-0/charm/venv/ops/framework.py", line 340, in emit
    framework._emit(event)
  File "/var/lib/juju/agents/unit-cos-proxy-0/charm/venv/ops/framework.py", line 842, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-cos-proxy-0/charm/venv/ops/framework.py", line 931, in _reemit
    custom_handler(event)
  File "/var/lib/juju/agents/unit-cos-proxy-0/charm/./src/charm.py", line 395, in _write_vector_config
    r = request.urlopen(dest)
  File "/usr/lib/python3.10/urllib/request.py", line 216, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python3.10/urllib/request.py", line 519, in open
    response = self._open(req, data)
  File "/usr/lib/python3.10/urllib/request.py", line 536, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
  File "/usr/lib/python3.10/urllib/request.py", line 496, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.10/urllib/request.py", line 1391, in https_open
    return self.do_open(http.client.HTTPSConnection, req,
  File "/usr/lib/python3.10/urllib/request.py", line 1351, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1007)>

Additional context

No response

[sed-i]

An aspect of this issue would go away with #118 merged, but we'd need the recv-ca-cert relation to be able to talk with loki behind tls. Would need to render ca_file in vector config. Alternatively, use "insecureSkipVerify" (ref).

[sed-i]

The specific problem described in this issue is now gone with #118 merged. Closing.