charm tracing fails over tls

[michaeldmitry]

Bug Description

When trying to integrate prometheus with tempo to send its charm traces over tls, it fails to do with an exception "Exception while exporting Span batch."

To Reproduce

juju deploy cos-lite --trust juju refresh prometheus --channel latest/edge juju deploy tempo-k8s tempo --channel latest/edge juju deploy self-signed-certificates juju integrate self-signed-certificates:certificates prometheus:certificates juju integrate self-signed-certificates:certificates tempo:certificates juju integrate tempo:tracing prometheus:tracing wait for an event on prometheus (e.g: update-status) look for error in juju logs

Environment

juju version: 3.4.2-genericlinux-amd64 microk8s version: MicroK8s v1.28.7 revision 6532 prometheus rev: 186

Relevant log output

unit-prometheus-0: 12:02:41 ERROR unit.prometheus/0.juju-log Exception while exporting Span batch.
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/urllib3/response.py", line 737, in _error_catcher
    yield
  File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/urllib3/response.py", line 862, in _raw_read
    data = self._fp_read(amt, read1=read1) if not fp_closed else b""
  File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/urllib3/response.py", line 845, in _fp_read
    return self._fp.read(amt) if amt is not None else self._fp.read()
  File "/usr/lib/python3.8/http/client.py", line 459, in read
    n = self.readinto(b)
  File "/usr/lib/python3.8/http/client.py", line 503, in readinto
    n = self.fp.readinto(b)
  File "/usr/lib/python3.8/socket.py", line 669, in readinto
    return self._sock.recv_into(b)
ConnectionResetError: [Errno 104] Connection reset by peer

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/requests/models.py", line 816, in generate
    yield from self.raw.stream(chunk_size, decode_content=True)
  File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/urllib3/response.py", line 1043, in stream
    data = self.read(amt=amt, decode_content=decode_content)
  File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/urllib3/response.py", line 935, in read
    data = self._raw_read(amt)
  File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/urllib3/response.py", line 891, in _raw_read
    self._fp.close()
  File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__
    self.gen.throw(type, value, traceback)
  File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/urllib3/response.py", line 764, in _error_catcher
    raise ProtocolError(f"Connection broken: {e!r}", e) from e
urllib3.exceptions.ProtocolError: ("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/opentelemetry/sdk/trace/export/__init__.py", line 368, in _export_batch
    self.span_exporter.export(self.spans_list[:idx])  # type: ignore
  File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/opentelemetry/exporter/otlp/proto/http/trace_exporter/__init__.py", line 145, in export
    resp = self._export(serialized_data)
  File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/opentelemetry/exporter/otlp/proto/http/trace_exporter/__init__.py", line 114, in _export
    return self._session.post(
  File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/requests/sessions.py", line 637, in post
    return self.request("POST", url, data=data, json=json, **kwargs)
  File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
  File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/requests/sessions.py", line 747, in send
    r.content
  File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/requests/models.py", line 899, in content
    self._content = b"".join(self.iter_content(CONTENT_CHUNK_SIZE)) or b""
  File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/requests/models.py", line 818, in generate
    raise ChunkedEncodingError(e)
requests.exceptions.ChunkedEncodingError: ("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer'))

Additional context

No response