When trying to integrate prometheus with tempo to send its charm traces over tls, it fails to do with an exception "Exception while exporting Span batch."
To Reproduce
juju deploy cos-lite --trust
juju refresh prometheus --channel latest/edge
juju deploy tempo-k8s tempo --channel latest/edge
juju deploy self-signed-certificates
juju integrate self-signed-certificates:certificates prometheus:certificates
juju integrate self-signed-certificates:certificates tempo:certificates
juju integrate tempo:tracing prometheus:tracing
wait for an event on prometheus (e.g: update-status)
look for error in juju logs
unit-prometheus-0: 12:02:41 ERROR unit.prometheus/0.juju-log Exception while exporting Span batch.
Traceback (most recent call last):
File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/urllib3/response.py", line 737, in _error_catcher
yield
File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/urllib3/response.py", line 862, in _raw_read
data = self._fp_read(amt, read1=read1) if not fp_closed else b""
File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/urllib3/response.py", line 845, in _fp_read
return self._fp.read(amt) if amt is not None else self._fp.read()
File "/usr/lib/python3.8/http/client.py", line 459, in read
n = self.readinto(b)
File "/usr/lib/python3.8/http/client.py", line 503, in readinto
n = self.fp.readinto(b)
File "/usr/lib/python3.8/socket.py", line 669, in readinto
return self._sock.recv_into(b)
ConnectionResetError: [Errno 104] Connection reset by peer
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/requests/models.py", line 816, in generate
yield from self.raw.stream(chunk_size, decode_content=True)
File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/urllib3/response.py", line 1043, in stream
data = self.read(amt=amt, decode_content=decode_content)
File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/urllib3/response.py", line 935, in read
data = self._raw_read(amt)
File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/urllib3/response.py", line 891, in _raw_read
self._fp.close()
File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__
self.gen.throw(type, value, traceback)
File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/urllib3/response.py", line 764, in _error_catcher
raise ProtocolError(f"Connection broken: {e!r}", e) from e
urllib3.exceptions.ProtocolError: ("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/opentelemetry/sdk/trace/export/__init__.py", line 368, in _export_batch
self.span_exporter.export(self.spans_list[:idx]) # type: ignore
File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/opentelemetry/exporter/otlp/proto/http/trace_exporter/__init__.py", line 145, in export
resp = self._export(serialized_data)
File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/opentelemetry/exporter/otlp/proto/http/trace_exporter/__init__.py", line 114, in _export
return self._session.post(
File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/requests/sessions.py", line 637, in post
return self.request("POST", url, data=data, json=json, **kwargs)
File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/requests/sessions.py", line 589, in request
resp = self.send(prep, **send_kwargs)
File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/requests/sessions.py", line 747, in send
r.content
File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/requests/models.py", line 899, in content
self._content = b"".join(self.iter_content(CONTENT_CHUNK_SIZE)) or b""
File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/requests/models.py", line 818, in generate
raise ChunkedEncodingError(e)
requests.exceptions.ChunkedEncodingError: ("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer'))
Bug Description
When trying to integrate prometheus with tempo to send its charm traces over tls, it fails to do with an exception "Exception while exporting Span batch."
To Reproduce
juju deploy cos-lite --trust juju refresh prometheus --channel latest/edge juju deploy tempo-k8s tempo --channel latest/edge juju deploy self-signed-certificates juju integrate self-signed-certificates:certificates prometheus:certificates juju integrate self-signed-certificates:certificates tempo:certificates juju integrate tempo:tracing prometheus:tracing wait for an event on prometheus (e.g: update-status) look for error in juju logs
Environment
juju version: 3.4.2-genericlinux-amd64 microk8s version: MicroK8s v1.28.7 revision 6532 prometheus rev: 186
Relevant log output
Additional context
No response