Error from remote write.

Bug Description

I got a backtrace in receive-remote-write-relation-created while relating to the grafana-agent machine charm over remote write.

To Reproduce

Not sure exactly but I was related cross-model to grafana-agent.

Environment

                       juju info v0.1

┌──────────────┬───────────────────────────────────────────────────┐ │ jhack │ 0.3.18.3 │ │ python │ 3.10.6 (/home/dylan/repos/jhack/venv/bin/python3) │ │ juju-* snaps │ juju │ 3.1.5 - 23354 (latest/stable) │ │ microk8s │ MicroK8s v1.27.2 revision 5372 │ │ lxd │ 5.0.2 │ │ multipass │ Not Installed. │ │ multipassd │ Not Installed. │ │ os │ Ubuntu 22.04.2 LTS │ │ kernel │ Linux 5.15.0-76-generic x86_64 │ └──────────────┴───────────────────────────────────────────────────┘

Relevant log output

unit-prometheus-0: 21:45:07 ERROR unit.prometheus/0.juju-log receive-remote-write:48: Uncaught exception while in charm code:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/ops/model.py", line 2693, in _run
    result = subprocess.run(args, **kwargs)  # type: ignore
  File "/usr/lib/python3.8/subprocess.py", line 516, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '('/var/lib/juju/tools/unit-prometheus-0/relation-set', '-r', '48', '--file', '-')' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./src/charm.py", line 797, in <module>
    main(PrometheusCharm)
  File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/ops/main.py", line 441, in main
    _emit_charm_event(charm, dispatcher.event_name)
  File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/ops/main.py", line 149, in _emit_charm_event
    event_to_emit.emit(*args, **kwargs)
  File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/ops/framework.py", line 344, in emit
    framework._emit(event)
  File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/ops/framework.py", line 833, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/ops/framework.py", line 922, in _reemit
    custom_handler(event)
  File "/var/lib/juju/agents/unit-prometheus-0/charm/lib/charms/prometheus_k8s/v0/prometheus_remote_write.py", line 860, in _on_consumers_changed
    self.update_endpoint(event.relation)
  File "/var/lib/juju/agents/unit-prometheus-0/charm/lib/charms/prometheus_k8s/v0/prometheus_remote_write.py", line 884, in update_endpoint
    self._set_endpoint_on_relation(relation)
  File "/var/lib/juju/agents/unit-prometheus-0/charm/lib/charms/prometheus_k8s/v0/prometheus_remote_write.py", line 902, in _set_endpoint_on_relation
    relation.data[self._charm.unit]["remote_write"] = json.dumps(
  File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/ops/model.py", line 1490, in __setitem__
    self._commit(key, value)
  File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/ops/model.py", line 1494, in _commit
    self._backend.update_relation_data(self.relation.id, self._entity, key, value)
  File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/ops/model.py", line 3029, in update_relation_data
    self.relation_set(relation_id, key, value, isinstance(_entity, Application))
  File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/ops/model.py", line 2798, in relation_set
    self._run(*args, input_stream=content)
  File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/ops/model.py", line 2695, in _run
    raise ModelError(e.stderr)
ops.model.ModelError: ERROR cannot read relation settings: unit "prometheus/0": settings not found

unit-prometheus-0: 21:45:08 ERROR juju.worker.uniter.operation hook "receive-remote-write-relation-created" (via hook dispatching script: dispatch) failed: exit status 1

Additional context

No response

canonical / prometheus-k8s-operator