Charm is happily "active/idle" when workload cannot resolve host

[sed-i]

Enhancement Proposal

If someone forgets to deploy COS charms with traefik,

graph LR

subgraph lxd
ubuntu --- grafana-agent
end

subgraph microk8s
prometheus
loki
end

grafana-agent --- loki
grafana-agent --- prometheus

then the URLs sent over a CMR would be k8s cluster URLs, such as http://prom-0.prom-endpoints.pebnote.svc.cluster.local:9090/api/v1/write, which won't be routeable.

Feb 27 21:54:44 juju-799803-2 grafana-agent.grafana-agent[8210]: ts=2024-02-27T21:54:44.643267725Z caller=dedupe.go:112 agent=prometheus instance=1bf1b94ab08a361769e96ef841afbe0e component=remote level=warn remote_name=1bf1b9-3b030a url=http://prom-0.prom-endpoints.pebnote.svc.cluster.local:9090/api/v1/write msg="Failed to send batch, retrying" err="Post \"http://prom-0.prom-endpoints.pebnote.svc.cluster.local:9090/api/v1/write\": dial tcp: lookup prom-0.prom-endpoints.pebnote.svc.cluster.local on 127.0.0.53:53: server misbehaving"

It could be handy if the charm blocks when the target URLs are not routeable. Some impl ideas:

• Check on update status, because at deploy time this check could be more racy than on update-status.
• Have a custom pebble notice running in the background that would wake up the charm on "rising/falling edge".

[mmkay]

There's a significant overlap between this issue and #57 - closing this one as a duplicate with a comment in #57 so that we keep the full context in there.