changing some charm configs does not push new alert rules into relation data with prometheus

[nishant-dash]

Bug Description

• After deploying cos-config and relating it to prometheus, I see new alert rules in prometheus.

• After changing the following charm configs,

  git_repo
  git_branch
  prometheus_alert_rules_path

the data bag gets emptied out

    "endpoint": "metrics-endpoint",
    "related-endpoint": "prometheus-config",
    "application-data": {
      "alert_rules": "{}"
    },

I checked the cos-configuration pod and I see it successfully pulled in the the data from github.

• Workaround

  juju remove-relation cos-configuration prometheus
  # wait till hooks are done executing
  juju integrate cos-configuration prometheus

To Reproduce

1. juju deploy cos and cos coniguration
2. set cos config charm configs
3. relate cos config and cos
4. change charm configs for

   git_repo
   git_branch
   prometheus_alert_rules_path

Environment

cos    manual-controller  microk8s-cloud/localhost  3.1.7    unsupported  16:05:57Z
...
cos-configuration  3.5.0    active      1  cos-configuration-k8s  latest/stable   42 ...
prometheus         2.47.2   active      1  prometheus-k8s         stable         159 ...
...

Relevant log output

N/A

Additional context

No response

[lucabello]

We should solve this by integrating COS Config with Prometheus via a metrics endpoint relation, and make sure to pass config_changed in the refresh events.

Not closing this yet, but the implementation should happen in #8.