canonical / grafana-agent-operator

https://charmhub.io/grafana-agent
Apache License 2.0
4 stars 11 forks source link

fix: configure prometheus_remote_write config for non-leader #168

Closed ca-scribner closed 3 months ago

ca-scribner commented 3 months ago

Issue

This PR fixes this issue where grafana-agent is not correctly setting its prometheus_remote_write settings for non-leader units, leading to non-leader units not properly writing their data to prometheus

Solution

The issue is caused by leadership guards in the grafana-cloud-integrator library's GrafanaCloudConfigRequirer relation handling. Specifically, it does nothing on relation changed/relation broken events.

The solution here is:

Context

Testing Instructions

juju add-model ga1
juju deploy ubuntu -n 2
juju deploy grafana-agent --channel=latest/edge
juju deploy grafana-cloud-integrator --channel latest/edge --config prometheus-url=http://some.domain.name:9090/api/v1/write
juju relate grafana-agent ubuntu
juju relate grafana-agent:grafana-cloud-config grafana-cloud-integrator:grafana-cloud-config

# Then compare the following:
juju ssh grafana-agent/0 -- cat /etc/grafana-agent.yaml | yq .integrations.prometheus_remote_write
juju ssh grafana-agent/1 -- cat /etc/grafana-agent.yaml | yq .integrations.prometheus_remote_write

where we should see both units having the same data:

- basic_auth:
    password: ''
    username: ''
  tls_config:
    insecure_skip_verify: false
  url: http://some.domain.name:9090/api/v1/write

Upgrade Notes