search

canonical / cos-configuration-k8s-operator

https://charmhub.io/cos-configuration-k8s
Apache License 2.0
4 stars 6 forks source link

Config change after deployment does not update Grafana dashboard #73

Closed cbartz closed 3 months ago

cbartz commented 12 months ago

Bug Description

Changing the configuration (e.g. grafana_dashboards) does not update the Grafana dashboard. This seems to be due to the fact that the dashboard is only updated during an install or upgrade event:

https://github.com/canonical/cos-configuration-k8s-operator/blob/051e1461430edb2d342f8b2088a79d363fd02140/lib/charms/grafana_k8s/v0/grafana_dashboard.py#L1051-L1052

As a user, I would expect the dashboard data to be updated when the configuration is changed.

To Reproduce

  1. juju deploy --trust grafana-k8s grafana
  2. juju deploy cos-configuration-k8s --config git_repo=https://github.com/canonical/github-runner-operator --config git_branch=main
  3. juju config cos-configuration-k8s grafana_dashboards_path=src/grafana_dashboard_metrics
  4. juju relate cos-configuration-k8s:grafana-dashboards grafana:grafana-dashboard
  5. juju show-unit grafana/0 --format json | jq '."grafana/0"."relation-info"'

The last command shows an empty dashboard in the integration data. It would not be empty if the configuration value grafana_dashboards_path=src/grafana_dashboard_metric had been passed at deployment time.

Environment

Juju (tested with 3.1 and 2.9) within multipass using microk8s.

Relevant log output

nit-cos-configuration-k8s-0: 14:14:25 INFO unit.cos-configuration-k8s/0.juju-log git-sync: I1012 12:14:24.881140     135 main.go:473] "level"=0 "msg"="starting up" "pid"=135 "args"=["/git-sync","--repo","https://github.com/canonical/github-runner-operator","--branch","main","--rev","HEAD","--depth","1","--root","/git","--dest","repo","--one-time"]
unit-cos-configuration-k8s-0: 14:14:25 INFO juju.worker.uniter.operation ran "config-changed" hook (via hook dispatching script: dispatch)
unit-cos-configuration-k8s-0: 14:14:40 INFO juju.worker.uniter.operation ran "grafana-dashboards-relation-created" hook (via hook dispatching script: dispatch)
unit-grafana-0: 14:14:42 INFO unit.grafana/0.juju-log grafana-dashboard:39: Restarted grafana-k8s
unit-grafana-0: 14:14:43 INFO juju.worker.uniter.operation ran "grafana-dashboard-relation-changed" hook (via hook dispatching script: dispatch)
unit-grafana-0: 14:14:43 INFO juju.worker.uniter.operation ran "grafana-dashboard-relation-joined" hook (via hook dispatching script: dispatch)
unit-grafana-0: 14:14:44 INFO juju.worker.uniter.operation ran "grafana-relation-changed" hook (via hook dispatching script: dispatch)
unit-grafana-0: 14:14:45 INFO unit.grafana/0.juju-log grafana-dashboard:39: HTTP Request: GET https://10.152.183.1/apis/apps/v1/namespaces/observability/statefulsets/grafana "HTTP/1.1 200 OK"
unit-grafana-0: 14:14:45 INFO unit.grafana/0.juju-log grafana-dashboard:39: HTTP Request: GET https://10.152.183.1/api/v1/namespaces/observability/pods/grafana-0 "HTTP/1.1 200 OK"
unit-grafana-0: 14:14:45 INFO unit.grafana/0.juju-log grafana-dashboard:39: reqs=ResourceRequirements(claims=None, limits={}, requests={'cpu': '0.25', 'memory': '200Mi'}), templated=ResourceRequirements(claims=None, limits=None, requests={'cpu': '250m', 'memory': '200Mi'}), actual=ResourceRequirements(claims=None, limits=None, requests={'cpu': '250m', 'memory': '200Mi'})
unit-grafana-0: 14:14:45 INFO unit.grafana/0.juju-log grafana-dashboard:39: HTTP Request: GET https://10.152.183.1/apis/apps/v1/namespaces/observability/statefulsets/grafana "HTTP/1.1 200 OK"
unit-grafana-0: 14:14:45 INFO unit.grafana/0.juju-log grafana-dashboard:39: HTTP Request: GET https://10.152.183.1/api/v1/namespaces/observability/pods/grafana-0 "HTTP/1.1 200 OK"
unit-grafana-0: 14:14:45 INFO unit.grafana/0.juju-log grafana-dashboard:39: Initializing dashboard provisioning path
unit-grafana-0: 14:14:46 INFO unit.grafana/0.juju-log grafana-dashboard:39: Restarted grafana-k8s
unit-grafana-0: 14:14:46 INFO juju.worker.uniter.operation ran "grafana-dashboard-relation-changed" hook (via hook dispatching script: dispatch)
unit-grafana-0: 14:14:47 INFO juju.worker.uniter.operation ran "grafana-relation-changed" hook (via hook dispatching script: dispatch)

Additional context

No response

PietroPasotti commented 12 months ago

I'm not sure that that is the issue, because while the relation provider class is in fact only doing that 'automatically' on install and upgrade, the charm itself is doing that in its common exit hook (https://github.com/canonical/cos-configuration-k8s-operator/blob/051e1461430edb2d342f8b2088a79d363fd02140/src/charm.py#L162C22-L162C22)

which is triggered, among other things, on config-changed.

So there must be something else going on. Perhaps a bug in the hashing?

cbartz commented 12 months ago

Thanks @PietroPasotti for the quick reply. It seems that the update of the boards is not triggered if the git hash does not change, which is indeed true in this case because the config grafana_dashboards is independent of the hash.

https://github.com/canonical/cos-configuration-k8s-operator/blob/051e1461430edb2d342f8b2088a79d363fd02140/src/charm.py#L363

dstathis commented 7 months ago

We should probably just always update on config-changed.

IbraAoad commented 3 months ago

The issue isn't reproducible when following the original steps and the updated dashboards path in the github-runner-operator (grafana_dashboards_path=src/grafana_dashboards).

However, the issue arises if step 4 is performed before step 3 in the reproduction steps (i.e., relating to Grafana before setting the grafana_dashboards_path config). The fix for this issue is addressed in https://github.com/canonical/grafana-k8s-operator/pull/335

IbraAoad commented 3 months ago

should be fixed by #98