search

canonical / grafana-agent-k8s-operator

https://charmhub.io/grafana-agent-k8s
Apache License 2.0
8 stars 18 forks source link

Grafana Dashboards not showing up on Grafana #231

Closed deusebio closed 1 year ago

deusebio commented 1 year ago

Bug Description

In Data Platform for k8s deployment, we used to relate components (e.g. Kafka) directly to cos, across models. I'm currently testing to provide relations mediated by grafana agent, as suggested in @simskij comment here. However, when doing this, Grafana dashboard are not appearing in Grafana.

To Reproduce

Deploy Kafka on Apps model

  1. juju add-model apps
  2. juju deploy kafka-k8s --channel edge
  3. juju deploy zookeeper-k8s --channel edge
  4. juju related kafka-k8s zookeeper-k8s
  5. juju deploy grafana-agent-k8s --channel candidate --trust

Deploy COS

  1. juju add-model cos
  2. juju deploy cos-lite --trust --overlay ./offers-overlay.yaml --overlay ./storage-small-overlay.yaml

Relate

  1. consume 
    juju consume micro:admin/cos.grafana-dashboards 
juju consume micro:admin/cos.prometheus-receive-remote-write 
juju consume micro:admin/cos.loki-logging
  2. relate agent to cos 
    juju relate grafana-dashboards agent
juju relate loki-logging agent
juju relate prometheus-receive-remote-write agent
  3. relate grafana agent and component 
    juju relate kafka-k8s:grafana-dashboards agent 
juju relate kafka-k8s:logging agent
juju relate kafka-k8s:metrics-endpoint agent

When I then log into grafana, dahsboard are not there. Note that dashboard are there when relating kafka-k8s directly to the cos offer (e.g. juju relate kafka:grafana-dashboards grafana-dashboards)

Environment

Juju 3.1.5 microk8s 1.27-strict/stable

COS MODEL

Model  Controller  Cloud/Region        Version  SLA          Timestamp
cos    micro       microk8s/localhost  3.1.5    unsupported  14:59:05Z

App           Version  Status  Scale  Charm             Channel  Rev  Address         Exposed  Message
alertmanager  0.25.0   active      1  alertmanager-k8s  stable    76  10.152.183.51   no       
catalogue              active      1  catalogue-k8s     stable    13  10.152.183.253  no       
grafana       9.2.1    active      1  grafana-k8s       stable    81  10.152.183.173  no       
loki          2.7.4    active      1  loki-k8s          stable    89  10.152.183.219  no       
prometheus    2.33.5   active      1  prometheus-k8s    stable   103  10.152.183.114  no       
traefik       2.9.6    active      1  traefik-k8s       stable   110  172.31.28.106   no

APPS MODEL

Model  Controller  Cloud/Region        Version  SLA          Timestamp
apps   micro       microk8s/localhost  3.1.5    unsupported  15:14:09Z

SAAS                             Status  Store  URL
grafana-dashboards               active  micro  admin/cos.grafana-dashboards
loki-logging                     active  micro  admin/cos.loki-logging
prometheus-receive-remote-write  active  micro  admin/cos.prometheus-receive-remote-write
prometheus-scrape                active  micro  admin/cos.prometheus-scrape

App            Version  Status  Scale  Charm              Channel  Rev  Address         Exposed  Message
agent          0.32.1   active      1  grafana-agent-k8s  stable    38  10.152.183.115  no       
kafka-k8s               active      3  kafka-k8s                     0  10.152.183.254  no       
zookeeper-k8s           active      3  zookeeper-k8s      3/edge    31  10.152.183.82   no   

### Relevant log output

```shell
N/A

Additional context

N/A

taurus-forever commented 1 year ago

The same issue reproduced with mysql-k8s and postgresql-k8s charms. No errors in logs/relations and no dashboards on Grafana WEB (if grafana-agent-k8s used).

P.S. To test locally using multipass: create a file DB_COS.yaml with content: https://pastebin.canonical.com/p/JgDJ4HrkxQ/ and run the command indicated in the first line of the file above.

simskij commented 1 year ago

Thanks for making us aware. We'll try to prioritize this issue as soon as possible.

sed-i commented 1 year ago

Related: https://github.com/canonical/grafana-k8s-operator/issues/221

simskij commented 1 year ago

Since we already have an issue for tracking dashboard transfer issues, I'll close this one as a dupe. Reasoning being that if dashboards don't make it over in the first place, they definitely won't be there after a restart either. 😅

deusebio commented 1 year ago

I see that the other ticket (https://github.com/canonical/grafana-k8s-operator/issues/221) was closed, but I have tested again the deployment above (using revision 42 for grafana-agent-k8s), and while alerting rules are now there, grafana dashboard are still not there. Is this expected or shall we re-open this ticket?