No grafana dashboard on WEB if postgresql:grafana-dashboard related with grafana-agent-k8s

[taurus-forever]

Bug Description

Hi,

@simskij hints us to use grafana-agent-k8s here. Unfortunately it doesn't work out of the box: all relations established without any errors, but no grafana dashboards shown on Grafana WEB.

During the private discussion, Simmon said:

I remember when we tested that the last time we ended up saying "go with direct grafana dashboard relation for now" - did we not? But we definitely need to fix that asap.

I have tried to relate dashboard directly, but then no metrics shown on WEB. I believe we need to trace this together to fix it and switch to grafana-agent-k8s in DB manuals. Tnx!

To Reproduce

Deploy COS:

juju add-model cos && juju deploy cos-lite --trust
juju offer grafana:grafana-dashboard
juju offer loki:logging loki-logging
juju offer prometheus:receive-remote-write

Deploy DB:

juju add-model postgresql && juju deploy postgresql-k8s --trust --channel 14/edge -n 3
juju deploy grafana-agent-k8s agent --trust
juju consume k8s:admin/cos.grafana
juju consume k8s:admin/cos.loki
juju consume k8s:admin/cos.prometheus

Failed case using grafana-agent-k8s :

juju relate agent grafana-dashboards
juju relate agent loki-logging
juju relate agent prometheus
juju relate postgresql-k8s:grafana-dashboard agent
juju relate postgresql-k8s:logging agent
juju relate postgresql-k8s:metrics-endpoint agent

Actual results: no DB dashboards on Grafana WEB, no any errors in logs.

Try to re-relate directly:

juju remove-relation postgresql-k8s:grafana-dashboard agent
juju relate postgresql-k8s:grafana-dashboard grafana-dashboards

Actual results: dashboards appeared, but no metrics shown on dashboards.

P.S. all works well, if postgresql-k8s related directly with SAAS (without grafana-agent-k8s, as described in manual).

Environment

juju 3.1.5, microk8s v1.28.0 cos:

App           Version  Status  Scale  Charm             Channel  Rev  Address         Exposed  Message
alertmanager  0.25.0   active      1  alertmanager-k8s  stable    76  10.152.183.247  no       
catalogue              active      1  catalogue-k8s     stable    13  10.152.183.166  no       
grafana       9.2.1    active      1  grafana-k8s       stable    81  10.152.183.60   no       
loki          2.7.4    active      1  loki-k8s          stable    89  10.152.183.89   no       
prometheus    2.33.5   active      1  prometheus-k8s    stable   103  10.152.183.87   no       
traefik       2.9.6    active      1  traefik-k8s       stable   110  10.76.203.19    no  

db:

App             Version  Status  Scale  Charm              Channel  Rev  Address         Exposed  Message
agent           0.32.1   active      1  grafana-agent-k8s  stable    38  10.152.183.127  no       
postgresql-k8s  14.9     active      3  postgresql-k8s     14/edge  128  10.152.183.111  no 

Relevant log output

No errors in log.

Additional context

Tnx!

[simskij]

@dstathis any thoughts on this?

[deusebio]

I believe I'm still facing the same issue with Kafka (only for dashboards, not alerts which now appears), as per the issue #231 . This may possibly have the same roots