canonical / cos-proxy-operator

https://charmhub.io/cos-proxy
Apache License 2.0
2 stars 12 forks source link

Grafana-Loki dashboard empty for 'openstack' model in new cos-lite deployment #132

Closed babyphilip closed 3 months ago

babyphilip commented 6 months ago

Bug Description

The newly deployed cos-lite bundle in stsstack unable to produce graphs for 'openstack' model in Loki dashboard. Deployed multiple times with the same result. Below 'openstack' model is 'myostmodel' Screenshot from 2024-04-08 13-20-35 v01

However the 'cos' model graphs are visible. Screenshot from 2024-04-08 13-23-55 v01

To Reproduce

  1. Deployed openstack in 'myostmodel' with Juju version 3.4.2, yoga/jammy

juju status (trimmed output)

Model Controller Cloud/Region Version SLA Timestamp myostmodel juju3-controller stsstack/stsstack 3.4.2 unsupported 09:19:40Z

SAAS Status Store URL alertmanager-karma-dashboard active microk8s-controller-new admin/cos.alertmanager-karma-dashboard grafana-dashboards active microk8s-controller-new admin/cos.grafana-dashboards loki-logging active microk8s-controller-new admin/cos.loki-logging prometheus-receive-remote-write active microk8s-controller-new admin/cos.prometheus-receive-remote-write

App Version Status Scale Charm Channel Rev Exposed Message ceph-mon 17.2.7 active 1 ceph-mon quincy/stable 204 no Unit is ready and clustered ceph-osd 17.2.7 active 3 ceph-osd quincy/stable 582 no Unit is ready (1 OSD) ceph-rgw 17.2.7 active 3 ceph-radosgw quincy/stable 564 no Unit is ready ceph-rgw-hacluster 2.1.2 active 3 hacluster 2.4/stable 131 no Unit is ready and clustered cinder 20.3.1 active 1 cinder yoga/stable 664 no Unit is ready cinder-ceph 20.3.1 active 1 cinder-ceph yoga/stable 527 no Unit is ready cinder-mysql-router 8.0.36 active 1 mysql-router 8.0/stable 154 no Unit is ready

  1. Deployed cos-lite in 'cos' model

    juju status

    Model Controller Cloud/Region Version SLA Timestamp cos microk8s-controller-new microk8s-cloud-new/localhost 3.4.2 unsupported 09:30:38Z

App Version Status Scale Charm Channel Rev Address Exposed Message alertmanager 0.26.0 active 1 alertmanager-k8s latest/stable 103 10.152.183.97 no
catalogue active 1 catalogue-k8s latest/stable 33 10.152.183.235 no
grafana 9.5.3 active 1 grafana-k8s latest/stable 105 10.152.183.120 no
loki 2.9.4 active 1 loki-k8s latest/stable 121 10.152.183.147 no
prometheus 2.49.1 active 1 prometheus-k8s latest/stable 170 10.152.183.185 no
traefik 2.10.5 active 1 traefik-k8s latest/stable 170 10.5.2.80 no

Unit Workload Agent Address Ports Message alertmanager/0 active idle 10.1.14.94
catalogue/0
active idle 10.1.14.84
grafana/0 active idle 10.1.14.95
loki/0
active idle 10.1.14.96
prometheus/0 active idle 10.1.14.97
traefik/0
active idle 10.1.14.93

Offer Application Charm Rev Connected Endpoint Interface Role alertmanager-karma-dashboard alertmanager alertmanager-k8s 103 0/0 karma-dashboard karma_dashboard provider grafana-dashboards grafana grafana-k8s 105 2/2 grafana-dashboard grafana_dashboard requirer loki-logging loki loki-k8s 121 2/2 logging loki_push_api provider prometheus-receive-remote-write prometheus prometheus-k8s 170 2/2 receive-remote-write prometheus_remote_write provider

  1. Added the offers curl -L https://raw.githubusercontent.com/canonical/cos-lite-bundle/main/overlays/offers-overlay.yaml -O juju deploy cos-lite --trust --overlay ./offers-overlay.yaml

  2. Consumed the offers in 'myostmodel' juju consume microk8s-controller-new:admin/cos.alertmanager-karma-dashboard juju consume microk8s-controller-new:admin/cos.grafana-dashboards juju consume microk8s-controller-new:admin/cos.loki-logging juju consume microk8s-controller-new:admin/cos.prometheus-receive-remote-write

  3. Integrated the offers with grafana-agent juju integrate grafana-agent-host grafana-dashboards juju integrate grafana-agent-container grafana-dashboards juju integrate grafana-agent-container loki-logging juju integrate grafana-agent-host loki-logging juju integrate grafana-agent-host prometheus-receive-remote-write juju integrate grafana-agent-container prometheus-receive-remote-write

  4. Integrated grafana-agent with mysql & keystone juju integrate grafana-agent-container:juju-info mysql:juju-info juju integrate grafana-agent-container:juju-info keystone:juju-info

Environment

Deployed in internal stsstack and have a customer facing the same issue.

cos charms - latest/stable grafana-agent-container - latest/edge
juju - 3.4.2

Relevant log output

Can provide any log

Additional context

No response

sed-i commented 6 months ago

Hi @babyphilip, @natalytvinova I'm looking at the two screenshots here, and it seems to make sense:

You're looking at the Loki self-monitoring dashboard. The dropdowns are filtering telemetry. The cos model happens to have loki, and the openstack model doesn't have loki. That's why when you set the model filter to openstack you don't see anything.

I imagine you were expecting to see everything in this dashboard, just like we do in "node-exporter" dashboards? You should probably be looking at a different dashboard?

ca-scribner commented 3 months ago

Hey @babyphilip, we're closing this as we think the comment sorts this out. But please reopen if we're mistaken!