kinvolk / lokomotive

🪦 DISCONTINUED Further Lokomotive development has been discontinued. Lokomotive is a 100% open-source, easy to use and secure Kubernetes distribution from the volks at Kinvolk
https://kinvolk.io/lokomotive-kubernetes/
Apache License 2.0
321 stars 49 forks source link

Rook Ceph Grafana Dashboard fails to show some info after mgr pod restarts #583

Open surajssd opened 4 years ago

surajssd commented 4 years ago

So mgr pod is the one which is exposing Ceph related metrics. And when this pod restarts the dashbaord shows certain panels with error:

Only queries that return single series/table is supported

See following screenshot:

Screenshot from 2020-06-09 00-27-34

Now this error occurs because these panels are expecting only one result for the query ceph_health_status and similar ones. But it gets two results(over the period) and hence it is complaining.

ceph_health_status{endpoint="http-metrics",instance="10.2.51.14:9283",job="rook-ceph-mgr",namespace="rook",pod="rook-ceph-mgr-a-6bc447cfff-sc85b",service="rook-ceph-mgr"}
ceph_health_status{endpoint="http-metrics",instance="10.2.153.5:9283",job="rook-ceph-mgr",namespace="rook",pod="rook-ceph-mgr-a-6bc447cfff-vplq6",service="rook-ceph-mgr"}

Above is obtained from Prometheus if you look at it in a Graph tab of:

Screenshot from 2020-06-09 00-28-12

But if you query for that "point in time", AKA Console tab, you see only one result:

Screenshot from 2020-06-09 00-32-18

invidian commented 3 years ago

@surajssd mind checking if this is still actual?