Some of node's metrics are missing on Nodes overview tab

devopsprodigy / kubegraf

Grafana-plugin for k8s' monitoring

MIT License

406 stars 45 forks source link

Some of node's metrics are missing on Nodes overview tab #9

Closed amravyan closed 5 years ago

amravyan commented 5 years ago

Hi!

Facing a strange issue: some of node's metrics (pod limits usage percent, for example) are missing on the tab Nodes overview (N/A NaN%). On the other hand, all these metrics are available in prometheus. Could you please check and investigate? All screenshots are attached.

Kubernetes v1.15.3 (on-premise, kubespray) Prometheus from helm chart stable/prometheus (9.1.1), node-exporter image prom/node-exporter:v0.18.0

prom_data_mem prom_data_cpu metrics

SergeiSporyshev commented 5 years ago

Hi! It's look like Prometheus cann't scrape metrics from kubelet (/metrics/cadvisor)

amravyan commented 5 years ago

In case you have Prometheus installed from prometheus chart (not prometheus operator), you have to add label replace rule listed below to two jobs: kubernetes-nodes and kubernetes-nodes-cadvisor.

          - source_labels: [__meta_kubernetes_node_name]
            action: replace
            target_label: node