search

devopsprodigy / kubegraf

Grafana-plugin for k8s' monitoring
MIT License
407 stars 45 forks source link

Node-exporter + kube-state-metrics configuration steps missing #40

Open rbuffi opened 4 years ago

rbuffi commented 4 years ago

Hello,

I deployed Prometheus + node-exporter + kube-state-metrics and followed the instructions but dashboards are not populating. I can see some metrics in the plugin itself like cluster status, node and applications overview. Is there some additional configuration I need to do to point prometheus to node-exporter / kube-state-metrics?

Regards,

Ronald

rbuffi commented 4 years ago

Allthough I replaced the token and certificate file in prometheus I see the following errors:

msg="/app/discovery/kubernetes/kubernetes.go:363: Failed to list v1.Pod: pods is forbidden: User \"system:serviceaccount:default:default\" cannot list resource \"pods\" in API group \"\" in the namespace \"monitoring\"" level=error ts=2020-08-25T08:39:16.118Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg="/app/discovery/kubernetes/kubernetes.go:363: Failed to list v1.Pod: pods is forbidden: User \"system:serviceaccount:default:default\" cannot list resource \"pods\" in API group \"\" in the namespace \"kube-system\"" level=error ts=2020-08-25T08:39:20.278Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg="/app/discovery/kubernetes/kubernetes.go:362: Failed to list *v1.Service: services is forbidden: User \"system:serviceaccount:default:default\" cannot list resource \"services\" in API group \"\" in the namespace \"default\""

rbuffi commented 4 years ago

After specifying the serviceaccount in the pometheus deployment the right account is used. I see some targets in prometheus but there are not a lot metrics being collected. Also the dashboards stay empty.

level=info ts=2020-08-25T23:40:33.097Z caller=head.go:804 component=tsdb msg="Head GC completed" duration=28.837431ms level=info ts=2020-08-25T23:40:33.424Z caller=head.go:804 component=tsdb msg="Head GC completed" duration=29.447403ms level=info ts=2020-08-25T23:40:33.425Z caller=checkpoint.go:96 component=tsdb msg="Creating checkpoint" from_segment=22 to_segment=23 mint=1598335200000 level=info ts=2020-08-25T23:40:33.814Z caller=head.go:884 component=tsdb msg="WAL checkpoint complete" first=22 last=23 duration=389.804627ms level=info ts=2020-08-25T23:40:34.145Z caller=head.go:804 component=tsdb msg="Head GC completed" duration=29.06915ms level=info ts=2020-08-25T23:40:34.480Z caller=head.go:804 component=tsdb msg="Head GC completed" duration=29.588383ms level=info ts=2020-08-25T23:40:34.481Z caller=checkpoint.go:96 component=tsdb msg="Creating checkpoint" from_segment=24 to_segment=25 mint=1598349600000 level=info ts=2020-08-25T23:40:34.886Z caller=head.go:884 component=tsdb msg="WAL checkpoint complete" first=24 last=25 duration=406.052433ms level=info ts=2020-08-25T23:40:35.235Z caller=head.go:804 component=tsdb msg="Head GC completed" duration=29.838507ms level=info ts=2020-08-25T23:40:35.588Z caller=head.go:804 component=tsdb msg="Head GC completed" duration=39.801732ms level=info ts=2020-08-25T23:40:35.590Z caller=checkpoint.go:96 component=tsdb msg="Creating checkpoint" from_segment=26 to_segment=27 mint=1598364000000 level=info ts=2020-08-25T23:40:35.988Z caller=head.go:884 component=tsdb msg="WAL checkpoint complete" first=26 last=27 duration=399.974385ms level=info ts=2020-08-25T23:40:36.349Z caller=head.go:804 component=tsdb msg="Head GC completed" duration=29.356627ms level=info ts=2020-08-25T23:40:36.679Z caller=head.go:804 component=tsdb msg="Head GC completed" duration=29.133853ms level=info ts=2020-08-25T23:40:36.680Z caller=checkpoint.go:96 component=tsdb msg="Creating checkpoint" from_segment=28 to_segment=29 mint=1598378400000 level=info ts=2020-08-25T23:40:37.060Z caller=head.go:884 component=tsdb msg="WAL checkpoint complete" first=28 last=29 duration=380.93007ms level=info ts=2020-08-25T23:40:37.384Z caller=head.go:804 component=tsdb msg="Head GC completed" duration=29.456885ms level=info ts=2020-08-25T23:40:37.755Z caller=head.go:804 component=tsdb msg="Head GC completed" duration=29.179204ms level=info ts=2020-08-25T23:40:37.756Z caller=checkpoint.go:96 component=tsdb msg="Creating checkpoint" from_segment=30 to_segment=31 mint=1598392800000 level=info ts=2020-08-25T23:40:38.121Z caller=head.go:884 component=tsdb msg="WAL checkpoint complete" first=30 last=31 duration=366.108777ms

rbuffi commented 4 years ago

I only see one active target. Can somebody help me out. I think I am close to a working set-up :)

monitoring/monitoring-prometheus-oper-alertmanager/0 (0/0 active targets) monitoring/monitoring-prometheus-oper-apiserver/0 (1/14 active targets) monitoring/monitoring-prometheus-oper-coredns/0 (0/18 active targets) monitoring/monitoring-prometheus-oper-grafana/0 (0/0 active targets) monitoring/monitoring-prometheus-oper-kube-controller-manager/0 (0/18 active targets) monitoring/monitoring-prometheus-oper-kube-etcd/0 (0/18 active targets) monitoring/monitoring-prometheus-oper-kube-proxy/0 (0/18 active targets) monitoring/monitoring-prometheus-oper-kube-scheduler/0 (0/18 active targets) monitoring/monitoring-prometheus-oper-kube-state-metrics/0 (0/0 active targets) monitoring/monitoring-prometheus-oper-kubelet/0 (0/18 active targets) monitoring/monitoring-prometheus-oper-kubelet/1 (0/18 active targets) monitoring/monitoring-prometheus-oper-node-exporter/0 (0/0 active targets) monitoring/monitoring-prometheus-oper-operator/0 (0/0 active targets) monitoring/monitoring-prometheus-oper-prometheus/0 (0/0 active targets)