search

sensu / sensu-kube-demo

The Sensu Kubernetes monitoring demo repository.
https://sensu.io
MIT License
13 stars 12 forks source link

Is the sensu-prometheus-collector check scraping the right kube-state-metrics exporter? #9

Open calebhailey opened 5 years ago

calebhailey commented 5 years ago

According to https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/#communicating-with-daemon-pods (regarding DNS for DaemonSet Pods), the documentation states that you can "Create a service with the same Pod selector, and use the service to reach a daemon on a random node. (No way to reach specific node.)"

The kube-state-metrics project is configured to use a Service, so the check-kube-state-metrics.json check definition in this project would be hitting a random kube-state-metrics endpoint. Is this OK? Does kube-state-metrics expose cluster-level metrics, or does it provide metrics on a per node basis?

calebhailey commented 5 years ago

I believe the answer is that kube-state-metrics does generate cluster-level metrics, so hitting a random DaemonSet pod is acceptable. From the readme:

kube-state-metrics is a simple service that listens to the Kubernetes API server and generates metrics about the state of the objects. (See examples in the Metrics section below.) It is not focused on the health of the individual Kubernetes components, but rather on the health of the various objects inside, such as deployments, nodes and pods.

https://github.com/kubernetes/kube-state-metrics

mattbeninghof commented 5 years ago

From the testing ran last night, it does appear that when the error code (9) is occurring, that no metrics are being fed. Could this be a timing issue?