Closed anton-johansson closed 5 years ago
Prometheus has a very nice Kubernetes auto discovery feature which will use in-cluster credentials using generated pod tokens. I usually use this as a base to deploy Prometheus and get most cluster metrics out.
Oh, that looks very interesting! I was already using the auto-discovery for pods with the scaping annotations, I did not know that it could autodiscover other things too.
Thanks a bunch for this!
By the way, @amimof: Do you know of any good "complete" dashboards for these components available on https://grafana.com/dashboards, or have you created your own?
I either create my own or use others on grafana.com as a base. One great dashboard to use with the node-exporter is https://grafana.com/dashboards/3320
Great, thanks a bunch!
Note: Technically not related to this repository other than the fact that I might need custom switches on other components than
kube-apiserver
. But I'll give it a go here anyway, maybe it's a good discussion topic. :)I noticed that all Kubernetes components and
etcd
exposes a/metrics
path with Prometheus metrics. So I was thinking that I should start scraping these, and see if I can find any pre-built dashboards for Grafana.I just have something to ask/discuss here.
kube-apiserver
should easily be accessible by my Prometheus pod, as long as I give the serviceaccount access to the/metrics
path (not sure how I do that, though, will need to investigate).Regarding
kube-scheduler
andkube-controller-manager
, I can access them over HTTP on ports 10259 and 10257 respectively. However, they have quite some strange CA certificates and I'm not able to use my own access token. I suppose switches--tls-cert-file
and--tls-private-key-file
will solve the strange CA certificate, but I'm not sure how to actually authenticate (avoiding 401 Unauthorized). Do you any ideas?When it comes to
etcd
, I can access that pretty easily. However, I need to use the client certificate and key stored on the masters (etcd.pem
andetcd-key.pem
), and I can't really access them from my Prometheus pod. I'm not sure I want to either. I guess this is something that is interesting here.kube-proxy
should be fairly simple. It only listens to127.0.0.1:10249
by defualt, but that's changable with a switch, so it should be fine.Finally: I wouldn't want to hardcode all server IPs in my Prometheus configuration file. It would be great if I could use Kubernetes services for this. I see that I have some endpoints (
kubectl get endpoints -n kube-system
), likekube-controller-manager
, but they're set to<none>
. I guess I could create my services manually (once) and utilize them. But I wouldn't want Prometheus to round-robin requests to them. I would want it to perform a DNS-lookup and scrape all targets of that DNS-lookup. Somehow... :) Ideas? For the worker nodes, it would be nice if I could utilizekubectl get nodes
to find IP adresses of nodes, and there reachkube-proxy
.Just close this if you feel it's too off-topic, and I'll try elsewhere.