Closed dcs3spp closed 3 years ago
simonpasquier
commented
3 years ago :wave: thanks for your report. It looks like your issue would be best answered by https://github.com/prometheus-community/helm-charts because this repository is only about the prometheus operator itself.
dcs3spp
commented
3 years ago Hi @simonpasquier, oops apologies.Yeah found links from hub.kubeapps.com - kube-prometheus-stack. That page linked me to https://github.com/prometheus-operator/kube-prometheus.
Ok, have raised the issue at prometheus-community/helm-charts. Apologies again.
Also asked as question on stack overflow.
What happened? The myrelease-name-prometheus-node-exporter service is failing with errors from the daemonset received after installation of the helm chart for kube-prometheus-stack is installed on Docker Desktop for Mac Kubernetes Cluster environment.
The scrape targets for
kube-scheduler:http://192.168.65.4:10251/metrics,kube-proxy:http://192.168.65.4:10249/metrics,kube-etcd:http://192.168.65.4:2379/metrics,kube-controller-manager:http://192.168.65.4:10252/metricsandnode-exporter:http://192.168.65.4:9100/metricsare marked as unhealthy. All show asconnection refused, except forkube-etcdwhich displaysconnection reset by peer.I have installed kube-prometheus-stack as a dependency in my helm chart on a local Docker for Mac Kubernetes cluster v1.19.7. I have also tried this on a minikube cluster using the hyperkit vm-driver, with the same result.
Chart.yaml
Values.yaml
Did you expect to see some different? All kubernetes start successfully and all scrape targets marked as healthy.
How to reproduce it (as minimally and precisely as possible): On Docker desktop for Mac OS environment install the helm chart for Kube-Prometheus-Stack v14.40 and inspect status of the aforementioned failed scrape targets and view the logs for myrelease-name-prometheus-node-exporter service pod(s).
Environment Mac OS Catalina 10.15.7 Docker Desktop For Mac 3.2.2(61853) with docker engine v20.10.5 Local Kubernetes 1.19.7 Cluster provided by Docker Desktop For Mac
Prometheus Operator version:
Insert image tag or Git SHA herekube-prometheus-stack (https://hub.kubeapps.com/charts/prometheus-community/kube-prometheus-stack#!) 14.4.0Kubernetes version information:
kubectl versionKubernetes cluster provided with Docker Desktop for Mac
?
release-name-prometheus-node-exporter error log
Anything else we need to know?:
kubectl get all
After updating
values.yamlto:The prometheus-node-exporter daemonset now starts based on earlier issue fix. However the scrape targets mentioned above still remain unhealthy with
Get "http://192.168.65.4:<port_num>/metrics": dial tcp 192.168.65.4:<port_num>: connect: connection refusederror.Tried further investigation of kube-scheduler by sending a port-forward and visiting http://localhost:10251/metrics. Log output from pod is shown below:
If I run in
minikubeon macOS with vm-driver hyperkit then thenode-exporterdaemonset is successful and with:Also the
kube-proxyscrape target appears to be available in minikube which I have verified using a port forwardkubectl -n kube-system port-forward kube-proxy-sxw8k 10249and visitinghttp://localhost:10249/metrics. This also appears to work with a port forward in docker-desktop cluster, but appears as failing in prometheus targets....So in minikube on macOS with hyperkit vm-driver and default helm chart values the following scrape targets are unavailable:
etcd-minikube logs
kube-controller-manager-minikube logs:
kube-scheduler-minikube logs
Questions Are these scrape targets dependent upon a successful
prometheus-node-exporter.hostRootFsMount? If so, then would the failed scrape targets mentioned in this issue be available if I tried to investigate and use alternative helm charts dependencies for prometheus and grafana from other providers?How do I enable
etcd,scheduler,controller-managerandkube-proxy(docker-desktop) scrape targets with a helm chart installation of kube-prometheus-stack on macOS Kubernetes cluster running on docker desktop and minikube?