Closed micdah closed 4 years ago
@micdah do you mind sharing
"minikube ip"
and also first enable the addon and then share output of minikube start --alsologtostderr -v=8
?
@medyagh of course! These commands have been executed with a clean slate (_entirely deleted ~/.minikube
folder before running commands).
Commands issued:
minikube addons enable metrics-server
minikube start --alsologtostderr -v=8
Output: pastebin.com/mcuXbw85 (too much to put in comment)
Output of minikube ip
:
192.168.64.48
Output of kubectl logs -n kube-system metrics-server-587f876775-fpf59
:
I1108 10:36:43.006107 1 heapster.go:71] /metrics-server --source=kubernetes.summary_api:''
I1108 10:36:43.006187 1 heapster.go:72] Metrics Server version v0.2.1
I1108 10:36:43.006572 1 configs.go:61] Using Kubernetes client with master "https://10.96.0.1:443" and version
I1108 10:36:43.006678 1 configs.go:62] Using kubelet port 10255
I1108 10:36:43.007640 1 heapster.go:128] Starting with Metric Sink
I1108 10:36:43.189888 1 serving.go:308] Generated self-signed cert (apiserver.local.config/certificates/apiserver.crt, apiserver.local.config/certificates/apiserver.key)
I1108 10:36:43.550421 1 heapster.go:101] Starting Heapster API server...
[restful] 2019/11/08 10:36:43 log.go:33: [restful/swagger] listing is available at https:///swaggerapi
[restful] 2019/11/08 10:36:43 log.go:33: [restful/swagger] https:///swaggerui/ is mapped to folder /swagger-ui/
I1108 10:36:43.551698 1 serve.go:85] Serving securely on 0.0.0.0:443
E1108 10:37:05.007912 1 summary.go:97] error while getting metrics summary from Kubelet minikube(192.168.64.48:10255): Get http://192.168.64.48:10255/stats/summary/: dial tcp 192.168.64.48:10255: getsockopt: connection refused
I1108 10:37:26.117263 1 reststorage.go:140] No metrics for container coredns in pod kube-system/coredns-5644d7b6d9-tbp8k
I1108 10:37:26.118248 1 reststorage.go:93] No metrics for pod kube-system/coredns-5644d7b6d9-tbp8k
I1108 10:37:26.118311 1 reststorage.go:140] No metrics for container metrics-server in pod kube-system/metrics-server-587f876775-fpf59
I1108 10:37:26.118574 1 reststorage.go:93] No metrics for pod kube-system/metrics-server-587f876775-fpf59
I1108 10:37:26.118582 1 reststorage.go:140] No metrics for container storage-provisioner in pod kube-system/storage-provisioner
I1108 10:37:26.118585 1 reststorage.go:93] No metrics for pod kube-system/storage-provisioner
I1108 10:37:26.118589 1 reststorage.go:140] No metrics for container kube-addon-manager in pod kube-system/kube-addon-manager-minikube
I1108 10:37:26.118592 1 reststorage.go:93] No metrics for pod kube-system/kube-addon-manager-minikube
I1108 10:37:26.118596 1 reststorage.go:140] No metrics for container kube-proxy in pod kube-system/kube-proxy-fsdzt
I1108 10:37:26.118835 1 reststorage.go:93] No metrics for pod kube-system/kube-proxy-fsdzt
I1108 10:37:26.118870 1 reststorage.go:140] No metrics for container coredns in pod kube-system/coredns-5644d7b6d9-qn2km
I1108 10:37:26.118875 1 reststorage.go:93] No metrics for pod kube-system/coredns-5644d7b6d9-qn2km
I1108 10:37:41.125699 1 reststorage.go:140] No metrics for container storage-provisioner in pod kube-system/storage-provisioner
I1108 10:37:41.125820 1 reststorage.go:93] No metrics for pod kube-system/storage-provisioner
I1108 10:37:41.125828 1 reststorage.go:140] No metrics for container kube-addon-manager in pod kube-system/kube-addon-manager-minikube
I1108 10:37:41.125831 1 reststorage.go:93] No metrics for pod kube-system/kube-addon-manager-minikube
I1108 10:37:41.125854 1 reststorage.go:140] No metrics for container kube-proxy in pod kube-system/kube-proxy-fsdzt
I1108 10:37:41.125859 1 reststorage.go:93] No metrics for pod kube-system/kube-proxy-fsdzt
I1108 10:37:41.125863 1 reststorage.go:140] No metrics for container coredns in pod kube-system/coredns-5644d7b6d9-qn2km
I1108 10:37:41.125866 1 reststorage.go:93] No metrics for pod kube-system/coredns-5644d7b6d9-qn2km
I1108 10:37:41.125870 1 reststorage.go:140] No metrics for container coredns in pod kube-system/coredns-5644d7b6d9-tbp8k
I1108 10:37:41.125873 1 reststorage.go:93] No metrics for pod kube-system/coredns-5644d7b6d9-tbp8k
I1108 10:37:41.125877 1 reststorage.go:140] No metrics for container metrics-server in pod kube-system/metrics-server-587f876775-fpf59
I1108 10:37:41.125882 1 reststorage.go:93] No metrics for pod kube-system/metrics-server-587f876775-fpf59
I1108 10:37:41.125886 1 reststorage.go:140] No metrics for container kube-scheduler in pod kube-system/kube-scheduler-minikube
I1108 10:37:41.125889 1 reststorage.go:93] No metrics for pod kube-system/kube-scheduler-minikube
I1108 10:37:41.125893 1 reststorage.go:140] No metrics for container kube-apiserver in pod kube-system/kube-apiserver-minikube
I1108 10:37:41.125895 1 reststorage.go:93] No metrics for pod kube-system/kube-apiserver-minikube
I1108 10:37:56.128740 1 reststorage.go:140] No metrics for container kube-scheduler in pod kube-system/kube-scheduler-minikube
I1108 10:37:56.128841 1 reststorage.go:93] No metrics for pod kube-system/kube-scheduler-minikube
I1108 10:37:56.128848 1 reststorage.go:140] No metrics for container kube-apiserver in pod kube-system/kube-apiserver-minikube
I1108 10:37:56.128852 1 reststorage.go:93] No metrics for pod kube-system/kube-apiserver-minikube
I1108 10:37:56.128856 1 reststorage.go:140] No metrics for container etcd in pod kube-system/etcd-minikube
I1108 10:37:56.128859 1 reststorage.go:93] No metrics for pod kube-system/etcd-minikube
I1108 10:37:56.128863 1 reststorage.go:140] No metrics for container storage-provisioner in pod kube-system/storage-provisioner
I1108 10:37:56.128866 1 reststorage.go:93] No metrics for pod kube-system/storage-provisioner
I1108 10:37:56.128871 1 reststorage.go:140] No metrics for container kube-proxy in pod kube-system/kube-proxy-fsdzt
I1108 10:37:56.128874 1 reststorage.go:93] No metrics for pod kube-system/kube-proxy-fsdzt
I1108 10:37:56.128878 1 reststorage.go:140] No metrics for container coredns in pod kube-system/coredns-5644d7b6d9-qn2km
I1108 10:37:56.128881 1 reststorage.go:93] No metrics for pod kube-system/coredns-5644d7b6d9-qn2km
I1108 10:37:56.128885 1 reststorage.go:140] No metrics for container coredns in pod kube-system/coredns-5644d7b6d9-tbp8k
I1108 10:37:56.128888 1 reststorage.go:93] No metrics for pod kube-system/coredns-5644d7b6d9-tbp8k
I1108 10:37:56.128892 1 reststorage.go:140] No metrics for container metrics-server in pod kube-system/metrics-server-587f876775-fpf59
I1108 10:37:56.128895 1 reststorage.go:93] No metrics for pod kube-system/metrics-server-587f876775-fpf59
I1108 10:37:56.128899 1 reststorage.go:140] No metrics for container kube-addon-manager in pod kube-system/kube-addon-manager-minikube
I1108 10:37:56.128902 1 reststorage.go:93] No metrics for pod kube-system/kube-addon-manager-minikube
I1108 10:37:56.128907 1 reststorage.go:140] No metrics for container kube-controller-manager in pod kube-system/kube-controller-manager-minikube
I1108 10:37:56.128956 1 reststorage.go:93] No metrics for pod kube-system/kube-controller-manager-minikube
E1108 10:38:05.001740 1 summary.go:97] error while getting metrics summary from Kubelet minikube(192.168.64.48:10255): Get http://192.168.64.48:10255/stats/summary/: dial tcp 192.168.64.48:10255: getsockopt: connection refused
I1108 10:38:11.134348 1 reststorage.go:140] No metrics for container kube-addon-manager in pod kube-system/kube-addon-manager-minikube
I1108 10:38:11.134363 1 reststorage.go:93] No metrics for pod kube-system/kube-addon-manager-minikube
I1108 10:38:11.134368 1 reststorage.go:140] No metrics for container kube-controller-manager in pod kube-system/kube-controller-manager-minikube
I1108 10:38:11.134372 1 reststorage.go:93] No metrics for pod kube-system/kube-controller-manager-minikube
I1108 10:38:11.134376 1 reststorage.go:140] No metrics for container storage-provisioner in pod kube-system/storage-provisioner
I1108 10:38:11.134378 1 reststorage.go:93] No metrics for pod kube-system/storage-provisioner
I1108 10:38:11.134383 1 reststorage.go:140] No metrics for container kube-proxy in pod kube-system/kube-proxy-fsdzt
I1108 10:38:11.134386 1 reststorage.go:93] No metrics for pod kube-system/kube-proxy-fsdzt
I1108 10:38:11.134390 1 reststorage.go:140] No metrics for container coredns in pod kube-system/coredns-5644d7b6d9-qn2km
I1108 10:38:11.134393 1 reststorage.go:93] No metrics for pod kube-system/coredns-5644d7b6d9-qn2km
I1108 10:38:11.134397 1 reststorage.go:140] No metrics for container coredns in pod kube-system/coredns-5644d7b6d9-tbp8k
I1108 10:38:11.134400 1 reststorage.go:93] No metrics for pod kube-system/coredns-5644d7b6d9-tbp8k
I1108 10:38:11.134404 1 reststorage.go:140] No metrics for container metrics-server in pod kube-system/metrics-server-587f876775-fpf59
I1108 10:38:11.134407 1 reststorage.go:93] No metrics for pod kube-system/metrics-server-587f876775-fpf59
I1108 10:38:11.134411 1 reststorage.go:140] No metrics for container kube-scheduler in pod kube-system/kube-scheduler-minikube
I1108 10:38:11.134414 1 reststorage.go:93] No metrics for pod kube-system/kube-scheduler-minikube
I1108 10:38:11.134418 1 reststorage.go:140] No metrics for container kube-apiserver in pod kube-system/kube-apiserver-minikube
I1108 10:38:11.134421 1 reststorage.go:93] No metrics for pod kube-system/kube-apiserver-minikube
I1108 10:38:11.134424 1 reststorage.go:140] No metrics for container etcd in pod kube-system/etcd-minikube
I1108 10:38:11.134427 1 reststorage.go:93] No metrics for pod kube-system/etcd-minikube
I1108 10:38:26.138300 1 reststorage.go:140] No metrics for container metrics-server in pod kube-system/metrics-server-587f876775-fpf59
I1108 10:38:26.138340 1 reststorage.go:93] No metrics for pod kube-system/metrics-server-587f876775-fpf59
I1108 10:38:26.138347 1 reststorage.go:140] No metrics for container kube-scheduler in pod kube-system/kube-scheduler-minikube
I1108 10:38:26.138350 1 reststorage.go:93] No metrics for pod kube-system/kube-scheduler-minikube
I1108 10:38:26.138355 1 reststorage.go:140] No metrics for container kube-apiserver in pod kube-system/kube-apiserver-minikube
I1108 10:38:26.138357 1 reststorage.go:93] No metrics for pod kube-system/kube-apiserver-minikube
I1108 10:38:26.138361 1 reststorage.go:140] No metrics for container etcd in pod kube-system/etcd-minikube
I1108 10:38:26.138364 1 reststorage.go:93] No metrics for pod kube-system/etcd-minikube
I1108 10:38:26.138368 1 reststorage.go:140] No metrics for container storage-provisioner in pod kube-system/storage-provisioner
I1108 10:38:26.138371 1 reststorage.go:93] No metrics for pod kube-system/storage-provisioner
I1108 10:38:26.138375 1 reststorage.go:140] No metrics for container kube-proxy in pod kube-system/kube-proxy-fsdzt
I1108 10:38:26.138378 1 reststorage.go:93] No metrics for pod kube-system/kube-proxy-fsdzt
I1108 10:38:26.138382 1 reststorage.go:140] No metrics for container coredns in pod kube-system/coredns-5644d7b6d9-qn2km
I1108 10:38:26.138384 1 reststorage.go:93] No metrics for pod kube-system/coredns-5644d7b6d9-qn2km
I1108 10:38:26.138389 1 reststorage.go:140] No metrics for container coredns in pod kube-system/coredns-5644d7b6d9-tbp8k
I1108 10:38:26.138392 1 reststorage.go:93] No metrics for pod kube-system/coredns-5644d7b6d9-tbp8k
I1108 10:38:26.138396 1 reststorage.go:140] No metrics for container kube-addon-manager in pod kube-system/kube-addon-manager-minikube
I1108 10:38:26.138407 1 reststorage.go:93] No metrics for pod kube-system/kube-addon-manager-minikube
I1108 10:38:26.138414 1 reststorage.go:140] No metrics for container kube-controller-manager in pod kube-system/kube-controller-manager-minikube
I1108 10:38:26.138416 1 reststorage.go:93] No metrics for pod kube-system/kube-controller-manager-minikube
I1108 10:38:41.144975 1 reststorage.go:140] No metrics for container kube-proxy in pod kube-system/kube-proxy-fsdzt
I1108 10:38:41.144989 1 reststorage.go:93] No metrics for pod kube-system/kube-proxy-fsdzt
I1108 10:38:41.144995 1 reststorage.go:140] No metrics for container coredns in pod kube-system/coredns-5644d7b6d9-qn2km
I1108 10:38:41.144998 1 reststorage.go:93] No metrics for pod kube-system/coredns-5644d7b6d9-qn2km
I1108 10:38:41.145002 1 reststorage.go:140] No metrics for container coredns in pod kube-system/coredns-5644d7b6d9-tbp8k
I1108 10:38:41.145005 1 reststorage.go:93] No metrics for pod kube-system/coredns-5644d7b6d9-tbp8k
I1108 10:38:41.145010 1 reststorage.go:140] No metrics for container metrics-server in pod kube-system/metrics-server-587f876775-fpf59
I1108 10:38:41.145013 1 reststorage.go:93] No metrics for pod kube-system/metrics-server-587f876775-fpf59
I1108 10:38:41.145017 1 reststorage.go:140] No metrics for container kube-scheduler in pod kube-system/kube-scheduler-minikube
I1108 10:38:41.145020 1 reststorage.go:93] No metrics for pod kube-system/kube-scheduler-minikube
I1108 10:38:41.145024 1 reststorage.go:140] No metrics for container kube-apiserver in pod kube-system/kube-apiserver-minikube
I1108 10:38:41.145027 1 reststorage.go:93] No metrics for pod kube-system/kube-apiserver-minikube
I1108 10:38:41.145032 1 reststorage.go:140] No metrics for container etcd in pod kube-system/etcd-minikube
I1108 10:38:41.145035 1 reststorage.go:93] No metrics for pod kube-system/etcd-minikube
I1108 10:38:41.145039 1 reststorage.go:140] No metrics for container storage-provisioner in pod kube-system/storage-provisioner
I1108 10:38:41.145042 1 reststorage.go:93] No metrics for pod kube-system/storage-provisioner
I1108 10:38:41.145046 1 reststorage.go:140] No metrics for container kube-controller-manager in pod kube-system/kube-controller-manager-minikube
I1108 10:38:41.145049 1 reststorage.go:93] No metrics for pod kube-system/kube-controller-manager-minikube
I1108 10:38:41.145054 1 reststorage.go:140] No metrics for container kube-addon-manager in pod kube-system/kube-addon-manager-minikube
I1108 10:38:41.145057 1 reststorage.go:93] No metrics for pod kube-system/kube-addon-manager-minikube
+1 Same issue here on OSX ;) Any updates?
I reproduce this(but I don't understand this root cause yet).
I digged this issue. The root cause is not getting stats summary via kubelet(port 10255).
E1107 10:41:05.007656 1 summary.go:97] error while getting metrics summary from Kubelet minikube(192.168.64.44:10255): Get http://192.168.64.44:10255/stats/summary/: dial tcp 192.168.64.44:10255: getsockopt: connection refused
kubelet has an option --read-only-port
(this is DEPRECATED).
--read-only-port
opens the port(default: 10255).
But now minikube v1.5.2(kubernetes v1.16.2), the kubelet port(10255) is not opened. So, metrics-server cannot get from kubelet stats summary api.
If metrics-server gets stat summary from reachable api-server, kubectl top
returns outputs.
I will send PR to fix this.
/assign
The exact command to reproduce the issue:
After upgrading from 1.4.0 to 1.5.x it seems metrics is no longer working. I have tried with all 1.5.x versions (1.5.0, 1.5.1 and 1.5.2), and it consistently fails with the below error message showing in the pod logs.
It auto-detects to use the
hyperkit
VM driver.The full output of the command that failed:
The output of the
minikube logs
command:The operating system version: macOS Catalina 10.15.1