search

kubernetes / minikube

Run Kubernetes locally
https://minikube.sigs.k8s.io/
Apache License 2.0
29.36k stars 4.88k forks source link

metrics-server doesn't seem to work in 1.5.x #5859

Closed micdah closed 4 years ago

micdah commented 4 years ago

The exact command to reproduce the issue:

minikube start
minikube addons enable metrics-server
kubectl logs -n kube-system metrics-server-587f876775-4d8jq

After upgrading from 1.4.0 to 1.5.x it seems metrics is no longer working. I have tried with all 1.5.x versions (1.5.0, 1.5.1 and 1.5.2), and it consistently fails with the below error message showing in the pod logs.

It auto-detects to use the hyperkit VM driver.

The full output of the command that failed:

I1107 10:40:32.667494       1 heapster.go:71] /metrics-server --source=kubernetes.summary_api:''
I1107 10:40:32.667545       1 heapster.go:72] Metrics Server version v0.2.1
I1107 10:40:32.667715       1 configs.go:61] Using Kubernetes client with master "https://10.96.0.1:443" and version
I1107 10:40:32.667723       1 configs.go:62] Using kubelet port 10255
I1107 10:40:32.668514       1 heapster.go:128] Starting with Metric Sink
I1107 10:40:33.181569       1 serving.go:308] Generated self-signed cert (apiserver.local.config/certificates/apiserver.crt, apiserver.local.config/certificates/apiserver.key)
I1107 10:40:33.326747       1 heapster.go:101] Starting Heapster API server...
[restful] 2019/11/07 10:40:33 log.go:33: [restful/swagger] listing is available at https:///swaggerapi
[restful] 2019/11/07 10:40:33 log.go:33: [restful/swagger] https:///swaggerui/ is mapped to folder /swagger-ui/
I1107 10:40:33.327930       1 serve.go:85] Serving securely on 0.0.0.0:443
E1107 10:41:05.007656       1 summary.go:97] error while getting metrics summary from Kubelet minikube(192.168.64.44:10255): Get http://192.168.64.44:10255/stats/summary/: dial tcp 192.168.64.44:10255: getsockopt: connection refused
E1107 10:42:05.002554       1 summary.go:97] error while getting metrics summary from Kubelet minikube(192.168.64.44:10255): Get http://192.168.64.44:10255/stats/summary/: dial tcp 192.168.64.44:10255: getsockopt: connection refused
E1107 10:43:05.006236       1 summary.go:97] error while getting metrics summary from Kubelet minikube(192.168.64.44:10255): Get http://192.168.64.44:10255/stats/summary/: dial tcp 192.168.64.44:10255: getsockopt: connection refused

The output of the minikube logs command:

==> Docker <==
-- Logs begin at Thu 2019-11-07 10:37:14 UTC, end at Thu 2019-11-07 10:44:50 UTC. --
Nov 07 10:37:18 minikube dockerd[1955]: time="2019-11-07T10:37:18.121514060Z" level=info msg="containerd successfully booted in 0.007222s"
Nov 07 10:37:18 minikube dockerd[1955]: time="2019-11-07T10:37:18.125074407Z" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc0000d2900, READY" module=grpc
Nov 07 10:37:18 minikube dockerd[1955]: time="2019-11-07T10:37:18.128628141Z" level=info msg="systemd-resolved is running, so using resolvconf: /run/systemd/resolve/resolv.conf"
Nov 07 10:37:18 minikube dockerd[1955]: time="2019-11-07T10:37:18.129264083Z" level=info msg="parsed scheme: \"unix\"" module=grpc
Nov 07 10:37:18 minikube dockerd[1955]: time="2019-11-07T10:37:18.129292005Z" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
Nov 07 10:37:18 minikube dockerd[1955]: time="2019-11-07T10:37:18.129407658Z" level=info msg="parsed scheme: \"unix\"" module=grpc
Nov 07 10:37:18 minikube dockerd[1955]: time="2019-11-07T10:37:18.129433053Z" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
Nov 07 10:37:18 minikube dockerd[1955]: time="2019-11-07T10:37:18.129500909Z" level=info msg="ccResolverWrapper: sending new addresses to cc: [{unix:///var/run/docker/containerd/containerd.sock 0  <nil>}]" module=grpc
Nov 07 10:37:18 minikube dockerd[1955]: time="2019-11-07T10:37:18.129542150Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
Nov 07 10:37:18 minikube dockerd[1955]: time="2019-11-07T10:37:18.129589503Z" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc0000d2e60, CONNECTING" module=grpc
Nov 07 10:37:18 minikube dockerd[1955]: time="2019-11-07T10:37:18.129848056Z" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc0000d2e60, READY" module=grpc
Nov 07 10:37:18 minikube dockerd[1955]: time="2019-11-07T10:37:18.129983720Z" level=info msg="ccResolverWrapper: sending new addresses to cc: [{unix:///var/run/docker/containerd/containerd.sock 0  <nil>}]" module=grpc
Nov 07 10:37:18 minikube dockerd[1955]: time="2019-11-07T10:37:18.130084554Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
Nov 07 10:37:18 minikube dockerd[1955]: time="2019-11-07T10:37:18.130135444Z" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc0000d3140, CONNECTING" module=grpc
Nov 07 10:37:18 minikube dockerd[1955]: time="2019-11-07T10:37:18.130241119Z" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc0000d3140, READY" module=grpc
Nov 07 10:37:18 minikube dockerd[1955]: time="2019-11-07T10:37:18.141204857Z" level=info msg="Graph migration to content-addressability took 0.00 seconds"
Nov 07 10:37:18 minikube dockerd[1955]: time="2019-11-07T10:37:18.141442803Z" level=warning msg="Your kernel does not support cgroup blkio weight"
Nov 07 10:37:18 minikube dockerd[1955]: time="2019-11-07T10:37:18.141472657Z" level=warning msg="Your kernel does not support cgroup blkio weight_device"
Nov 07 10:37:18 minikube dockerd[1955]: time="2019-11-07T10:37:18.141481648Z" level=warning msg="Your kernel does not support cgroup blkio throttle.read_bps_device"
Nov 07 10:37:18 minikube dockerd[1955]: time="2019-11-07T10:37:18.141488023Z" level=warning msg="Your kernel does not support cgroup blkio throttle.write_bps_device"
Nov 07 10:37:18 minikube dockerd[1955]: time="2019-11-07T10:37:18.141494173Z" level=warning msg="Your kernel does not support cgroup blkio throttle.read_iops_device"
Nov 07 10:37:18 minikube dockerd[1955]: time="2019-11-07T10:37:18.141500460Z" level=warning msg="Your kernel does not support cgroup blkio throttle.write_iops_device"
Nov 07 10:37:18 minikube dockerd[1955]: time="2019-11-07T10:37:18.141740337Z" level=info msg="Loading containers: start."
Nov 07 10:37:18 minikube dockerd[1955]: time="2019-11-07T10:37:18.195671945Z" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address"
Nov 07 10:37:18 minikube dockerd[1955]: time="2019-11-07T10:37:18.233395188Z" level=info msg="Loading containers: done."
Nov 07 10:37:18 minikube dockerd[1955]: time="2019-11-07T10:37:18.241141102Z" level=warning msg="failed to retrieve runc version: unknown output format: runc version commit: 425e105d5a03fabd737a126ad93d62a9eeede87f\nspec: 1.0.1-dev\n"
Nov 07 10:37:18 minikube dockerd[1955]: time="2019-11-07T10:37:18.252900587Z" level=info msg="Docker daemon" commit=039a7df9ba graphdriver(s)=overlay2 version=18.09.9
Nov 07 10:37:18 minikube dockerd[1955]: time="2019-11-07T10:37:18.253011613Z" level=info msg="Daemon has completed initialization"
Nov 07 10:37:18 minikube dockerd[1955]: time="2019-11-07T10:37:18.267158643Z" level=info msg="API listen on /var/run/docker.sock"
Nov 07 10:37:18 minikube systemd[1]: Started Docker Application Container Engine.
Nov 07 10:37:18 minikube dockerd[1955]: time="2019-11-07T10:37:18.267579611Z" level=info msg="API listen on [::]:2376"
Nov 07 10:37:59 minikube dockerd[1955]: time="2019-11-07T10:37:59.363247129Z" level=warning msg="failed to retrieve runc version: unknown output format: runc version commit: 425e105d5a03fabd737a126ad93d62a9eeede87f\nspec: 1.0.1-dev\n"
Nov 07 10:37:59 minikube dockerd[1955]: time="2019-11-07T10:37:59.410274495Z" level=warning msg="failed to retrieve runc version: unknown output format: runc version commit: 425e105d5a03fabd737a126ad93d62a9eeede87f\nspec: 1.0.1-dev\n"
Nov 07 10:37:59 minikube dockerd[1955]: time="2019-11-07T10:37:59.441848599Z" level=warning msg="failed to retrieve runc version: unknown output format: runc version commit: 425e105d5a03fabd737a126ad93d62a9eeede87f\nspec: 1.0.1-dev\n"
Nov 07 10:37:59 minikube dockerd[1955]: time="2019-11-07T10:37:59.707014532Z" level=warning msg="failed to retrieve runc version: unknown output format: runc version commit: 425e105d5a03fabd737a126ad93d62a9eeede87f\nspec: 1.0.1-dev\n"
Nov 07 10:38:01 minikube dockerd[1955]: time="2019-11-07T10:38:01.827870858Z" level=warning msg="failed to retrieve runc version: unknown output format: runc version commit: 425e105d5a03fabd737a126ad93d62a9eeede87f\nspec: 1.0.1-dev\n"
Nov 07 10:38:01 minikube dockerd[1955]: time="2019-11-07T10:38:01.840406579Z" level=warning msg="failed to retrieve runc version: unknown output format: runc version commit: 425e105d5a03fabd737a126ad93d62a9eeede87f\nspec: 1.0.1-dev\n"
Nov 07 10:38:01 minikube dockerd[1955]: time="2019-11-07T10:38:01.870048622Z" level=warning msg="failed to retrieve runc version: unknown output format: runc version commit: 425e105d5a03fabd737a126ad93d62a9eeede87f\nspec: 1.0.1-dev\n"
Nov 07 10:38:01 minikube dockerd[1955]: time="2019-11-07T10:38:01.881412541Z" level=warning msg="failed to retrieve runc version: unknown output format: runc version commit: 425e105d5a03fabd737a126ad93d62a9eeede87f\nspec: 1.0.1-dev\n"
Nov 07 10:38:10 minikube dockerd[1955]: time="2019-11-07T10:38:10.042790601Z" level=warning msg="failed to retrieve runc version: unknown output format: runc version commit: 425e105d5a03fabd737a126ad93d62a9eeede87f\nspec: 1.0.1-dev\n"
Nov 07 10:38:10 minikube dockerd[1955]: time="2019-11-07T10:38:10.840714023Z" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/80ef83f1384465ddc769792e849271113ef1b7978d269ffe7dc10f6ff8a91e5f/shim.sock" debug=false pid=2992
Nov 07 10:38:10 minikube dockerd[1955]: time="2019-11-07T10:38:10.845414850Z" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/a44116db4d09c7c401c34e50b106016a4bbcf94a9d81d165a3dc5d8b35ec8abf/shim.sock" debug=false pid=2994
Nov 07 10:38:10 minikube dockerd[1955]: time="2019-11-07T10:38:10.848096860Z" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/ba8c7713eee8a9032e32735d006bbb5cab61cffd2eeb7432ccbb825431031554/shim.sock" debug=false pid=3001
Nov 07 10:38:10 minikube dockerd[1955]: time="2019-11-07T10:38:10.862172302Z" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/a94e99305cd8ae9f3ec94e2959d6a062283f487628b136904c894386059cd0b0/shim.sock" debug=false pid=3015
Nov 07 10:38:10 minikube dockerd[1955]: time="2019-11-07T10:38:10.867688009Z" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/503743188ff644848776ddf8afbbd1206b1b8f7c308cfa342c89a64f58659647/shim.sock" debug=false pid=3027
Nov 07 10:38:11 minikube dockerd[1955]: time="2019-11-07T10:38:11.182523298Z" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/dd1eabc1b9ee9ce9debc1a9b3bbcac133508c4e8d0ab4f96329c787db0899fad/shim.sock" debug=false pid=3219
Nov 07 10:38:11 minikube dockerd[1955]: time="2019-11-07T10:38:11.207911209Z" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/c60248781cea4c0c6c2e68a951e6f7c65eaa1e488641483ba21c1542fa34e4b2/shim.sock" debug=false pid=3230
Nov 07 10:38:11 minikube dockerd[1955]: time="2019-11-07T10:38:11.276317723Z" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/678b8f5e56ab85dbde92d831c031cd9552ae77752bd8613f2d83c417d1542d23/shim.sock" debug=false pid=3258
Nov 07 10:38:11 minikube dockerd[1955]: time="2019-11-07T10:38:11.294248539Z" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/ddf13220dcf3fc5e359e5cd17e96f35b8ab5bb23984ae84d1790801b982ee5d5/shim.sock" debug=false pid=3267
Nov 07 10:38:15 minikube dockerd[1955]: time="2019-11-07T10:38:15.488640321Z" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/4f025cf7b783bc82f4aa522dcd8b5fef518d07f35a7c14866f7b6d0ace01e745/shim.sock" debug=false pid=3452
Nov 07 10:38:28 minikube dockerd[1955]: time="2019-11-07T10:38:28.615367387Z" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/47ccdd6d5a24a88741782e9e589cae004abe9891efa6019a27fbb8b2e363fe48/shim.sock" debug=false pid=3725
Nov 07 10:38:28 minikube dockerd[1955]: time="2019-11-07T10:38:28.716101102Z" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/57f434d8428170ae867b79885bae87a2d1db1c9e9a39deaaf4aa60287735820c/shim.sock" debug=false pid=3750
Nov 07 10:38:28 minikube dockerd[1955]: time="2019-11-07T10:38:28.850565388Z" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/52d0ebf101e3b65c6c84eede1005e7922a3a7b67edf40029c6bc1616fa6f8532/shim.sock" debug=false pid=3821
Nov 07 10:38:28 minikube dockerd[1955]: time="2019-11-07T10:38:28.864199660Z" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/e301ff8cab442960a4810e3f58ebd9af44546a6ca6b2b96eef59f1a53b04bc85/shim.sock" debug=false pid=3826
Nov 07 10:38:28 minikube dockerd[1955]: time="2019-11-07T10:38:28.995033238Z" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/1a183a38a1624f08ac2cb81a327488860ebd6367ebc22d098a139460162e2e66/shim.sock" debug=false pid=3897
Nov 07 10:38:29 minikube dockerd[1955]: time="2019-11-07T10:38:29.019130599Z" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/fde8c6cad14b493781fccdb37bc794e9c72bb06c4dc84f4d02cb9f92cd20fdbe/shim.sock" debug=false pid=3908
Nov 07 10:38:29 minikube dockerd[1955]: time="2019-11-07T10:38:29.397135839Z" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/d7179ca2ee18244a6f07e42aab0f6025f888407165b6c62c5f4475c09da6e9ca/shim.sock" debug=false pid=4056
Nov 07 10:38:29 minikube dockerd[1955]: time="2019-11-07T10:38:29.449494931Z" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/15c02ccc41e0aeed9bfc5f6217ad4544dda5199c6f4ba18c032f6247b28a4510/shim.sock" debug=false pid=4075
Nov 07 10:40:29 minikube dockerd[1955]: time="2019-11-07T10:40:29.613954025Z" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/640aed2b96584d9ab11f8e1b0a947aec550b83a903461712f6aa4ae7b88d2861/shim.sock" debug=false pid=5204
Nov 07 10:40:32 minikube dockerd[1955]: time="2019-11-07T10:40:32.495969417Z" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/3ba75987e3edf39a4a27a041a81e3a3ca866f3c612192deab6d27d3f3d0203b6/shim.sock" debug=false pid=5292

==> container status <==
CONTAINER           IMAGE                                                                                                     CREATED             STATE               NAME                      ATTEMPT             POD ID
3ba75987e3edf       k8s.gcr.io/metrics-server-amd64@sha256:49a9f12f7067d11f42c803dbe61ed2c1299959ad85cb315b25ff7eef8e6b8892   4 minutes ago       Running             metrics-server            0                   640aed2b96584
15c02ccc41e0a       bf261d1579144                                                                                             6 minutes ago       Running             coredns                   0                   e301ff8cab442
d7179ca2ee182       bf261d1579144                                                                                             6 minutes ago       Running             coredns                   0                   52d0ebf101e3b
fde8c6cad14b4       4689081edb103                                                                                             6 minutes ago       Running             storage-provisioner       0                   47ccdd6d5a24a
1a183a38a1624       8454cbe08dc9f                                                                                             6 minutes ago       Running             kube-proxy                0                   57f434d842817
4f025cf7b783b       k8s.gcr.io/kube-addon-manager@sha256:3e315022a842d782a28e729720f21091dde21f1efea28868d65ec595ad871616     6 minutes ago       Running             kube-addon-manager        0                   80ef83f138446
ddf13220dcf3f       6e4bffa46d70b                                                                                             6 minutes ago       Running             kube-controller-manager   0                   503743188ff64
678b8f5e56ab8       b2756210eeabf                                                                                             6 minutes ago       Running             etcd                      0                   a94e99305cd8a
c60248781cea4       ebac1ae204a2c                                                                                             6 minutes ago       Running             kube-scheduler            0                   a44116db4d09c
dd1eabc1b9ee9       c2c9a0406787c                                                                                             6 minutes ago       Running             kube-apiserver            0                   ba8c7713eee8a

==> coredns ["15c02ccc41e0"] <==
E1107 10:38:59.636993       1 reflector.go:126] pkg/mod/k8s.io/client-go@v11.0.0+incompatible/tools/cache/reflector.go:94: Failed to list *v1.Namespace: Get https://10.96.0.1:443/api/v1/namespaces?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout
.:E3
107 10:38:59.6374201       1 reflector.go:126] pkg/modi/nk8r.iol/caientRgun@n11.0.0+incompatible/tools/cache/aeflecion gM:D94:  ailed to4c lbi9s7t7c 7*vfc1.S8rvcice:fGatb1https://60.96.0.1:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.96.6.3:743: i/o timeouD
NS-1.6.2
E1107 10:38:59.637708       1 reflector.go:126] pkg/mod/k8s.io/clie3t7-go[v11F0.0 ilcompa/iabmle/t ols1ca1che./re,flec5a3.go:94: Failed to list *v1.Endpoints: GDt hSttp1s:6//10.96.l.1n4u4x/api/614/,enopoints?,i mit=5905&raesourceVersi0on=0: -i0al tc: 18:.96.00.21:4Z4 3:INiFOo]t ipmuogut
n/ready: Still waiting on: "kubernetes"
2019-11-07T10:38:49.021Z [INFO] plugin/ready: Still waiting on: "kubernetes"
2019-11-07T10:38:59.019Z [INFO] plugin/ready: Still waiting on: "kubernetes"
I1107 10:38:59.636745       1 trace.go:82] Trace[1955056402]: "Reflector pkg/mod/k8s.io/client-go@v11.0.0+incompatible/tools/cache/reflector.go:94 ListAndWatch" (started: 2019-11-07 10:38:29.636114877 +0000 UTC m=+0.021589822) (total time: 30.000575796s):
Trace[1955056402]: [30.000575796s] [30.000575796s] END
E1107 10:38:59.636993       1 reflector.go:126] pkg/mod/k8s.io/client-go@v11.0.0+incompatible/tools/cache/reflector.go:94: Failed to list *v1.Namespace: Get https://10.96.0.1:443/api/v1/namespaces?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout
E1107 10:38:59.636993       1 reflector.go:126] pkg/mod/k8s.io/client-go@v11.0.0+incompatible/tools/cache/reflector.go:94: Failed to list *v1.Namespace: Get https://10.96.0.1:443/api/v1/namespaces?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout
E1107 10:38:59.636993       1 reflector.go:126] pkg/mod/k8s.io/client-go@v11.0.0+incompatible/tools/cache/reflector.go:94: Failed to list *v1.Namespace: Get https://10.96.0.1:443/api/v1/namespaces?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout
I1107 10:38:59.637032       1 trace.go:82] Trace[1723824775]: "Reflector pkg/mod/k8s.io/client-go@v11.0.0+incompatible/tools/cache/reflector.go:94 ListAndWatch" (started: 2019-11-07 10:38:29.636585028 +0000 UTC m=+0.022059955) (total time: 30.00021973s):
Trace[1723824775]: [30.00021973s] [30.00021973s] END
E1107 10:38:59.637401       1 reflector.go:126] pkg/mod/k8s.io/client-go@v11.0.0+incompatible/tools/cache/reflector.go:94: Failed to list *v1.Service: Get https://10.96.0.1:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout
E1107 10:38:59.637401       1 reflector.go:126] pkg/mod/k8s.io/client-go@v11.0.0+incompatible/tools/cache/reflector.go:94: Failed to list *v1.Service: Get https://10.96.0.1:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout
E1107 10:38:59.637401       1 reflector.go:126] pkg/mod/k8s.io/client-go@v11.0.0+incompatible/tools/cache/reflector.go:94: Failed to list *v1.Service: Get https://10.96.0.1:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout
I1107 10:38:59.637658       1 trace.go:82] Trace[2142836995]: "Reflector pkg/mod/k8s.io/client-go@v11.0.0+incompatible/tools/cache/reflector.go:94 ListAndWatch" (started: 2019-11-07 10:38:29.636783089 +0000 UTC m=+0.022258008) (total time: 30.000864232s):
Trace[2142836995]: [30.000864232s] [30.000864232s] END
E1107 10:38:59.637708       1 reflector.go:126] pkg/mod/k8s.io/client-go@v11.0.0+incompatible/tools/cache/reflector.go:94: Failed to list *v1.Endpoints: Get https://10.96.0.1:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout
E1107 10:38:59.637708       1 reflector.go:126] pkg/mod/k8s.io/client-go@v11.0.0+incompatible/tools/cache/reflector.go:94: Failed to list *v1.Endpoints: Get https://10.96.0.1:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout
E1107 10:38:59.637708       1 reflector.go:126] pkg/mod/k8s.io/client-go@v11.0.0+incompatible/tools/cache/reflector.go:94: Failed to list *v1.Endpoints: Get https://10.96.0.1:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout

==> coredns ["d7179ca2ee18"] <==
E1107 10:38:59.610515       1 reflector.go:126] pkg/mod/k8s.io/client-go@v11.0.0+incompatible/tools/cache/reflector.go:94: Failed to list *v1.Service: Get https://10.96.0.1:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout
E1107 10:38:59.611395       1 reflector.go:126] pkg/mod/k8s.io/client-go@v11.0.0+incompatible/tools/cache/reflector.go:94: Failed to list *v1.Namespace: Get https://10.96.0.1:443/api/v1/namespaces?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeo.t:53
2019-11-07T10:38:34.613Z [INFO] plugin/reload: Running configuratEon 1MD05  10:38:c9b961b1967 c  d fc  a5 r8ecf4fecbo1r0.go:1a67] 
kg/mod2k08s9i-o/c-i07ntT-1go:v311:0.0+in4.613Z [INFO] CoreDNS-1compatible/tools/cache/re.flector.go:94: Failed to list *v1.Endpoints: Get https://10.96.0.1:443/api/v1/en2po1i9ts?l-m07tT51000&res3u4.ceVer i[In=0:]dial ucp 10d6., .g0o.1:2.8443: ieob timeout
DNS-1.6.2
linux/amd64, go1.12.8, 795a3eb
2019-11-07T10:38:35.587Z [INFO] plugin/ready: Still waiting on: "kubernetes"
2019-11-07T10:38:45.587Z [INFO] plugin/ready: Still waiting on: "kubernetes"
2019-11-07T10:38:55.588Z [INFO] plugin/ready: Still waiting on: "kubernetes"
I1107 10:38:59.610273       1 trace.go:82] Trace[1464848933]: "Reflector pkg/mod/k8s.io/client-go@v11.0.0+incompatible/tools/cache/reflector.go:94 ListAndWatch" (started: 2019-11-07 10:38:29.609732738 +0000 UTC m=+0.042608353) (total time: 30.000512741s):
Trace[1464848933]: [30.000512741s] [30.000512741s] END
E1107 10:38:59.610515       1 reflector.go:126] pkg/mod/k8s.io/client-go@v11.0.0+incompatible/tools/cache/reflector.go:94: Failed to list *v1.Service: Get https://10.96.0.1:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout
E1107 10:38:59.610515       1 reflector.go:126] pkg/mod/k8s.io/client-go@v11.0.0+incompatible/tools/cache/reflector.go:94: Failed to list *v1.Service: Get https://10.96.0.1:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout
E1107 10:38:59.610515       1 reflector.go:126] pkg/mod/k8s.io/client-go@v11.0.0+incompatible/tools/cache/reflector.go:94: Failed to list *v1.Service: Get https://10.96.0.1:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout
I1107 10:38:59.610270       1 trace.go:82] Trace[1741398676]: "Reflector pkg/mod/k8s.io/client-go@v11.0.0+incompatible/tools/cache/reflector.go:94 ListAndWatch" (started: 2019-11-07 10:38:29.609732639 +0000 UTC m=+0.042608229) (total time: 30.000513259s):
Trace[1741398676]: [30.000513259s] [30.000513259s] END
E1107 10:38:59.611395       1 reflector.go:126] pkg/mod/k8s.io/client-go@v11.0.0+incompatible/tools/cache/reflector.go:94: Failed to list *v1.Namespace: Get https://10.96.0.1:443/api/v1/namespaces?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout
E1107 10:38:59.611395       1 reflector.go:126] pkg/mod/k8s.io/client-go@v11.0.0+incompatible/tools/cache/reflector.go:94: Failed to list *v1.Namespace: Get https://10.96.0.1:443/api/v1/namespaces?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout
E1107 10:38:59.611395       1 reflector.go:126] pkg/mod/k8s.io/client-go@v11.0.0+incompatible/tools/cache/reflector.go:94: Failed to list *v1.Namespace: Get https://10.96.0.1:443/api/v1/namespaces?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout
I1107 10:38:59.610975       1 trace.go:82] Trace[1346997222]: "Reflector pkg/mod/k8s.io/client-go@v11.0.0+incompatible/tools/cache/reflector.go:94 ListAndWatch" (started: 2019-11-07 10:38:29.61043046 +0000 UTC m=+0.043306049) (total time: 30.000526081s):
Trace[1346997222]: [30.000526081s] [30.000526081s] END
E1107 10:38:59.611965       1 reflector.go:126] pkg/mod/k8s.io/client-go@v11.0.0+incompatible/tools/cache/reflector.go:94: Failed to list *v1.Endpoints: Get https://10.96.0.1:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout
E1107 10:38:59.611965       1 reflector.go:126] pkg/mod/k8s.io/client-go@v11.0.0+incompatible/tools/cache/reflector.go:94: Failed to list *v1.Endpoints: Get https://10.96.0.1:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout
E1107 10:38:59.611965       1 reflector.go:126] pkg/mod/k8s.io/client-go@v11.0.0+incompatible/tools/cache/reflector.go:94: Failed to list *v1.Endpoints: Get https://10.96.0.1:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout

==> dmesg <==
[Nov 7 10:36] ERROR: earlyprintk= earlyser already used
[  +0.000000] You have booted with nomodeset. This means your GPU drivers are DISABLED
[  +0.000001] Any video related functionality will be severely degraded, and you may not even be able to suspend the system properly
[  +0.000000] Unless you actually understand what nomodeset does, you should reboot without enabling it
[  +0.005710] ACPI BIOS Warning (bug): Incorrect checksum in table [DSDT] - 0xC0, should be 0x1D (20180810/tbprint-177)
[Nov 7 10:37] ACPI Error: Could not enable RealTimeClock event (20180810/evxfevnt-184)
[  +0.000002] ACPI Warning: Could not enable fixed event - RealTimeClock (4) (20180810/evxface-620)
[  +0.006220] platform regulatory.0: Direct firmware load for regulatory.db failed with error -2
[  +1.823904] systemd[1]: Failed to bump fs.file-max, ignoring: Invalid argument
[  +0.006334] systemd-fstab-generator[1082]: Ignoring "noauto" for root device
[  +0.001110] systemd[1]: File /usr/lib/systemd/system/systemd-journald.service:12 configures an IP firewall (IPAddressDeny=any), but the local system does not support BPF/cgroup based firewalling.
[  +0.000001] systemd[1]: Proceeding WITHOUT firewalling in effect! (This warning is only shown for the first loaded unit using IP firewalling.)
[  +0.928881] NFSD: the nfsdcld client tracking upcall will be removed in 3.10. Please transition to using nfsdcltrack.
[  +0.550773] vboxguest: loading out-of-tree module taints kernel.
[  +0.003256] vboxguest: PCI device not found, probably running on physical hardware.
[  +1.113883] systemd-fstab-generator[1871]: Ignoring "noauto" for root device
[ +29.917905] systemd-fstab-generator[2512]: Ignoring "noauto" for root device
[ +11.893933] systemd-fstab-generator[2862]: Ignoring "noauto" for root device
[Nov 7 10:38] kauditd_printk_skb: 68 callbacks suppressed
[ +18.752926] kauditd_printk_skb: 20 callbacks suppressed
[ +30.800173] kauditd_printk_skb: 35 callbacks suppressed
[Nov 7 10:39] NFSD: Unable to end grace period: -110
[ +19.590601] kauditd_printk_skb: 5 callbacks suppressed
[Nov 7 10:41] kauditd_printk_skb: 11 callbacks suppressed

==> kernel <==
 10:44:50 up 7 min,  0 users,  load average: 0.84, 0.75, 0.39
Linux minikube 4.19.76 #1 SMP Tue Oct 29 14:56:42 PDT 2019 x86_64 GNU/Linux
PRETTY_NAME="Buildroot 2019.02.6"

==> kube-addon-manager ["4f025cf7b783"] <==
error: no objects passed to apply
error: no objects passed to apply
error: no objects passed to apply
INFO: Leader election disabled.
INFO: == Kubernetes addon ensure completed at 2019-11-07T10:44:22+00:00 ==
INFO: == Reconciling with deprecated label ==
INFO: == Reconciling with addon-manager label ==
apiservice.apiregistration.k8s.io/v1beta1.metrics.k8s.io unchanged
deployment.apps/metrics-server unchanged
service/metrics-server unchanged
serviceaccount/storage-provisioner unchanged
INFO: == Kubernetes addon reconcile completed at 2019-11-07T10:44:24+00:00 ==
INFO: Leader election disabled.
INFO: == Kubernetes addon ensure completed at 2019-11-07T10:44:27+00:00 ==
INFO: == Reconciling with deprecated label ==
INFO: == Reconciling with addon-manager label ==
apiservice.apiregistration.k8s.io/v1beta1.metrics.k8s.io unchanged
deployment.apps/metrics-server unchanged
service/metrics-server unchanged
serviceaccount/storage-provisioner unchanged
INFO: == Kubernetes addon reconcile completed at 2019-11-07T10:44:28+00:00 ==
INFO: Leader election disabled.
INFO: == Kubernetes addon ensure completed at 2019-11-07T10:44:32+00:00 ==
INFO: == Reconciling with depeerrtord:  lnob eob j=ec=t
 pasNseOd: t=o  Reconciling apiptlhy addon-managereraroerl: ==o
 obpjesctrsv ipcas.sapi reogiastpraytion.rk8sr.i on/ov1obejtac1.m etrsicesd.kt8s.aiop luyn
changed
deployment.apps/metrics-server unchanged
service/metrics-server unchanged
serviceaccount/storage-provisioner unchanged
INFO: == Kubernetes addon reconcile completed at 2019-11-07T10:44:34+00:00 ==
INFO: Leader election disabled.
INFO: == Kubernetes addon ensure completed at 2019-11-07T10:44:37+00:00 ==
INFO: == Reconciling with deprecated label ==
INFO: == Reconciling with addon-manager label ==
apiservice.apiregistration.k8s.io/v1beta1.metrics.k8s.io unchanged
deployment.apps/metrics-server unchanged
service/metrics-server unchanged
serviceaccount/storage-provisioner unchanged
INFO: == Kubernetes addon reconcile completed at 2019-11-07T10:44:39+00:00 ==
INFO: Leader election disabled.
INFO: == Kubernetes addon ensure completed at 2019-11-07T10:44:42+00:00 ==
INFO: == Reconciling with deprecated label ==
INFO: == Reconciling with addon-manager label ==
apiservice.apiregistration.k8s.io/v1beta1.metrics.k8s.io unchanged
deployment.apps/metrics-server unchanged
service/metrics-server unchanged
serviceaccount/storage-provisioner unchanged
INFO: == Kubernetes addon reconcile completed at 2019-11-07T10:44:43+00:00 ==
INFO: Leader election disabled.
INFO: == Kubernetes addon ensure completed at 2019-11-07T10:44:47+00:00 ==
INFO: == Reconciling with deprecated label ==
INFO: == Reconciling with addon-manager label ==
apiservice.apiregistration.k8s.io/v1beta1.metrics.k8s.io unchanged
deployment.apps/metrics-server unchanged
service/metrics-server unchanged
serviceaccount/storage-provisioner unchanged
INFO: == Kubernetes addon reconcile completed at 2019-11-07T10:44:49+00:00 ==

==> kube-apiserver ["dd1eabc1b9ee"] <==
I1107 10:38:15.891146       1 controller.go:85] Starting OpenAPI controller
I1107 10:38:15.891459       1 customresource_discovery_controller.go:208] Starting DiscoveryController
I1107 10:38:15.891721       1 naming_controller.go:288] Starting NamingConditionController
I1107 10:38:15.891884       1 establishing_controller.go:73] Starting EstablishingController
I1107 10:38:15.891950       1 nonstructuralschema_controller.go:191] Starting NonStructuralSchemaConditionController
I1107 10:38:15.892002       1 apiapproval_controller.go:185] Starting KubernetesAPIApprovalPolicyConformantConditionController
I1107 10:38:15.894583       1 apiservice_controller.go:94] Starting APIServiceRegistrationController
I1107 10:38:15.894735       1 cache.go:32] Waiting for caches to sync for APIServiceRegistrationController controller
I1107 10:38:15.896318       1 available_controller.go:383] Starting AvailableConditionController
I1107 10:38:15.896344       1 cache.go:32] Waiting for caches to sync for AvailableConditionController controller
I1107 10:38:15.896360       1 controller.go:81] Starting OpenAPI AggregationController
E1107 10:38:15.932010       1 controller.go:154] Unable to remove old endpoints from kubernetes service: StorageError: key not found, Code: 1, Key: /registry/masterleases/192.168.64.44, ResourceVersion: 0, AdditionalErrorMsg: 
I1107 10:38:15.993519       1 cache.go:39] Caches are synced for autoregister controller
I1107 10:38:15.995838       1 cache.go:39] Caches are synced for APIServiceRegistrationController controller
I1107 10:38:15.998481       1 shared_informer.go:204] Caches are synced for crd-autoregister 
I1107 10:38:15.999022       1 cache.go:39] Caches are synced for AvailableConditionController controller
I1107 10:38:16.890599       1 controller.go:107] OpenAPI AggregationController: Processing item 
I1107 10:38:16.890642       1 controller.go:130] OpenAPI AggregationController: action for item : Nothing (removed from the queue).
I1107 10:38:16.890777       1 controller.go:130] OpenAPI AggregationController: action for item k8s_internal_local_delegation_chain_0000000000: Nothing (removed from the queue).
I1107 10:38:16.926270       1 storage_scheduling.go:139] created PriorityClass system-node-critical with value 2000001000
I1107 10:38:16.941377       1 storage_scheduling.go:139] created PriorityClass system-cluster-critical with value 2000000000
I1107 10:38:16.941639       1 storage_scheduling.go:148] all system priority classes are created successfully or already exist.
I1107 10:38:17.207849       1 controller.go:606] quota admission added evaluator for: roles.rbac.authorization.k8s.io
I1107 10:38:17.233554       1 controller.go:606] quota admission added evaluator for: rolebindings.rbac.authorization.k8s.io
W1107 10:38:17.299927       1 lease.go:222] Resetting endpoints for master service "kubernetes" to [192.168.64.44]
I1107 10:38:17.301013       1 controller.go:606] quota admission added evaluator for: endpoints
I1107 10:38:18.354517       1 controller.go:606] quota admission added evaluator for: serviceaccounts
I1107 10:38:18.370448       1 controller.go:606] quota admission added evaluator for: deployments.apps
I1107 10:38:18.712098       1 controller.go:606] quota admission added evaluator for: daemonsets.apps
I1107 10:38:19.263709       1 controller.go:606] quota admission added evaluator for: leases.coordination.k8s.io
I1107 10:38:27.396427       1 controller.go:606] quota admission added evaluator for: controllerrevisions.apps
I1107 10:38:27.421502       1 controller.go:606] quota admission added evaluator for: events.events.k8s.io
I1107 10:38:27.454740       1 controller.go:606] quota admission added evaluator for: replicasets.apps
I1107 10:40:29.831858       1 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
W1107 10:40:29.831898       1 handler_proxy.go:99] no RequestInfo found in the context
E1107 10:40:29.831942       1 controller.go:114] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
, Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
I1107 10:40:29.831948       1 controller.go:127] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
I1107 10:40:35.216162       1 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E1107 10:40:35.232497       1 controller.go:114] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: OpenAPI spec does not exist
I1107 10:40:35.232619       1 controller.go:127] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
I1107 10:41:05.799578       1 trace.go:116] Trace[1419279147]: "Get" url:/api/v1/namespaces/kube-system/pods/metrics-server-587f876775-4d8jq/log (started: 2019-11-07 10:40:35.077638848 +0000 UTC m=+143.604168291) (total time: 30.721895666s):
Trace[1419279147]: [30.72189388s] [30.719624571s] Transformed response object
I1107 10:41:11.526004       1 trace.go:116] Trace[661815294]: "Get" url:/api/v1/namespaces/kube-system/pods/metrics-server-587f876775-4d8jq/log (started: 2019-11-07 10:41:08.118909552 +0000 UTC m=+176.645438995) (total time: 3.407000949s):
Trace[661815294]: [3.406998986s] [3.404356742s] Transformed response object
I1107 10:41:27.828073       1 trace.go:116] Trace[1050020735]: "Get" url:/api/v1/namespaces/kube-system/pods/metrics-server-587f876775-4d8jq/log (started: 2019-11-07 10:41:21.405035621 +0000 UTC m=+189.931565069) (total time: 6.423013259s):
Trace[1050020735]: [6.423012075s] [6.420922081s] Transformed response object
I1107 10:41:35.233326       1 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E1107 10:41:35.235786       1 controller.go:114] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: OpenAPI spec does not exist
I1107 10:41:35.235943       1 controller.go:127] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
I1107 10:41:40.111974       1 trace.go:116] Trace[1787699228]: "Get" url:/api/v1/namespaces/kube-system/pods/metrics-server-587f876775-4d8jq/log (started: 2019-11-07 10:41:29.666832142 +0000 UTC m=+198.193361592) (total time: 10.445113067s):
Trace[1787699228]: [10.445111394s] [10.442754297s] Transformed response object
I1107 10:43:16.956351       1 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E1107 10:43:16.959818       1 controller.go:114] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: OpenAPI spec does not exist
I1107 10:43:16.960051       1 controller.go:127] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
I1107 10:43:39.504173       1 trace.go:116] Trace[809237406]: "Get" url:/api/v1/namespaces/kube-system/pods/metrics-server-587f876775-4d8jq/log (started: 2019-11-07 10:42:15.575614397 +0000 UTC m=+244.102143837) (total time: 1m23.928404061s):
Trace[809237406]: [1m23.928402458s] [1m23.926732155s] Transformed response object
I1107 10:44:16.960472       1 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E1107 10:44:16.963582       1 controller.go:114] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: OpenAPI spec does not exist
I1107 10:44:16.963623       1 controller.go:127] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.

==> kube-controller-manager ["ddf13220dcf3"] <==
I1107 10:38:26.255308       1 controllermanager.go:534] Started "podgc"
I1107 10:38:26.255369       1 gc_controller.go:75] Starting GC controller
I1107 10:38:26.255383       1 shared_informer.go:197] Waiting for caches to sync for GC
I1107 10:38:26.503493       1 controllermanager.go:534] Started "tokencleaner"
I1107 10:38:26.503628       1 tokencleaner.go:117] Starting token cleaner controller
I1107 10:38:26.503799       1 shared_informer.go:197] Waiting for caches to sync for token_cleaner
I1107 10:38:26.503850       1 shared_informer.go:204] Caches are synced for token_cleaner 
W1107 10:38:26.754507       1 probe.go:268] Flexvolume plugin directory at /usr/libexec/kubernetes/kubelet-plugins/volume/exec/ does not exist. Recreating.
I1107 10:38:26.755452       1 controllermanager.go:534] Started "attachdetach"
I1107 10:38:26.755687       1 attach_detach_controller.go:334] Starting attach detach controller
I1107 10:38:26.755900       1 shared_informer.go:197] Waiting for caches to sync for attach detach
I1107 10:38:27.003597       1 controllermanager.go:534] Started "replicationcontroller"
I1107 10:38:27.003888       1 shared_informer.go:197] Waiting for caches to sync for resource quota
I1107 10:38:27.004064       1 replica_set.go:182] Starting replicationcontroller controller
I1107 10:38:27.004121       1 shared_informer.go:197] Waiting for caches to sync for ReplicationController
W1107 10:38:27.010868       1 actual_state_of_world.go:506] Failed to update statusUpdateNeeded field in actual state of world: Failed to set statusUpdateNeeded to needed true, because nodeName="minikube" does not exist
I1107 10:38:27.054340       1 shared_informer.go:204] Caches are synced for ClusterRoleAggregator 
I1107 10:38:27.057061       1 shared_informer.go:204] Caches are synced for TTL 
I1107 10:38:27.057210       1 shared_informer.go:204] Caches are synced for certificate 
I1107 10:38:27.057250       1 shared_informer.go:204] Caches are synced for PV protection 
I1107 10:38:27.057259       1 shared_informer.go:204] Caches are synced for certificate 
I1107 10:38:27.078873       1 log.go:172] [INFO] signed certificate with serial number 668992698187447208126476588932421869414867040165
E1107 10:38:27.084115       1 clusterroleaggregation_controller.go:180] view failed with : Operation cannot be fulfilled on clusterroles.rbac.authorization.k8s.io "view": the object has been modified; please apply your changes to the latest version and try again
I1107 10:38:27.105483       1 shared_informer.go:204] Caches are synced for bootstrap_signer 
I1107 10:38:27.204321       1 shared_informer.go:204] Caches are synced for expand 
I1107 10:38:27.367497       1 shared_informer.go:204] Caches are synced for PVC protection 
I1107 10:38:27.374816       1 shared_informer.go:204] Caches are synced for taint 
I1107 10:38:27.375035       1 node_lifecycle_controller.go:1208] Initializing eviction metric for zone: 
W1107 10:38:27.375226       1 node_lifecycle_controller.go:903] Missing timestamp for Node minikube. Assuming now as a timestamp.
I1107 10:38:27.375381       1 node_lifecycle_controller.go:1108] Controller detected that zone  is now in state Normal.
I1107 10:38:27.375592       1 event.go:255] Event(v1.ObjectReference{Kind:"Node", Namespace:"", Name:"minikube", UID:"a7d2e139-b0ec-409c-ba56-41393e13da42", APIVersion:"", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RegisteredNode' Node minikube event: Registered Node minikube in Controller
I1107 10:38:27.375776       1 taint_manager.go:186] Starting NoExecuteTaintManager
I1107 10:38:27.392302       1 shared_informer.go:204] Caches are synced for daemon sets 
I1107 10:38:27.403753       1 event.go:255] Event(v1.ObjectReference{Kind:"DaemonSet", Namespace:"kube-system", Name:"kube-proxy", UID:"66940beb-bae2-41b9-b371-0ad432570d57", APIVersion:"apps/v1", ResourceVersion:"178", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: kube-proxy-tkxr8
I1107 10:38:27.406071       1 shared_informer.go:204] Caches are synced for disruption 
I1107 10:38:27.406143       1 disruption.go:341] Sending events to api server.
I1107 10:38:27.406391       1 shared_informer.go:204] Caches are synced for persistent volume 
I1107 10:38:27.406658       1 shared_informer.go:204] Caches are synced for endpoint 
I1107 10:38:27.406825       1 shared_informer.go:204] Caches are synced for ReplicationController 
I1107 10:38:27.406730       1 shared_informer.go:204] Caches are synced for stateful set 
I1107 10:38:27.409037       1 shared_informer.go:197] Waiting for caches to sync for garbage collector
I1107 10:38:27.409376       1 shared_informer.go:204] Caches are synced for namespace 
I1107 10:38:27.425769       1 shared_informer.go:204] Caches are synced for job 
I1107 10:38:27.429248       1 shared_informer.go:204] Caches are synced for ReplicaSet 
E1107 10:38:27.436591       1 daemon_controller.go:302] kube-system/kube-proxy failed with : error storing status for daemon set &v1.DaemonSet{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"kube-proxy", GenerateName:"", Namespace:"kube-system", SelfLink:"/apis/apps/v1/namespaces/kube-system/daemonsets/kube-proxy", UID:"66940beb-bae2-41b9-b371-0ad432570d57", ResourceVersion:"178", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63708719898, loc:(*time.Location)(0x7777fc0)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"k8s-app":"kube-proxy"}, Annotations:map[string]string{"deprecated.daemonset.template.generation":"1"}, OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.DaemonSetSpec{Selector:(*v1.LabelSelector)(0xc00171eb60), Template:v1.PodTemplateSpec{ObjectMeta:v1.ObjectMeta{Name:"", GenerateName:"", Namespace:"", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"k8s-app":"kube-proxy"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.PodSpec{Volumes:[]v1.Volume{v1.Volume{Name:"kube-proxy", VolumeSource:v1.VolumeSource{HostPath:(*v1.HostPathVolumeSource)(nil), EmptyDir:(*v1.EmptyDirVolumeSource)(nil), GCEPersistentDisk:(*v1.GCEPersistentDiskVolumeSource)(nil), AWSElasticBlockStore:(*v1.AWSElasticBlockStoreVolumeSource)(nil), GitRepo:(*v1.GitRepoVolumeSource)(nil), Secret:(*v1.SecretVolumeSource)(nil), NFS:(*v1.NFSVolumeSource)(nil), ISCSI:(*v1.ISCSIVolumeSource)(nil), Glusterfs:(*v1.GlusterfsVolumeSource)(nil), PersistentVolumeClaim:(*v1.PersistentVolumeClaimVolumeSource)(nil), RBD:(*v1.RBDVolumeSource)(nil), FlexVolume:(*v1.FlexVolumeSource)(nil), Cinder:(*v1.CinderVolumeSource)(nil), CephFS:(*v1.CephFSVolumeSource)(nil), Flocker:(*v1.FlockerVolumeSource)(nil), DownwardAPI:(*v1.DownwardAPIVolumeSource)(nil), FC:(*v1.FCVolumeSource)(nil), AzureFile:(*v1.AzureFileVolumeSource)(nil), ConfigMap:(*v1.ConfigMapVolumeSource)(0xc001697d40), VsphereVolume:(*v1.VsphereVirtualDiskVolumeSource)(nil), Quobyte:(*v1.QuobyteVolumeSource)(nil), AzureDisk:(*v1.AzureDiskVolumeSource)(nil), PhotonPersistentDisk:(*v1.PhotonPersistentDiskVolumeSource)(nil), Projected:(*v1.ProjectedVolumeSource)(nil), PortworxVolume:(*v1.PortworxVolumeSource)(nil), ScaleIO:(*v1.ScaleIOVolumeSource)(nil), StorageOS:(*v1.StorageOSVolumeSource)(nil), CSI:(*v1.CSIVolumeSource)(nil)}}, v1.Volume{Name:"xtables-lock", VolumeSource:v1.VolumeSource{HostPath:(*v1.HostPathVolumeSource)(0xc00171eb80), EmptyDir:(*v1.EmptyDirVolumeSource)(nil), GCEPersistentDisk:(*v1.GCEPersistentDiskVolumeSource)(nil), AWSElasticBlockStore:(*v1.AWSElasticBlockStoreVolumeSource)(nil), GitRepo:(*v1.GitRepoVolumeSource)(nil), Secret:(*v1.SecretVolumeSource)(nil), NFS:(*v1.NFSVolumeSource)(nil), ISCSI:(*v1.ISCSIVolumeSource)(nil), Glusterfs:(*v1.GlusterfsVolumeSource)(nil), PersistentVolumeClaim:(*v1.PersistentVolumeClaimVolumeSource)(nil), RBD:(*v1.RBDVolumeSource)(nil), FlexVolume:(*v1.FlexVolumeSource)(nil), Cinder:(*v1.CinderVolumeSource)(nil), CephFS:(*v1.CephFSVolumeSource)(nil), Flocker:(*v1.FlockerVolumeSource)(nil), DownwardAPI:(*v1.DownwardAPIVolumeSource)(nil), FC:(*v1.FCVolumeSource)(nil), AzureFile:(*v1.AzureFileVolumeSource)(nil), ConfigMap:(*v1.ConfigMapVolumeSource)(nil), VsphereVolume:(*v1.VsphereVirtualDiskVolumeSource)(nil), Quobyte:(*v1.QuobyteVolumeSource)(nil), AzureDisk:(*v1.AzureDiskVolumeSource)(nil), PhotonPersistentDisk:(*v1.PhotonPersistentDiskVolumeSource)(nil), Projected:(*v1.ProjectedVolumeSource)(nil), PortworxVolume:(*v1.PortworxVolumeSource)(nil), ScaleIO:(*v1.ScaleIOVolumeSource)(nil), StorageOS:(*v1.StorageOSVolumeSource)(nil), CSI:(*v1.CSIVolumeSource)(nil)}}, v1.Volume{Name:"lib-modules", VolumeSource:v1.VolumeSource{HostPath:(*v1.HostPathVolumeSource)(0xc00171eba0), EmptyDir:(*v1.EmptyDirVolumeSource)(nil), GCEPersistentDisk:(*v1.GCEPersistentDiskVolumeSource)(nil), AWSElasticBlockStore:(*v1.AWSElasticBlockStoreVolumeSource)(nil), GitRepo:(*v1.GitRepoVolumeSource)(nil), Secret:(*v1.SecretVolumeSource)(nil), NFS:(*v1.NFSVolumeSource)(nil), ISCSI:(*v1.ISCSIVolumeSource)(nil), Glusterfs:(*v1.GlusterfsVolumeSource)(nil), PersistentVolumeClaim:(*v1.PersistentVolumeClaimVolumeSource)(nil), RBD:(*v1.RBDVolumeSource)(nil), FlexVolume:(*v1.FlexVolumeSource)(nil), Cinder:(*v1.CinderVolumeSource)(nil), CephFS:(*v1.CephFSVolumeSource)(nil), Flocker:(*v1.FlockerVolumeSource)(nil), DownwardAPI:(*v1.DownwardAPIVolumeSource)(nil), FC:(*v1.FCVolumeSource)(nil), AzureFile:(*v1.AzureFileVolumeSource)(nil), ConfigMap:(*v1.ConfigMapVolumeSource)(nil), VsphereVolume:(*v1.VsphereVirtualDiskVolumeSource)(nil), Quobyte:(*v1.QuobyteVolumeSource)(nil), AzureDisk:(*v1.AzureDiskVolumeSource)(nil), PhotonPersistentDisk:(*v1.PhotonPersistentDiskVolumeSource)(nil), Projected:(*v1.ProjectedVolumeSource)(nil), PortworxVolume:(*v1.PortworxVolumeSource)(nil), ScaleIO:(*v1.ScaleIOVolumeSource)(nil), StorageOS:(*v1.StorageOSVolumeSource)(nil), CSI:(*v1.CSIVolumeSource)(nil)}}}, InitContainers:[]v1.Container(nil), Containers:[]v1.Container{v1.Container{Name:"kube-proxy", Image:"k8s.gcr.io/kube-proxy:v1.16.2", Command:[]string{"/usr/local/bin/kube-proxy", "--config=/var/lib/kube-proxy/config.conf", "--hostname-override=$(NODE_NAME)"}, Args:[]string(nil), WorkingDir:"", Ports:[]v1.ContainerPort(nil), EnvFrom:[]v1.EnvFromSource(nil), Env:[]v1.EnvVar{v1.EnvVar{Name:"NODE_NAME", Value:"", ValueFrom:(*v1.EnvVarSource)(0xc00171ebe0)}}, Resources:v1.ResourceRequirements{Limits:v1.ResourceList(nil), Requests:v1.ResourceList(nil)}, VolumeMounts:[]v1.VolumeMount{v1.VolumeMount{Name:"kube-proxy", ReadOnly:false, MountPath:"/var/lib/kube-proxy", SubPath:"", MountPropagation:(*v1.MountPropagationMode)(nil), SubPathExpr:""}, v1.VolumeMount{Name:"xtables-lock", ReadOnly:false, MountPath:"/run/xtables.lock", SubPath:"", MountPropagation:(*v1.MountPropagationMode)(nil), SubPathExpr:""}, v1.VolumeMount{Name:"lib-modules", ReadOnly:true, MountPath:"/lib/modules", SubPath:"", MountPropagation:(*v1.MountPropagationMode)(nil), SubPathExpr:""}}, VolumeDevices:[]v1.VolumeDevice(nil), LivenessProbe:(*v1.Probe)(nil), ReadinessProbe:(*v1.Probe)(nil), StartupProbe:(*v1.Probe)(nil), Lifecycle:(*v1.Lifecycle)(nil), TerminationMessagePath:"/dev/termination-log", TerminationMessagePolicy:"File", ImagePullPolicy:"IfNotPresent", SecurityContext:(*v1.SecurityContext)(0xc001387d10), Stdin:false, StdinOnce:false, TTY:false}}, EphemeralContainers:[]v1.EphemeralContainer(nil), RestartPolicy:"Always", TerminationGracePeriodSeconds:(*int64)(0xc0016b7708), ActiveDeadlineSeconds:(*int64)(nil), DNSPolicy:"ClusterFirst", NodeSelector:map[string]string{"beta.kubernetes.io/os":"linux"}, ServiceAccountName:"kube-proxy", DeprecatedServiceAccount:"kube-proxy", AutomountServiceAccountToken:(*bool)(nil), NodeName:"", HostNetwork:true, HostPID:false, HostIPC:false, ShareProcessNamespace:(*bool)(nil), SecurityContext:(*v1.PodSecurityContext)(0xc0015edd40), ImagePullSecrets:[]v1.LocalObjectReference(nil), Hostname:"", Subdomain:"", Affinity:(*v1.Affinity)(nil), SchedulerName:"default-scheduler", Tolerations:[]v1.Toleration{v1.Toleration{Key:"CriticalAddonsOnly", Operator:"Exists", Value:"", Effect:"", TolerationSeconds:(*int64)(nil)}, v1.Toleration{Key:"", Operator:"Exists", Value:"", Effect:"", TolerationSeconds:(*int64)(nil)}}, HostAliases:[]v1.HostAlias(nil), PriorityClassName:"system-node-critical", Priority:(*int32)(nil), DNSConfig:(*v1.PodDNSConfig)(nil), ReadinessGates:[]v1.PodReadinessGate(nil), RuntimeClassName:(*string)(nil), EnableServiceLinks:(*bool)(nil), PreemptionPolicy:(*v1.PreemptionPolicy)(nil), Overhead:v1.ResourceList(nil), TopologySpreadConstraints:[]v1.TopologySpreadConstraint(nil)}}, UpdateStrategy:v1.DaemonSetUpdateStrategy{Type:"RollingUpdate", RollingUpdate:(*v1.RollingUpdateDaemonSet)(0xc00000f198)}, MinReadySeconds:0, RevisionHistoryLimit:(*int32)(0xc0016b7748)}, Status:v1.DaemonSetStatus{CurrentNumberScheduled:0, NumberMisscheduled:0, DesiredNumberScheduled:0, NumberReady:0, ObservedGeneration:0, UpdatedNumberScheduled:0, NumberAvailable:0, NumberUnavailable:0, CollisionCount:(*int32)(nil), Conditions:[]v1.DaemonSetCondition(nil)}}: Operation cannot be fulfilled on daemonsets.apps "kube-proxy": the object has been modified; please apply your changes to the latest version and try again
I1107 10:38:27.452966       1 shared_informer.go:204] Caches are synced for deployment 
I1107 10:38:27.455089       1 shared_informer.go:204] Caches are synced for HPA 
I1107 10:38:27.457647       1 shared_informer.go:204] Caches are synced for attach detach 
I1107 10:38:27.458903       1 event.go:255] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"kube-system", Name:"coredns", UID:"d7bf1502-446f-4da3-b156-bd5957e0285c", APIVersion:"apps/v1", ResourceVersion:"173", FieldPath:""}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled up replica set coredns-5644d7b6d9 to 2
I1107 10:38:27.459004       1 shared_informer.go:204] Caches are synced for GC 
I1107 10:38:27.464035       1 event.go:255] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"kube-system", Name:"coredns-5644d7b6d9", UID:"7b9b0068-7948-426d-8ba0-705913707e26", APIVersion:"apps/v1", ResourceVersion:"341", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: coredns-5644d7b6d9-84dsl
I1107 10:38:27.497319       1 event.go:255] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"kube-system", Name:"coredns-5644d7b6d9", UID:"7b9b0068-7948-426d-8ba0-705913707e26", APIVersion:"apps/v1", ResourceVersion:"341", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: coredns-5644d7b6d9-lhtz2
I1107 10:38:27.504096       1 shared_informer.go:204] Caches are synced for service account 
I1107 10:38:27.604651       1 shared_informer.go:204] Caches are synced for resource quota 
I1107 10:38:27.609795       1 shared_informer.go:204] Caches are synced for garbage collector 
I1107 10:38:27.609839       1 garbagecollector.go:139] Garbage collector: all resource monitors have synced. Proceeding to collect garbage
I1107 10:38:27.610151       1 shared_informer.go:204] Caches are synced for garbage collector 
I1107 10:38:27.612000       1 shared_informer.go:204] Caches are synced for resource quota 
I1107 10:40:28.295834       1 event.go:255] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"kube-system", Name:"metrics-server", UID:"22fe556a-e353-415c-a5bb-ed24bdecbd4b", APIVersion:"apps/v1", ResourceVersion:"569", FieldPath:""}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled up replica set metrics-server-587f876775 to 1
I1107 10:40:28.306430       1 event.go:255] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"kube-system", Name:"metrics-server-587f876775", UID:"6f01ea5e-0c2d-4d2c-96ef-3aac40e0e671", APIVersion:"apps/v1", ResourceVersion:"570", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: metrics-server-587f876775-4d8jq

==> kube-proxy ["1a183a38a162"] <==
W1107 10:38:29.496577       1 server_others.go:329] Flag proxy-mode="" unknown, assuming iptables proxy
I1107 10:38:29.513405       1 node.go:135] Successfully retrieved node IP: 192.168.64.44
I1107 10:38:29.513447       1 server_others.go:149] Using iptables Proxier.
W1107 10:38:29.513658       1 proxier.go:287] clusterCIDR not specified, unable to distinguish between internal and external traffic
I1107 10:38:29.516830       1 server.go:529] Version: v1.16.2
I1107 10:38:29.536063       1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_max' to 131072
I1107 10:38:29.536381       1 conntrack.go:52] Setting nf_conntrack_max to 131072
I1107 10:38:29.538105       1 conntrack.go:83] Setting conntrack hashsize to 32768
I1107 10:38:29.544504       1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400
I1107 10:38:29.544583       1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600
I1107 10:38:29.544776       1 config.go:131] Starting endpoints config controller
I1107 10:38:29.544792       1 shared_informer.go:197] Waiting for caches to sync for endpoints config
I1107 10:38:29.544841       1 config.go:313] Starting service config controller
I1107 10:38:29.544851       1 shared_informer.go:197] Waiting for caches to sync for service config
I1107 10:38:29.645164       1 shared_informer.go:204] Caches are synced for service config 
I1107 10:38:29.645171       1 shared_informer.go:204] Caches are synced for endpoints config 

==> kube-scheduler ["c60248781cea"] <==
I1107 10:38:12.288905       1 serving.go:319] Generated self-signed cert in-memory
W1107 10:38:15.946951       1 authentication.go:262] Unable to get configmap/extension-apiserver-authentication in kube-system.  Usually fixed by 'kubectl create rolebinding -n kube-system ROLEBINDING_NAME --role=extension-apiserver-authentication-reader --serviceaccount=YOUR_NS:YOUR_SA'
W1107 10:38:15.947018       1 authentication.go:199] Error looking up in-cluster authentication configuration: configmaps "extension-apiserver-authentication" is forbidden: User "system:kube-scheduler" cannot get resource "configmaps" in API group "" in the namespace "kube-system"
W1107 10:38:15.947047       1 authentication.go:200] Continuing without authentication configuration. This may treat all requests as anonymous.
W1107 10:38:15.947053       1 authentication.go:201] To require authentication configuration lookup to succeed, set --authentication-tolerate-lookup-failure=false
I1107 10:38:15.964734       1 server.go:143] Version: v1.16.2
I1107 10:38:15.965870       1 defaults.go:91] TaintNodesByCondition is enabled, PodToleratesNodeTaints predicate is mandatory
W1107 10:38:15.971810       1 authorization.go:47] Authorization is disabled
W1107 10:38:15.972008       1 authentication.go:79] Authentication is disabled
I1107 10:38:15.972152       1 deprecated_insecure_serving.go:51] Serving healthz insecurely on [::]:10251
I1107 10:38:15.972714       1 secure_serving.go:123] Serving securely on 127.0.0.1:10259
E1107 10:38:16.028586       1 reflector.go:123] k8s.io/kubernetes/cmd/kube-scheduler/app/server.go:236: Failed to list *v1.Pod: pods is forbidden: User "system:kube-scheduler" cannot list resource "pods" in API group "" at the cluster scope
E1107 10:38:16.032302       1 reflector.go:123] k8s.io/client-go/informers/factory.go:134: Failed to list *v1.Node: nodes is forbidden: User "system:kube-scheduler" cannot list resource "nodes" in API group "" at the cluster scope
E1107 10:38:16.032504       1 reflector.go:123] k8s.io/client-go/informers/factory.go:134: Failed to list *v1.PersistentVolume: persistentvolumes is forbidden: User "system:kube-scheduler" cannot list resource "persistentvolumes" in API group "" at the cluster scope
E1107 10:38:16.032664       1 reflector.go:123] k8s.io/client-go/informers/factory.go:134: Failed to list *v1beta1.CSINode: csinodes.storage.k8s.io is forbidden: User "system:kube-scheduler" cannot list resource "csinodes" in API group "storage.k8s.io" at the cluster scope
E1107 10:38:16.034406       1 reflector.go:123] k8s.io/client-go/informers/factory.go:134: Failed to list *v1.StatefulSet: statefulsets.apps is forbidden: User "system:kube-scheduler" cannot list resource "statefulsets" in API group "apps" at the cluster scope
E1107 10:38:16.035177       1 reflector.go:123] k8s.io/client-go/informers/factory.go:134: Failed to list *v1.ReplicaSet: replicasets.apps is forbidden: User "system:kube-scheduler" cannot list resource "replicasets" in API group "apps" at the cluster scope
E1107 10:38:16.035744       1 reflector.go:123] k8s.io/client-go/informers/factory.go:134: Failed to list *v1.StorageClass: storageclasses.storage.k8s.io is forbidden: User "system:kube-scheduler" cannot list resource "storageclasses" in API group "storage.k8s.io" at the cluster scope
E1107 10:38:16.036011       1 reflector.go:123] k8s.io/client-go/informers/factory.go:134: Failed to list *v1.Service: services is forbidden: User "system:kube-scheduler" cannot list resource "services" in API group "" at the cluster scope
E1107 10:38:16.036271       1 reflector.go:123] k8s.io/client-go/informers/factory.go:134: Failed to list *v1.ReplicationController: replicationcontrollers is forbidden: User "system:kube-scheduler" cannot list resource "replicationcontrollers" in API group "" at the cluster scope
E1107 10:38:16.038282       1 reflector.go:123] k8s.io/client-go/informers/factory.go:134: Failed to list *v1beta1.PodDisruptionBudget: poddisruptionbudgets.policy is forbidden: User "system:kube-scheduler" cannot list resource "poddisruptionbudgets" in API group "policy" at the cluster scope
E1107 10:38:16.041331       1 reflector.go:123] k8s.io/client-go/informers/factory.go:134: Failed to list *v1.PersistentVolumeClaim: persistentvolumeclaims is forbidden: User "system:kube-scheduler" cannot list resource "persistentvolumeclaims" in API group "" at the cluster scope
E1107 10:38:17.029478       1 reflector.go:123] k8s.io/kubernetes/cmd/kube-scheduler/app/server.go:236: Failed to list *v1.Pod: pods is forbidden: User "system:kube-scheduler" cannot list resource "pods" in API group "" at the cluster scope
E1107 10:38:17.033427       1 reflector.go:123] k8s.io/client-go/informers/factory.go:134: Failed to list *v1.Node: nodes is forbidden: User "system:kube-scheduler" cannot list resource "nodes" in API group "" at the cluster scope
E1107 10:38:17.041712       1 reflector.go:123] k8s.io/client-go/informers/factory.go:134: Failed to list *v1.PersistentVolume: persistentvolumes is forbidden: User "system:kube-scheduler" cannot list resource "persistentvolumes" in API group "" at the cluster scope
E1107 10:38:17.043009       1 reflector.go:123] k8s.io/client-go/informers/factory.go:134: Failed to list *v1beta1.CSINode: csinodes.storage.k8s.io is forbidden: User "system:kube-scheduler" cannot list resource "csinodes" in API group "storage.k8s.io" at the cluster scope
E1107 10:38:17.044370       1 reflector.go:123] k8s.io/client-go/informers/factory.go:134: Failed to list *v1.StatefulSet: statefulsets.apps is forbidden: User "system:kube-scheduler" cannot list resource "statefulsets" in API group "apps" at the cluster scope
E1107 10:38:17.045725       1 reflector.go:123] k8s.io/client-go/informers/factory.go:134: Failed to list *v1.ReplicaSet: replicasets.apps is forbidden: User "system:kube-scheduler" cannot list resource "replicasets" in API group "apps" at the cluster scope
E1107 10:38:17.046747       1 reflector.go:123] k8s.io/client-go/informers/factory.go:134: Failed to list *v1.StorageClass: storageclasses.storage.k8s.io is forbidden: User "system:kube-scheduler" cannot list resource "storageclasses" in API group "storage.k8s.io" at the cluster scope
E1107 10:38:17.049324       1 reflector.go:123] k8s.io/client-go/informers/factory.go:134: Failed to list *v1.Service: services is forbidden: User "system:kube-scheduler" cannot list resource "services" in API group "" at the cluster scope
E1107 10:38:17.050477       1 reflector.go:123] k8s.io/client-go/informers/factory.go:134: Failed to list *v1.ReplicationController: replicationcontrollers is forbidden: User "system:kube-scheduler" cannot list resource "replicationcontrollers" in API group "" at the cluster scope
E1107 10:38:17.051969       1 reflector.go:123] k8s.io/client-go/informers/factory.go:134: Failed to list *v1beta1.PodDisruptionBudget: poddisruptionbudgets.policy is forbidden: User "system:kube-scheduler" cannot list resource "poddisruptionbudgets" in API group "policy" at the cluster scope
E1107 10:38:17.055595       1 reflector.go:123] k8s.io/client-go/informers/factory.go:134: Failed to list *v1.PersistentVolumeClaim: persistentvolumeclaims is forbidden: User "system:kube-scheduler" cannot list resource "persistentvolumeclaims" in API group "" at the cluster scope
I1107 10:38:18.073413       1 leaderelection.go:241] attempting to acquire leader lease  kube-system/kube-scheduler...
I1107 10:38:18.079832       1 leaderelection.go:251] successfully acquired lease kube-system/kube-scheduler

==> kubelet <==
-- Logs begin at Thu 2019-11-07 10:37:14 UTC, end at Thu 2019-11-07 10:44:50 UTC. --
Nov 07 10:38:13 minikube kubelet[2895]: E1107 10:38:13.748628    2895 kubelet.go:2267] node "minikube" not found
Nov 07 10:38:13 minikube kubelet[2895]: E1107 10:38:13.848812    2895 kubelet.go:2267] node "minikube" not found
Nov 07 10:38:13 minikube kubelet[2895]: E1107 10:38:13.949068    2895 kubelet.go:2267] node "minikube" not found
Nov 07 10:38:14 minikube kubelet[2895]: E1107 10:38:14.049267    2895 kubelet.go:2267] node "minikube" not found
Nov 07 10:38:14 minikube kubelet[2895]: E1107 10:38:14.149454    2895 kubelet.go:2267] node "minikube" not found
Nov 07 10:38:14 minikube kubelet[2895]: E1107 10:38:14.249618    2895 kubelet.go:2267] node "minikube" not found
Nov 07 10:38:14 minikube kubelet[2895]: E1107 10:38:14.349804    2895 kubelet.go:2267] node "minikube" not found
Nov 07 10:38:14 minikube kubelet[2895]: E1107 10:38:14.449916    2895 kubelet.go:2267] node "minikube" not found
Nov 07 10:38:14 minikube kubelet[2895]: E1107 10:38:14.550217    2895 kubelet.go:2267] node "minikube" not found
Nov 07 10:38:14 minikube kubelet[2895]: E1107 10:38:14.650331    2895 kubelet.go:2267] node "minikube" not found
Nov 07 10:38:14 minikube kubelet[2895]: E1107 10:38:14.750512    2895 kubelet.go:2267] node "minikube" not found
Nov 07 10:38:14 minikube kubelet[2895]: E1107 10:38:14.850694    2895 kubelet.go:2267] node "minikube" not found
Nov 07 10:38:14 minikube kubelet[2895]: E1107 10:38:14.951090    2895 kubelet.go:2267] node "minikube" not found
Nov 07 10:38:15 minikube kubelet[2895]: E1107 10:38:15.051742    2895 kubelet.go:2267] node "minikube" not found
Nov 07 10:38:15 minikube kubelet[2895]: E1107 10:38:15.152067    2895 kubelet.go:2267] node "minikube" not found
Nov 07 10:38:15 minikube kubelet[2895]: E1107 10:38:15.252235    2895 kubelet.go:2267] node "minikube" not found
Nov 07 10:38:15 minikube kubelet[2895]: E1107 10:38:15.352439    2895 kubelet.go:2267] node "minikube" not found
Nov 07 10:38:15 minikube kubelet[2895]: E1107 10:38:15.452685    2895 kubelet.go:2267] node "minikube" not found
Nov 07 10:38:15 minikube kubelet[2895]: E1107 10:38:15.552858    2895 kubelet.go:2267] node "minikube" not found
Nov 07 10:38:15 minikube kubelet[2895]: E1107 10:38:15.653049    2895 kubelet.go:2267] node "minikube" not found
Nov 07 10:38:15 minikube kubelet[2895]: I1107 10:38:15.664674    2895 kubelet_node_status.go:286] Setting node annotation to enable volume controller attach/detach
Nov 07 10:38:15 minikube kubelet[2895]: E1107 10:38:15.753217    2895 kubelet.go:2267] node "minikube" not found
Nov 07 10:38:15 minikube kubelet[2895]: E1107 10:38:15.853474    2895 kubelet.go:2267] node "minikube" not found
Nov 07 10:38:15 minikube kubelet[2895]: E1107 10:38:15.953670    2895 kubelet.go:2267] node "minikube" not found
Nov 07 10:38:15 minikube kubelet[2895]: E1107 10:38:15.974501    2895 controller.go:220] failed to get node "minikube" when trying to set owner ref to the node lease: nodes "minikube" not found
Nov 07 10:38:16 minikube kubelet[2895]: I1107 10:38:16.048818    2895 kubelet_node_status.go:75] Successfully registered node minikube
Nov 07 10:38:16 minikube kubelet[2895]: I1107 10:38:16.053981    2895 reconciler.go:154] Reconciler: start to sync state
Nov 07 10:38:16 minikube kubelet[2895]: E1107 10:38:16.055099    2895 controller.go:135] failed to ensure node lease exists, will retry in 3.2s, error: namespaces "kube-node-lease" not found
Nov 07 10:38:27 minikube kubelet[2895]: E1107 10:38:27.425189    2895 reflector.go:123] object-"kube-system"/"kube-proxy-token-92gsl": Failed to list *v1.Secret: secrets "kube-proxy-token-92gsl" is forbidden: User "system:node:minikube" cannot list resource "secrets" in API group "" in the namespace "kube-system": no relationship found between node "minikube" and this object
Nov 07 10:38:27 minikube kubelet[2895]: E1107 10:38:27.477511    2895 reflector.go:123] object-"kube-system"/"coredns-token-7wbk9": Failed to list *v1.Secret: secrets "coredns-token-7wbk9" is forbidden: User "system:node:minikube" cannot list resource "secrets" in API group "" in the namespace "kube-system": no relationship found between node "minikube" and this object
Nov 07 10:38:27 minikube kubelet[2895]: E1107 10:38:27.477744    2895 reflector.go:123] object-"kube-system"/"coredns": Failed to list *v1.ConfigMap: configmaps "coredns" is forbidden: User "system:node:minikube" cannot list resource "configmaps" in API group "" in the namespace "kube-system": no relationship found between node "minikube" and this object
Nov 07 10:38:27 minikube kubelet[2895]: I1107 10:38:27.510976    2895 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "kube-proxy" (UniqueName: "kubernetes.io/configmap/15f486ba-e908-4cac-b915-44c68b521337-kube-proxy") pod "kube-proxy-tkxr8" (UID: "15f486ba-e908-4cac-b915-44c68b521337")
Nov 07 10:38:27 minikube kubelet[2895]: I1107 10:38:27.511043    2895 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "lib-modules" (UniqueName: "kubernetes.io/host-path/15f486ba-e908-4cac-b915-44c68b521337-lib-modules") pod "kube-proxy-tkxr8" (UID: "15f486ba-e908-4cac-b915-44c68b521337")
Nov 07 10:38:27 minikube kubelet[2895]: I1107 10:38:27.511068    2895 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "xtables-lock" (UniqueName: "kubernetes.io/host-path/15f486ba-e908-4cac-b915-44c68b521337-xtables-lock") pod "kube-proxy-tkxr8" (UID: "15f486ba-e908-4cac-b915-44c68b521337")
Nov 07 10:38:27 minikube kubelet[2895]: I1107 10:38:27.511088    2895 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "kube-proxy-token-92gsl" (UniqueName: "kubernetes.io/secret/15f486ba-e908-4cac-b915-44c68b521337-kube-proxy-token-92gsl") pod "kube-proxy-tkxr8" (UID: "15f486ba-e908-4cac-b915-44c68b521337")
Nov 07 10:38:27 minikube kubelet[2895]: I1107 10:38:27.611351    2895 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "config-volume" (UniqueName: "kubernetes.io/configmap/1f674e2b-a336-4715-8dce-e8dd3901e962-config-volume") pod "coredns-5644d7b6d9-lhtz2" (UID: "1f674e2b-a336-4715-8dce-e8dd3901e962")
Nov 07 10:38:27 minikube kubelet[2895]: I1107 10:38:27.611490    2895 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "coredns-token-7wbk9" (UniqueName: "kubernetes.io/secret/1f674e2b-a336-4715-8dce-e8dd3901e962-coredns-token-7wbk9") pod "coredns-5644d7b6d9-lhtz2" (UID: "1f674e2b-a336-4715-8dce-e8dd3901e962")
Nov 07 10:38:27 minikube kubelet[2895]: I1107 10:38:27.611595    2895 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "coredns-token-7wbk9" (UniqueName: "kubernetes.io/secret/a2054f46-4e00-43e8-ba3c-e2045df6c8ee-coredns-token-7wbk9") pod "coredns-5644d7b6d9-84dsl" (UID: "a2054f46-4e00-43e8-ba3c-e2045df6c8ee")
Nov 07 10:38:27 minikube kubelet[2895]: I1107 10:38:27.611676    2895 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "config-volume" (UniqueName: "kubernetes.io/configmap/a2054f46-4e00-43e8-ba3c-e2045df6c8ee-config-volume") pod "coredns-5644d7b6d9-84dsl" (UID: "a2054f46-4e00-43e8-ba3c-e2045df6c8ee")
Nov 07 10:38:28 minikube kubelet[2895]: I1107 10:38:28.313916    2895 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "tmp" (UniqueName: "kubernetes.io/host-path/491fb0ce-36d7-4969-b185-27f491dbd3a3-tmp") pod "storage-provisioner" (UID: "491fb0ce-36d7-4969-b185-27f491dbd3a3")
Nov 07 10:38:28 minikube kubelet[2895]: I1107 10:38:28.314149    2895 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "storage-provisioner-token-z6qhc" (UniqueName: "kubernetes.io/secret/491fb0ce-36d7-4969-b185-27f491dbd3a3-storage-provisioner-token-z6qhc") pod "storage-provisioner" (UID: "491fb0ce-36d7-4969-b185-27f491dbd3a3")
Nov 07 10:38:28 minikube kubelet[2895]: W1107 10:38:28.892022    2895 pod_container_deletor.go:75] Container "47ccdd6d5a24a88741782e9e589cae004abe9891efa6019a27fbb8b2e363fe48" not found in pod's containers
Nov 07 10:38:29 minikube kubelet[2895]: W1107 10:38:29.272004    2895 docker_sandbox.go:394] failed to read pod IP from plugin/docker: Couldn't find network status for kube-system/coredns-5644d7b6d9-84dsl through plugin: invalid network status for
Nov 07 10:38:29 minikube kubelet[2895]: W1107 10:38:29.272027    2895 docker_sandbox.go:394] failed to read pod IP from plugin/docker: Couldn't find network status for kube-system/coredns-5644d7b6d9-84dsl through plugin: invalid network status for
Nov 07 10:38:29 minikube kubelet[2895]: W1107 10:38:29.273443    2895 pod_container_deletor.go:75] Container "52d0ebf101e3b65c6c84eede1005e7922a3a7b67edf40029c6bc1616fa6f8532" not found in pod's containers
Nov 07 10:38:29 minikube kubelet[2895]: W1107 10:38:29.298772    2895 pod_container_deletor.go:75] Container "57f434d8428170ae867b79885bae87a2d1db1c9e9a39deaaf4aa60287735820c" not found in pod's containers
Nov 07 10:38:29 minikube kubelet[2895]: W1107 10:38:29.315330    2895 docker_sandbox.go:394] failed to read pod IP from plugin/docker: Couldn't find network status for kube-system/coredns-5644d7b6d9-lhtz2 through plugin: invalid network status for
Nov 07 10:38:30 minikube kubelet[2895]: W1107 10:38:30.310859    2895 docker_sandbox.go:394] failed to read pod IP from plugin/docker: Couldn't find network status for kube-system/coredns-5644d7b6d9-84dsl through plugin: invalid network status for
Nov 07 10:38:30 minikube kubelet[2895]: W1107 10:38:30.326794    2895 docker_sandbox.go:394] failed to read pod IP from plugin/docker: Couldn't find network status for kube-system/coredns-5644d7b6d9-lhtz2 through plugin: invalid network status for
Nov 07 10:38:31 minikube kubelet[2895]: I1107 10:38:31.811107    2895 transport.go:132] certificate rotation detected, shutting down client connections to start using new credentials
Nov 07 10:40:28 minikube kubelet[2895]: E1107 10:40:28.344106    2895 reflector.go:123] object-"kube-system"/"default-token-56bfg": Failed to list *v1.Secret: secrets "default-token-56bfg" is forbidden: User "system:node:minikube" cannot list resource "secrets" in API group "" in the namespace "kube-system": no relationship found between node "minikube" and this object
Nov 07 10:40:28 minikube kubelet[2895]: I1107 10:40:28.420743    2895 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "default-token-56bfg" (UniqueName: "kubernetes.io/secret/217a1569-45ab-4981-931e-063c89f1a6ac-default-token-56bfg") pod "metrics-server-587f876775-4d8jq" (UID: "217a1569-45ab-4981-931e-063c89f1a6ac")
Nov 07 10:40:29 minikube kubelet[2895]: W1107 10:40:29.827506    2895 docker_sandbox.go:394] failed to read pod IP from plugin/docker: Couldn't find network status for kube-system/metrics-server-587f876775-4d8jq through plugin: invalid network status for
Nov 07 10:40:30 minikube kubelet[2895]: W1107 10:40:30.151759    2895 docker_sandbox.go:394] failed to read pod IP from plugin/docker: Couldn't find network status for kube-system/metrics-server-587f876775-4d8jq through plugin: invalid network status for
Nov 07 10:40:33 minikube kubelet[2895]: W1107 10:40:33.169549    2895 docker_sandbox.go:394] failed to read pod IP from plugin/docker: Couldn't find network status for kube-system/metrics-server-587f876775-4d8jq through plugin: invalid network status for
Nov 07 10:41:05 minikube kubelet[2895]: I1107 10:41:05.803362    2895 log.go:172] http: superfluous response.WriteHeader call from k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog.(*respLogger).WriteHeader (httplog.go:197)
Nov 07 10:41:11 minikube kubelet[2895]: I1107 10:41:11.529427    2895 log.go:172] http: superfluous response.WriteHeader call from k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog.(*respLogger).WriteHeader (httplog.go:197)
Nov 07 10:41:27 minikube kubelet[2895]: I1107 10:41:27.831700    2895 log.go:172] http: superfluous response.WriteHeader call from k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog.(*respLogger).WriteHeader (httplog.go:197)
Nov 07 10:41:40 minikube kubelet[2895]: I1107 10:41:40.114284    2895 log.go:172] http: superfluous response.WriteHeader call from k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog.(*respLogger).WriteHeader (httplog.go:197)
Nov 07 10:43:39 minikube kubelet[2895]: I1107 10:43:39.507275    2895 log.go:172] http: superfluous response.WriteHeader call from k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog.(*respLogger).WriteHeader (httplog.go:197)

==> storage-provisioner ["fde8c6cad14b"] <==

The operating system version: macOS Catalina 10.15.1

medyagh commented 4 years ago

@micdah do you mind sharing

"minikube ip"

and also first enable the addon and then share output of minikube start --alsologtostderr -v=8 ?

micdah commented 4 years ago

@medyagh of course! These commands have been executed with a clean slate (_entirely deleted ~/.minikube folder before running commands).

Commands issued:

minikube addons enable metrics-server
minikube start --alsologtostderr -v=8

Output: pastebin.com/mcuXbw85 (too much to put in comment)

Output of minikube ip:

192.168.64.48

Output of kubectl logs -n kube-system metrics-server-587f876775-fpf59:

I1108 10:36:43.006107       1 heapster.go:71] /metrics-server --source=kubernetes.summary_api:''
I1108 10:36:43.006187       1 heapster.go:72] Metrics Server version v0.2.1
I1108 10:36:43.006572       1 configs.go:61] Using Kubernetes client with master "https://10.96.0.1:443" and version
I1108 10:36:43.006678       1 configs.go:62] Using kubelet port 10255
I1108 10:36:43.007640       1 heapster.go:128] Starting with Metric Sink
I1108 10:36:43.189888       1 serving.go:308] Generated self-signed cert (apiserver.local.config/certificates/apiserver.crt, apiserver.local.config/certificates/apiserver.key)
I1108 10:36:43.550421       1 heapster.go:101] Starting Heapster API server...
[restful] 2019/11/08 10:36:43 log.go:33: [restful/swagger] listing is available at https:///swaggerapi
[restful] 2019/11/08 10:36:43 log.go:33: [restful/swagger] https:///swaggerui/ is mapped to folder /swagger-ui/
I1108 10:36:43.551698       1 serve.go:85] Serving securely on 0.0.0.0:443
E1108 10:37:05.007912       1 summary.go:97] error while getting metrics summary from Kubelet minikube(192.168.64.48:10255): Get http://192.168.64.48:10255/stats/summary/: dial tcp 192.168.64.48:10255: getsockopt: connection refused
I1108 10:37:26.117263       1 reststorage.go:140] No metrics for container coredns in pod kube-system/coredns-5644d7b6d9-tbp8k
I1108 10:37:26.118248       1 reststorage.go:93] No metrics for pod kube-system/coredns-5644d7b6d9-tbp8k
I1108 10:37:26.118311       1 reststorage.go:140] No metrics for container metrics-server in pod kube-system/metrics-server-587f876775-fpf59
I1108 10:37:26.118574       1 reststorage.go:93] No metrics for pod kube-system/metrics-server-587f876775-fpf59
I1108 10:37:26.118582       1 reststorage.go:140] No metrics for container storage-provisioner in pod kube-system/storage-provisioner
I1108 10:37:26.118585       1 reststorage.go:93] No metrics for pod kube-system/storage-provisioner
I1108 10:37:26.118589       1 reststorage.go:140] No metrics for container kube-addon-manager in pod kube-system/kube-addon-manager-minikube
I1108 10:37:26.118592       1 reststorage.go:93] No metrics for pod kube-system/kube-addon-manager-minikube
I1108 10:37:26.118596       1 reststorage.go:140] No metrics for container kube-proxy in pod kube-system/kube-proxy-fsdzt
I1108 10:37:26.118835       1 reststorage.go:93] No metrics for pod kube-system/kube-proxy-fsdzt
I1108 10:37:26.118870       1 reststorage.go:140] No metrics for container coredns in pod kube-system/coredns-5644d7b6d9-qn2km
I1108 10:37:26.118875       1 reststorage.go:93] No metrics for pod kube-system/coredns-5644d7b6d9-qn2km
I1108 10:37:41.125699       1 reststorage.go:140] No metrics for container storage-provisioner in pod kube-system/storage-provisioner
I1108 10:37:41.125820       1 reststorage.go:93] No metrics for pod kube-system/storage-provisioner
I1108 10:37:41.125828       1 reststorage.go:140] No metrics for container kube-addon-manager in pod kube-system/kube-addon-manager-minikube
I1108 10:37:41.125831       1 reststorage.go:93] No metrics for pod kube-system/kube-addon-manager-minikube
I1108 10:37:41.125854       1 reststorage.go:140] No metrics for container kube-proxy in pod kube-system/kube-proxy-fsdzt
I1108 10:37:41.125859       1 reststorage.go:93] No metrics for pod kube-system/kube-proxy-fsdzt
I1108 10:37:41.125863       1 reststorage.go:140] No metrics for container coredns in pod kube-system/coredns-5644d7b6d9-qn2km
I1108 10:37:41.125866       1 reststorage.go:93] No metrics for pod kube-system/coredns-5644d7b6d9-qn2km
I1108 10:37:41.125870       1 reststorage.go:140] No metrics for container coredns in pod kube-system/coredns-5644d7b6d9-tbp8k
I1108 10:37:41.125873       1 reststorage.go:93] No metrics for pod kube-system/coredns-5644d7b6d9-tbp8k
I1108 10:37:41.125877       1 reststorage.go:140] No metrics for container metrics-server in pod kube-system/metrics-server-587f876775-fpf59
I1108 10:37:41.125882       1 reststorage.go:93] No metrics for pod kube-system/metrics-server-587f876775-fpf59
I1108 10:37:41.125886       1 reststorage.go:140] No metrics for container kube-scheduler in pod kube-system/kube-scheduler-minikube
I1108 10:37:41.125889       1 reststorage.go:93] No metrics for pod kube-system/kube-scheduler-minikube
I1108 10:37:41.125893       1 reststorage.go:140] No metrics for container kube-apiserver in pod kube-system/kube-apiserver-minikube
I1108 10:37:41.125895       1 reststorage.go:93] No metrics for pod kube-system/kube-apiserver-minikube
I1108 10:37:56.128740       1 reststorage.go:140] No metrics for container kube-scheduler in pod kube-system/kube-scheduler-minikube
I1108 10:37:56.128841       1 reststorage.go:93] No metrics for pod kube-system/kube-scheduler-minikube
I1108 10:37:56.128848       1 reststorage.go:140] No metrics for container kube-apiserver in pod kube-system/kube-apiserver-minikube
I1108 10:37:56.128852       1 reststorage.go:93] No metrics for pod kube-system/kube-apiserver-minikube
I1108 10:37:56.128856       1 reststorage.go:140] No metrics for container etcd in pod kube-system/etcd-minikube
I1108 10:37:56.128859       1 reststorage.go:93] No metrics for pod kube-system/etcd-minikube
I1108 10:37:56.128863       1 reststorage.go:140] No metrics for container storage-provisioner in pod kube-system/storage-provisioner
I1108 10:37:56.128866       1 reststorage.go:93] No metrics for pod kube-system/storage-provisioner
I1108 10:37:56.128871       1 reststorage.go:140] No metrics for container kube-proxy in pod kube-system/kube-proxy-fsdzt
I1108 10:37:56.128874       1 reststorage.go:93] No metrics for pod kube-system/kube-proxy-fsdzt
I1108 10:37:56.128878       1 reststorage.go:140] No metrics for container coredns in pod kube-system/coredns-5644d7b6d9-qn2km
I1108 10:37:56.128881       1 reststorage.go:93] No metrics for pod kube-system/coredns-5644d7b6d9-qn2km
I1108 10:37:56.128885       1 reststorage.go:140] No metrics for container coredns in pod kube-system/coredns-5644d7b6d9-tbp8k
I1108 10:37:56.128888       1 reststorage.go:93] No metrics for pod kube-system/coredns-5644d7b6d9-tbp8k
I1108 10:37:56.128892       1 reststorage.go:140] No metrics for container metrics-server in pod kube-system/metrics-server-587f876775-fpf59
I1108 10:37:56.128895       1 reststorage.go:93] No metrics for pod kube-system/metrics-server-587f876775-fpf59
I1108 10:37:56.128899       1 reststorage.go:140] No metrics for container kube-addon-manager in pod kube-system/kube-addon-manager-minikube
I1108 10:37:56.128902       1 reststorage.go:93] No metrics for pod kube-system/kube-addon-manager-minikube
I1108 10:37:56.128907       1 reststorage.go:140] No metrics for container kube-controller-manager in pod kube-system/kube-controller-manager-minikube
I1108 10:37:56.128956       1 reststorage.go:93] No metrics for pod kube-system/kube-controller-manager-minikube
E1108 10:38:05.001740       1 summary.go:97] error while getting metrics summary from Kubelet minikube(192.168.64.48:10255): Get http://192.168.64.48:10255/stats/summary/: dial tcp 192.168.64.48:10255: getsockopt: connection refused
I1108 10:38:11.134348       1 reststorage.go:140] No metrics for container kube-addon-manager in pod kube-system/kube-addon-manager-minikube
I1108 10:38:11.134363       1 reststorage.go:93] No metrics for pod kube-system/kube-addon-manager-minikube
I1108 10:38:11.134368       1 reststorage.go:140] No metrics for container kube-controller-manager in pod kube-system/kube-controller-manager-minikube
I1108 10:38:11.134372       1 reststorage.go:93] No metrics for pod kube-system/kube-controller-manager-minikube
I1108 10:38:11.134376       1 reststorage.go:140] No metrics for container storage-provisioner in pod kube-system/storage-provisioner
I1108 10:38:11.134378       1 reststorage.go:93] No metrics for pod kube-system/storage-provisioner
I1108 10:38:11.134383       1 reststorage.go:140] No metrics for container kube-proxy in pod kube-system/kube-proxy-fsdzt
I1108 10:38:11.134386       1 reststorage.go:93] No metrics for pod kube-system/kube-proxy-fsdzt
I1108 10:38:11.134390       1 reststorage.go:140] No metrics for container coredns in pod kube-system/coredns-5644d7b6d9-qn2km
I1108 10:38:11.134393       1 reststorage.go:93] No metrics for pod kube-system/coredns-5644d7b6d9-qn2km
I1108 10:38:11.134397       1 reststorage.go:140] No metrics for container coredns in pod kube-system/coredns-5644d7b6d9-tbp8k
I1108 10:38:11.134400       1 reststorage.go:93] No metrics for pod kube-system/coredns-5644d7b6d9-tbp8k
I1108 10:38:11.134404       1 reststorage.go:140] No metrics for container metrics-server in pod kube-system/metrics-server-587f876775-fpf59
I1108 10:38:11.134407       1 reststorage.go:93] No metrics for pod kube-system/metrics-server-587f876775-fpf59
I1108 10:38:11.134411       1 reststorage.go:140] No metrics for container kube-scheduler in pod kube-system/kube-scheduler-minikube
I1108 10:38:11.134414       1 reststorage.go:93] No metrics for pod kube-system/kube-scheduler-minikube
I1108 10:38:11.134418       1 reststorage.go:140] No metrics for container kube-apiserver in pod kube-system/kube-apiserver-minikube
I1108 10:38:11.134421       1 reststorage.go:93] No metrics for pod kube-system/kube-apiserver-minikube
I1108 10:38:11.134424       1 reststorage.go:140] No metrics for container etcd in pod kube-system/etcd-minikube
I1108 10:38:11.134427       1 reststorage.go:93] No metrics for pod kube-system/etcd-minikube
I1108 10:38:26.138300       1 reststorage.go:140] No metrics for container metrics-server in pod kube-system/metrics-server-587f876775-fpf59
I1108 10:38:26.138340       1 reststorage.go:93] No metrics for pod kube-system/metrics-server-587f876775-fpf59
I1108 10:38:26.138347       1 reststorage.go:140] No metrics for container kube-scheduler in pod kube-system/kube-scheduler-minikube
I1108 10:38:26.138350       1 reststorage.go:93] No metrics for pod kube-system/kube-scheduler-minikube
I1108 10:38:26.138355       1 reststorage.go:140] No metrics for container kube-apiserver in pod kube-system/kube-apiserver-minikube
I1108 10:38:26.138357       1 reststorage.go:93] No metrics for pod kube-system/kube-apiserver-minikube
I1108 10:38:26.138361       1 reststorage.go:140] No metrics for container etcd in pod kube-system/etcd-minikube
I1108 10:38:26.138364       1 reststorage.go:93] No metrics for pod kube-system/etcd-minikube
I1108 10:38:26.138368       1 reststorage.go:140] No metrics for container storage-provisioner in pod kube-system/storage-provisioner
I1108 10:38:26.138371       1 reststorage.go:93] No metrics for pod kube-system/storage-provisioner
I1108 10:38:26.138375       1 reststorage.go:140] No metrics for container kube-proxy in pod kube-system/kube-proxy-fsdzt
I1108 10:38:26.138378       1 reststorage.go:93] No metrics for pod kube-system/kube-proxy-fsdzt
I1108 10:38:26.138382       1 reststorage.go:140] No metrics for container coredns in pod kube-system/coredns-5644d7b6d9-qn2km
I1108 10:38:26.138384       1 reststorage.go:93] No metrics for pod kube-system/coredns-5644d7b6d9-qn2km
I1108 10:38:26.138389       1 reststorage.go:140] No metrics for container coredns in pod kube-system/coredns-5644d7b6d9-tbp8k
I1108 10:38:26.138392       1 reststorage.go:93] No metrics for pod kube-system/coredns-5644d7b6d9-tbp8k
I1108 10:38:26.138396       1 reststorage.go:140] No metrics for container kube-addon-manager in pod kube-system/kube-addon-manager-minikube
I1108 10:38:26.138407       1 reststorage.go:93] No metrics for pod kube-system/kube-addon-manager-minikube
I1108 10:38:26.138414       1 reststorage.go:140] No metrics for container kube-controller-manager in pod kube-system/kube-controller-manager-minikube
I1108 10:38:26.138416       1 reststorage.go:93] No metrics for pod kube-system/kube-controller-manager-minikube
I1108 10:38:41.144975       1 reststorage.go:140] No metrics for container kube-proxy in pod kube-system/kube-proxy-fsdzt
I1108 10:38:41.144989       1 reststorage.go:93] No metrics for pod kube-system/kube-proxy-fsdzt
I1108 10:38:41.144995       1 reststorage.go:140] No metrics for container coredns in pod kube-system/coredns-5644d7b6d9-qn2km
I1108 10:38:41.144998       1 reststorage.go:93] No metrics for pod kube-system/coredns-5644d7b6d9-qn2km
I1108 10:38:41.145002       1 reststorage.go:140] No metrics for container coredns in pod kube-system/coredns-5644d7b6d9-tbp8k
I1108 10:38:41.145005       1 reststorage.go:93] No metrics for pod kube-system/coredns-5644d7b6d9-tbp8k
I1108 10:38:41.145010       1 reststorage.go:140] No metrics for container metrics-server in pod kube-system/metrics-server-587f876775-fpf59
I1108 10:38:41.145013       1 reststorage.go:93] No metrics for pod kube-system/metrics-server-587f876775-fpf59
I1108 10:38:41.145017       1 reststorage.go:140] No metrics for container kube-scheduler in pod kube-system/kube-scheduler-minikube
I1108 10:38:41.145020       1 reststorage.go:93] No metrics for pod kube-system/kube-scheduler-minikube
I1108 10:38:41.145024       1 reststorage.go:140] No metrics for container kube-apiserver in pod kube-system/kube-apiserver-minikube
I1108 10:38:41.145027       1 reststorage.go:93] No metrics for pod kube-system/kube-apiserver-minikube
I1108 10:38:41.145032       1 reststorage.go:140] No metrics for container etcd in pod kube-system/etcd-minikube
I1108 10:38:41.145035       1 reststorage.go:93] No metrics for pod kube-system/etcd-minikube
I1108 10:38:41.145039       1 reststorage.go:140] No metrics for container storage-provisioner in pod kube-system/storage-provisioner
I1108 10:38:41.145042       1 reststorage.go:93] No metrics for pod kube-system/storage-provisioner
I1108 10:38:41.145046       1 reststorage.go:140] No metrics for container kube-controller-manager in pod kube-system/kube-controller-manager-minikube
I1108 10:38:41.145049       1 reststorage.go:93] No metrics for pod kube-system/kube-controller-manager-minikube
I1108 10:38:41.145054       1 reststorage.go:140] No metrics for container kube-addon-manager in pod kube-system/kube-addon-manager-minikube
I1108 10:38:41.145057       1 reststorage.go:93] No metrics for pod kube-system/kube-addon-manager-minikube

derailed commented 4 years ago

+1 Same issue here on OSX ;) Any updates?

govargo commented 4 years ago

I reproduce this(but I don't understand this root cause yet).

govargo commented 4 years ago

I digged this issue. The root cause is not getting stats summary via kubelet(port 10255).

E1107 10:41:05.007656       1 summary.go:97] error while getting metrics summary from Kubelet minikube(192.168.64.44:10255): Get http://192.168.64.44:10255/stats/summary/: dial tcp 192.168.64.44:10255: getsockopt: connection refused

kubelet has an option --read-only-port(this is DEPRECATED). --read-only-port opens the port(default: 10255).

But now minikube v1.5.2(kubernetes v1.16.2), the kubelet port(10255) is not opened. So, metrics-server cannot get from kubelet stats summary api.

If metrics-server gets stat summary from reachable api-server, kubectl top returns outputs.

govargo commented 4 years ago

I will send PR to fix this.

/assign