search

kubernetes / minikube

Run Kubernetes locally
https://minikube.sigs.k8s.io/
Apache License 2.0
29.28k stars 4.87k forks source link

localkube-v1.8.0 changed between Nov 21 and today, and breaks minikube #2277

Closed bbakersmith closed 6 years ago

bbakersmith commented 6 years ago

Environment:

Minikube version: v0.23.0

What happened:

I installed Minikube and it worked. 2 weeks later I installed Minikube on a new machine (identical hardware and OS) and the system containers failed due to a TLS handshake error. All the tool versions and cached image names in ~/.minikube/cache are the same between both machines.

What you expected to happen:

I expect Minikube to run on both machines given the same versions of all software and images.

How to reproduce it:

From a fresh install with no ~/.minikube directory today:

minikube start
minikube dashboard

The dashboard never comes up, TLS errors show up in the logs as the system containers rebuild forever.

If I copy an older version of localkube-v1.8.0 (same name and version but from November 21) into ~/.minikube/cache/localkube and

minikube delete
minikube start
minikube dashboard

Everything works.

So, something changed in localkube-v1.8.0 between November 21 and today that breaks minikube. Diffing the older and newer binaries confirms that they did change. This should not happen, versioned binaries should be immutable.

This behavior has been confirmed on multiple laptops.

Output of minikube logs (if applicable):

-- Logs begin at Tue 2017-12-05 16:27:56 UTC, end at Tue 2017-12-05 16:40:20 UTC. --
Dec 05 16:28:14 minikube systemd[1]: Starting Localkube...
Dec 05 16:28:14 minikube localkube[3327]: listening for peers on http://localhost:2380
Dec 05 16:28:14 minikube localkube[3327]: listening for client requests on localhost:2379
Dec 05 16:28:14 minikube localkube[3327]: name = default
Dec 05 16:28:14 minikube localkube[3327]: data dir = /var/lib/localkube/etcd
Dec 05 16:28:14 minikube localkube[3327]: member dir = /var/lib/localkube/etcd/member
Dec 05 16:28:14 minikube localkube[3327]: heartbeat = 100ms
Dec 05 16:28:14 minikube localkube[3327]: election = 1000ms
Dec 05 16:28:14 minikube localkube[3327]: snapshot count = 10000
Dec 05 16:28:14 minikube localkube[3327]: advertise client URLs = http://localhost:2379
Dec 05 16:28:14 minikube localkube[3327]: initial advertise peer URLs = http://localhost:2380
Dec 05 16:28:14 minikube localkube[3327]: initial cluster = default=http://localhost:2380
Dec 05 16:28:14 minikube localkube[3327]: starting member 8e9e05c52164694d in cluster cdf818194e3a8c32
Dec 05 16:28:14 minikube localkube[3327]: 8e9e05c52164694d became follower at term 0
Dec 05 16:28:14 minikube localkube[3327]: newRaft 8e9e05c52164694d [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]
Dec 05 16:28:14 minikube localkube[3327]: 8e9e05c52164694d became follower at term 1
Dec 05 16:28:14 minikube localkube[3327]: starting server... [version: 3.1.10, cluster version: to_be_decided]
Dec 05 16:28:14 minikube localkube[3327]: added member 8e9e05c52164694d [http://localhost:2380] to cluster cdf818194e3a8c32
Dec 05 16:28:15 minikube localkube[3327]: 8e9e05c52164694d is starting a new election at term 1
Dec 05 16:28:15 minikube localkube[3327]: 8e9e05c52164694d became candidate at term 2
Dec 05 16:28:15 minikube localkube[3327]: 8e9e05c52164694d received MsgVoteResp from 8e9e05c52164694d at term 2
Dec 05 16:28:15 minikube localkube[3327]: 8e9e05c52164694d became leader at term 2
Dec 05 16:28:15 minikube localkube[3327]: raft.node: 8e9e05c52164694d elected leader 8e9e05c52164694d at term 2
Dec 05 16:28:15 minikube localkube[3327]: setting up the initial cluster version to 3.1
Dec 05 16:28:15 minikube localkube[3327]: set the initial cluster version to 3.1
Dec 05 16:28:15 minikube localkube[3327]: I1205 16:28:15.429580    3327 etcd.go:58] Etcd server is ready
Dec 05 16:28:15 minikube localkube[3327]: localkube host ip address: 10.0.2.15
Dec 05 16:28:15 minikube localkube[3327]: Starting apiserver...
Dec 05 16:28:15 minikube localkube[3327]: Waiting for apiserver to be healthy...
Dec 05 16:28:15 minikube localkube[3327]: I1205 16:28:15.430107    3327 server.go:114] Version: v1.8.0
Dec 05 16:28:15 minikube localkube[3327]: W1205 16:28:15.430307    3327 authentication.go:380] AnonymousAuth is not allowed with the AllowAll authorizer.  Resetting AnonymousAuth to false. You should use a different authorizer
Dec 05 16:28:15 minikube localkube[3327]: enabled capabilities for version 3.1
Dec 05 16:28:15 minikube localkube[3327]: published {Name:default ClientURLs:[http://localhost:2379]} to cluster cdf818194e3a8c32
Dec 05 16:28:15 minikube localkube[3327]: ready to serve client requests
Dec 05 16:28:15 minikube localkube[3327]: serving insecure client requests on 127.0.0.1:2379, this is strongly discouraged!
Dec 05 16:28:15 minikube localkube[3327]: I1205 16:28:15.432183    3327 plugins.go:101] No cloud provider specified.
Dec 05 16:28:16 minikube localkube[3327]: [restful] 2017/12/05 16:28:16 log.go:33: [restful/swagger] listing is available at https://10.0.2.15:8443/swaggerapi
Dec 05 16:28:16 minikube localkube[3327]: [restful] 2017/12/05 16:28:16 log.go:33: [restful/swagger] https://10.0.2.15:8443/swaggerui/ is mapped to folder /swagger-ui/
Dec 05 16:28:16 minikube localkube[3327]: I1205 16:28:16.430096    3327 ready.go:30] Performing healthcheck on https://localhost:8443/healthz
Dec 05 16:28:16 minikube localkube[3327]: E1205 16:28:16.431452    3327 ready.go:40] Error performing healthcheck: Get https://localhost:8443/healthz: dial tcp 127.0.0.1:8443: getsockopt: connection refused
Dec 05 16:28:17 minikube localkube[3327]: [restful] 2017/12/05 16:28:17 log.go:33: [restful/swagger] listing is available at https://10.0.2.15:8443/swaggerapi
Dec 05 16:28:17 minikube localkube[3327]: [restful] 2017/12/05 16:28:17 log.go:33: [restful/swagger] https://10.0.2.15:8443/swaggerui/ is mapped to folder /swagger-ui/
Dec 05 16:28:17 minikube localkube[3327]: I1205 16:28:17.431464    3327 ready.go:30] Performing healthcheck on https://localhost:8443/healthz
Dec 05 16:28:17 minikube localkube[3327]: E1205 16:28:17.432375    3327 ready.go:40] Error performing healthcheck: Get https://localhost:8443/healthz: dial tcp 127.0.0.1:8443: getsockopt: connection refused
Dec 05 16:28:18 minikube localkube[3327]: I1205 16:28:18.430164    3327 ready.go:30] Performing healthcheck on https://localhost:8443/healthz
Dec 05 16:28:18 minikube localkube[3327]: E1205 16:28:18.431164    3327 ready.go:40] Error performing healthcheck: Get https://localhost:8443/healthz: dial tcp 127.0.0.1:8443: getsockopt: connection refused
Dec 05 16:28:19 minikube localkube[3327]: I1205 16:28:19.430714    3327 ready.go:30] Performing healthcheck on https://localhost:8443/healthz
Dec 05 16:28:19 minikube localkube[3327]: E1205 16:28:19.431711    3327 ready.go:40] Error performing healthcheck: Get https://localhost:8443/healthz: dial tcp 127.0.0.1:8443: getsockopt: connection refused
Dec 05 16:28:20 minikube localkube[3327]: I1205 16:28:20.430236    3327 ready.go:30] Performing healthcheck on https://localhost:8443/healthz
Dec 05 16:28:20 minikube localkube[3327]: E1205 16:28:20.431529    3327 ready.go:40] Error performing healthcheck: Get https://localhost:8443/healthz: dial tcp 127.0.0.1:8443: getsockopt: connection refused
Dec 05 16:28:21 minikube localkube[3327]: I1205 16:28:21.165648    3327 aggregator.go:138] Skipping APIService creation for scheduling.k8s.io/v1alpha1
Dec 05 16:28:21 minikube localkube[3327]: I1205 16:28:21.166474    3327 serve.go:85] Serving securely on 0.0.0.0:8443
Dec 05 16:28:21 minikube systemd[1]: Started Localkube.
Dec 05 16:28:21 minikube localkube[3327]: I1205 16:28:21.169876    3327 apiservice_controller.go:112] Starting APIServiceRegistrationController
Dec 05 16:28:21 minikube localkube[3327]: I1205 16:28:21.170191    3327 cache.go:32] Waiting for caches to sync for APIServiceRegistrationController controller
Dec 05 16:28:21 minikube localkube[3327]: I1205 16:28:21.170435    3327 available_controller.go:192] Starting AvailableConditionController
Dec 05 16:28:21 minikube localkube[3327]: I1205 16:28:21.170596    3327 cache.go:32] Waiting for caches to sync for AvailableConditionController controller
Dec 05 16:28:21 minikube localkube[3327]: I1205 16:28:21.170762    3327 controller.go:84] Starting OpenAPI AggregationController
Dec 05 16:28:21 minikube localkube[3327]: I1205 16:28:21.170921    3327 crdregistration_controller.go:112] Starting crd-autoregister controller
Dec 05 16:28:21 minikube localkube[3327]: I1205 16:28:21.171032    3327 controller_utils.go:1041] Waiting for caches to sync for crd-autoregister controller
Dec 05 16:28:21 minikube localkube[3327]: I1205 16:28:21.179317    3327 crd_finalizer.go:242] Starting CRDFinalizer
Dec 05 16:28:21 minikube localkube[3327]: I1205 16:28:21.185023    3327 customresource_discovery_controller.go:152] Starting DiscoveryController
Dec 05 16:28:21 minikube localkube[3327]: I1205 16:28:21.185112    3327 naming_controller.go:277] Starting NamingConditionController
Dec 05 16:28:21 minikube localkube[3327]: I1205 16:28:21.273785    3327 controller_utils.go:1048] Caches are synced for crd-autoregister controller
Dec 05 16:28:21 minikube localkube[3327]: I1205 16:28:21.273852    3327 cache.go:39] Caches are synced for APIServiceRegistrationController controller
Dec 05 16:28:21 minikube localkube[3327]: I1205 16:28:21.273873    3327 cache.go:39] Caches are synced for AvailableConditionController controller
Dec 05 16:28:21 minikube localkube[3327]: I1205 16:28:21.273898    3327 autoregister_controller.go:136] Starting autoregister controller
Dec 05 16:28:21 minikube localkube[3327]: I1205 16:28:21.273906    3327 cache.go:32] Waiting for caches to sync for autoregister controller
Dec 05 16:28:21 minikube localkube[3327]: I1205 16:28:21.374080    3327 cache.go:39] Caches are synced for autoregister controller
Dec 05 16:28:21 minikube localkube[3327]: I1205 16:28:21.432111    3327 ready.go:30] Performing healthcheck on https://localhost:8443/healthz
Dec 05 16:28:21 minikube localkube[3327]: I1205 16:28:21.450616    3327 ready.go:49] Got healthcheck response: [+]ping ok
Dec 05 16:28:21 minikube localkube[3327]: [+]etcd ok
Dec 05 16:28:21 minikube localkube[3327]: [+]poststarthook/generic-apiserver-start-informers ok
Dec 05 16:28:21 minikube localkube[3327]: [+]poststarthook/start-apiextensions-informers ok
Dec 05 16:28:21 minikube localkube[3327]: [+]poststarthook/start-apiextensions-controllers ok
Dec 05 16:28:21 minikube localkube[3327]: [+]poststarthook/bootstrap-controller ok
Dec 05 16:28:21 minikube localkube[3327]: [-]poststarthook/ca-registration failed: reason withheld
Dec 05 16:28:21 minikube localkube[3327]: [+]poststarthook/start-kube-apiserver-informers ok
Dec 05 16:28:21 minikube localkube[3327]: [+]poststarthook/start-kube-aggregator-informers ok
Dec 05 16:28:21 minikube localkube[3327]: [+]poststarthook/apiservice-registration-controller ok
Dec 05 16:28:21 minikube localkube[3327]: [+]poststarthook/apiservice-status-available-controller ok
Dec 05 16:28:21 minikube localkube[3327]: [+]poststarthook/apiservice-openapi-controller ok
Dec 05 16:28:21 minikube localkube[3327]: [+]poststarthook/kube-apiserver-autoregistration ok
Dec 05 16:28:21 minikube localkube[3327]: [-]autoregister-completion failed: reason withheld
Dec 05 16:28:21 minikube localkube[3327]: healthz check failed
Dec 05 16:28:22 minikube localkube[3327]: I1205 16:28:22.378177    3327 trace.go:76] Trace[1626706478]: "Create /apis/apiregistration.k8s.io/v1beta1/apiservices" (started: 2017-12-05 16:28:21.375173214 +0000 UTC m=+7.256289351) (total time: 1.00296862s):
Dec 05 16:28:22 minikube localkube[3327]: Trace[1626706478]: [1.00285974s] [1.002789605s] Object stored in database
Dec 05 16:28:22 minikube localkube[3327]: I1205 16:28:22.430497    3327 ready.go:30] Performing healthcheck on https://localhost:8443/healthz
Dec 05 16:28:22 minikube localkube[3327]: I1205 16:28:22.440656    3327 ready.go:49] Got healthcheck response: ok
Dec 05 16:28:22 minikube localkube[3327]: apiserver is ready!
Dec 05 16:28:22 minikube localkube[3327]: Starting controller-manager...
Dec 05 16:28:22 minikube localkube[3327]: Waiting for controller-manager to be healthy...
Dec 05 16:28:22 minikube localkube[3327]: I1205 16:28:22.441655    3327 controllermanager.go:109] Version: v1.8.0
Dec 05 16:28:22 minikube localkube[3327]: I1205 16:28:22.447144    3327 leaderelection.go:174] attempting to acquire leader lease...
Dec 05 16:28:22 minikube localkube[3327]: I1205 16:28:22.460075    3327 leaderelection.go:184] successfully acquired lease kube-system/kube-controller-manager
Dec 05 16:28:22 minikube localkube[3327]: I1205 16:28:22.465278    3327 event.go:218] Event(v1.ObjectReference{Kind:"Endpoints", Namespace:"kube-system", Name:"kube-controller-manager", UID:"4f97f05d-d9d9-11e7-8bd9-0800270d97b1", APIVersion:"v1", ResourceVersion:"35", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' minikube became leader
Dec 05 16:28:22 minikube localkube[3327]: I1205 16:28:22.487707    3327 plugins.go:101] No cloud provider specified.
Dec 05 16:28:22 minikube localkube[3327]: I1205 16:28:22.490197    3327 controllermanager.go:487] Started "serviceaccount"
Dec 05 16:28:22 minikube localkube[3327]: I1205 16:28:22.493381    3327 controllermanager.go:487] Started "replicaset"
Dec 05 16:28:22 minikube localkube[3327]: I1205 16:28:22.494811    3327 controllermanager.go:487] Started "statefulset"
Dec 05 16:28:22 minikube localkube[3327]: I1205 16:28:22.495329    3327 stateful_set.go:146] Starting stateful set controller
Dec 05 16:28:22 minikube localkube[3327]: I1205 16:28:22.496502    3327 controller_utils.go:1041] Waiting for caches to sync for stateful set controller
Dec 05 16:28:22 minikube localkube[3327]: I1205 16:28:22.493588    3327 controller_utils.go:1041] Waiting for caches to sync for tokens controller
Dec 05 16:28:22 minikube localkube[3327]: I1205 16:28:22.493662    3327 serviceaccounts_controller.go:113] Starting service account controller
Dec 05 16:28:22 minikube localkube[3327]: I1205 16:28:22.496523    3327 controller_utils.go:1041] Waiting for caches to sync for service account controller
Dec 05 16:28:22 minikube localkube[3327]: I1205 16:28:22.493542    3327 replica_set.go:156] Starting replica set controller
Dec 05 16:28:22 minikube localkube[3327]: I1205 16:28:22.496531    3327 controller_utils.go:1041] Waiting for caches to sync for replica set controller
Dec 05 16:28:22 minikube localkube[3327]: I1205 16:28:22.499255    3327 controllermanager.go:487] Started "cronjob"
Dec 05 16:28:22 minikube localkube[3327]: I1205 16:28:22.499490    3327 cronjob_controller.go:98] Starting CronJob Manager
Dec 05 16:28:22 minikube localkube[3327]: I1205 16:28:22.501569    3327 controllermanager.go:487] Started "ttl"
Dec 05 16:28:22 minikube localkube[3327]: E1205 16:28:22.502501    3327 core.go:70] Failed to start service controller: WARNING: no cloud provider provided, services of type LoadBalancer will fail.
Dec 05 16:28:22 minikube localkube[3327]: W1205 16:28:22.502688    3327 controllermanager.go:484] Skipping "service"
Dec 05 16:28:22 minikube localkube[3327]: W1205 16:28:22.502817    3327 core.go:128] Unsuccessful parsing of cluster CIDR : invalid CIDR address:
Dec 05 16:28:22 minikube localkube[3327]: I1205 16:28:22.503152    3327 core.go:131] Will not configure cloud provider routes for allocate-node-cidrs: false, configure-cloud-routes: true.
Dec 05 16:28:22 minikube localkube[3327]: W1205 16:28:22.503318    3327 controllermanager.go:484] Skipping "route"
Dec 05 16:28:22 minikube localkube[3327]: I1205 16:28:22.502581    3327 ttl_controller.go:116] Starting TTL controller
Dec 05 16:28:22 minikube localkube[3327]: I1205 16:28:22.503621    3327 controller_utils.go:1041] Waiting for caches to sync for TTL controller
Dec 05 16:28:22 minikube localkube[3327]: I1205 16:28:22.505855    3327 controllermanager.go:487] Started "persistentvolume-binder"
Dec 05 16:28:22 minikube localkube[3327]: I1205 16:28:22.506308    3327 pv_controller_base.go:259] Starting persistent volume controller
Dec 05 16:28:22 minikube localkube[3327]: I1205 16:28:22.506499    3327 controller_utils.go:1041] Waiting for caches to sync for persistent volume controller
Dec 05 16:28:22 minikube localkube[3327]: I1205 16:28:22.508398    3327 controllermanager.go:487] Started "endpoint"
Dec 05 16:28:22 minikube localkube[3327]: I1205 16:28:22.508585    3327 endpoints_controller.go:153] Starting endpoint controller
Dec 05 16:28:22 minikube localkube[3327]: I1205 16:28:22.508915    3327 controller_utils.go:1041] Waiting for caches to sync for endpoint controller
Dec 05 16:28:22 minikube localkube[3327]: W1205 16:28:22.510760    3327 shared_informer.go:304] resyncPeriod 65089258457378 is smaller than resyncCheckPeriod 71316131012880 and the informer has already started. Changing it to 71316131012880
Dec 05 16:28:22 minikube localkube[3327]: I1205 16:28:22.511304    3327 controllermanager.go:487] Started "resourcequota"
Dec 05 16:28:22 minikube localkube[3327]: I1205 16:28:22.512012    3327 resource_quota_controller.go:238] Starting resource quota controller
Dec 05 16:28:22 minikube localkube[3327]: I1205 16:28:22.512312    3327 controller_utils.go:1041] Waiting for caches to sync for resource quota controller
Dec 05 16:28:22 minikube localkube[3327]: I1205 16:28:22.535878    3327 controllermanager.go:487] Started "namespace"
Dec 05 16:28:22 minikube localkube[3327]: I1205 16:28:22.536520    3327 namespace_controller.go:186] Starting namespace controller
Dec 05 16:28:22 minikube localkube[3327]: I1205 16:28:22.536696    3327 controller_utils.go:1041] Waiting for caches to sync for namespace controller
Dec 05 16:28:22 minikube localkube[3327]: I1205 16:28:22.536814    3327 controllermanager.go:487] Started "deployment"
Dec 05 16:28:22 minikube localkube[3327]: I1205 16:28:22.540743    3327 controllermanager.go:487] Started "horizontalpodautoscaling"
Dec 05 16:28:22 minikube localkube[3327]: E1205 16:28:22.541881    3327 certificates.go:48] Failed to start certificate controller: error reading CA cert file "/etc/kubernetes/ca/ca.pem": open /etc/kubernetes/ca/ca.pem: no such file or directory
Dec 05 16:28:22 minikube localkube[3327]: W1205 16:28:22.542046    3327 controllermanager.go:484] Skipping "csrsigning"
Dec 05 16:28:22 minikube localkube[3327]: I1205 16:28:22.543748    3327 controllermanager.go:487] Started "csrapproving"
Dec 05 16:28:22 minikube localkube[3327]: W1205 16:28:22.543992    3327 controllermanager.go:471] "tokencleaner" is disabled
Dec 05 16:28:22 minikube localkube[3327]: I1205 16:28:22.543930    3327 horizontal.go:145] Starting HPA controller
Dec 05 16:28:22 minikube localkube[3327]: I1205 16:28:22.544448    3327 controller_utils.go:1041] Waiting for caches to sync for HPA controller
Dec 05 16:28:22 minikube localkube[3327]: I1205 16:28:22.544025    3327 certificate_controller.go:109] Starting certificate controller
Dec 05 16:28:22 minikube localkube[3327]: I1205 16:28:22.544484    3327 controller_utils.go:1041] Waiting for caches to sync for certificate controller
Dec 05 16:28:22 minikube localkube[3327]: I1205 16:28:22.536931    3327 deployment_controller.go:151] Starting deployment controller
Dec 05 16:28:22 minikube localkube[3327]: I1205 16:28:22.544500    3327 controller_utils.go:1041] Waiting for caches to sync for deployment controller
Dec 05 16:28:22 minikube localkube[3327]: I1205 16:28:22.546979    3327 node_controller.go:249] Sending events to api server.
Dec 05 16:28:22 minikube localkube[3327]: I1205 16:28:22.548213    3327 taint_controller.go:158] Sending events to api server.
Dec 05 16:28:22 minikube localkube[3327]: I1205 16:28:22.548438    3327 controllermanager.go:487] Started "node"
Dec 05 16:28:22 minikube localkube[3327]: I1205 16:28:22.548525    3327 node_controller.go:516] Starting node controller
Dec 05 16:28:22 minikube localkube[3327]: I1205 16:28:22.548717    3327 controller_utils.go:1041] Waiting for caches to sync for node controller
Dec 05 16:28:22 minikube localkube[3327]: W1205 16:28:22.549644    3327 probe.go:215] Flexvolume plugin directory at /usr/libexec/kubernetes/kubelet-plugins/volume/exec/ does not exist. Recreating.
Dec 05 16:28:22 minikube localkube[3327]: I1205 16:28:22.549987    3327 controllermanager.go:487] Started "attachdetach"
Dec 05 16:28:22 minikube localkube[3327]: I1205 16:28:22.551559    3327 controllermanager.go:487] Started "replicationcontroller"
Dec 05 16:28:22 minikube localkube[3327]: I1205 16:28:22.552575    3327 replication_controller.go:151] Starting RC controller
Dec 05 16:28:22 minikube localkube[3327]: I1205 16:28:22.552747    3327 controller_utils.go:1041] Waiting for caches to sync for RC controller
Dec 05 16:28:22 minikube localkube[3327]: I1205 16:28:22.552982    3327 attach_detach_controller.go:255] Starting attach detach controller
Dec 05 16:28:22 minikube localkube[3327]: I1205 16:28:22.553077    3327 controller_utils.go:1041] Waiting for caches to sync for attach detach controller
Dec 05 16:28:22 minikube localkube[3327]: I1205 16:28:22.597895    3327 controller_utils.go:1048] Caches are synced for tokens controller
Dec 05 16:28:23 minikube localkube[3327]: controller-manager is ready!
Dec 05 16:28:23 minikube localkube[3327]: Starting scheduler...
Dec 05 16:28:23 minikube localkube[3327]: Waiting for scheduler to be healthy...
Dec 05 16:28:23 minikube localkube[3327]: E1205 16:28:23.445118    3327 server.go:173] unable to register configz: register config "componentconfig" twice
Dec 05 16:28:23 minikube localkube[3327]: I1205 16:28:23.754752    3327 controllermanager.go:487] Started "garbagecollector"
Dec 05 16:28:23 minikube localkube[3327]: I1205 16:28:23.755248    3327 garbagecollector.go:136] Starting garbage collector controller
Dec 05 16:28:23 minikube localkube[3327]: I1205 16:28:23.757097    3327 controller_utils.go:1041] Waiting for caches to sync for garbage collector controller
Dec 05 16:28:23 minikube localkube[3327]: I1205 16:28:23.757528    3327 graph_builder.go:321] GraphBuilder running
Dec 05 16:28:23 minikube localkube[3327]: I1205 16:28:23.758155    3327 controllermanager.go:487] Started "daemonset"
Dec 05 16:28:23 minikube localkube[3327]: I1205 16:28:23.758318    3327 daemon_controller.go:230] Starting daemon sets controller
Dec 05 16:28:23 minikube localkube[3327]: I1205 16:28:23.758532    3327 controller_utils.go:1041] Waiting for caches to sync for daemon sets controller
Dec 05 16:28:23 minikube localkube[3327]: I1205 16:28:23.760465    3327 controllermanager.go:487] Started "disruption"
Dec 05 16:28:23 minikube localkube[3327]: W1205 16:28:23.760734    3327 controllermanager.go:471] "bootstrapsigner" is disabled
Dec 05 16:28:23 minikube localkube[3327]: I1205 16:28:23.760512    3327 disruption.go:288] Starting disruption controller
Dec 05 16:28:23 minikube localkube[3327]: I1205 16:28:23.761089    3327 controller_utils.go:1041] Waiting for caches to sync for disruption controller
Dec 05 16:28:23 minikube localkube[3327]: I1205 16:28:23.763143    3327 controllermanager.go:487] Started "podgc"
Dec 05 16:28:23 minikube localkube[3327]: I1205 16:28:23.763222    3327 gc_controller.go:76] Starting GC controller
Dec 05 16:28:23 minikube localkube[3327]: I1205 16:28:23.763438    3327 controller_utils.go:1041] Waiting for caches to sync for GC controller
Dec 05 16:28:23 minikube localkube[3327]: I1205 16:28:23.764894    3327 controllermanager.go:487] Started "job"
Dec 05 16:28:23 minikube localkube[3327]: W1205 16:28:23.765141    3327 controllermanager.go:484] Skipping "persistentvolume-expander"
Dec 05 16:28:23 minikube localkube[3327]: I1205 16:28:23.764975    3327 job_controller.go:138] Starting job controller
Dec 05 16:28:23 minikube localkube[3327]: I1205 16:28:23.765645    3327 controller_utils.go:1041] Waiting for caches to sync for job controller
Dec 05 16:28:23 minikube localkube[3327]: I1205 16:28:23.796798    3327 controller_utils.go:1048] Caches are synced for replica set controller
Dec 05 16:28:23 minikube localkube[3327]: I1205 16:28:23.796870    3327 controller_utils.go:1048] Caches are synced for stateful set controller
Dec 05 16:28:23 minikube localkube[3327]: I1205 16:28:23.796898    3327 controller_utils.go:1048] Caches are synced for service account controller
Dec 05 16:28:23 minikube localkube[3327]: I1205 16:28:23.804536    3327 controller_utils.go:1048] Caches are synced for TTL controller
Dec 05 16:28:23 minikube localkube[3327]: I1205 16:28:23.812180    3327 controller_utils.go:1048] Caches are synced for persistent volume controller
Dec 05 16:28:23 minikube localkube[3327]: I1205 16:28:23.812715    3327 controller_utils.go:1048] Caches are synced for endpoint controller
Dec 05 16:28:23 minikube localkube[3327]: I1205 16:28:23.816655    3327 controller_utils.go:1048] Caches are synced for resource quota controller
Dec 05 16:28:23 minikube localkube[3327]: I1205 16:28:23.837691    3327 controller_utils.go:1048] Caches are synced for namespace controller
Dec 05 16:28:23 minikube localkube[3327]: I1205 16:28:23.844773    3327 controller_utils.go:1048] Caches are synced for deployment controller
Dec 05 16:28:23 minikube localkube[3327]: I1205 16:28:23.844865    3327 controller_utils.go:1048] Caches are synced for HPA controller
Dec 05 16:28:23 minikube localkube[3327]: I1205 16:28:23.844882    3327 controller_utils.go:1048] Caches are synced for certificate controller
Dec 05 16:28:23 minikube localkube[3327]: I1205 16:28:23.849157    3327 controller_utils.go:1048] Caches are synced for node controller
Dec 05 16:28:23 minikube localkube[3327]: I1205 16:28:23.849252    3327 taint_controller.go:181] Starting NoExecuteTaintManager
Dec 05 16:28:23 minikube localkube[3327]: I1205 16:28:23.853289    3327 controller_utils.go:1048] Caches are synced for attach detach controller
Dec 05 16:28:23 minikube localkube[3327]: I1205 16:28:23.853289    3327 controller_utils.go:1048] Caches are synced for RC controller
Dec 05 16:28:23 minikube localkube[3327]: I1205 16:28:23.858769    3327 controller_utils.go:1048] Caches are synced for daemon sets controller
Dec 05 16:28:23 minikube localkube[3327]: I1205 16:28:23.863670    3327 controller_utils.go:1048] Caches are synced for GC controller
Dec 05 16:28:23 minikube localkube[3327]: I1205 16:28:23.867452    3327 controller_utils.go:1048] Caches are synced for job controller
Dec 05 16:28:23 minikube localkube[3327]: I1205 16:28:23.961589    3327 controller_utils.go:1048] Caches are synced for disruption controller
Dec 05 16:28:23 minikube localkube[3327]: I1205 16:28:23.961820    3327 disruption.go:296] Sending events to api server.
Dec 05 16:28:24 minikube localkube[3327]: I1205 16:28:24.146925    3327 controller_utils.go:1041] Waiting for caches to sync for scheduler controller
Dec 05 16:28:24 minikube localkube[3327]: I1205 16:28:24.157957    3327 controller_utils.go:1048] Caches are synced for garbage collector controller
Dec 05 16:28:24 minikube localkube[3327]: I1205 16:28:24.157985    3327 garbagecollector.go:145] Garbage collector: all resource monitors have synced. Proceeding to collect garbage
Dec 05 16:28:24 minikube localkube[3327]: I1205 16:28:24.247069    3327 controller_utils.go:1048] Caches are synced for scheduler controller
Dec 05 16:28:24 minikube localkube[3327]: I1205 16:28:24.247299    3327 leaderelection.go:174] attempting to acquire leader lease...
Dec 05 16:28:24 minikube localkube[3327]: I1205 16:28:24.253170    3327 leaderelection.go:184] successfully acquired lease kube-system/kube-scheduler
Dec 05 16:28:24 minikube localkube[3327]: I1205 16:28:24.253451    3327 event.go:218] Event(v1.ObjectReference{Kind:"Endpoints", Namespace:"kube-system", Name:"kube-scheduler", UID:"50a9a9de-d9d9-11e7-8bd9-0800270d97b1", APIVersion:"v1", ResourceVersion:"46", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' minikube became leader
Dec 05 16:28:24 minikube localkube[3327]: scheduler is ready!
Dec 05 16:28:24 minikube localkube[3327]: Starting kubelet...
Dec 05 16:28:24 minikube localkube[3327]: Waiting for kubelet to be healthy...
Dec 05 16:28:24 minikube localkube[3327]: I1205 16:28:24.443587    3327 feature_gate.go:156] feature gates: map[]
Dec 05 16:28:24 minikube localkube[3327]: W1205 16:28:24.443717    3327 server.go:276] --require-kubeconfig is deprecated. Set --kubeconfig without using --require-kubeconfig.
Dec 05 16:28:24 minikube localkube[3327]: I1205 16:28:24.873687    3327 client.go:75] Connecting to docker on unix:///var/run/docker.sock
Dec 05 16:28:24 minikube localkube[3327]: I1205 16:28:24.873923    3327 client.go:95] Start docker client with request timeout=2m0s
Dec 05 16:28:24 minikube localkube[3327]: W1205 16:28:24.879435    3327 server.go:289] --cloud-provider=auto-detect is deprecated. The desired cloud provider should be set explicitly
Dec 05 16:28:24 minikube localkube[3327]: I1205 16:28:24.921527    3327 manager.go:149] cAdvisor running in container: "/sys/fs/cgroup/cpu,cpuacct/system.slice/localkube.service"
Dec 05 16:28:24 minikube localkube[3327]: I1205 16:28:24.968806    3327 fs.go:139] Filesystem UUIDs: map[2017-10-19-17-24-41-00:/dev/sr0 2b17b85d-fc02-47fa-ac9f-46373c49d907:/dev/sda2 f4fff947-dbac-42e7-8b18-5fcdddcfc777:/dev/sda1]
Dec 05 16:28:24 minikube localkube[3327]: I1205 16:28:24.968837    3327 fs.go:140] Filesystem partitions: map[tmpfs:{mountpoint:/dev/shm major:0 minor:17 fsType:tmpfs blockSize:0} /dev/sda1:{mountpoint:/mnt/sda1 major:8 minor:1 fsType:ext4 blockSize:0}]
Dec 05 16:28:24 minikube localkube[3327]: I1205 16:28:24.969881    3327 manager.go:216] Machine: {NumCores:2 CpuFrequency:2495998 MemoryCapacity:12599300096 HugePages:[{PageSize:2048 NumPages:0}] MachineID:74ab50bf248e4860aabd8fa5cdfaee06 SystemUUID:5E406D27-BFE9-4495-9B7C-69BA9785B0ED BootID:64d2312a-7a47-43e2-97ab-9040f8722e1b Filesystems:[{Device:rootfs DeviceMajor:0 DeviceMinor:1 Capacity:0 Type:vfs Inodes:0 HasInodes:true} {Device:tmpfs DeviceMajor:0 DeviceMinor:17 Capacity:6299648000 Type:vfs Inodes:1538000 HasInodes:true} {Device:/dev/sda1 DeviceMajor:8 DeviceMinor:1 Capacity:17293533184 Type:vfs Inodes:9732096 HasInodes:true}] DiskMap:map[8:0:{Name:sda Major:8 Minor:0 Size:20971520000 Scheduler:cfq}] NetworkDevices:[{Name:eth0 MacAddress:08:00:27:0d:97:b1 Speed:1000 Mtu:1500} {Name:eth1 MacAddress:08:00:27:ce:65:30 Speed:1000 Mtu:1500} {Name:sit0 MacAddress:00:00:00:00 Speed:0 Mtu:1480}] Topology:[{Id:0 Memory:12599300096 Cores:[{Id:0 Threads:[0] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:262144 Type:Unified Level:2} {Size:4194304 Type:Unified Level:3}]} {Id:1 Threads:[1] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:262144 Type:Unified Level:2} {Size:4194304 Type:Unified Level:3}]}] Caches:[]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Dec 05 16:28:24 minikube localkube[3327]: I1205 16:28:24.972803    3327 manager.go:222] Version: {KernelVersion:4.9.13 ContainerOsVersion:Buildroot 2017.02 DockerVersion:17.06.0-ce DockerAPIVersion:1.30 CadvisorVersion: CadvisorRevision:}
Dec 05 16:28:24 minikube localkube[3327]: I1205 16:28:24.973188    3327 server.go:422] --cgroups-per-qos enabled, but --cgroup-root was not specified.  defaulting to /
Dec 05 16:28:24 minikube localkube[3327]: I1205 16:28:24.975755    3327 container_manager_linux.go:252] container manager verified user specified cgroup-root exists: /
Dec 05 16:28:24 minikube localkube[3327]: I1205 16:28:24.975781    3327 container_manager_linux.go:257] Creating Container Manager object based on Node Config: {RuntimeCgroupsName: SystemCgroupsName: KubeletCgroupsName: ContainerRuntime:docker CgroupsPerQOS:true CgroupRoot:/ CgroupDriver:cgroupfs ProtectKernelDefaults:false NodeAllocatableConfig:{KubeReservedCgroupName: SystemReservedCgroupName: EnforceNodeAllocatable:map[pods:{}] KubeReserved:map[] SystemReserved:map[] HardEvictionThresholds:[{Signal:memory.available Operator:LessThan Value:{Quantity:100Mi Percentage:0} GracePeriod:0s MinReclaim:<nil>} {Signal:nodefs.available Operator:LessThan Value:{Quantity:<nil> Percentage:0.1} GracePeriod:0s MinReclaim:<nil>} {Signal:nodefs.inodesFree Operator:LessThan Value:{Quantity:<nil> Percentage:0.05} GracePeriod:0s MinReclaim:<nil>}]} ExperimentalQOSReserved:map[] ExperimentalCPUManagerPolicy:none ExperimentalCPUManagerReconcilePeriod:10s}
Dec 05 16:28:24 minikube localkube[3327]: I1205 16:28:24.975898    3327 container_manager_linux.go:288] Creating device plugin handler: false
Dec 05 16:28:24 minikube localkube[3327]: I1205 16:28:24.976098    3327 kubelet.go:273] Adding manifest file: /etc/kubernetes/manifests
Dec 05 16:28:24 minikube localkube[3327]: I1205 16:28:24.976120    3327 kubelet.go:283] Watching apiserver
Dec 05 16:28:24 minikube localkube[3327]: W1205 16:28:24.991441    3327 kubelet_network.go:69] Hairpin mode set to "promiscuous-bridge" but kubenet is not enabled, falling back to "hairpin-veth"
Dec 05 16:28:24 minikube localkube[3327]: I1205 16:28:24.991479    3327 kubelet.go:517] Hairpin mode set to "hairpin-veth"
Dec 05 16:28:24 minikube localkube[3327]: I1205 16:28:24.998416    3327 docker_service.go:207] Docker cri networking managed by kubernetes.io/no-op
Dec 05 16:28:25 minikube localkube[3327]: I1205 16:28:25.008520    3327 docker_service.go:224] Setting cgroupDriver to cgroupfs
Dec 05 16:28:25 minikube localkube[3327]: I1205 16:28:25.021396    3327 remote_runtime.go:43] Connecting to runtime service unix:///var/run/dockershim.sock
Dec 05 16:28:25 minikube localkube[3327]: I1205 16:28:25.023062    3327 kuberuntime_manager.go:174] Container runtime docker initialized, version: 17.06.0-ce, apiVersion: 1.30.0
Dec 05 16:28:25 minikube localkube[3327]: I1205 16:28:25.023211    3327 kuberuntime_manager.go:898] updating runtime config through cri with podcidr 10.180.1.0/24
Dec 05 16:28:25 minikube localkube[3327]: I1205 16:28:25.023326    3327 docker_service.go:306] docker cri received runtime config &RuntimeConfig{NetworkConfig:&NetworkConfig{PodCidr:10.180.1.0/24,},}
Dec 05 16:28:25 minikube localkube[3327]: I1205 16:28:25.024627    3327 kubelet_network.go:276] Setting Pod CIDR:  -> 10.180.1.0/24
Dec 05 16:28:25 minikube localkube[3327]: I1205 16:28:25.029139    3327 server.go:718] Started kubelet v1.8.0
Dec 05 16:28:25 minikube localkube[3327]: E1205 16:28:25.029608    3327 kubelet.go:1234] Image garbage collection failed once. Stats initialization may not have completed yet: unable to find data for container /
Dec 05 16:28:25 minikube localkube[3327]: I1205 16:28:25.030226    3327 kubelet_node_status.go:276] Setting node annotation to enable volume controller attach/detach
Dec 05 16:28:25 minikube localkube[3327]: I1205 16:28:25.032314    3327 server.go:128] Starting to listen on 0.0.0.0:10250
Dec 05 16:28:25 minikube localkube[3327]: I1205 16:28:25.032986    3327 server.go:296] Adding debug handlers to kubelet server.
Dec 05 16:28:25 minikube localkube[3327]: I1205 16:28:25.073024    3327 fs_resource_analyzer.go:66] Starting FS ResourceAnalyzer
Dec 05 16:28:25 minikube localkube[3327]: I1205 16:28:25.073052    3327 status_manager.go:140] Starting to sync pod status with apiserver
Dec 05 16:28:25 minikube localkube[3327]: I1205 16:28:25.073062    3327 kubelet.go:1768] Starting kubelet main sync loop.
Dec 05 16:28:25 minikube localkube[3327]: I1205 16:28:25.073080    3327 kubelet.go:1779] skipping pod synchronization - [container runtime is down PLEG is not healthy: pleg was last seen active 2562047h47m16.854775807s ago; threshold is 3m0s]
Dec 05 16:28:25 minikube localkube[3327]: E1205 16:28:25.073502    3327 container_manager_linux.go:603] [ContainerManager]: Fail to get rootfs information unable to find data for container /
Dec 05 16:28:25 minikube localkube[3327]: I1205 16:28:25.073526    3327 volume_manager.go:246] Starting Kubelet Volume Manager
Dec 05 16:28:25 minikube localkube[3327]: I1205 16:28:25.098678    3327 factory.go:355] Registering Docker factory
Dec 05 16:28:25 minikube localkube[3327]: I1205 16:28:25.100615    3327 factory.go:89] Registering Rkt factory
Dec 05 16:28:25 minikube localkube[3327]: I1205 16:28:25.102324    3327 factory.go:157] Registering CRI-O factory
Dec 05 16:28:25 minikube localkube[3327]: I1205 16:28:25.103338    3327 factory.go:54] Registering systemd factory
Dec 05 16:28:25 minikube localkube[3327]: I1205 16:28:25.104402    3327 factory.go:86] Registering Raw factory
Dec 05 16:28:25 minikube localkube[3327]: I1205 16:28:25.104636    3327 manager.go:1140] Started watching for new ooms in manager
Dec 05 16:28:25 minikube localkube[3327]: I1205 16:28:25.105059    3327 manager.go:311] Starting recovery of all containers
Dec 05 16:28:25 minikube localkube[3327]: I1205 16:28:25.163548    3327 manager.go:316] Recovery completed
Dec 05 16:28:25 minikube localkube[3327]: I1205 16:28:25.170311    3327 rkt.go:56] starting detectRktContainers thread
Dec 05 16:28:25 minikube localkube[3327]: I1205 16:28:25.173695    3327 kubelet_node_status.go:276] Setting node annotation to enable volume controller attach/detach
Dec 05 16:28:25 minikube localkube[3327]: I1205 16:28:25.176621    3327 kubelet_node_status.go:83] Attempting to register node minikube
Dec 05 16:28:25 minikube localkube[3327]: I1205 16:28:25.182637    3327 kubelet_node_status.go:86] Successfully registered node minikube
Dec 05 16:28:25 minikube localkube[3327]: E1205 16:28:25.184042    3327 actual_state_of_world.go:483] Failed to set statusUpdateNeeded to needed true because nodeName="minikube"  does not exist
Dec 05 16:28:25 minikube localkube[3327]: E1205 16:28:25.184103    3327 actual_state_of_world.go:497] Failed to update statusUpdateNeeded field in actual state of world: Failed to set statusUpdateNeeded to needed true because nodeName="minikube"  does not exist
Dec 05 16:28:25 minikube localkube[3327]: I1205 16:28:25.191118    3327 kuberuntime_manager.go:898] updating runtime config through cri with podcidr
Dec 05 16:28:25 minikube localkube[3327]: I1205 16:28:25.192608    3327 docker_service.go:306] docker cri received runtime config &RuntimeConfig{NetworkConfig:&NetworkConfig{PodCidr:,},}
Dec 05 16:28:25 minikube localkube[3327]: I1205 16:28:25.193377    3327 kubelet_network.go:276] Setting Pod CIDR: 10.180.1.0/24 ->
Dec 05 16:28:25 minikube localkube[3327]: E1205 16:28:25.234664    3327 helpers.go:832] Could not find capacity information for resource ephemeral-storage
Dec 05 16:28:25 minikube localkube[3327]: W1205 16:28:25.234707    3327 helpers.go:843] eviction manager: no observation found for eviction signal allocatableNodeFs.available
Dec 05 16:28:25 minikube localkube[3327]: kubelet is ready!
Dec 05 16:28:25 minikube localkube[3327]: Starting proxy...
Dec 05 16:28:25 minikube localkube[3327]: Waiting for proxy to be healthy...
Dec 05 16:28:25 minikube localkube[3327]: W1205 16:28:25.444673    3327 server_others.go:63] unable to register configz: register config "componentconfig" twice
Dec 05 16:28:25 minikube localkube[3327]: I1205 16:28:25.464801    3327 server_others.go:117] Using iptables Proxier.
Dec 05 16:28:25 minikube localkube[3327]: W1205 16:28:25.480357    3327 proxier.go:473] clusterCIDR not specified, unable to distinguish between internal and external traffic
Dec 05 16:28:25 minikube localkube[3327]: I1205 16:28:25.480786    3327 server_others.go:152] Tearing down inactive rules.
Dec 05 16:28:25 minikube localkube[3327]: E1205 16:28:25.553424    3327 healthcheck.go:317] Failed to start node healthz on 0: listen tcp: address 0: missing port in address
Dec 05 16:28:25 minikube localkube[3327]: I1205 16:28:25.553470    3327 config.go:202] Starting service config controller
Dec 05 16:28:25 minikube localkube[3327]: I1205 16:28:25.553478    3327 controller_utils.go:1041] Waiting for caches to sync for service config controller
Dec 05 16:28:25 minikube localkube[3327]: I1205 16:28:25.553512    3327 config.go:102] Starting endpoints config controller
Dec 05 16:28:25 minikube localkube[3327]: I1205 16:28:25.553518    3327 controller_utils.go:1041] Waiting for caches to sync for endpoints config controller
Dec 05 16:28:25 minikube localkube[3327]: I1205 16:28:25.654705    3327 controller_utils.go:1048] Caches are synced for service config controller
Dec 05 16:28:25 minikube localkube[3327]: I1205 16:28:25.654718    3327 controller_utils.go:1048] Caches are synced for endpoints config controller
Dec 05 16:28:26 minikube localkube[3327]: proxy is ready!
Dec 05 16:28:28 minikube localkube[3327]: I1205 16:28:28.850870    3327 node_controller.go:563] Initializing eviction metric for zone:
Dec 05 16:28:28 minikube localkube[3327]: W1205 16:28:28.851298    3327 node_controller.go:916] Missing timestamp for Node minikube. Assuming now as a timestamp.
Dec 05 16:28:28 minikube localkube[3327]: I1205 16:28:28.851325    3327 node_controller.go:782] Controller detected that all Nodes are not-Ready. Entering master disruption mode.
Dec 05 16:28:28 minikube localkube[3327]: I1205 16:28:28.851198    3327 event.go:218] Event(v1.ObjectReference{Kind:"Node", Namespace:"", Name:"minikube", UID:"513702e8-d9d9-11e7-8bd9-0800270d97b1", APIVersion:"", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RegisteredNode' Node minikube event: Registered Node minikube in Controller
Dec 05 16:28:30 minikube localkube[3327]: E1205 16:28:30.075837    3327 helpers.go:468] PercpuUsage had 0 cpus, but the actual number is 2; ignoring extra CPUs
Dec 05 16:28:30 minikube localkube[3327]: I1205 16:28:30.174526    3327 reconciler.go:212] operationExecutor.VerifyControllerAttachedVolume started for volume "addons" (UniqueName: "kubernetes.io/host-path/7b19c3ba446df5355649563d32723e4f-addons") pod "kube-addon-manager-minikube" (UID: "7b19c3ba446df5355649563d32723e4f")
Dec 05 16:28:30 minikube localkube[3327]: I1205 16:28:30.174613    3327 reconciler.go:212] operationExecutor.VerifyControllerAttachedVolume started for volume "kubeconfig" (UniqueName: "kubernetes.io/host-path/7b19c3ba446df5355649563d32723e4f-kubeconfig") pod "kube-addon-manager-minikube" (UID: "7b19c3ba446df5355649563d32723e4f")
Dec 05 16:28:32 minikube localkube[3327]: I1205 16:28:32.771045    3327 event.go:218] Event(v1.ObjectReference{Kind:"ReplicationController", Namespace:"kube-system", Name:"kubernetes-dashboard", UID:"55bbf7f6-d9d9-11e7-8bd9-0800270d97b1", APIVersion:"v1", ResourceVersion:"79", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: kubernetes-dashboard-8dh4m
Dec 05 16:28:32 minikube localkube[3327]: I1205 16:28:32.772252    3327 event.go:218] Event(v1.ObjectReference{Kind:"Pod", Namespace:"kube-system", Name:"kubernetes-dashboard-8dh4m", UID:"55bc513d-d9d9-11e7-8bd9-0800270d97b1", APIVersion:"v1", ResourceVersion:"80", FieldPath:""}): type: 'Warning' reason: 'FailedScheduling' No nodes are available that match all of the predicates: NodeNotReady (1).
Dec 05 16:28:32 minikube localkube[3327]: I1205 16:28:32.775115    3327 event.go:218] Event(v1.ObjectReference{Kind:"Pod", Namespace:"kube-system", Name:"kubernetes-dashboard-8dh4m", UID:"55bc513d-d9d9-11e7-8bd9-0800270d97b1", APIVersion:"v1", ResourceVersion:"81", FieldPath:""}): type: 'Warning' reason: 'FailedScheduling' No nodes are available that match all of the predicates: NodeNotReady (1).
Dec 05 16:28:32 minikube localkube[3327]: W1205 16:28:32.772727    3327 factory.go:928] Request for pod kube-system/kubernetes-dashboard-8dh4m already in flight, abandoning
Dec 05 16:28:32 minikube localkube[3327]: I1205 16:28:32.926033    3327 event.go:218] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"kube-system", Name:"kube-dns", UID:"55d41ef9-d9d9-11e7-8bd9-0800270d97b1", APIVersion:"extensions", ResourceVersion:"91", FieldPath:""}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled up replica set kube-dns-86f6f55dd5 to 1
Dec 05 16:28:32 minikube localkube[3327]: I1205 16:28:32.935327    3327 event.go:218] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"kube-system", Name:"kube-dns-86f6f55dd5", UID:"55d4dc72-d9d9-11e7-8bd9-0800270d97b1", APIVersion:"extensions", ResourceVersion:"92", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: kube-dns-86f6f55dd5-k6sn9
Dec 05 16:28:32 minikube localkube[3327]: I1205 16:28:32.940185    3327 event.go:218] Event(v1.ObjectReference{Kind:"Pod", Namespace:"kube-system", Name:"kube-dns-86f6f55dd5-k6sn9", UID:"55d57dc7-d9d9-11e7-8bd9-0800270d97b1", APIVersion:"v1", ResourceVersion:"93", FieldPath:""}): type: 'Warning' reason: 'FailedScheduling' No nodes are available that match all of the predicates: NodeNotReady (1).
Dec 05 16:28:32 minikube localkube[3327]: I1205 16:28:32.946819    3327 event.go:218] Event(v1.ObjectReference{Kind:"Pod", Namespace:"kube-system", Name:"kube-dns-86f6f55dd5-k6sn9", UID:"55d57dc7-d9d9-11e7-8bd9-0800270d97b1", APIVersion:"v1", ResourceVersion:"96", FieldPath:""}): type: 'Warning' reason: 'FailedScheduling' No nodes are available that match all of the predicates: NodeNotReady (1).
Dec 05 16:28:32 minikube localkube[3327]: W1205 16:28:32.946853    3327 factory.go:928] Request for pod kube-system/kube-dns-86f6f55dd5-k6sn9 already in flight, abandoning
Dec 05 16:28:33 minikube localkube[3327]: I1205 16:28:33.031602    3327 event.go:218] Event(v1.ObjectReference{Kind:"Pod", Namespace:"kube-system", Name:"registry-creds-fmlqd", UID:"55e4c0f6-d9d9-11e7-8bd9-0800270d97b1", APIVersion:"v1", ResourceVersion:"105", FieldPath:""}): type: 'Warning' reason: 'FailedScheduling' No nodes are available that match all of the predicates: NodeNotReady (1).
Dec 05 16:28:33 minikube localkube[3327]: I1205 16:28:33.031879    3327 event.go:218] Event(v1.ObjectReference{Kind:"ReplicationController", Namespace:"kube-system", Name:"registry-creds", UID:"55e4457b-d9d9-11e7-8bd9-0800270d97b1", APIVersion:"v1", ResourceVersion:"104", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: registry-creds-fmlqd
Dec 05 16:28:33 minikube localkube[3327]: I1205 16:28:33.042839    3327 event.go:218] Event(v1.ObjectReference{Kind:"Pod", Namespace:"kube-system", Name:"registry-creds-fmlqd", UID:"55e4c0f6-d9d9-11e7-8bd9-0800270d97b1", APIVersion:"v1", ResourceVersion:"106", FieldPath:""}): type: 'Warning' reason: 'FailedScheduling' No nodes are available that match all of the predicates: NodeNotReady (1).
Dec 05 16:28:33 minikube localkube[3327]: W1205 16:28:33.043378    3327 factory.go:928] Request for pod kube-system/registry-creds-fmlqd already in flight, abandoning
Dec 05 16:28:33 minikube localkube[3327]: I1205 16:28:33.773196    3327 event.go:218] Event(v1.ObjectReference{Kind:"Pod", Namespace:"kube-system", Name:"kubernetes-dashboard-8dh4m", UID:"55bc513d-d9d9-11e7-8bd9-0800270d97b1", APIVersion:"v1", ResourceVersion:"81", FieldPath:""}): type: 'Warning' reason: 'FailedScheduling' No nodes are available that match all of the predicates: NodeNotReady (1).
Dec 05 16:28:33 minikube localkube[3327]: I1205 16:28:33.941919    3327 event.go:218] Event(v1.ObjectReference{Kind:"Pod", Namespace:"kube-system", Name:"kube-dns-86f6f55dd5-k6sn9", UID:"55d57dc7-d9d9-11e7-8bd9-0800270d97b1", APIVersion:"v1", ResourceVersion:"96", FieldPath:""}): type: 'Warning' reason: 'FailedScheduling' No nodes are available that match all of the predicates: NodeNotReady (1).
Dec 05 16:28:34 minikube localkube[3327]: I1205 16:28:34.034121    3327 event.go:218] Event(v1.ObjectReference{Kind:"Pod", Namespace:"kube-system", Name:"registry-creds-fmlqd", UID:"55e4c0f6-d9d9-11e7-8bd9-0800270d97b1", APIVersion:"v1", ResourceVersion:"106", FieldPath:""}): type: 'Warning' reason: 'FailedScheduling' No nodes are available that match all of the predicates: NodeNotReady (1).
Dec 05 16:28:35 minikube localkube[3327]: I1205 16:28:35.778574    3327 event.go:218] Event(v1.ObjectReference{Kind:"Pod", Namespace:"kube-system", Name:"kubernetes-dashboard-8dh4m", UID:"55bc513d-d9d9-11e7-8bd9-0800270d97b1", APIVersion:"v1", ResourceVersion:"81", FieldPath:""}): type: 'Normal' reason: 'Scheduled' Successfully assigned kubernetes-dashboard-8dh4m to minikube
Dec 05 16:28:35 minikube localkube[3327]: I1205 16:28:35.932164    3327 reconciler.go:212] operationExecutor.VerifyControllerAttachedVolume started for volume "default-token-4rvkm" (UniqueName: "kubernetes.io/secret/55bc513d-d9d9-11e7-8bd9-0800270d97b1-default-token-4rvkm") pod "kubernetes-dashboard-8dh4m" (UID: "55bc513d-d9d9-11e7-8bd9-0800270d97b1")
Dec 05 16:28:35 minikube localkube[3327]: I1205 16:28:35.949099    3327 event.go:218] Event(v1.ObjectReference{Kind:"Pod", Namespace:"kube-system", Name:"kube-dns-86f6f55dd5-k6sn9", UID:"55d57dc7-d9d9-11e7-8bd9-0800270d97b1", APIVersion:"v1", ResourceVersion:"96", FieldPath:""}): type: 'Normal' reason: 'Scheduled' Successfully assigned kube-dns-86f6f55dd5-k6sn9 to minikube
Dec 05 16:28:35 minikube localkube[3327]: E1205 16:28:35.955519    3327 helpers.go:468] PercpuUsage had 0 cpus, but the actual number is 2; ignoring extra CPUs
Dec 05 16:28:36 minikube localkube[3327]: I1205 16:28:36.058732    3327 event.go:218] Event(v1.ObjectReference{Kind:"Pod", Namespace:"kube-system", Name:"registry-creds-fmlqd", UID:"55e4c0f6-d9d9-11e7-8bd9-0800270d97b1", APIVersion:"v1", ResourceVersion:"106", FieldPath:""}): type: 'Normal' reason: 'Scheduled' Successfully assigned registry-creds-fmlqd to minikube
Dec 05 16:28:36 minikube localkube[3327]: I1205 16:28:36.134035    3327 reconciler.go:212] operationExecutor.VerifyControllerAttachedVolume started for volume "kube-dns-config" (UniqueName: "kubernetes.io/configmap/55d57dc7-d9d9-11e7-8bd9-0800270d97b1-kube-dns-config") pod "kube-dns-86f6f55dd5-k6sn9" (UID: "55d57dc7-d9d9-11e7-8bd9-0800270d97b1")
Dec 05 16:28:36 minikube localkube[3327]: I1205 16:28:36.134234    3327 reconciler.go:212] operationExecutor.VerifyControllerAttachedVolume started for volume "default-token-4rvkm" (UniqueName: "kubernetes.io/secret/55d57dc7-d9d9-11e7-8bd9-0800270d97b1-default-token-4rvkm") pod "kube-dns-86f6f55dd5-k6sn9" (UID: "55d57dc7-d9d9-11e7-8bd9-0800270d97b1")
Dec 05 16:28:36 minikube localkube[3327]: I1205 16:28:36.236050    3327 reconciler.go:212] operationExecutor.VerifyControllerAttachedVolume started for volume "gcr-creds" (UniqueName: "kubernetes.io/secret/55e4c0f6-d9d9-11e7-8bd9-0800270d97b1-gcr-creds") pod "registry-creds-fmlqd" (UID: "55e4c0f6-d9d9-11e7-8bd9-0800270d97b1")
Dec 05 16:28:36 minikube localkube[3327]: I1205 16:28:36.236094    3327 reconciler.go:212] operationExecutor.VerifyControllerAttachedVolume started for volume "default-token-4rvkm" (UniqueName: "kubernetes.io/secret/55e4c0f6-d9d9-11e7-8bd9-0800270d97b1-default-token-4rvkm") pod "registry-creds-fmlqd" (UID: "55e4c0f6-d9d9-11e7-8bd9-0800270d97b1")
Dec 05 16:28:36 minikube localkube[3327]: E1205 16:28:36.421274    3327 secret.go:201] Couldn't get secret kube-system/registry-creds-gcr: secrets "registry-creds-gcr" not found
Dec 05 16:28:36 minikube localkube[3327]: E1205 16:28:36.421498    3327 nestedpendingoperations.go:263] Operation for "\"kubernetes.io/secret/55e4c0f6-d9d9-11e7-8bd9-0800270d97b1-gcr-creds\" (\"55e4c0f6-d9d9-11e7-8bd9-0800270d97b1\")" failed. No retries permitted until 2017-12-05 16:28:36.921398771 +0000 UTC m=+22.802514911 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "gcr-creds" (UniqueName: "kubernetes.io/secret/55e4c0f6-d9d9-11e7-8bd9-0800270d97b1-gcr-creds") pod "registry-creds-fmlqd" (UID: "55e4c0f6-d9d9-11e7-8bd9-0800270d97b1") : secrets "registry-creds-gcr" not found
Dec 05 16:28:36 minikube localkube[3327]: W1205 16:28:36.597428    3327 container.go:354] Failed to create summary reader for "/system.slice/run-rcfd20eb9dcb94f29bdfd2ac7030e255b.scope": none of the resources are being tracked.
Dec 05 16:28:36 minikube localkube[3327]: W1205 16:28:36.599456    3327 container.go:354] Failed to create summary reader for "/system.slice/run-r58339578235d4737941072054eb42f3a.scope": none of the resources are being tracked.
Dec 05 16:28:36 minikube localkube[3327]: W1205 16:28:36.599752    3327 container.go:354] Failed to create summary reader for "/system.slice/run-r9280b0104c8d4904be7a05e72f4afa02.scope": none of the resources are being tracked.
Dec 05 16:28:36 minikube localkube[3327]: E1205 16:28:36.952641    3327 secret.go:201] Couldn't get secret kube-system/registry-creds-gcr: secrets "registry-creds-gcr" not found
Dec 05 16:28:36 minikube localkube[3327]: E1205 16:28:36.952732    3327 nestedpendingoperations.go:263] Operation for "\"kubernetes.io/secret/55e4c0f6-d9d9-11e7-8bd9-0800270d97b1-gcr-creds\" (\"55e4c0f6-d9d9-11e7-8bd9-0800270d97b1\")" failed. No retries permitted until 2017-12-05 16:28:37.952705686 +0000 UTC m=+23.833821842 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "gcr-creds" (UniqueName: "kubernetes.io/secret/55e4c0f6-d9d9-11e7-8bd9-0800270d97b1-gcr-creds") pod "registry-creds-fmlqd" (UID: "55e4c0f6-d9d9-11e7-8bd9-0800270d97b1") : secrets "registry-creds-gcr" not found
Dec 05 16:28:37 minikube localkube[3327]: http: TLS handshake error from 172.17.0.3:59574: remote error: tls: bad certificate
Dec 05 16:28:37 minikube localkube[3327]: http: TLS handshake error from 172.17.0.3:59578: remote error: tls: bad certificate
Dec 05 16:28:37 minikube localkube[3327]: http: TLS handshake error from 172.17.0.3:59582: remote error: tls: bad certificate
Dec 05 16:28:37 minikube localkube[3327]: http: TLS handshake error from 172.17.0.3:59576: remote error: tls: bad certificate
Dec 05 16:28:37 minikube localkube[3327]: W1205 16:28:37.217379    3327 kuberuntime_container.go:191] Non-root verification doesn't support non-numeric user (nobody)
Dec 05 16:28:37 minikube localkube[3327]: E1205 16:28:37.961373    3327 secret.go:201] Couldn't get secret kube-system/registry-creds-gcr: secrets "registry-creds-gcr" not found
Dec 05 16:28:37 minikube localkube[3327]: E1205 16:28:37.961457    3327 nestedpendingoperations.go:263] Operation for "\"kubernetes.io/secret/55e4c0f6-d9d9-11e7-8bd9-0800270d97b1-gcr-creds\" (\"55e4c0f6-d9d9-11e7-8bd9-0800270d97b1\")" failed. No retries permitted until 2017-12-05 16:28:39.961437415 +0000 UTC m=+25.842553554 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "gcr-creds" (UniqueName: "kubernetes.io/secret/55e4c0f6-d9d9-11e7-8bd9-0800270d97b1-gcr-creds") pod "registry-creds-fmlqd" (UID: "55e4c0f6-d9d9-11e7-8bd9-0800270d97b1") : secrets "registry-creds-gcr" not found
Dec 05 16:28:38 minikube localkube[3327]: http: TLS handshake error from 172.17.0.3:59592: remote error: tls: bad certificate
Dec 05 16:28:38 minikube localkube[3327]: http: TLS handshake error from 172.17.0.3:59594: remote error: tls: bad certificate
Dec 05 16:28:38 minikube localkube[3327]: http: TLS handshake error from 172.17.0.3:59596: remote error: tls: bad certificate
Dec 05 16:28:38 minikube localkube[3327]: I1205 16:28:38.852467    3327 node_controller.go:809] Controller detected that some Nodes are Ready. Exiting master disruption mode.
Dec 05 16:28:39 minikube localkube[3327]: http: TLS handshake error from 172.17.0.3:59598: remote error: tls: bad certificate
Dec 05 16:28:39 minikube localkube[3327]: http: TLS handshake error from 172.17.0.3:59600: remote error: tls: bad certificate
Dec 05 16:28:39 minikube localkube[3327]: http: TLS handshake error from 172.17.0.3:59602: remote error: tls: bad certificate
Dec 05 16:28:39 minikube localkube[3327]: E1205 16:28:39.977410    3327 secret.go:201] Couldn't get secret kube-system/registry-creds-gcr: secrets "registry-creds-gcr" not found
Dec 05 16:28:39 minikube localkube[3327]: E1205 16:28:39.977490    3327 nestedpendingoperations.go:263] Operation for "\"kubernetes.io/secret/55e4c0f6-d9d9-11e7-8bd9-0800270d97b1-gcr-creds\" (\"55e4c0f6-d9d9-11e7-8bd9-0800270d97b1\")" failed. No retries permitted until 2017-12-05 16:28:43.977472085 +0000 UTC m=+29.858588218 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "gcr-creds" (UniqueName: "kubernetes.io/secret/55e4c0f6-d9d9-11e7-8bd9-0800270d97b1-gcr-creds") pod "registry-creds-fmlqd" (UID: "55e4c0f6-d9d9-11e7-8bd9-0800270d97b1") : secrets "registry-creds-gcr" not found
Dec 05 16:28:40 minikube localkube[3327]: http: TLS handshake error from 172.17.0.3:59604: remote error: tls: bad certificate
Dec 05 16:28:40 minikube localkube[3327]: http: TLS handshake error from 172.17.0.3:59606: remote error: tls: bad certificate
Dec 05 16:28:40 minikube localkube[3327]: http: TLS handshake error from 172.17.0.3:59608: remote error: tls: bad certificate
Dec 05 16:28:41 minikube localkube[3327]: http: TLS handshake error from 172.17.0.3:59610: remote error: tls: bad certificate
Dec 05 16:28:41 minikube localkube[3327]: http: TLS handshake error from 172.17.0.3:59614: remote error: tls: bad certificate
Dec 05 16:28:41 minikube localkube[3327]: http: TLS handshake error from 172.17.0.3:59612: remote error: tls: bad certificate
Dec 05 16:28:42 minikube localkube[3327]: http: TLS handshake error from 172.17.0.3:59616: remote error: tls: bad certificate
Dec 05 16:28:42 minikube localkube[3327]: http: TLS handshake error from 172.17.0.3:59618: remote error: tls: bad certificate
Dec 05 16:28:42 minikube localkube[3327]: http: TLS handshake error from 172.17.0.3:59620: remote error: tls: bad certificate
Dec 05 16:28:43 minikube localkube[3327]: http: TLS handshake error from 172.17.0.3:59622: remote error: tls: bad certificate
Dec 05 16:28:43 minikube localkube[3327]: http: TLS handshake error from 172.17.0.3:59624: remote error: tls: bad certificate
Dec 05 16:28:43 minikube localkube[3327]: http: TLS handshake error from 172.17.0.3:59626: remote error: tls: bad certificate
Dec 05 16:28:44 minikube localkube[3327]: E1205 16:28:44.008878    3327 secret.go:201] Couldn't get secret kube-system/registry-creds-gcr: secrets "registry-creds-gcr" not found
Dec 05 16:28:44 minikube localkube[3327]: E1205 16:28:44.008938    3327 nestedpendingoperations.go:263] Operation for "\"kubernetes.io/secret/55e4c0f6-d9d9-11e7-8bd9-0800270d97b1-gcr-creds\" (\"55e4c0f6-d9d9-11e7-8bd9-0800270d97b1\")" failed. No retries permitted until 2017-12-05 16:28:52.008918349 +0000 UTC m=+37.890034490 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "gcr-creds" (UniqueName: "kubernetes.io/secret/55e4c0f6-d9d9-11e7-8bd9-0800270d97b1-gcr-creds") pod "registry-creds-fmlqd" (UID: "55e4c0f6-d9d9-11e7-8bd9-0800270d97b1") : secrets "registry-creds-gcr" not found
Dec 05 16:28:44 minikube localkube[3327]: http: TLS handshake error from 172.17.0.3:59628: remote error: tls: bad certificate
Dec 05 16:28:44 minikube localkube[3327]: http: TLS handshake error from 172.17.0.3:59630: remote error: tls: bad certificate
Dec 05 16:28:44 minikube localkube[3327]: http: TLS handshake error from 172.17.0.3:59632: remote error: tls: bad certificate
Dec 05 16:28:45 minikube localkube[3327]: http: TLS handshake error from 172.17.0.3:59634: remote error: tls: bad certificate
Dec 05 16:28:45 minikube localkube[3327]: http: TLS handshake error from 172.17.0.3:59638: remote error: tls: bad certificate
Dec 05 16:28:45 minikube localkube[3327]: http: TLS handshake error from 172.17.0.3:59636: remote error: tls: bad certificate
Dec 05 16:28:46 minikube localkube[3327]: http: TLS handshake error from 172.17.0.3:59640: remote error: tls: bad certificate
Dec 05 16:28:46 minikube localkube[3327]: http: TLS handshake error from 172.17.0.3:59642: remote error: tls: bad certificate
Dec 05 16:28:46 minikube localkube[3327]: http: TLS handshake error from 172.17.0.3:59644: remote error: tls: bad certificate
Dec 05 16:28:47 minikube localkube[3327]: http: TLS handshake error from 172.17.0.3:59646: remote error: tls: bad certificate
Dec 05 16:28:47 minikube localkube[3327]: http: TLS handshake error from 172.17.0.3:59648: remote error: tls: bad certificate
Dec 05 16:28:47 minikube localkube[3327]: http: TLS handshake error from 172.17.0.3:59650: remote error: tls: bad certificate
Dec 05 16:28:48 minikube localkube[3327]: http: TLS handshake error from 172.17.0.3:59654: remote error: tls: bad certificate
Dec 05 16:28:48 minikube localkube[3327]: http: TLS handshake error from 172.17.0.3:59656: remote error: tls: bad certificate
Dec 05 16:28:48 minikube localkube[3327]: http: TLS handshake error from 172.17.0.3:59658: remote error: tls: bad certificate
Dec 05 16:28:49 minikube localkube[3327]: http: TLS handshake error from 172.17.0.3:59660: remote error: tls: bad certificate
Dec 05 16:28:49 minikube localkube[3327]: http: TLS handshake error from 172.17.0.3:59662: remote error: tls: bad certificate
Dec 05 16:28:49 minikube localkube[3327]: http: TLS handshake error from 172.17.0.3:59664: remote error: tls: bad certificate
Dec 05 16:28:49 minikube localkube[3327]: http: TLS handshake error from 172.17.0.2:54482: remote error: tls: bad certificate
Dec 05 16:28:50 minikube localkube[3327]: http: TLS handshake error from 172.17.0.3:59668: remote error: tls: bad certificate
Dec 05 16:28:50 minikube localkube[3327]: http: TLS handshake error from 172.17.0.3:59670: remote error: tls: bad certificate

Additional Questions:

bbakersmith commented 6 years ago

I hunted down the binary storage location: https://storage.googleapis.com/minikube

I see that the v1.8.0 binary was indeed modified on November 30

<Contents>
<Key>k8sReleases/v1.8.0/localkube-linux-amd64</Key>
<Generation>1512063866146054</Generation>
<MetaGeneration>1</MetaGeneration>
<LastModified>2017-11-30T17:44:26.081Z</LastModified>
<ETag>"56630c50366efa492d36d0b885ee7d3d"</ETag>
<Size>155452040</Size>
</Contents>
r2d4 commented 6 years ago

Sorry about this.

Yes, this was due to a breaking api change we introduced in 0.24.0 for localkube.

You can either

  1. use --bootstrapper kubeadm, which will be the default in future versions
  2. or, update to 0.24.0 or 0.24.1

Obviously its pretty bad to overwrite binaries, but we needed to do so to not break future versions since we found this issue after we had released 0.24.0. We're in a odd in-between state before we fully deprecate localkube, and the transition could have been a little cleaner, but we haven't had the bandwidth.

bbakersmith commented 6 years ago

Thanks for the prompt reply! Upgrading to 0.24.1 fixed the issue.