VM instance of MiniKube hangs on provisioning ingress addon

[happy-machine]

Steps to reproduce the issue:

minikube start —driver=docker --cpus=4 --memory='8200m' --vm=true minikube addons enable ingress

Full output of failed command: 🔎 Verifying ingress addon...

💣 enable failed: run callbacks: running callbacks: [verifying ingress addon pods : timed out waiting for the condition: timed out waiting for the condition]

😿 minikube is exiting due to an error. If the above message is not useful, open an issue: 👉 https://github.com/kubernetes/minikube/issues/new/choose

Full output of minikube start command used, if not already included: Have tried both latest stable (1.18) which gives:

😄 minikube v1.12.0 on Darwin 10.15.4 ✨ Automatically selected the hyperkit driver 👍 Starting control plane node minikube in cluster minikube 🔥 Creating hyperkit VM (CPUs=4, Memory=8200MB, Disk=20000MB) ... ❗ This VM is having trouble accessing https://k8s.gcr.io 💡 To pull new external images, you may need to configure a proxy: https://minikube.sigs.k8s.io/docs/reference/networking/proxy/ 🐳 Preparing Kubernetes v1.18.3 on Docker 19.03.12 ... 🔎 Verifying Kubernetes components... 🌟 Enabled addons: default-storageclass, storage-provisioner 🏄 Done! kubectl is now configured to use "minikube"

and 1.16.6-beta.0:

😄 minikube v1.12.0 on Darwin 10.15.4 ✨ Automatically selected the hyperkit driver 👍 Starting control plane node minikube in cluster minikube 🔥 Creating hyperkit VM (CPUs=4, Memory=8200MB, Disk=20000MB) ... ❗ This VM is having trouble accessing https://k8s.gcr.io 💡 To pull new external images, you may need to configure a proxy: https://minikube.sigs.k8s.io/docs/reference/networking/proxy/ 🐳 Preparing Kubernetes v1.18.3 on Docker 19.03.12 ... 🔎 Verifying Kubernetes components... 🌟 Enabled addons: default-storageclass, storage-provisioner 🏄 Done! kubectl is now configured to use "minikube"

❗ /usr/local/bin/kubectl is version 1.16.6-beta.0, which may be incompatible with Kubernetes 1.18.3.

Optional: Full output of minikube logs command:

==> Docker <== -- Logs begin at Tue 2020-07-14 18:44:23 UTC, end at Tue 2020-07-14 18:48:08 UTC. -- Jul 14 18:44:43 minikube dockerd[2160]: time="2020-07-14T18:44:43.734698603Z" level=info msg="parsed scheme: \"unix\"" module=grpc Jul 14 18:44:43 minikube dockerd[2160]: time="2020-07-14T18:44:43.734740204Z" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc Jul 14 18:44:43 minikube dockerd[2160]: time="2020-07-14T18:44:43.734762202Z" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///var/run/docker/containerd/containerd.sock 0 }] }" module=grpc Jul 14 18:44:43 minikube dockerd[2160]: time="2020-07-14T18:44:43.734772095Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc Jul 14 18:44:43 minikube dockerd[2160]: time="2020-07-14T18:44:43.736110521Z" level=info msg="parsed scheme: \"unix\"" module=grpc Jul 14 18:44:43 minikube dockerd[2160]: time="2020-07-14T18:44:43.736190484Z" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc Jul 14 18:44:43 minikube dockerd[2160]: time="2020-07-14T18:44:43.736207030Z" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///var/run/docker/containerd/containerd.sock 0 }] }" module=grpc Jul 14 18:44:43 minikube dockerd[2160]: time="2020-07-14T18:44:43.736221079Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc Jul 14 18:44:44 minikube dockerd[2160]: time="2020-07-14T18:44:44.919571127Z" level=warning msg="Your kernel does not support cgroup blkio weight" Jul 14 18:44:44 minikube dockerd[2160]: time="2020-07-14T18:44:44.919986217Z" level=warning msg="Your kernel does not support cgroup blkio weight_device" Jul 14 18:44:44 minikube dockerd[2160]: time="2020-07-14T18:44:44.920706661Z" level=warning msg="Your kernel does not support cgroup blkio throttle.read_bps_device" Jul 14 18:44:44 minikube dockerd[2160]: time="2020-07-14T18:44:44.920754127Z" level=warning msg="Your kernel does not support cgroup blkio throttle.write_bps_device" Jul 14 18:44:44 minikube dockerd[2160]: time="2020-07-14T18:44:44.920834762Z" level=warning msg="Your kernel does not support cgroup blkio throttle.read_iops_device" Jul 14 18:44:44 minikube dockerd[2160]: time="2020-07-14T18:44:44.920900459Z" level=warning msg="Your kernel does not support cgroup blkio throttle.write_iops_device" Jul 14 18:44:44 minikube dockerd[2160]: time="2020-07-14T18:44:44.921924551Z" level=info msg="Loading containers: start." Jul 14 18:44:45 minikube dockerd[2160]: time="2020-07-14T18:44:45.054139698Z" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address" Jul 14 18:44:45 minikube dockerd[2160]: time="2020-07-14T18:44:45.096330502Z" level=info msg="Loading containers: done." Jul 14 18:44:45 minikube dockerd[2160]: time="2020-07-14T18:44:45.125749082Z" level=info msg="Docker daemon" commit=48a66213fe graphdriver(s)=overlay2 version=19.03.12 Jul 14 18:44:45 minikube dockerd[2160]: time="2020-07-14T18:44:45.125890760Z" level=info msg="Daemon has completed initialization" Jul 14 18:44:45 minikube dockerd[2160]: time="2020-07-14T18:44:45.141570894Z" level=info msg="API listen on /var/run/docker.sock" Jul 14 18:44:45 minikube systemd[1]: Started Docker Application Container Engine. Jul 14 18:44:45 minikube dockerd[2160]: time="2020-07-14T18:44:45.141584979Z" level=info msg="API listen on [::]:2376" Jul 14 18:44:56 minikube dockerd[2160]: time="2020-07-14T18:44:56.391637890Z" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/c2a48dccfae685f8cf95b2b147865d12772f633c79fef00aeb30d621bdbef7a5/shim.sock" debug=false pid=3183 Jul 14 18:44:56 minikube dockerd[2160]: time="2020-07-14T18:44:56.393504252Z" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/eadbe0d43a369c7e83b6b9ffae2b5775b6008ba7e9179c11cfc1cc780afc0c74/shim.sock" debug=false pid=3184 Jul 14 18:44:56 minikube dockerd[2160]: time="2020-07-14T18:44:56.400400511Z" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/b3eb18792b9d262539c526787ef1b4ca5464b409238783d458465c3039f57631/shim.sock" debug=false pid=3185 Jul 14 18:44:56 minikube dockerd[2160]: time="2020-07-14T18:44:56.415705762Z" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/1f0b17433a6dc29fb322f8940fffc988755042cde404912490ae8fa1cd01d08e/shim.sock" debug=false pid=3217 Jul 14 18:44:56 minikube dockerd[2160]: time="2020-07-14T18:44:56.716264332Z" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/34dc3cd4f7674de0a2996b6410ba15daf12f5387e13cd2f942d2fbbe16433b35/shim.sock" debug=false pid=3379 Jul 14 18:44:56 minikube dockerd[2160]: time="2020-07-14T18:44:56.721956776Z" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/e8b85292454bc2bcdb78dece1cd9f730dc49a8927c3a015c3159974ee27606bd/shim.sock" debug=false pid=3383 Jul 14 18:44:56 minikube dockerd[2160]: time="2020-07-14T18:44:56.726195424Z" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/70fbde97d58cf17ebe741447fd528deb09faa3855b744717e3e3c57cbc41840c/shim.sock" debug=false pid=3388 Jul 14 18:44:56 minikube dockerd[2160]: time="2020-07-14T18:44:56.727435106Z" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/92a1447dbd23008c1441c9258a43f2e34a0a7e9cce77a43d49913ec26d40afae/shim.sock" debug=false pid=3402 Jul 14 18:45:12 minikube dockerd[2160]: time="2020-07-14T18:45:12.082861937Z" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/4b98f990ca6e43e6e7042cf788cc6d8683c057e62860f6d41a5ad17ca11392c7/shim.sock" debug=false pid=4241 Jul 14 18:45:12 minikube dockerd[2160]: time="2020-07-14T18:45:12.331831656Z" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/124edbb8c84e6f5c19d7134788b4ab4fe9e84570bf2eb852f7f938d4b36ed1be/shim.sock" debug=false pid=4282 Jul 14 18:45:17 minikube dockerd[2160]: time="2020-07-14T18:45:17.135256232Z" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/e11e2c5487fe72b4e9fff110e4e6208cd26c3cb1716a9d83194149689be8ec0f/shim.sock" debug=false pid=4425 Jul 14 18:45:17 minikube dockerd[2160]: time="2020-07-14T18:45:17.172786944Z" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/3fc695db226840c83a6933547a6670a608314b6d8cdfc927034aa04d56e4644a/shim.sock" debug=false pid=4451 Jul 14 18:45:17 minikube dockerd[2160]: time="2020-07-14T18:45:17.332956643Z" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/fab3d6332d44a3c54dcac1c3402677a8b5238076ff404a903dc3464a4c148496/shim.sock" debug=false pid=4515 Jul 14 18:45:17 minikube dockerd[2160]: time="2020-07-14T18:45:17.466500483Z" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/baeafde973dbcf3c88788691f15a813ea34748d94d1b698e1f3a5ac3683ce6a7/shim.sock" debug=false pid=4566 Jul 14 18:45:32 minikube dockerd[2160]: time="2020-07-14T18:45:32.930301967Z" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/fab5f84fc13c67103847a8e239e19618866b6f2352a0e3aa587241104ca7ee20/shim.sock" debug=false pid=4784 Jul 14 18:45:33 minikube dockerd[2160]: time="2020-07-14T18:45:33.504287987Z" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/d9b75176f1186be612d5a9f98ea2664cc6f8a91a1c0a9755b4799db00b77c60e/shim.sock" debug=false pid=4839 Jul 14 18:45:33 minikube dockerd[2160]: time="2020-07-14T18:45:33.915769146Z" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/b806c5cdd41af170583e4a1ed5007cbb876259bb027723f4a26e407802de1526/shim.sock" debug=false pid=4889 Jul 14 18:45:34 minikube dockerd[2160]: time="2020-07-14T18:45:34.213172781Z" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/a09c578d276e7de01e57cb55c38a3b81a3e542b9f29bded63dc1fe6d91257b02/shim.sock" debug=false pid=4944 Jul 14 18:45:54 minikube dockerd[2160]: time="2020-07-14T18:45:54.589421188Z" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/22c1f02b671d2831d0fa74b4989498a46443cc60c1c5d32bea1f9522658c2902/shim.sock" debug=false pid=5192 Jul 14 18:45:54 minikube dockerd[2160]: time="2020-07-14T18:45:54.609261861Z" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/33c44ec8fed20aca3b4a2eaf64dd5f3e6d250a228d922a9c1c348f36a65f7446/shim.sock" debug=false pid=5202 Jul 14 18:46:09 minikube dockerd[2160]: time="2020-07-14T18:46:09.890010117Z" level=warning msg="Error getting v2 registry: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Jul 14 18:46:09 minikube dockerd[2160]: time="2020-07-14T18:46:09.890882948Z" level=info msg="Attempting next endpoint for pull after error: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Jul 14 18:46:09 minikube dockerd[2160]: time="2020-07-14T18:46:09.891959113Z" level=error msg="Handler for POST /images/create returned error: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Jul 14 18:46:24 minikube dockerd[2160]: time="2020-07-14T18:46:24.896951452Z" level=warning msg="Error getting v2 registry: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Jul 14 18:46:24 minikube dockerd[2160]: time="2020-07-14T18:46:24.897305455Z" level=info msg="Attempting next endpoint for pull after error: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Jul 14 18:46:24 minikube dockerd[2160]: time="2020-07-14T18:46:24.897469906Z" level=error msg="Handler for POST /images/create returned error: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Jul 14 18:46:40 minikube dockerd[2160]: time="2020-07-14T18:46:40.242231186Z" level=warning msg="Error getting v2 registry: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Jul 14 18:46:40 minikube dockerd[2160]: time="2020-07-14T18:46:40.242307051Z" level=info msg="Attempting next endpoint for pull after error: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Jul 14 18:46:40 minikube dockerd[2160]: time="2020-07-14T18:46:40.242388045Z" level=error msg="Handler for POST /images/create returned error: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Jul 14 18:46:55 minikube dockerd[2160]: time="2020-07-14T18:46:55.246246752Z" level=warning msg="Error getting v2 registry: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Jul 14 18:46:55 minikube dockerd[2160]: time="2020-07-14T18:46:55.247337199Z" level=info msg="Attempting next endpoint for pull after error: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Jul 14 18:46:55 minikube dockerd[2160]: time="2020-07-14T18:46:55.247481686Z" level=error msg="Handler for POST /images/create returned error: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Jul 14 18:47:23 minikube dockerd[2160]: time="2020-07-14T18:47:23.236157162Z" level=warning msg="Error getting v2 registry: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Jul 14 18:47:23 minikube dockerd[2160]: time="2020-07-14T18:47:23.236283105Z" level=info msg="Attempting next endpoint for pull after error: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Jul 14 18:47:23 minikube dockerd[2160]: time="2020-07-14T18:47:23.236997584Z" level=error msg="Handler for POST /images/create returned error: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Jul 14 18:47:38 minikube dockerd[2160]: time="2020-07-14T18:47:38.250789172Z" level=warning msg="Error getting v2 registry: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Jul 14 18:47:38 minikube dockerd[2160]: time="2020-07-14T18:47:38.250890744Z" level=info msg="Attempting next endpoint for pull after error: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Jul 14 18:47:38 minikube dockerd[2160]: time="2020-07-14T18:47:38.251049726Z" level=error msg="Handler for POST /images/create returned error: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" ==> container status <== CONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID a09c578d276e7 85d666cddd043 2 minutes ago Running kubernetes-dashboard 0 b806c5cdd41af d9b75176f1186 86262685d9abb 2 minutes ago Running dashboard-metrics-scraper 0 fab5f84fc13c6 baeafde973dbc 67da37a9a360e 2 minutes ago Running coredns 0 3fc695db22684 fab3d6332d44a 4689081edb103 2 minutes ago Running storage-provisioner 0 e11e2c5487fe7 124edbb8c84e6 3439b7546f29b 2 minutes ago Running kube-proxy 0 4b98f990ca6e4 e8b85292454bc 303ce5db0e90d 3 minutes ago Running etcd 0 1f0b17433a6dc 92a1447dbd230 da26705ccb4b5 3 minutes ago Running kube-controller-manager 0 b3eb18792b9d2 34dc3cd4f7674 7e28efa976bd1 3 minutes ago Running kube-apiserver 0 eadbe0d43a369 70fbde97d58cf 76216c34ed0c7 3 minutes ago Running kube-scheduler 0 c2a48dccfae68 ==> coredns [baeafde973db] <== .:53 [INFO] plugin/reload: Running configuration MD5 = 4e235fcc3696966e76816bcd9034ebc7 CoreDNS-1.6.7 linux/amd64, go1.13.6, da7f65b ==> describe nodes <== Name: minikube Roles: master Labels: beta.kubernetes.io/arch=amd64 beta.kubernetes.io/os=linux kubernetes.io/arch=amd64 kubernetes.io/hostname=minikube kubernetes.io/os=linux minikube.k8s.io/commit=c83e6c47124b71190e138dbc687d2556d31488d6 minikube.k8s.io/name=minikube minikube.k8s.io/updated_at=2020_07_14T19_45_03_0700 minikube.k8s.io/version=v1.12.0 node-role.kubernetes.io/master= Annotations: kubeadm.alpha.kubernetes.io/cri-socket: /var/run/dockershim.sock node.alpha.kubernetes.io/ttl: 0 volumes.kubernetes.io/controller-managed-attach-detach: true CreationTimestamp: Tue, 14 Jul 2020 18:45:00 +0000 Taints: Unschedulable: false Lease: HolderIdentity: minikube AcquireTime: RenewTime: Tue, 14 Jul 2020 18:48:04 +0000 Conditions: Type Status LastHeartbeatTime LastTransitionTime Reason Message ---- ------ ----------------- ------------------ ------ ------- MemoryPressure False Tue, 14 Jul 2020 18:45:14 +0000 Tue, 14 Jul 2020 18:44:57 +0000 KubeletHasSufficientMemory kubelet has sufficient memory available DiskPressure False Tue, 14 Jul 2020 18:45:14 +0000 Tue, 14 Jul 2020 18:44:57 +0000 KubeletHasNoDiskPressure kubelet has no disk pressure PIDPressure False Tue, 14 Jul 2020 18:45:14 +0000 Tue, 14 Jul 2020 18:44:57 +0000 KubeletHasSufficientPID kubelet has sufficient PID available Ready True Tue, 14 Jul 2020 18:45:14 +0000 Tue, 14 Jul 2020 18:45:14 +0000 KubeletReady kubelet is posting ready status Addresses: InternalIP: 192.168.64.5 Hostname: minikube Capacity: cpu: 4 ephemeral-storage: 16954224Ki hugepages-2Mi: 0 memory: 8167756Ki pods: 110 Allocatable: cpu: 4 ephemeral-storage: 16954224Ki hugepages-2Mi: 0 memory: 8167756Ki pods: 110 System Info: Machine ID: bd429a90db104e6cb9b5bd7fe61d8fda System UUID: 02fa11ea-0000-0000-a31d-acde48001122 Boot ID: dc4dd3f9-beb7-47e6-b337-783a1fc1c0c3 Kernel Version: 4.19.114 OS Image: Buildroot 2019.02.11 Operating System: linux Architecture: amd64 Container Runtime Version: docker://19.3.12 Kubelet Version: v1.18.3 Kube-Proxy Version: v1.18.3 Non-terminated Pods: (12 in total) Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits AGE --------- ---- ------------ ---------- --------------- ------------- --- kube-system coredns-66bff467f8-hw82w 100m (2%) 0 (0%) 70Mi (0%) 170Mi (2%) 2m59s kube-system etcd-minikube 0 (0%) 0 (0%) 0 (0%) 0 (0%) 3m5s kube-system ingress-nginx-admission-create-h5njb 0 (0%) 0 (0%) 0 (0%) 0 (0%) 2m15s kube-system ingress-nginx-admission-patch-5ktvb 0 (0%) 0 (0%) 0 (0%) 0 (0%) 2m15s kube-system ingress-nginx-controller-7bb4c67d67-2ntwj 100m (2%) 0 (0%) 90Mi (1%) 0 (0%) 2m16s kube-system kube-apiserver-minikube 250m (6%) 0 (0%) 0 (0%) 0 (0%) 3m5s kube-system kube-controller-manager-minikube 200m (5%) 0 (0%) 0 (0%) 0 (0%) 3m5s kube-system kube-proxy-76s5h 0 (0%) 0 (0%) 0 (0%) 0 (0%) 2m59s kube-system kube-scheduler-minikube 100m (2%) 0 (0%) 0 (0%) 0 (0%) 3m5s kube-system storage-provisioner 0 (0%) 0 (0%) 0 (0%) 0 (0%) 3m5s kubernetes-dashboard dashboard-metrics-scraper-dc6947fbf-wxrw5 0 (0%) 0 (0%) 0 (0%) 0 (0%) 2m37s kubernetes-dashboard kubernetes-dashboard-6dbb54fd95-zwd9r 0 (0%) 0 (0%) 0 (0%) 0 (0%) 2m36s Allocated resources: (Total limits may be over 100 percent, i.e., overcommitted.) Resource Requests Limits -------- -------- ------ cpu 750m (18%) 0 (0%) memory 160Mi (2%) 170Mi (2%) ephemeral-storage 0 (0%) 0 (0%) hugepages-2Mi 0 (0%) 0 (0%) Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Starting 3m5s kubelet, minikube Starting kubelet. Normal NodeHasSufficientMemory 3m5s kubelet, minikube Node minikube status is now: NodeHasSufficientMemory Normal NodeHasNoDiskPressure 3m5s kubelet, minikube Node minikube status is now: NodeHasNoDiskPressure Normal NodeHasSufficientPID 3m5s kubelet, minikube Node minikube status is now: NodeHasSufficientPID Normal NodeNotReady 3m5s kubelet, minikube Node minikube status is now: NodeNotReady Normal NodeAllocatableEnforced 3m5s kubelet, minikube Updated Node Allocatable limit across pods Normal Starting 2m57s kube-proxy, minikube Starting kube-proxy. Normal NodeReady 2m55s kubelet, minikube Node minikube status is now: NodeReady ==> dmesg <== [Jul14 18:44] ERROR: earlyprintk= earlyser already used [ +0.000000] You have booted with nomodeset. This means your GPU drivers are DISABLED [ +0.000000] Any video related functionality will be severely degraded, and you may not even be able to suspend the system properly [ +0.000000] Unless you actually understand what nomodeset does, you should reboot without enabling it [ +0.147802] ACPI BIOS Warning (bug): Incorrect checksum in table [DSDT] - 0xBE, should be 0x1B (20180810/tbprint-177) [ +4.338135] ACPI Error: Could not enable RealTimeClock event (20180810/evxfevnt-184) [ +0.000002] ACPI Warning: Could not enable fixed event - RealTimeClock (4) (20180810/evxface-620) [ +0.011746] platform regulatory.0: Direct firmware load for regulatory.db failed with error -2 [ +2.512045] systemd[1]: Failed to bump fs.file-max, ignoring: Invalid argument [ +0.003551] systemd-fstab-generator[1127]: Ignoring "noauto" for root device [ +0.004252] systemd[1]: File /usr/lib/systemd/system/systemd-journald.service:12 configures an IP firewall (IPAddressDeny=any), but the local system does not support BPF/cgroup based firewalling. [ +0.000002] systemd[1]: Proceeding WITHOUT firewalling in effect! (This warning is only shown for the first loaded unit using IP firewalling.) [ +0.805375] NFSD: the nfsdcld client tracking upcall will be removed in 3.10. Please transition to using nfsdcltrack. [ +0.633582] vboxguest: loading out-of-tree module taints kernel. [ +0.003746] vboxguest: PCI device not found, probably running on physical hardware. [ +1.494161] systemd-fstab-generator[1934]: Ignoring "noauto" for root device [ +0.070899] systemd-fstab-generator[1944]: Ignoring "noauto" for root device [ +16.529095] systemd-fstab-generator[2147]: Ignoring "noauto" for root device [ +2.534736] kauditd_printk_skb: 65 callbacks suppressed [ +0.285633] systemd-fstab-generator[2313]: Ignoring "noauto" for root device [ +0.843701] systemd-fstab-generator[2396]: Ignoring "noauto" for root device [ +1.831028] systemd-fstab-generator[2639]: Ignoring "noauto" for root device [ +7.587217] kauditd_printk_skb: 107 callbacks suppressed [Jul14 18:45] systemd-fstab-generator[3908]: Ignoring "noauto" for root device [ +9.169015] kauditd_printk_skb: 32 callbacks suppressed [ +12.075583] kauditd_printk_skb: 41 callbacks suppressed [ +29.419504] kauditd_printk_skb: 20 callbacks suppressed [Jul14 18:46] NFSD: Unable to end grace period: -110 ==> etcd [e8b85292454b] <== [WARNING] Deprecated '--logger=capnslog' flag is set; use '--logger=zap' flag instead 2020-07-14 18:44:56.893397 I | etcdmain: etcd Version: 3.4.3 2020-07-14 18:44:56.893427 I | etcdmain: Git SHA: 3cf2f69b5 2020-07-14 18:44:56.893430 I | etcdmain: Go Version: go1.12.12 2020-07-14 18:44:56.893433 I | etcdmain: Go OS/Arch: linux/amd64 2020-07-14 18:44:56.893436 I | etcdmain: setting maximum number of CPUs to 4, total number of available CPUs is 4 [WARNING] Deprecated '--logger=capnslog' flag is set; use '--logger=zap' flag instead 2020-07-14 18:44:56.893533 I | embed: peerTLS: cert = /var/lib/minikube/certs/etcd/peer.crt, key = /var/lib/minikube/certs/etcd/peer.key, trusted-ca = /var/lib/minikube/certs/etcd/ca.crt, client-cert-auth = true, crl-file = 2020-07-14 18:44:56.895217 I | embed: name = minikube 2020-07-14 18:44:56.895227 I | embed: data dir = /var/lib/minikube/etcd 2020-07-14 18:44:56.895231 I | embed: member dir = /var/lib/minikube/etcd/member 2020-07-14 18:44:56.895233 I | embed: heartbeat = 100ms 2020-07-14 18:44:56.895235 I | embed: election = 1000ms 2020-07-14 18:44:56.895238 I | embed: snapshot count = 10000 2020-07-14 18:44:56.895244 I | embed: advertise client URLs = https://192.168.64.5:2379 2020-07-14 18:44:56.905290 I | etcdserver: starting member d59b486478f5e7d4 in cluster 6b4c8bb4940320a5 raft2020/07/14 18:44:56 INFO: d59b486478f5e7d4 switched to configuration voters=() raft2020/07/14 18:44:56 INFO: d59b486478f5e7d4 became follower at term 0 raft2020/07/14 18:44:56 INFO: newRaft d59b486478f5e7d4 [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0] raft2020/07/14 18:44:56 INFO: d59b486478f5e7d4 became follower at term 1 raft2020/07/14 18:44:56 INFO: d59b486478f5e7d4 switched to configuration voters=(15391975747832113108) 2020-07-14 18:44:56.918502 W | auth: simple token is not cryptographically signed 2020-07-14 18:44:56.924740 I | etcdserver: starting server... [version: 3.4.3, cluster version: to_be_decided] 2020-07-14 18:44:56.948350 I | etcdserver: d59b486478f5e7d4 as single-node; fast-forwarding 9 ticks (election ticks 10) raft2020/07/14 18:44:56 INFO: d59b486478f5e7d4 switched to configuration voters=(15391975747832113108) 2020-07-14 18:44:56.948748 I | etcdserver/membership: added member d59b486478f5e7d4 [https://192.168.64.5:2380] to cluster 6b4c8bb4940320a5 2020-07-14 18:44:56.952229 I | embed: ClientTLS: cert = /var/lib/minikube/certs/etcd/server.crt, key = /var/lib/minikube/certs/etcd/server.key, trusted-ca = /var/lib/minikube/certs/etcd/ca.crt, client-cert-auth = true, crl-file = 2020-07-14 18:44:56.952386 I | embed: listening for metrics on http://127.0.0.1:2381 2020-07-14 18:44:56.952437 I | embed: listening for peers on 192.168.64.5:2380 raft2020/07/14 18:44:57 INFO: d59b486478f5e7d4 is starting a new election at term 1 raft2020/07/14 18:44:57 INFO: d59b486478f5e7d4 became candidate at term 2 raft2020/07/14 18:44:57 INFO: d59b486478f5e7d4 received MsgVoteResp from d59b486478f5e7d4 at term 2 raft2020/07/14 18:44:57 INFO: d59b486478f5e7d4 became leader at term 2 raft2020/07/14 18:44:57 INFO: raft.node: d59b486478f5e7d4 elected leader d59b486478f5e7d4 at term 2 2020-07-14 18:44:57.912466 I | etcdserver: published {Name:minikube ClientURLs:[https://192.168.64.5:2379]} to cluster 6b4c8bb4940320a5 2020-07-14 18:44:57.912573 I | embed: ready to serve client requests 2020-07-14 18:44:57.913259 I | etcdserver: setting up the initial cluster version to 3.4 2020-07-14 18:44:57.913579 I | embed: ready to serve client requests 2020-07-14 18:44:57.915623 I | embed: serving client requests on 127.0.0.1:2379 2020-07-14 18:44:57.922862 I | embed: serving client requests on 192.168.64.5:2379 2020-07-14 18:44:57.929014 N | etcdserver/membership: set the initial cluster version to 3.4 2020-07-14 18:44:57.929468 I | etcdserver/api: enabled capabilities for version 3.4 ==> kernel <== 18:48:09 up 3 min, 0 users, load average: 0.39, 0.34, 0.16 Linux minikube 4.19.114 #1 SMP Mon Jul 6 11:11:02 PDT 2020 x86_64 GNU/Linux PRETTY_NAME="Buildroot 2019.02.11" ==> kube-apiserver [34dc3cd4f767] <== W0714 18:44:58.809353 1 genericapiserver.go:409] Skipping API node.k8s.io/v1alpha1 because it has no resources. W0714 18:44:58.821862 1 genericapiserver.go:409] Skipping API rbac.authorization.k8s.io/v1alpha1 because it has no resources. W0714 18:44:58.824348 1 genericapiserver.go:409] Skipping API scheduling.k8s.io/v1alpha1 because it has no resources. W0714 18:44:58.835810 1 genericapiserver.go:409] Skipping API storage.k8s.io/v1alpha1 because it has no resources. W0714 18:44:58.852591 1 genericapiserver.go:409] Skipping API apps/v1beta2 because it has no resources. W0714 18:44:58.852629 1 genericapiserver.go:409] Skipping API apps/v1beta1 because it has no resources. I0714 18:44:58.862045 1 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook. I0714 18:44:58.862082 1 plugins.go:161] Loaded 10 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota. I0714 18:44:58.863509 1 client.go:361] parsed scheme: "endpoint" I0714 18:44:58.863562 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://127.0.0.1:2379 0 }] I0714 18:44:58.870182 1 client.go:361] parsed scheme: "endpoint" I0714 18:44:58.870206 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://127.0.0.1:2379 0 }] I0714 18:45:00.733512 1 dynamic_cafile_content.go:167] Starting request-header::/var/lib/minikube/certs/front-proxy-ca.crt I0714 18:45:00.733576 1 dynamic_cafile_content.go:167] Starting client-ca-bundle::/var/lib/minikube/certs/ca.crt I0714 18:45:00.733817 1 dynamic_serving_content.go:130] Starting serving-cert::/var/lib/minikube/certs/apiserver.crt::/var/lib/minikube/certs/apiserver.key I0714 18:45:00.733912 1 secure_serving.go:178] Serving securely on [::]:8443 I0714 18:45:00.734226 1 apiservice_controller.go:94] Starting APIServiceRegistrationController I0714 18:45:00.734267 1 cache.go:32] Waiting for caches to sync for APIServiceRegistrationController controller I0714 18:45:00.734291 1 tlsconfig.go:240] Starting DynamicServingCertificateController I0714 18:45:00.734309 1 cluster_authentication_trust_controller.go:440] Starting cluster_authentication_trust_controller controller I0714 18:45:00.734347 1 shared_informer.go:223] Waiting for caches to sync for cluster_authentication_trust_controller I0714 18:45:00.734427 1 autoregister_controller.go:141] Starting autoregister controller I0714 18:45:00.734432 1 cache.go:32] Waiting for caches to sync for autoregister controller I0714 18:45:00.735492 1 dynamic_cafile_content.go:167] Starting request-header::/var/lib/minikube/certs/front-proxy-ca.crt I0714 18:45:00.735569 1 available_controller.go:387] Starting AvailableConditionController I0714 18:45:00.735595 1 cache.go:32] Waiting for caches to sync for AvailableConditionController controller I0714 18:45:00.735695 1 crd_finalizer.go:266] Starting CRDFinalizer I0714 18:45:00.736212 1 dynamic_cafile_content.go:167] Starting client-ca-bundle::/var/lib/minikube/certs/ca.crt I0714 18:45:00.736305 1 controller.go:86] Starting OpenAPI controller I0714 18:45:00.736319 1 customresource_discovery_controller.go:209] Starting DiscoveryController I0714 18:45:00.736350 1 naming_controller.go:291] Starting NamingConditionController I0714 18:45:00.736361 1 establishing_controller.go:76] Starting EstablishingController I0714 18:45:00.736435 1 nonstructuralschema_controller.go:186] Starting NonStructuralSchemaConditionController I0714 18:45:00.736467 1 apiapproval_controller.go:186] Starting KubernetesAPIApprovalPolicyConformantConditionController I0714 18:45:00.737107 1 controller.go:81] Starting OpenAPI AggregationController I0714 18:45:00.738134 1 crdregistration_controller.go:111] Starting crd-autoregister controller I0714 18:45:00.738141 1 shared_informer.go:223] Waiting for caches to sync for crd-autoregister E0714 18:45:00.738940 1 controller.go:152] Unable to remove old endpoints from kubernetes service: StorageError: key not found, Code: 1, Key: /registry/masterleases/192.168.64.5, ResourceVersion: 0, AdditionalErrorMsg: I0714 18:45:00.834696 1 cache.go:39] Caches are synced for autoregister controller I0714 18:45:00.835309 1 shared_informer.go:230] Caches are synced for cluster_authentication_trust_controller I0714 18:45:00.836060 1 cache.go:39] Caches are synced for AvailableConditionController controller I0714 18:45:00.838055 1 cache.go:39] Caches are synced for APIServiceRegistrationController controller I0714 18:45:00.838362 1 shared_informer.go:230] Caches are synced for crd-autoregister I0714 18:45:01.733841 1 controller.go:130] OpenAPI AggregationController: action for item : Nothing (removed from the queue). I0714 18:45:01.733941 1 controller.go:130] OpenAPI AggregationController: action for item k8s_internal_local_delegation_chain_0000000000: Nothing (removed from the queue). I0714 18:45:01.743706 1 storage_scheduling.go:134] created PriorityClass system-node-critical with value 2000001000 I0714 18:45:01.750681 1 storage_scheduling.go:134] created PriorityClass system-cluster-critical with value 2000000000 I0714 18:45:01.750743 1 storage_scheduling.go:143] all system priority classes are created successfully or already exist. I0714 18:45:02.128203 1 controller.go:606] quota admission added evaluator for: roles.rbac.authorization.k8s.io I0714 18:45:02.173149 1 controller.go:606] quota admission added evaluator for: rolebindings.rbac.authorization.k8s.io W0714 18:45:02.339816 1 lease.go:224] Resetting endpoints for master service "kubernetes" to [192.168.64.5] I0714 18:45:02.341448 1 controller.go:606] quota admission added evaluator for: endpoints I0714 18:45:02.350374 1 controller.go:606] quota admission added evaluator for: endpointslices.discovery.k8s.io I0714 18:45:03.071506 1 controller.go:606] quota admission added evaluator for: serviceaccounts I0714 18:45:03.617005 1 controller.go:606] quota admission added evaluator for: deployments.apps I0714 18:45:03.795980 1 controller.go:606] quota admission added evaluator for: daemonsets.apps I0714 18:45:04.189618 1 controller.go:606] quota admission added evaluator for: leases.coordination.k8s.io I0714 18:45:10.047923 1 controller.go:606] quota admission added evaluator for: replicasets.apps I0714 18:45:10.180517 1 controller.go:606] quota admission added evaluator for: controllerrevisions.apps I0714 18:45:54.036433 1 controller.go:606] quota admission added evaluator for: jobs.batch ==> kube-controller-manager [92a1447dbd23] <== I0714 18:45:10.051767 1 event.go:278] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"kube-system", Name:"coredns", UID:"fae20924-cfdf-449f-9337-81fb168c2fbc", APIVersion:"apps/v1", ResourceVersion:"225", FieldPath:""}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled up replica set coredns-66bff467f8 to 1 I0714 18:45:10.058692 1 shared_informer.go:230] Caches are synced for ReplicationController I0714 18:45:10.058665 1 shared_informer.go:230] Caches are synced for TTL I0714 18:45:10.058713 1 shared_informer.go:230] Caches are synced for expand I0714 18:45:10.059357 1 shared_informer.go:230] Caches are synced for certificate-csrapproving I0714 18:45:10.059654 1 shared_informer.go:230] Caches are synced for endpoint I0714 18:45:10.064876 1 shared_informer.go:230] Caches are synced for certificate-csrsigning I0714 18:45:10.065987 1 shared_informer.go:230] Caches are synced for job I0714 18:45:10.066929 1 shared_informer.go:230] Caches are synced for ClusterRoleAggregator I0714 18:45:10.076225 1 shared_informer.go:230] Caches are synced for namespace I0714 18:45:10.089236 1 shared_informer.go:230] Caches are synced for PVC protection I0714 18:45:10.110611 1 shared_informer.go:230] Caches are synced for bootstrap_signer I0714 18:45:10.111169 1 shared_informer.go:230] Caches are synced for ReplicaSet I0714 18:45:10.111246 1 shared_informer.go:230] Caches are synced for attach detach I0714 18:45:10.118076 1 shared_informer.go:230] Caches are synced for PV protection I0714 18:45:10.118876 1 shared_informer.go:230] Caches are synced for endpoint_slice I0714 18:45:10.123980 1 shared_informer.go:230] Caches are synced for service account I0714 18:45:10.124821 1 shared_informer.go:230] Caches are synced for GC I0714 18:45:10.128754 1 event.go:278] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"kube-system", Name:"coredns-66bff467f8", UID:"10affa91-22a3-4e49-8231-e199dbdd0d6a", APIVersion:"apps/v1", ResourceVersion:"302", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: coredns-66bff467f8-hw82w I0714 18:45:10.129648 1 shared_informer.go:230] Caches are synced for persistent volume I0714 18:45:10.173485 1 shared_informer.go:230] Caches are synced for daemon sets I0714 18:45:10.198553 1 event.go:278] Event(v1.ObjectReference{Kind:"DaemonSet", Namespace:"kube-system", Name:"kube-proxy", UID:"be771dd0-9dcd-45b4-b2ea-bce498650b5d", APIVersion:"apps/v1", ResourceVersion:"214", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: kube-proxy-76s5h I0714 18:45:10.220205 1 shared_informer.go:230] Caches are synced for taint I0714 18:45:10.220414 1 node_lifecycle_controller.go:1433] Initializing eviction metric for zone: I0714 18:45:10.220511 1 taint_manager.go:187] Starting NoExecuteTaintManager W0714 18:45:10.220707 1 node_lifecycle_controller.go:1048] Missing timestamp for Node minikube. Assuming now as a timestamp. I0714 18:45:10.220783 1 node_lifecycle_controller.go:1199] Controller detected that all Nodes are not-Ready. Entering master disruption mode. I0714 18:45:10.221143 1 event.go:278] Event(v1.ObjectReference{Kind:"Node", Namespace:"", Name:"minikube", UID:"d88bab69-3afa-4e95-9088-d7d2b14cc1d9", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RegisteredNode' Node minikube event: Registered Node minikube in Controller E0714 18:45:10.229820 1 daemon_controller.go:292] kube-system/kube-proxy failed with : error storing status for daemon set &v1.DaemonSet{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"kube-proxy", GenerateName:"", Namespace:"kube-system", SelfLink:"/apis/apps/v1/namespaces/kube-system/daemonsets/kube-proxy", UID:"be771dd0-9dcd-45b4-b2ea-bce498650b5d", ResourceVersion:"214", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63730349103, loc:(*time.Location)(0x6d09200)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"k8s-app":"kube-proxy"}, Annotations:map[string]string{"deprecated.daemonset.template.generation":"1"}, OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry{v1.ManagedFieldsEntry{Manager:"kubeadm", Operation:"Update", APIVersion:"apps/v1", Time:(*v1.Time)(0xc0017e1980), FieldsType:"FieldsV1", FieldsV1:(*v1.FieldsV1)(0xc0017e19a0)}}}, Spec:v1.DaemonSetSpec{Selector:(*v1.LabelSelector)(0xc0017e19c0), Template:v1.PodTemplateSpec{ObjectMeta:v1.ObjectMeta{Name:"", GenerateName:"", Namespace:"", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"k8s-app":"kube-proxy"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.PodSpec{Volumes:[]v1.Volume{v1.Volume{Name:"kube-proxy", VolumeSource:v1.VolumeSource{HostPath:(*v1.HostPathVolumeSource)(nil), EmptyDir:(*v1.EmptyDirVolumeSource)(nil), GCEPersistentDisk:(*v1.GCEPersistentDiskVolumeSource)(nil), AWSElasticBlockStore:(*v1.AWSElasticBlockStoreVolumeSource)(nil), GitRepo:(*v1.GitRepoVolumeSource)(nil), Secret:(*v1.SecretVolumeSource)(nil), NFS:(*v1.NFSVolumeSource)(nil), ISCSI:(*v1.ISCSIVolumeSource)(nil), Glusterfs:(*v1.GlusterfsVolumeSource)(nil), PersistentVolumeClaim:(*v1.PersistentVolumeClaimVolumeSource)(nil), RBD:(*v1.RBDVolumeSource)(nil), FlexVolume:(*v1.FlexVolumeSource)(nil), Cinder:(*v1.CinderVolumeSource)(nil), CephFS:(*v1.CephFSVolumeSource)(nil), Flocker:(*v1.FlockerVolumeSource)(nil), DownwardAPI:(*v1.DownwardAPIVolumeSource)(nil), FC:(*v1.FCVolumeSource)(nil), AzureFile:(*v1.AzureFileVolumeSource)(nil), ConfigMap:(*v1.ConfigMapVolumeSource)(0xc0018a6940), VsphereVolume:(*v1.VsphereVirtualDiskVolumeSource)(nil), Quobyte:(*v1.QuobyteVolumeSource)(nil), AzureDisk:(*v1.AzureDiskVolumeSource)(nil), PhotonPersistentDisk:(*v1.PhotonPersistentDiskVolumeSource)(nil), Projected:(*v1.ProjectedVolumeSource)(nil), PortworxVolume:(*v1.PortworxVolumeSource)(nil), ScaleIO:(*v1.ScaleIOVolumeSource)(nil), StorageOS:(*v1.StorageOSVolumeSource)(nil), CSI:(*v1.CSIVolumeSource)(nil)}}, v1.Volume{Name:"xtables-lock", VolumeSource:v1.VolumeSource{HostPath:(*v1.HostPathVolumeSource)(0xc0017e19e0), EmptyDir:(*v1.EmptyDirVolumeSource)(nil), GCEPersistentDisk:(*v1.GCEPersistentDiskVolumeSource)(nil), AWSElasticBlockStore:(*v1.AWSElasticBlockStoreVolumeSource)(nil), GitRepo:(*v1.GitRepoVolumeSource)(nil), Secret:(*v1.SecretVolumeSource)(nil), NFS:(*v1.NFSVolumeSource)(nil), ISCSI:(*v1.ISCSIVolumeSource)(nil), Glusterfs:(*v1.GlusterfsVolumeSource)(nil), PersistentVolumeClaim:(*v1.PersistentVolumeClaimVolumeSource)(nil), RBD:(*v1.RBDVolumeSource)(nil), FlexVolume:(*v1.FlexVolumeSource)(nil), Cinder:(*v1.CinderVolumeSource)(nil), CephFS:(*v1.CephFSVolumeSource)(nil), Flocker:(*v1.FlockerVolumeSource)(nil), DownwardAPI:(*v1.DownwardAPIVolumeSource)(nil), FC:(*v1.FCVolumeSource)(nil), AzureFile:(*v1.AzureFileVolumeSource)(nil), ConfigMap:(*v1.ConfigMapVolumeSource)(nil), VsphereVolume:(*v1.VsphereVirtualDiskVolumeSource)(nil), Quobyte:(*v1.QuobyteVolumeSource)(nil), AzureDisk:(*v1.AzureDiskVolumeSource)(nil), PhotonPersistentDisk:(*v1.PhotonPersistentDiskVolumeSource)(nil), Projected:(*v1.ProjectedVolumeSource)(nil), PortworxVolume:(*v1.PortworxVolumeSource)(nil), ScaleIO:(*v1.ScaleIOVolumeSource)(nil), StorageOS:(*v1.StorageOSVolumeSource)(nil), CSI:(*v1.CSIVolumeSource)(nil)}}, v1.Volume{Name:"lib-modules", VolumeSource:v1.VolumeSource{HostPath:(*v1.HostPathVolumeSource)(0xc0017e1a00), EmptyDir:(*v1.EmptyDirVolumeSource)(nil), GCEPersistentDisk:(*v1.GCEPersistentDiskVolumeSource)(nil), AWSElasticBlockStore:(*v1.AWSElasticBlockStoreVolumeSource)(nil), GitRepo:(*v1.GitRepoVolumeSource)(nil), Secret:(*v1.SecretVolumeSource)(nil), NFS:(*v1.NFSVolumeSource)(nil), ISCSI:(*v1.ISCSIVolumeSource)(nil), Glusterfs:(*v1.GlusterfsVolumeSource)(nil), PersistentVolumeClaim:(*v1.PersistentVolumeClaimVolumeSource)(nil), RBD:(*v1.RBDVolumeSource)(nil), FlexVolume:(*v1.FlexVolumeSource)(nil), Cinder:(*v1.CinderVolumeSource)(nil), CephFS:(*v1.CephFSVolumeSource)(nil), Flocker:(*v1.FlockerVolumeSource)(nil), DownwardAPI:(*v1.DownwardAPIVolumeSource)(nil), FC:(*v1.FCVolumeSource)(nil), AzureFile:(*v1.AzureFileVolumeSource)(nil), ConfigMap:(*v1.ConfigMapVolumeSource)(nil), VsphereVolume:(*v1.VsphereVirtualDiskVolumeSource)(nil), Quobyte:(*v1.QuobyteVolumeSource)(nil), AzureDisk:(*v1.AzureDiskVolumeSource)(nil), PhotonPersistentDisk:(*v1.PhotonPersistentDiskVolumeSource)(nil), Projected:(*v1.ProjectedVolumeSource)(nil), PortworxVolume:(*v1.PortworxVolumeSource)(nil), ScaleIO:(*v1.ScaleIOVolumeSource)(nil), StorageOS:(*v1.StorageOSVolumeSource)(nil), CSI:(*v1.CSIVolumeSource)(nil)}}}, InitContainers:[]v1.Container(nil), Containers:[]v1.Container{v1.Container{Name:"kube-proxy", Image:"k8s.gcr.io/kube-proxy:v1.18.3", Command:[]string{"/usr/local/bin/kube-proxy", "--config=/var/lib/kube-proxy/config.conf", "--hostname-override=$(NODE_NAME)"}, Args:[]string(nil), WorkingDir:"", Ports:[]v1.ContainerPort(nil), EnvFrom:[]v1.EnvFromSource(nil), Env:[]v1.EnvVar{v1.EnvVar{Name:"NODE_NAME", Value:"", ValueFrom:(*v1.EnvVarSource)(0xc0017e1a40)}}, Resources:v1.ResourceRequirements{Limits:v1.ResourceList(nil), Requests:v1.ResourceList(nil)}, VolumeMounts:[]v1.VolumeMount{v1.VolumeMount{Name:"kube-proxy", ReadOnly:false, MountPath:"/var/lib/kube-proxy", SubPath:"", MountPropagation:(*v1.MountPropagationMode)(nil), SubPathExpr:""}, v1.VolumeMount{Name:"xtables-lock", ReadOnly:false, MountPath:"/run/xtables.lock", SubPath:"", MountPropagation:(*v1.MountPropagationMode)(nil), SubPathExpr:""}, v1.VolumeMount{Name:"lib-modules", ReadOnly:true, MountPath:"/lib/modules", SubPath:"", MountPropagation:(*v1.MountPropagationMode)(nil), SubPathExpr:""}}, VolumeDevices:[]v1.VolumeDevice(nil), LivenessProbe:(*v1.Probe)(nil), ReadinessProbe:(*v1.Probe)(nil), StartupProbe:(*v1.Probe)(nil), Lifecycle:(*v1.Lifecycle)(nil), TerminationMessagePath:"/dev/termination-log", TerminationMessagePolicy:"File", ImagePullPolicy:"IfNotPresent", SecurityContext:(*v1.SecurityContext)(0xc001982780), Stdin:false, StdinOnce:false, TTY:false}}, EphemeralContainers:[]v1.EphemeralContainer(nil), RestartPolicy:"Always", TerminationGracePeriodSeconds:(*int64)(0xc0018738d8), ActiveDeadlineSeconds:(*int64)(nil), DNSPolicy:"ClusterFirst", NodeSelector:map[string]string{"kubernetes.io/os":"linux"}, ServiceAccountName:"kube-proxy", DeprecatedServiceAccount:"kube-proxy", AutomountServiceAccountToken:(*bool)(nil), NodeName:"", HostNetwork:true, HostPID:false, HostIPC:false, ShareProcessNamespace:(*bool)(nil), SecurityContext:(*v1.PodSecurityContext)(0xc000454cb0), ImagePullSecrets:[]v1.LocalObjectReference(nil), Hostname:"", Subdomain:"", Affinity:(*v1.Affinity)(nil), SchedulerName:"default-scheduler", Tolerations:[]v1.Toleration{v1.Toleration{Key:"CriticalAddonsOnly", Operator:"Exists", Value:"", Effect:"", TolerationSeconds:(*int64)(nil)}, v1.Toleration{Key:"", Operator:"Exists", Value:"", Effect:"", TolerationSeconds:(*int64)(nil)}}, HostAliases:[]v1.HostAlias(nil), PriorityClassName:"system-node-critical", Priority:(*int32)(nil), DNSConfig:(*v1.PodDNSConfig)(nil), ReadinessGates:[]v1.PodReadinessGate(nil), RuntimeClassName:(*string)(nil), EnableServiceLinks:(*bool)(nil), PreemptionPolicy:(*v1.PreemptionPolicy)(nil), Overhead:v1.ResourceList(nil), TopologySpreadConstraints:[]v1.TopologySpreadConstraint(nil)}}, UpdateStrategy:v1.DaemonSetUpdateStrategy{Type:"RollingUpdate", RollingUpdate:(*v1.RollingUpdateDaemonSet)(0xc00000eb80)}, MinReadySeconds:0, RevisionHistoryLimit:(*int32)(0xc001873928)}, Status:v1.DaemonSetStatus{CurrentNumberScheduled:0, NumberMisscheduled:0, DesiredNumberScheduled:0, NumberReady:0, ObservedGeneration:0, UpdatedNumberScheduled:0, NumberAvailable:0, NumberUnavailable:0, CollisionCount:(*int32)(nil), Conditions:[]v1.DaemonSetCondition(nil)}}: Operation cannot be fulfilled on daemonsets.apps "kube-proxy": the object has been modified; please apply your changes to the latest version and try again I0714 18:45:10.443501 1 shared_informer.go:230] Caches are synced for stateful set I0714 18:45:10.462021 1 shared_informer.go:230] Caches are synced for disruption I0714 18:45:10.462241 1 disruption.go:339] Sending events to api server. I0714 18:45:10.627573 1 shared_informer.go:230] Caches are synced for garbage collector I0714 18:45:10.647320 1 shared_informer.go:230] Caches are synced for garbage collector I0714 18:45:10.647374 1 garbagecollector.go:142] Garbage collector: all resource monitors have synced. Proceeding to collect garbage I0714 18:45:10.661630 1 shared_informer.go:230] Caches are synced for HPA I0714 18:45:10.718662 1 shared_informer.go:230] Caches are synced for resource quota I0714 18:45:11.761645 1 shared_informer.go:223] Waiting for caches to sync for resource quota I0714 18:45:11.761744 1 shared_informer.go:230] Caches are synced for resource quota I0714 18:45:15.221919 1 node_lifecycle_controller.go:1226] Controller detected that some Nodes are Ready. Exiting master disruption mode. I0714 18:45:32.335944 1 event.go:278] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"kubernetes-dashboard", Name:"dashboard-metrics-scraper", UID:"8f17b2bc-c434-48d7-bfde-77ced409fba0", APIVersion:"apps/v1", ResourceVersion:"398", FieldPath:""}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled up replica set dashboard-metrics-scraper-dc6947fbf to 1 I0714 18:45:32.359793 1 event.go:278] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"kubernetes-dashboard", Name:"dashboard-metrics-scraper-dc6947fbf", UID:"5fbf0929-b16b-45ae-a1d9-fb9809bdc000", APIVersion:"apps/v1", ResourceVersion:"399", FieldPath:""}): type: 'Warning' reason: 'FailedCreate' Error creating: pods "dashboard-metrics-scraper-dc6947fbf-" is forbidden: error looking up service account kubernetes-dashboard/kubernetes-dashboard: serviceaccount "kubernetes-dashboard" not found E0714 18:45:32.372994 1 replica_set.go:535] sync "kubernetes-dashboard/dashboard-metrics-scraper-dc6947fbf" failed with pods "dashboard-metrics-scraper-dc6947fbf-" is forbidden: error looking up service account kubernetes-dashboard/kubernetes-dashboard: serviceaccount "kubernetes-dashboard" not found I0714 18:45:32.373320 1 event.go:278] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"kubernetes-dashboard", Name:"kubernetes-dashboard", UID:"6fa5cf50-c564-4cfd-afd2-ae464b2c1e0b", APIVersion:"apps/v1", ResourceVersion:"403", FieldPath:""}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled up replica set kubernetes-dashboard-6dbb54fd95 to 1 E0714 18:45:32.379443 1 replica_set.go:535] sync "kubernetes-dashboard/dashboard-metrics-scraper-dc6947fbf" failed with pods "dashboard-metrics-scraper-dc6947fbf-" is forbidden: error looking up service account kubernetes-dashboard/kubernetes-dashboard: serviceaccount "kubernetes-dashboard" not found I0714 18:45:32.379975 1 event.go:278] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"kubernetes-dashboard", Name:"dashboard-metrics-scraper-dc6947fbf", UID:"5fbf0929-b16b-45ae-a1d9-fb9809bdc000", APIVersion:"apps/v1", ResourceVersion:"406", FieldPath:""}): type: 'Warning' reason: 'FailedCreate' Error creating: pods "dashboard-metrics-scraper-dc6947fbf-" is forbidden: error looking up service account kubernetes-dashboard/kubernetes-dashboard: serviceaccount "kubernetes-dashboard" not found I0714 18:45:32.385562 1 event.go:278] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"kubernetes-dashboard", Name:"kubernetes-dashboard-6dbb54fd95", UID:"99a624ba-e3a8-49e8-bb8e-4bd39ca3bc70", APIVersion:"apps/v1", ResourceVersion:"404", FieldPath:""}): type: 'Warning' reason: 'FailedCreate' Error creating: pods "kubernetes-dashboard-6dbb54fd95-" is forbidden: error looking up service account kubernetes-dashboard/kubernetes-dashboard: serviceaccount "kubernetes-dashboard" not found E0714 18:45:32.388443 1 replica_set.go:535] sync "kubernetes-dashboard/dashboard-metrics-scraper-dc6947fbf" failed with pods "dashboard-metrics-scraper-dc6947fbf-" is forbidden: error looking up service account kubernetes-dashboard/kubernetes-dashboard: serviceaccount "kubernetes-dashboard" not found I0714 18:45:32.388522 1 event.go:278] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"kubernetes-dashboard", Name:"dashboard-metrics-scraper-dc6947fbf", UID:"5fbf0929-b16b-45ae-a1d9-fb9809bdc000", APIVersion:"apps/v1", ResourceVersion:"406", FieldPath:""}): type: 'Warning' reason: 'FailedCreate' Error creating: pods "dashboard-metrics-scraper-dc6947fbf-" is forbidden: error looking up service account kubernetes-dashboard/kubernetes-dashboard: serviceaccount "kubernetes-dashboard" not found E0714 18:45:32.396809 1 replica_set.go:535] sync "kubernetes-dashboard/dashboard-metrics-scraper-dc6947fbf" failed with pods "dashboard-metrics-scraper-dc6947fbf-" is forbidden: error looking up service account kubernetes-dashboard/kubernetes-dashboard: serviceaccount "kubernetes-dashboard" not found I0714 18:45:32.396924 1 event.go:278] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"kubernetes-dashboard", Name:"dashboard-metrics-scraper-dc6947fbf", UID:"5fbf0929-b16b-45ae-a1d9-fb9809bdc000", APIVersion:"apps/v1", ResourceVersion:"406", FieldPath:""}): type: 'Warning' reason: 'FailedCreate' Error creating: pods "dashboard-metrics-scraper-dc6947fbf-" is forbidden: error looking up service account kubernetes-dashboard/kubernetes-dashboard: serviceaccount "kubernetes-dashboard" not found E0714 18:45:32.403425 1 replica_set.go:535] sync "kubernetes-dashboard/kubernetes-dashboard-6dbb54fd95" failed with pods "kubernetes-dashboard-6dbb54fd95-" is forbidden: error looking up service account kubernetes-dashboard/kubernetes-dashboard: serviceaccount "kubernetes-dashboard" not found I0714 18:45:32.408791 1 event.go:278] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"kubernetes-dashboard", Name:"kubernetes-dashboard-6dbb54fd95", UID:"99a624ba-e3a8-49e8-bb8e-4bd39ca3bc70", APIVersion:"apps/v1", ResourceVersion:"412", FieldPath:""}): type: 'Warning' reason: 'FailedCreate' Error creating: pods "kubernetes-dashboard-6dbb54fd95-" is forbidden: error looking up service account kubernetes-dashboard/kubernetes-dashboard: serviceaccount "kubernetes-dashboard" not found E0714 18:45:32.409271 1 replica_set.go:535] sync "kubernetes-dashboard/kubernetes-dashboard-6dbb54fd95" failed with pods "kubernetes-dashboard-6dbb54fd95-" is forbidden: error looking up service account kubernetes-dashboard/kubernetes-dashboard: serviceaccount "kubernetes-dashboard" not found I0714 18:45:32.442059 1 event.go:278] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"kubernetes-dashboard", Name:"dashboard-metrics-scraper-dc6947fbf", UID:"5fbf0929-b16b-45ae-a1d9-fb9809bdc000", APIVersion:"apps/v1", ResourceVersion:"406", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: dashboard-metrics-scraper-dc6947fbf-wxrw5 I0714 18:45:33.433426 1 event.go:278] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"kubernetes-dashboard", Name:"kubernetes-dashboard-6dbb54fd95", UID:"99a624ba-e3a8-49e8-bb8e-4bd39ca3bc70", APIVersion:"apps/v1", ResourceVersion:"412", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: kubernetes-dashboard-6dbb54fd95-zwd9r I0714 18:45:53.973903 1 event.go:278] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"kube-system", Name:"ingress-nginx-controller", UID:"e5708b4e-7756-40ec-bd8d-72d4b12a6bd4", APIVersion:"apps/v1", ResourceVersion:"492", FieldPath:""}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled up replica set ingress-nginx-controller-7bb4c67d67 to 1 I0714 18:45:53.988811 1 event.go:278] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"kube-system", Name:"ingress-nginx-controller-7bb4c67d67", UID:"ff0e40da-bdca-4c78-9c75-47392aed29a1", APIVersion:"apps/v1", ResourceVersion:"493", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: ingress-nginx-controller-7bb4c67d67-2ntwj I0714 18:45:54.055830 1 event.go:278] Event(v1.ObjectReference{Kind:"Job", Namespace:"kube-system", Name:"ingress-nginx-admission-patch", UID:"64f9e034-ecf5-40a2-b7fb-121aebd7b239", APIVersion:"batch/v1", ResourceVersion:"509", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: ingress-nginx-admission-patch-5ktvb I0714 18:45:54.057476 1 event.go:278] Event(v1.ObjectReference{Kind:"Job", Namespace:"kube-system", Name:"ingress-nginx-admission-create", UID:"6df9bd82-1f1a-4468-aa10-fd679f78457e", APIVersion:"batch/v1", ResourceVersion:"508", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: ingress-nginx-admission-create-h5njb ==> kube-proxy [124edbb8c84e] <== W0714 18:45:12.611834 1 server_others.go:559] Unknown proxy mode "", assuming iptables proxy I0714 18:45:12.623852 1 node.go:136] Successfully retrieved node IP: 192.168.64.5 I0714 18:45:12.623912 1 server_others.go:186] Using iptables Proxier. W0714 18:45:12.623935 1 server_others.go:436] detect-local-mode set to ClusterCIDR, but no cluster CIDR defined I0714 18:45:12.623940 1 server_others.go:447] detect-local-mode: ClusterCIDR , defaulting to no-op detect-local I0714 18:45:12.624515 1 server.go:583] Version: v1.18.3 I0714 18:45:12.625618 1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_max' to 131072 I0714 18:45:12.625671 1 conntrack.go:52] Setting nf_conntrack_max to 131072 I0714 18:45:12.625852 1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400 I0714 18:45:12.625960 1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600 I0714 18:45:12.626555 1 config.go:315] Starting service config controller I0714 18:45:12.626580 1 shared_informer.go:223] Waiting for caches to sync for service config I0714 18:45:12.627794 1 config.go:133] Starting endpoints config controller I0714 18:45:12.627835 1 shared_informer.go:223] Waiting for caches to sync for endpoints config I0714 18:45:12.728210 1 shared_informer.go:230] Caches are synced for endpoints config I0714 18:45:12.728223 1 shared_informer.go:230] Caches are synced for service config ==> kube-scheduler [70fbde97d58c] <== I0714 18:44:57.005495 1 registry.go:150] Registering EvenPodsSpread predicate and priority function I0714 18:44:57.005605 1 registry.go:150] Registering EvenPodsSpread predicate and priority function I0714 18:44:57.342903 1 serving.go:313] Generated self-signed cert in-memory W0714 18:45:00.793482 1 authentication.go:349] Unable to get configmap/extension-apiserver-authentication in kube-system. Usually fixed by 'kubectl create rolebinding -n kube-system ROLEBINDING_NAME --role=extension-apiserver-authentication-reader --serviceaccount=YOUR_NS:YOUR_SA' W0714 18:45:00.793538 1 authentication.go:297] Error looking up in-cluster authentication configuration: configmaps "extension-apiserver-authentication" is forbidden: User "system:kube-scheduler" cannot get resource "configmaps" in API group "" in the namespace "kube-system" W0714 18:45:00.793547 1 authentication.go:298] Continuing without authentication configuration. This may treat all requests as anonymous. W0714 18:45:00.793557 1 authentication.go:299] To require authentication configuration lookup to succeed, set --authentication-tolerate-lookup-failure=false I0714 18:45:00.812138 1 registry.go:150] Registering EvenPodsSpread predicate and priority function I0714 18:45:00.812268 1 registry.go:150] Registering EvenPodsSpread predicate and priority function W0714 18:45:00.814301 1 authorization.go:47] Authorization is disabled W0714 18:45:00.814332 1 authentication.go:40] Authentication is disabled I0714 18:45:00.814352 1 deprecated_insecure_serving.go:51] Serving healthz insecurely on [::]:10251 I0714 18:45:00.815591 1 secure_serving.go:178] Serving securely on 127.0.0.1:10259 I0714 18:45:00.815966 1 configmap_cafile_content.go:202] Starting client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0714 18:45:00.816024 1 shared_informer.go:223] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0714 18:45:00.816071 1 tlsconfig.go:240] Starting DynamicServingCertificateController E0714 18:45:00.818656 1 reflector.go:178] k8s.io/client-go/informers/factory.go:135: Failed to list *v1.PersistentVolume: persistentvolumes is forbidden: User "system:kube-scheduler" cannot list resource "persistentvolumes" in API group "" at the cluster scope E0714 18:45:00.819034 1 reflector.go:178] k8s.io/client-go/informers/factory.go:135: Failed to list *v1.Node: nodes is forbidden: User "system:kube-scheduler" cannot list resource "nodes" in API group "" at the cluster scope E0714 18:45:00.819563 1 reflector.go:178] k8s.io/client-go/informers/factory.go:135: Failed to list *v1.CSINode: csinodes.storage.k8s.io is forbidden: User "system:kube-scheduler" cannot list resource "csinodes" in API group "storage.k8s.io" at the cluster scope E0714 18:45:00.820523 1 reflector.go:178] k8s.io/client-go/informers/factory.go:135: Failed to list *v1.StorageClass: storageclasses.storage.k8s.io is forbidden: User "system:kube-scheduler" cannot list resource "storageclasses" in API group "storage.k8s.io" at the cluster scope E0714 18:45:00.820540 1 reflector.go:178] k8s.io/apiserver/pkg/server/dynamiccertificates/configmap_cafile_content.go:206: Failed to list *v1.ConfigMap: configmaps "extension-apiserver-authentication" is forbidden: User "system:kube-scheduler" cannot list resource "configmaps" in API group "" in the namespace "kube-system" E0714 18:45:00.820912 1 reflector.go:178] k8s.io/client-go/informers/factory.go:135: Failed to list *v1.PersistentVolumeClaim: persistentvolumeclaims is forbidden: User "system:kube-scheduler" cannot list resource "persistentvolumeclaims" in API group "" at the cluster scope E0714 18:45:00.821010 1 reflector.go:178] k8s.io/client-go/informers/factory.go:135: Failed to list *v1beta1.PodDisruptionBudget: poddisruptionbudgets.policy is forbidden: User "system:kube-scheduler" cannot list resource "poddisruptionbudgets" in API group "policy" at the cluster scope E0714 18:45:00.821160 1 reflector.go:178] k8s.io/client-go/informers/factory.go:135: Failed to list *v1.Service: services is forbidden: User "system:kube-scheduler" cannot list resource "services" in API group "" at the cluster scope E0714 18:45:00.822017 1 reflector.go:178] k8s.io/kubernetes/cmd/kube-scheduler/app/server.go:233: Failed to list *v1.Pod: pods is forbidden: User "system:kube-scheduler" cannot list resource "pods" in API group "" at the cluster scope E0714 18:45:01.632421 1 reflector.go:178] k8s.io/client-go/informers/factory.go:135: Failed to list *v1.CSINode: csinodes.storage.k8s.io is forbidden: User "system:kube-scheduler" cannot list resource "csinodes" in API group "storage.k8s.io" at the cluster scope E0714 18:45:01.770533 1 reflector.go:178] k8s.io/client-go/informers/factory.go:135: Failed to list *v1beta1.PodDisruptionBudget: poddisruptionbudgets.policy is forbidden: User "system:kube-scheduler" cannot list resource "poddisruptionbudgets" in API group "policy" at the cluster scope E0714 18:45:01.812081 1 reflector.go:178] k8s.io/apiserver/pkg/server/dynamiccertificates/configmap_cafile_content.go:206: Failed to list *v1.ConfigMap: configmaps "extension-apiserver-authentication" is forbidden: User "system:kube-scheduler" cannot list resource "configmaps" in API group "" in the namespace "kube-system" E0714 18:45:01.858606 1 reflector.go:178] k8s.io/client-go/informers/factory.go:135: Failed to list *v1.PersistentVolumeClaim: persistentvolumeclaims is forbidden: User "system:kube-scheduler" cannot list resource "persistentvolumeclaims" in API group "" at the cluster scope E0714 18:45:01.883705 1 reflector.go:178] k8s.io/client-go/informers/factory.go:135: Failed to list *v1.Service: services is forbidden: User "system:kube-scheduler" cannot list resource "services" in API group "" at the cluster scope E0714 18:45:04.734431 1 factory.go:503] pod: kube-system/storage-provisioner is already present in the active queue I0714 18:45:04.916317 1 shared_informer.go:230] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file E0714 18:45:10.152465 1 factory.go:503] pod: kube-system/coredns-66bff467f8-hw82w is already present in the active queue ==> kubelet <== -- Logs begin at Tue 2020-07-14 18:44:23 UTC, end at Tue 2020-07-14 18:48:09 UTC. -- Jul 14 18:45:54 minikube kubelet[3917]: I0714 18:45:54.091012 3917 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/7f2c72a2-cf51-4a12-837e-22ad3a410cfb-webhook-cert") pod "ingress-nginx-controller-7bb4c67d67-2ntwj" (UID: "7f2c72a2-cf51-4a12-837e-22ad3a410cfb") Jul 14 18:45:54 minikube kubelet[3917]: I0714 18:45:54.091149 3917 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "ingress-nginx-admission-token-jq9cn" (UniqueName: "kubernetes.io/secret/fb45013b-134d-4e11-84c0-f3ac72169f32-ingress-nginx-admission-token-jq9cn") pod "ingress-nginx-admission-patch-5ktvb" (UID: "fb45013b-134d-4e11-84c0-f3ac72169f32") Jul 14 18:45:54 minikube kubelet[3917]: I0714 18:45:54.091282 3917 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "ingress-nginx-admission-token-jq9cn" (UniqueName: "kubernetes.io/secret/23c647b4-1296-42a7-b7a8-7b693b044054-ingress-nginx-admission-token-jq9cn") pod "ingress-nginx-admission-create-h5njb" (UID: "23c647b4-1296-42a7-b7a8-7b693b044054") Jul 14 18:45:54 minikube kubelet[3917]: E0714 18:45:54.192086 3917 secret.go:195] Couldn't get secret kube-system/ingress-nginx-admission: secret "ingress-nginx-admission" not found Jul 14 18:45:54 minikube kubelet[3917]: E0714 18:45:54.192250 3917 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/secret/7f2c72a2-cf51-4a12-837e-22ad3a410cfb-webhook-cert podName:7f2c72a2-cf51-4a12-837e-22ad3a410cfb nodeName:}" failed. No retries permitted until 2020-07-14 18:45:54.692188018 +0000 UTC m=+51.117575668 (durationBeforeRetry 500ms). Error: "MountVolume.SetUp failed for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7f2c72a2-cf51-4a12-837e-22ad3a410cfb-webhook-cert\") pod \"ingress-nginx-controller-7bb4c67d67-2ntwj\" (UID: \"7f2c72a2-cf51-4a12-837e-22ad3a410cfb\") : secret \"ingress-nginx-admission\" not found" Jul 14 18:45:54 minikube kubelet[3917]: E0714 18:45:54.701222 3917 secret.go:195] Couldn't get secret kube-system/ingress-nginx-admission: secret "ingress-nginx-admission" not found Jul 14 18:45:54 minikube kubelet[3917]: E0714 18:45:54.701305 3917 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/secret/7f2c72a2-cf51-4a12-837e-22ad3a410cfb-webhook-cert podName:7f2c72a2-cf51-4a12-837e-22ad3a410cfb nodeName:}" failed. No retries permitted until 2020-07-14 18:45:55.701285379 +0000 UTC m=+52.126673013 (durationBeforeRetry 1s). Error: "MountVolume.SetUp failed for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7f2c72a2-cf51-4a12-837e-22ad3a410cfb-webhook-cert\") pod \"ingress-nginx-controller-7bb4c67d67-2ntwj\" (UID: \"7f2c72a2-cf51-4a12-837e-22ad3a410cfb\") : secret \"ingress-nginx-admission\" not found" Jul 14 18:45:54 minikube kubelet[3917]: W0714 18:45:54.874237 3917 docker_sandbox.go:400] failed to read pod IP from plugin/docker: Couldn't find network status for kube-system/ingress-nginx-admission-create-h5njb through plugin: invalid network status for Jul 14 18:45:54 minikube kubelet[3917]: W0714 18:45:54.903745 3917 docker_sandbox.go:400] failed to read pod IP from plugin/docker: Couldn't find network status for kube-system/ingress-nginx-admission-patch-5ktvb through plugin: invalid network status for Jul 14 18:45:55 minikube kubelet[3917]: W0714 18:45:55.152007 3917 docker_sandbox.go:400] failed to read pod IP from plugin/docker: Couldn't find network status for kube-system/ingress-nginx-admission-create-h5njb through plugin: invalid network status for Jul 14 18:45:55 minikube kubelet[3917]: W0714 18:45:55.154772 3917 docker_sandbox.go:400] failed to read pod IP from plugin/docker: Couldn't find network status for kube-system/ingress-nginx-admission-patch-5ktvb through plugin: invalid network status for Jul 14 18:45:55 minikube kubelet[3917]: E0714 18:45:55.717014 3917 secret.go:195] Couldn't get secret kube-system/ingress-nginx-admission: secret "ingress-nginx-admission" not found Jul 14 18:45:55 minikube kubelet[3917]: E0714 18:45:55.717389 3917 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/secret/7f2c72a2-cf51-4a12-837e-22ad3a410cfb-webhook-cert podName:7f2c72a2-cf51-4a12-837e-22ad3a410cfb nodeName:}" failed. No retries permitted until 2020-07-14 18:45:57.717352168 +0000 UTC m=+54.142739837 (durationBeforeRetry 2s). Error: "MountVolume.SetUp failed for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7f2c72a2-cf51-4a12-837e-22ad3a410cfb-webhook-cert\") pod \"ingress-nginx-controller-7bb4c67d67-2ntwj\" (UID: \"7f2c72a2-cf51-4a12-837e-22ad3a410cfb\") : secret \"ingress-nginx-admission\" not found" Jul 14 18:45:57 minikube kubelet[3917]: E0714 18:45:57.737752 3917 secret.go:195] Couldn't get secret kube-system/ingress-nginx-admission: secret "ingress-nginx-admission" not found Jul 14 18:45:57 minikube kubelet[3917]: E0714 18:45:57.737946 3917 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/secret/7f2c72a2-cf51-4a12-837e-22ad3a410cfb-webhook-cert podName:7f2c72a2-cf51-4a12-837e-22ad3a410cfb nodeName:}" failed. No retries permitted until 2020-07-14 18:46:01.737924149 +0000 UTC m=+58.163311794 (durationBeforeRetry 4s). Error: "MountVolume.SetUp failed for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7f2c72a2-cf51-4a12-837e-22ad3a410cfb-webhook-cert\") pod \"ingress-nginx-controller-7bb4c67d67-2ntwj\" (UID: \"7f2c72a2-cf51-4a12-837e-22ad3a410cfb\") : secret \"ingress-nginx-admission\" not found" Jul 14 18:46:01 minikube kubelet[3917]: E0714 18:46:01.778192 3917 secret.go:195] Couldn't get secret kube-system/ingress-nginx-admission: secret "ingress-nginx-admission" not found Jul 14 18:46:01 minikube kubelet[3917]: E0714 18:46:01.778574 3917 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/secret/7f2c72a2-cf51-4a12-837e-22ad3a410cfb-webhook-cert podName:7f2c72a2-cf51-4a12-837e-22ad3a410cfb nodeName:}" failed. No retries permitted until 2020-07-14 18:46:09.778521994 +0000 UTC m=+66.203909687 (durationBeforeRetry 8s). Error: "MountVolume.SetUp failed for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7f2c72a2-cf51-4a12-837e-22ad3a410cfb-webhook-cert\") pod \"ingress-nginx-controller-7bb4c67d67-2ntwj\" (UID: \"7f2c72a2-cf51-4a12-837e-22ad3a410cfb\") : secret \"ingress-nginx-admission\" not found" Jul 14 18:46:09 minikube kubelet[3917]: E0714 18:46:09.853938 3917 secret.go:195] Couldn't get secret kube-system/ingress-nginx-admission: secret "ingress-nginx-admission" not found Jul 14 18:46:09 minikube kubelet[3917]: E0714 18:46:09.854264 3917 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/secret/7f2c72a2-cf51-4a12-837e-22ad3a410cfb-webhook-cert podName:7f2c72a2-cf51-4a12-837e-22ad3a410cfb nodeName:}" failed. No retries permitted until 2020-07-14 18:46:25.854227861 +0000 UTC m=+82.279615614 (durationBeforeRetry 16s). Error: "MountVolume.SetUp failed for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7f2c72a2-cf51-4a12-837e-22ad3a410cfb-webhook-cert\") pod \"ingress-nginx-controller-7bb4c67d67-2ntwj\" (UID: \"7f2c72a2-cf51-4a12-837e-22ad3a410cfb\") : secret \"ingress-nginx-admission\" not found" Jul 14 18:46:09 minikube kubelet[3917]: E0714 18:46:09.892805 3917 remote_image.go:113] PullImage "jettech/kube-webhook-certgen:v1.2.0" from image service failed: rpc error: code = Unknown desc = Error response from daemon: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) Jul 14 18:46:09 minikube kubelet[3917]: E0714 18:46:09.892913 3917 kuberuntime_image.go:50] Pull image "jettech/kube-webhook-certgen:v1.2.0" failed: rpc error: code = Unknown desc = Error response from daemon: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) Jul 14 18:46:09 minikube kubelet[3917]: E0714 18:46:09.893362 3917 kuberuntime_manager.go:801] container start failed: ErrImagePull: rpc error: code = Unknown desc = Error response from daemon: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) Jul 14 18:46:09 minikube kubelet[3917]: E0714 18:46:09.894227 3917 pod_workers.go:191] Error syncing pod 23c647b4-1296-42a7-b7a8-7b693b044054 ("ingress-nginx-admission-create-h5njb_kube-system(23c647b4-1296-42a7-b7a8-7b693b044054)"), skipping: failed to "StartContainer" for "create" with ErrImagePull: "rpc error: code = Unknown desc = Error response from daemon: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Jul 14 18:46:10 minikube kubelet[3917]: E0714 18:46:10.347604 3917 pod_workers.go:191] Error syncing pod 23c647b4-1296-42a7-b7a8-7b693b044054 ("ingress-nginx-admission-create-h5njb_kube-system(23c647b4-1296-42a7-b7a8-7b693b044054)"), skipping: failed to "StartContainer" for "create" with ImagePullBackOff: "Back-off pulling image \"jettech/kube-webhook-certgen:v1.2.0\"" Jul 14 18:46:24 minikube kubelet[3917]: E0714 18:46:24.898460 3917 remote_image.go:113] PullImage "jettech/kube-webhook-certgen:v1.2.0" from image service failed: rpc error: code = Unknown desc = Error response from daemon: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) Jul 14 18:46:24 minikube kubelet[3917]: E0714 18:46:24.898612 3917 kuberuntime_image.go:50] Pull image "jettech/kube-webhook-certgen:v1.2.0" failed: rpc error: code = Unknown desc = Error response from daemon: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) Jul 14 18:46:24 minikube kubelet[3917]: E0714 18:46:24.898819 3917 kuberuntime_manager.go:801] container start failed: ErrImagePull: rpc error: code = Unknown desc = Error response from daemon: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) Jul 14 18:46:24 minikube kubelet[3917]: E0714 18:46:24.898880 3917 pod_workers.go:191] Error syncing pod fb45013b-134d-4e11-84c0-f3ac72169f32 ("ingress-nginx-admission-patch-5ktvb_kube-system(fb45013b-134d-4e11-84c0-f3ac72169f32)"), skipping: failed to "StartContainer" for "patch" with ErrImagePull: "rpc error: code = Unknown desc = Error response from daemon: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Jul 14 18:46:25 minikube kubelet[3917]: E0714 18:46:25.492775 3917 pod_workers.go:191] Error syncing pod fb45013b-134d-4e11-84c0-f3ac72169f32 ("ingress-nginx-admission-patch-5ktvb_kube-system(fb45013b-134d-4e11-84c0-f3ac72169f32)"), skipping: failed to "StartContainer" for "patch" with ImagePullBackOff: "Back-off pulling image \"jettech/kube-webhook-certgen:v1.2.0\"" Jul 14 18:46:25 minikube kubelet[3917]: E0714 18:46:25.943628 3917 secret.go:195] Couldn't get secret kube-system/ingress-nginx-admission: secret "ingress-nginx-admission" not found Jul 14 18:46:25 minikube kubelet[3917]: E0714 18:46:25.943792 3917 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/secret/7f2c72a2-cf51-4a12-837e-22ad3a410cfb-webhook-cert podName:7f2c72a2-cf51-4a12-837e-22ad3a410cfb nodeName:}" failed. No retries permitted until 2020-07-14 18:46:57.94376101 +0000 UTC m=+114.369148666 (durationBeforeRetry 32s). Error: "MountVolume.SetUp failed for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7f2c72a2-cf51-4a12-837e-22ad3a410cfb-webhook-cert\") pod \"ingress-nginx-controller-7bb4c67d67-2ntwj\" (UID: \"7f2c72a2-cf51-4a12-837e-22ad3a410cfb\") : secret \"ingress-nginx-admission\" not found" Jul 14 18:46:40 minikube kubelet[3917]: E0714 18:46:40.243759 3917 remote_image.go:113] PullImage "jettech/kube-webhook-certgen:v1.2.0" from image service failed: rpc error: code = Unknown desc = Error response from daemon: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) Jul 14 18:46:40 minikube kubelet[3917]: E0714 18:46:40.244538 3917 kuberuntime_image.go:50] Pull image "jettech/kube-webhook-certgen:v1.2.0" failed: rpc error: code = Unknown desc = Error response from daemon: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) Jul 14 18:46:40 minikube kubelet[3917]: E0714 18:46:40.245017 3917 kuberuntime_manager.go:801] container start failed: ErrImagePull: rpc error: code = Unknown desc = Error response from daemon: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) Jul 14 18:46:40 minikube kubelet[3917]: E0714 18:46:40.245192 3917 pod_workers.go:191] Error syncing pod 23c647b4-1296-42a7-b7a8-7b693b044054 ("ingress-nginx-admission-create-h5njb_kube-system(23c647b4-1296-42a7-b7a8-7b693b044054)"), skipping: failed to "StartContainer" for "create" with ErrImagePull: "rpc error: code = Unknown desc = Error response from daemon: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Jul 14 18:46:54 minikube kubelet[3917]: E0714 18:46:54.235956 3917 pod_workers.go:191] Error syncing pod 23c647b4-1296-42a7-b7a8-7b693b044054 ("ingress-nginx-admission-create-h5njb_kube-system(23c647b4-1296-42a7-b7a8-7b693b044054)"), skipping: failed to "StartContainer" for "create" with ImagePullBackOff: "Back-off pulling image \"jettech/kube-webhook-certgen:v1.2.0\"" Jul 14 18:46:55 minikube kubelet[3917]: E0714 18:46:55.248209 3917 remote_image.go:113] PullImage "jettech/kube-webhook-certgen:v1.2.0" from image service failed: rpc error: code = Unknown desc = Error response from daemon: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) Jul 14 18:46:55 minikube kubelet[3917]: E0714 18:46:55.249132 3917 kuberuntime_image.go:50] Pull image "jettech/kube-webhook-certgen:v1.2.0" failed: rpc error: code = Unknown desc = Error response from daemon: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) Jul 14 18:46:55 minikube kubelet[3917]: E0714 18:46:55.249296 3917 kuberuntime_manager.go:801] container start failed: ErrImagePull: rpc error: code = Unknown desc = Error response from daemon: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) Jul 14 18:46:55 minikube kubelet[3917]: E0714 18:46:55.249427 3917 pod_workers.go:191] Error syncing pod fb45013b-134d-4e11-84c0-f3ac72169f32 ("ingress-nginx-admission-patch-5ktvb_kube-system(fb45013b-134d-4e11-84c0-f3ac72169f32)"), skipping: failed to "StartContainer" for "patch" with ErrImagePull: "rpc error: code = Unknown desc = Error response from daemon: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Jul 14 18:46:57 minikube kubelet[3917]: E0714 18:46:57.992006 3917 secret.go:195] Couldn't get secret kube-system/ingress-nginx-admission: secret "ingress-nginx-admission" not found Jul 14 18:46:57 minikube kubelet[3917]: E0714 18:46:57.992451 3917 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/secret/7f2c72a2-cf51-4a12-837e-22ad3a410cfb-webhook-cert podName:7f2c72a2-cf51-4a12-837e-22ad3a410cfb nodeName:}" failed. No retries permitted until 2020-07-14 18:48:01.992400606 +0000 UTC m=+178.417788299 (durationBeforeRetry 1m4s). Error: "MountVolume.SetUp failed for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7f2c72a2-cf51-4a12-837e-22ad3a410cfb-webhook-cert\") pod \"ingress-nginx-controller-7bb4c67d67-2ntwj\" (UID: \"7f2c72a2-cf51-4a12-837e-22ad3a410cfb\") : secret \"ingress-nginx-admission\" not found" Jul 14 18:47:10 minikube kubelet[3917]: E0714 18:47:10.237314 3917 pod_workers.go:191] Error syncing pod fb45013b-134d-4e11-84c0-f3ac72169f32 ("ingress-nginx-admission-patch-5ktvb_kube-system(fb45013b-134d-4e11-84c0-f3ac72169f32)"), skipping: failed to "StartContainer" for "patch" with ImagePullBackOff: "Back-off pulling image \"jettech/kube-webhook-certgen:v1.2.0\"" Jul 14 18:47:23 minikube kubelet[3917]: E0714 18:47:23.237529 3917 remote_image.go:113] PullImage "jettech/kube-webhook-certgen:v1.2.0" from image service failed: rpc error: code = Unknown desc = Error response from daemon: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) Jul 14 18:47:23 minikube kubelet[3917]: E0714 18:47:23.237575 3917 kuberuntime_image.go:50] Pull image "jettech/kube-webhook-certgen:v1.2.0" failed: rpc error: code = Unknown desc = Error response from daemon: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) Jul 14 18:47:23 minikube kubelet[3917]: E0714 18:47:23.237744 3917 kuberuntime_manager.go:801] container start failed: ErrImagePull: rpc error: code = Unknown desc = Error response from daemon: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) Jul 14 18:47:23 minikube kubelet[3917]: E0714 18:47:23.237788 3917 pod_workers.go:191] Error syncing pod 23c647b4-1296-42a7-b7a8-7b693b044054 ("ingress-nginx-admission-create-h5njb_kube-system(23c647b4-1296-42a7-b7a8-7b693b044054)"), skipping: failed to "StartContainer" for "create" with ErrImagePull: "rpc error: code = Unknown desc = Error response from daemon: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Jul 14 18:47:35 minikube kubelet[3917]: E0714 18:47:35.232846 3917 pod_workers.go:191] Error syncing pod 23c647b4-1296-42a7-b7a8-7b693b044054 ("ingress-nginx-admission-create-h5njb_kube-system(23c647b4-1296-42a7-b7a8-7b693b044054)"), skipping: failed to "StartContainer" for "create" with ImagePullBackOff: "Back-off pulling image \"jettech/kube-webhook-certgen:v1.2.0\"" Jul 14 18:47:38 minikube kubelet[3917]: E0714 18:47:38.251770 3917 remote_image.go:113] PullImage "jettech/kube-webhook-certgen:v1.2.0" from image service failed: rpc error: code = Unknown desc = Error response from daemon: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) Jul 14 18:47:38 minikube kubelet[3917]: E0714 18:47:38.251816 3917 kuberuntime_image.go:50] Pull image "jettech/kube-webhook-certgen:v1.2.0" failed: rpc error: code = Unknown desc = Error response from daemon: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) Jul 14 18:47:38 minikube kubelet[3917]: E0714 18:47:38.251887 3917 kuberuntime_manager.go:801] container start failed: ErrImagePull: rpc error: code = Unknown desc = Error response from daemon: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) Jul 14 18:47:38 minikube kubelet[3917]: E0714 18:47:38.251915 3917 pod_workers.go:191] Error syncing pod fb45013b-134d-4e11-84c0-f3ac72169f32 ("ingress-nginx-admission-patch-5ktvb_kube-system(fb45013b-134d-4e11-84c0-f3ac72169f32)"), skipping: failed to "StartContainer" for "patch" with ErrImagePull: "rpc error: code = Unknown desc = Error response from daemon: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Jul 14 18:47:47 minikube kubelet[3917]: E0714 18:47:47.234809 3917 pod_workers.go:191] Error syncing pod 23c647b4-1296-42a7-b7a8-7b693b044054 ("ingress-nginx-admission-create-h5njb_kube-system(23c647b4-1296-42a7-b7a8-7b693b044054)"), skipping: failed to "StartContainer" for "create" with ImagePullBackOff: "Back-off pulling image \"jettech/kube-webhook-certgen:v1.2.0\"" Jul 14 18:47:50 minikube kubelet[3917]: E0714 18:47:50.237581 3917 pod_workers.go:191] Error syncing pod fb45013b-134d-4e11-84c0-f3ac72169f32 ("ingress-nginx-admission-patch-5ktvb_kube-system(fb45013b-134d-4e11-84c0-f3ac72169f32)"), skipping: failed to "StartContainer" for "patch" with ImagePullBackOff: "Back-off pulling image \"jettech/kube-webhook-certgen:v1.2.0\"" Jul 14 18:47:57 minikube kubelet[3917]: E0714 18:47:57.012541 3917 kubelet.go:1681] Unable to attach or mount volumes for pod "ingress-nginx-controller-7bb4c67d67-2ntwj_kube-system(7f2c72a2-cf51-4a12-837e-22ad3a410cfb)": unmounted volumes=[webhook-cert], unattached volumes=[webhook-cert ingress-nginx-token-bvdpx]: timed out waiting for the condition; skipping pod Jul 14 18:47:57 minikube kubelet[3917]: E0714 18:47:57.012687 3917 pod_workers.go:191] Error syncing pod 7f2c72a2-cf51-4a12-837e-22ad3a410cfb ("ingress-nginx-controller-7bb4c67d67-2ntwj_kube-system(7f2c72a2-cf51-4a12-837e-22ad3a410cfb)"), skipping: unmounted volumes=[webhook-cert], unattached volumes=[webhook-cert ingress-nginx-token-bvdpx]: timed out waiting for the condition Jul 14 18:48:01 minikube kubelet[3917]: E0714 18:48:01.233333 3917 pod_workers.go:191] Error syncing pod fb45013b-134d-4e11-84c0-f3ac72169f32 ("ingress-nginx-admission-patch-5ktvb_kube-system(fb45013b-134d-4e11-84c0-f3ac72169f32)"), skipping: failed to "StartContainer" for "patch" with ImagePullBackOff: "Back-off pulling image \"jettech/kube-webhook-certgen:v1.2.0\"" Jul 14 18:48:02 minikube kubelet[3917]: E0714 18:48:02.060797 3917 secret.go:195] Couldn't get secret kube-system/ingress-nginx-admission: secret "ingress-nginx-admission" not found Jul 14 18:48:02 minikube kubelet[3917]: E0714 18:48:02.060994 3917 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/secret/7f2c72a2-cf51-4a12-837e-22ad3a410cfb-webhook-cert podName:7f2c72a2-cf51-4a12-837e-22ad3a410cfb nodeName:}" failed. No retries permitted until 2020-07-14 18:50:04.060954863 +0000 UTC m=+300.486342527 (durationBeforeRetry 2m2s). Error: "MountVolume.SetUp failed for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7f2c72a2-cf51-4a12-837e-22ad3a410cfb-webhook-cert\") pod \"ingress-nginx-controller-7bb4c67d67-2ntwj\" (UID: \"7f2c72a2-cf51-4a12-837e-22ad3a410cfb\") : secret \"ingress-nginx-admission\" not found" Jul 14 18:48:02 minikube kubelet[3917]: E0714 18:48:02.241149 3917 pod_workers.go:191] Error syncing pod 23c647b4-1296-42a7-b7a8-7b693b044054 ("ingress-nginx-admission-create-h5njb_kube-system(23c647b4-1296-42a7-b7a8-7b693b044054)"), skipping: failed to "StartContainer" for "create" with ImagePullBackOff: "Back-off pulling image \"jettech/kube-webhook-certgen:v1.2.0\"" ==> kubernetes-dashboard [a09c578d276e] <== 2020/07/14 18:45:39 Getting list persistent volumes claims 2020/07/14 18:45:39 Internal error occurred: No metric client provided. Skipping metrics. 2020/07/14 18:45:39 [2020-07-14T18:45:39Z] Outcoming response to 172.17.0.1:51772 with 200 status code 2020/07/14 18:45:39 Internal error occurred: No metric client provided. Skipping metrics. 2020/07/14 18:45:39 Internal error occurred: No metric client provided. Skipping metrics. 2020/07/14 18:45:39 [2020-07-14T18:45:39Z] Outcoming response to 172.17.0.1:51774 with 200 status code 2020/07/14 18:45:39 [2020-07-14T18:45:39Z] Outcoming response to 172.17.0.1:51766 with 200 status code 2020/07/14 18:45:39 [2020-07-14T18:45:39Z] Outcoming response to 172.17.0.1:51776 with 200 status code 2020/07/14 18:45:39 [2020-07-14T18:45:39Z] Outcoming response to 172.17.0.1:51744 with 200 status code 2020/07/14 18:45:44 [2020-07-14T18:45:44Z] Incoming HTTP/1.1 GET /api/v1/namespace request from 172.17.0.1:51766: 2020/07/14 18:45:44 Getting list of namespaces 2020/07/14 18:45:44 [2020-07-14T18:45:44Z] Incoming HTTP/1.1 GET /api/v1/job/default?itemsPerPage=10&page=1&sortBy=d,creationTimestamp request from 172.17.0.1:51794: 2020/07/14 18:45:44 Getting list of all jobs in the cluster 2020/07/14 18:45:44 [2020-07-14T18:45:44Z] Incoming HTTP/1.1 GET /api/v1/cronjob/default?itemsPerPage=10&page=1&sortBy=d,creationTimestamp request from 172.17.0.1:51790: 2020/07/14 18:45:44 Getting list of all cron jobs in the cluster 2020/07/14 18:45:44 [2020-07-14T18:45:44Z] Incoming HTTP/1.1 GET /api/v1/daemonset/default?itemsPerPage=10&page=1&sortBy=d,creationTimestamp request from 172.17.0.1:51770: 2020/07/14 18:45:44 [2020-07-14T18:45:44Z] Outcoming response to 172.17.0.1:51766 with 200 status code 2020/07/14 18:45:44 [2020-07-14T18:45:44Z] Incoming HTTP/1.1 GET /api/v1/deployment/default?itemsPerPage=10&page=1&sortBy=d,creationTimestamp request from 172.17.0.1:51792: 2020/07/14 18:45:44 Getting list of all deployments in the cluster 2020/07/14 18:45:44 [2020-07-14T18:45:44Z] Incoming HTTP/1.1 GET /api/v1/pod/default?itemsPerPage=10&page=1&sortBy=d,creationTimestamp request from 172.17.0.1:51796: 2020/07/14 18:45:44 Getting list of all pods in the cluster 2020/07/14 18:45:44 Internal error occurred: No metric client provided. Skipping metrics. 2020/07/14 18:45:44 [2020-07-14T18:45:44Z] Outcoming response to 172.17.0.1:51790 with 200 status code 2020/07/14 18:45:44 [2020-07-14T18:45:44Z] Incoming HTTP/1.1 GET /api/v1/replicaset/default?itemsPerPage=10&page=1&sortBy=d,creationTimestamp request from 172.17.0.1:51766: 2020/07/14 18:45:44 Getting list of all replica sets in the cluster 2020/07/14 18:45:44 Internal error occurred: No metric client provided. Skipping metrics. 2020/07/14 18:45:44 Internal error occurred: No metric client provided. Skipping metrics. 2020/07/14 18:45:44 [2020-07-14T18:45:44Z] Outcoming response to 172.17.0.1:51770 with 200 status code 2020/07/14 18:45:44 [2020-07-14T18:45:44Z] Outcoming response to 172.17.0.1:51794 with 200 status code 2020/07/14 18:45:44 Internal error occurred: No metric client provided. Skipping metrics. 2020/07/14 18:45:44 [2020-07-14T18:45:44Z] Outcoming response to 172.17.0.1:51792 with 200 status code 2020/07/14 18:45:44 Internal error occurred: No metric client provided. Skipping metrics. 2020/07/14 18:45:44 Getting pod metrics 2020/07/14 18:45:44 Internal error occurred: No metric client provided. Skipping metrics. 2020/07/14 18:45:44 [2020-07-14T18:45:44Z] Incoming HTTP/1.1 GET /api/v1/replicationcontroller/default?itemsPerPage=10&page=1&sortBy=d,creationTimestamp request from 172.17.0.1:51770: 2020/07/14 18:45:44 Getting list of all replication controllers in the cluster 2020/07/14 18:45:44 [2020-07-14T18:45:44Z] Outcoming response to 172.17.0.1:51796 with 200 status code 2020/07/14 18:45:44 [2020-07-14T18:45:44Z] Incoming HTTP/1.1 GET /api/v1/statefulset/default?itemsPerPage=10&page=1&sortBy=d,creationTimestamp request from 172.17.0.1:51790: 2020/07/14 18:45:44 Getting list of all pet sets in the cluster 2020/07/14 18:45:44 [2020-07-14T18:45:44Z] Incoming HTTP/1.1 GET /api/v1/ingress/default?itemsPerPage=10&page=1&sortBy=d,creationTimestamp request from 172.17.0.1:51796: 2020/07/14 18:45:44 [2020-07-14T18:45:44Z] Incoming HTTP/1.1 GET /api/v1/configmap/default?itemsPerPage=10&page=1&sortBy=d,creationTimestamp request from 172.17.0.1:51800: 2020/07/14 18:45:44 Getting list config maps in the namespace default 2020/07/14 18:45:44 [2020-07-14T18:45:44Z] Incoming HTTP/1.1 GET /api/v1/service/default?itemsPerPage=10&page=1&sortBy=d,creationTimestamp request from 172.17.0.1:51798: 2020/07/14 18:45:44 Getting list of all services in the cluster 2020/07/14 18:45:44 [2020-07-14T18:45:44Z] Outcoming response to 172.17.0.1:51796 with 200 status code 2020/07/14 18:45:44 Internal error occurred: No metric client provided. Skipping metrics. 2020/07/14 18:45:44 [2020-07-14T18:45:44Z] Outcoming response to 172.17.0.1:51766 with 200 status code 2020/07/14 18:45:44 [2020-07-14T18:45:44Z] Outcoming response to 172.17.0.1:51800 with 200 status code 2020/07/14 18:45:44 Internal error occurred: No metric client provided. Skipping metrics. 2020/07/14 18:45:44 [2020-07-14T18:45:44Z] Outcoming response to 172.17.0.1:51770 with 200 status code 2020/07/14 18:45:44 [2020-07-14T18:45:44Z] Outcoming response to 172.17.0.1:51798 with 200 status code 2020/07/14 18:45:44 Internal error occurred: No metric client provided. Skipping metrics. 2020/07/14 18:45:44 [2020-07-14T18:45:44Z] Outcoming response to 172.17.0.1:51790 with 200 status code 2020/07/14 18:45:44 [2020-07-14T18:45:44Z] Incoming HTTP/1.1 GET /api/v1/persistentvolumeclaim/default?itemsPerPage=10&page=1&sortBy=d,creationTimestamp request from 172.17.0.1:51766: 2020/07/14 18:45:44 Getting list persistent volumes claims 2020/07/14 18:45:44 [2020-07-14T18:45:44Z] Incoming HTTP/1.1 GET /api/v1/secret/default?itemsPerPage=10&page=1&sortBy=d,creationTimestamp request from 172.17.0.1:51796: 2020/07/14 18:45:44 Getting list of secrets in &{[default]} namespace 2020/07/14 18:45:44 [2020-07-14T18:45:44Z] Outcoming response to 172.17.0.1:51796 with 200 status code 2020/07/14 18:45:44 [2020-07-14T18:45:44Z] Outcoming response to 172.17.0.1:51766 with 200 status code 2020/07/14 18:46:04 Successful request to sidecar

[priyawadhwa]

Hey @happy-machine thanks for opening this issue.

I noticed you have both the --driver=docker and --vm=true flags appended to minikube start. Those are opposing flags, since --driver=docker tells minikube to run in a docker container, while --vm=true tells minikube to run in a VM. I'd remove whichever one you don't want.

Could you also run:

minikube delete
minikube start <additonal flags>

and then provide the output of:

minikube addons enable ingress --alsologtostderr
kubectl get po -A

[priyawadhwa]

Hopefully it's OK if I close this - there wasn't enough information to make it actionable, and some time has already passed. If you are able to provide additional details, you may reopen it at any point by adding /reopen to your comment.

Here is additional information that may be helpful to us:

• Whether the issue occurs with the latest minikube release
• The output of the suggestions in my previous comment

Thank you for sharing your experience!

[manas-mukherjee]

Hi @priyawadhwa,

I am facing the same timeout issue.

Here is the log which you requested above.

$ minikube addons enable ingress --alsologtostderr I1214 10:35:45.501549 50344 out.go:185] Setting OutFile to fd 1 ... I1214 10:35:45.502042 50344 out.go:237] isatty.IsTerminal(1) = true I1214 10:35:45.502050 50344 out.go:198] Setting ErrFile to fd 2... I1214 10:35:45.502055 50344 out.go:237] isatty.IsTerminal(2) = true I1214 10:35:45.502143 50344 root.go:279] Updating PATH: /Users/mmukherjee/.minikube/bin I1214 10:35:45.502935 50344 addons.go:55] Setting ingress=true in profile "minikube" I1214 10:35:45.502950 50344 addons.go:131] Setting addon ingress=true in "minikube" I1214 10:35:45.502963 50344 host.go:66] Checking if "minikube" exists ... I1214 10:35:45.503691 50344 main.go:119] libmachine: COMMAND: /usr/local/bin/VBoxManage showvminfo minikube --machinereadable I1214 10:35:45.611827 50344 main.go:119] libmachine: STDOUT: { name="minikube" groups="/" ostype="Linux 2.6 / 3.x / 4.x (64-bit)" UUID="4c9440b5-e810-4977-8211-ccbb70c9e119" CfgFile="/Users/mmukherjee/.minikube/machines/minikube/minikube/minikube.vbox" SnapFldr="/Users/mmukherjee/.minikube/machines/minikube/minikube/Snapshots" LogFldr="/Users/mmukherjee/.minikube/machines/minikube/minikube/Logs" hardwareuuid="4c9440b5-e810-4977-8211-ccbb70c9e119" memory=6000 pagefusion="off" vram=8 cpuexecutioncap=100 hpet="on" cpu-profile="host" chipset="piix3" firmware="BIOS" cpus=2 pae="on" longmode="on" triplefaultreset="off" apic="on" x2apic="off" nested-hw-virt="off" cpuid-portability-level=0 bootmenu="disabled" boot1="dvd" boot2="dvd" boot3="disk" boot4="none" acpi="on" ioapic="on" biosapic="apic" biossystemtimeoffset=0 rtcuseutc="on" hwvirtex="on" nestedpaging="on" largepages="on" vtxvpid="on" vtxux="on" paravirtprovider="default" effparavirtprovider="kvm" VMState="running" VMStateChangeTime="2020-12-14T04:55:58.868000000" graphicscontroller="vboxvga" monitorcount=1 accelerate3d="off" accelerate2dvideo="off" teleporterenabled="off" teleporterport=0 teleporteraddress="" teleporterpassword="" tracing-enabled="off" tracing-allow-vm-access="off" tracing-config="" autostart-enabled="off" autostart-delay=0 defaultfrontend="" vmprocpriority="default" storagecontrollername0="SATA" storagecontrollertype0="IntelAhci" storagecontrollerinstance0="0" storagecontrollermaxportcount0="30" storagecontrollerportcount0="30" storagecontrollerbootable0="on" "SATA-0-0"="/Users/mmukherjee/.minikube/machines/minikube/boot2docker.iso" "SATA-ImageUUID-0-0"="b0885204-54f8-4065-93f5-0cabe634d643" "SATA-tempeject"="off" "SATA-IsEjected"="off" "SATA-1-0"="/Users/mmukherjee/.minikube/machines/minikube/disk.vmdk" "SATA-ImageUUID-1-0"="388287f8-6cb2-4c6c-bc8b-999b55b3bf9b" "SATA-2-0"="none" "SATA-3-0"="none" "SATA-4-0"="none" "SATA-5-0"="none" "SATA-6-0"="none" "SATA-7-0"="none" "SATA-8-0"="none" "SATA-9-0"="none" "SATA-10-0"="none" "SATA-11-0"="none" "SATA-12-0"="none" "SATA-13-0"="none" "SATA-14-0"="none" "SATA-15-0"="none" "SATA-16-0"="none" "SATA-17-0"="none" "SATA-18-0"="none" "SATA-19-0"="none" "SATA-20-0"="none" "SATA-21-0"="none" "SATA-22-0"="none" "SATA-23-0"="none" "SATA-24-0"="none" "SATA-25-0"="none" "SATA-26-0"="none" "SATA-27-0"="none" "SATA-28-0"="none" "SATA-29-0"="none" natnet1="nat" macaddress1="080027F15FB4" cableconnected1="on" nic1="nat" nictype1="virtio" nicspeed1="0" mtu="0" sockSnd="64" sockRcv="64" tcpWndSnd="64" tcpWndRcv="64" Forwarding(0)="ssh,tcp,127.0.0.1,61511,,22" hostonlyadapter2="vboxnet0" macaddress2="0800278ECE50" cableconnected2="on" nic2="hostonly" nictype2="virtio" nicspeed2="0" nic3="none" nic4="none" nic5="none" nic6="none" nic7="none" nic8="none" hidpointing="ps2mouse" hidkeyboard="ps2kbd" uart1="off" uart2="off" uart3="off" uart4="off" lpt1="off" lpt2="off" audio="coreaudio" audio_out="off" audio_in="off" clipboard="disabled" draganddrop="disabled" SessionName="headless" VideoMode="720,400,0"@0,0 1 vrde="off" usb="off" ehci="off" xhci="off" SharedFolderNameMachineMapping1="Users" SharedFolderPathMachineMapping1="/Users" VRDEActiveConnection="off" VRDEClients==0 videocap="off" videocapaudio="off" capturescreens="" capturefilename="/Users/mmukherjee/.minikube/machines/minikube/minikube/minikube.webm" captureres="1024x768" capturevideorate=512 capturevideofps=25 captureopts="" GuestMemoryBalloon=0 GuestOSType="Linux26_64" GuestAdditionsRunLevel=2 GuestAdditionsVersion="5.2.42 r137960" GuestAdditionsFacility_VirtualBox Base Driver=50,1607921778174 GuestAdditionsFacility_VirtualBox System Service=50,1607921778502 GuestAdditionsFacility_Seamless Mode=0,1607921778978 GuestAdditionsFacility_Graphics Mode=0,1607921778173 } I1214 10:35:45.611922 50344 main.go:119] libmachine: STDERR: { } I1214 10:35:45.612023 50344 addons.go:243] installing /etc/kubernetes/addons/ingress-configmap.yaml I1214 10:35:45.612034 50344 ssh_runner.go:215] scp deploy/addons/ingress/ingress-configmap.yaml.tmpl --> /etc/kubernetes/addons/ingress-configmap.yaml (1251 bytes) I1214 10:35:45.612054 50344 sshutil.go:45] new ssh client: &{IP:127.0.0.1 Port:61511 SSHKeyPath:/Users/mmukherjee/.minikube/machines/minikube/id_rsa Username:docker} I1214 10:35:45.659295 50344 addons.go:243] installing /etc/kubernetes/addons/ingress-rbac.yaml I1214 10:35:45.659315 50344 ssh_runner.go:215] scp deploy/addons/ingress/ingress-rbac.yaml.tmpl --> /etc/kubernetes/addons/ingress-rbac.yaml (4828 bytes) I1214 10:35:45.666446 50344 addons.go:243] installing /etc/kubernetes/addons/ingress-dp.yaml I1214 10:35:45.666457 50344 ssh_runner.go:215] scp memory --> /etc/kubernetes/addons/ingress-dp.yaml (8520 bytes) I1214 10:35:45.673294 50344 ssh_runner.go:148] Run: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.19.4/kubectl apply -f /etc/kubernetes/addons/ingress-configmap.yaml -f /etc/kubernetes/addons/ingress-rbac.yaml -f /etc/kubernetes/addons/ingress-dp.yaml I1214 10:35:45.897502 50344 addons.go:342] Verifying addon ingress=true in "minikube" I1214 10:35:45.906579 50344 out.go:110] 🔎 Verifying ingress addon... 🔎 Verifying ingress addon... I1214 10:35:45.913249 50344 kapi.go:75] Waiting for pod with label "app.kubernetes.io/name=ingress-nginx" in ns "kube-system" ... I1214 10:35:45.925648 50344 kapi.go:86] Found 3 Pods for label selector app.kubernetes.io/name=ingress-nginx I1214 10:35:45.925661 50344 kapi.go:96] waiting for pod "app.kubernetes.io/name=ingress-nginx", current state: Pending: [] I1214 10:35:46.432463 50344 kapi.go:96] waiting for pod "app.kubernetes.io/name=ingress-nginx", current state: Pending: [] I1214 10:35:46.933501 50344 kapi.go:96] waiting for pod "app.kubernetes.io/name=ingress-nginx", current state: Pending: []

Thanks

[manas-mukherjee]

Here is the log from the minikube dashboard.

MountVolume.SetUp failed for volume "webhook-cert" : secret "ingress-nginx-admission" not found
Failed to pull image "us.gcr.io/k8s-artifacts-prod/ingress-nginx/controller:v0.40.2": rpc error: code = Unknown desc = error pulling image configuration: Get https://storage.googleapis.com/us.artifacts.k8s-artifacts-prod.appspot.com/containers/images/sha256:4b26fa2d90ae3bd47a8b9b49cf6a49cf195234ed32e4e0fef4787cd679218ff4: x509: certificate signed by unknown authority

Somehow, my system is not able to download the ingress addon from us.gcr.io. Please let me know if there is any workaround.

[manas-mukherjee]

Solved :) Here is what I did.

1. Manually downloaded the image in my host system. $ docker pull us.gcr.io/k8s-artifacts-prod/ingress-nginx/controller:v0.40.2

2. Created a tar file of the downloaded image $ docker save us.gcr.io/k8s-artifacts-prod/ingress-nginx/controller:v0.40.2 > ingressnginx.tar

3. Copied the tar to the minikube node $ scp -i $(minikube ssh-key) ${PWD}/ingressnginx.tar docker@$(minikube ip):/home/docker/

4. Logged into the minikube node $ minikube ssh

5. Verified the uploaded file in minikube $ pwd $ ls -ltr

6. Load an image from a tar $ docker load < ingressnginx.tar

7. From the host machine, enabled the ingress add-ons. $ minikube addons enable ingress 🔎 Verifying ingress addon... 🌟 The 'ingress' addon is enabled

Hope it will be useful.

[studyzhanglei]

Solved :) Here is what I did.

1. Manually downloaded the image in my host system. $ docker pull us.gcr.io/k8s-artifacts-prod/ingress-nginx/controller:v0.40.2
2. Created a tar file of the downloaded image $ docker save us.gcr.io/k8s-artifacts-prod/ingress-nginx/controller:v0.40.2 > ingressnginx.tar
3. Copied the tar to the minikube node $ scp -i $(minikube ssh-key) ${PWD}/ingressnginx.tar docker@$(minikube ip):/home/docker/
4. Logged into the minikube node $ minikube ssh
5. Verified the uploaded file in minikube $ pwd $ ls -ltr
6. Load an image from a tar $ docker load < ingressnginx.tar
7. From the host machine, enabled the ingress add-ons. $ minikube addons enable ingress 🔎 Verifying ingress addon... 🌟 The 'ingress' addon is enabled

Hope it will be useful.

Thank you, it works on my machine

[InclinedScorpio]

Just in case for someone it still doesn't work

> minikube stop 
> minikube start
> minikube addons enable ingress

Now wait for sometime ! It will install :)

[Nghiauet]

Hey @happy-machine thanks for opening this issue.

I noticed you have both the --driver=docker and --vm=true flags appended to minikube start. Those are opposing flags, since --driver=docker tells minikube to run in a docker container, while --vm=true tells minikube to run in a VM. I'd remove whichever one you don't want.

Could you also run:

minikube delete
minikube start <additonal flags>

and then provide the output of:

minikube addons enable ingress --alsologtostderr
kubectl get po -A

Thank you so muck I solved my problems <3