PodSecurityPolicy: no providers available to validate pod request

[Madhu-1]

Steps to reproduce the issue:

1.download minikube 1.9.0

2. start minikube with `minikube start -alsologtostderr --v=1 -b kubeadm --kubernetes-version="v1.18.0" --vm-driver="kvm2" --feature-gates="BlockVolume=true,CSIBlockVolume=true,VolumeSnapshotDataSource=true,ExpandCSIVolumes=true" --extra-config=apiserver.enable-admission-plugins=PodSecurityPolicy

Full output of failed command: >

Full output of minikube start command used, if not already included:

attaching logs in a file minikube.log

Optional: Full output of minikube logs command:

minikube logs ==> Docker <== -- Logs begin at Mon 2020-03-30 04:07:01 UTC, end at Mon 2020-03-30 04:16:46 UTC. -- Mar 30 04:08:59 minikube dockerd[2201]: time="2020-03-30T04:08:59.285398033Z" level=info msg="loading plugin "io.containerd.differ.v1.walking"..." type=io.containerd.differ.v1 Mar 30 04:08:59 minikube dockerd[2201]: time="2020-03-30T04:08:59.285456623Z" level=info msg="loading plugin "io.containerd.gc.v1.scheduler"..." type=io.containerd.gc.v1 Mar 30 04:08:59 minikube dockerd[2201]: time="2020-03-30T04:08:59.285537137Z" level=info msg="loading plugin "io.containerd.service.v1.containers-service"..." type=io.containerd.service.v1 Mar 30 04:08:59 minikube dockerd[2201]: time="2020-03-30T04:08:59.285594646Z" level=info msg="loading plugin "io.containerd.service.v1.content-service"..." type=io.containerd.service.v1 Mar 30 04:08:59 minikube dockerd[2201]: time="2020-03-30T04:08:59.285642243Z" level=info msg="loading plugin "io.containerd.service.v1.diff-service"..." type=io.containerd.service.v1 Mar 30 04:08:59 minikube dockerd[2201]: time="2020-03-30T04:08:59.285690911Z" level=info msg="loading plugin "io.containerd.service.v1.images-service"..." type=io.containerd.service.v1 Mar 30 04:08:59 minikube dockerd[2201]: time="2020-03-30T04:08:59.285745441Z" level=info msg="loading plugin "io.containerd.service.v1.leases-service"..." type=io.containerd.service.v1 Mar 30 04:08:59 minikube dockerd[2201]: time="2020-03-30T04:08:59.285795642Z" level=info msg="loading plugin "io.containerd.service.v1.namespaces-service"..." type=io.containerd.service.v1 Mar 30 04:08:59 minikube dockerd[2201]: time="2020-03-30T04:08:59.285843466Z" level=info msg="loading plugin "io.containerd.service.v1.snapshots-service"..." type=io.containerd.service.v1 Mar 30 04:08:59 minikube dockerd[2201]: time="2020-03-30T04:08:59.285890865Z" level=info msg="loading plugin "io.containerd.runtime.v1.linux"..." type=io.containerd.runtime.v1 Mar 30 04:08:59 minikube dockerd[2201]: time="2020-03-30T04:08:59.286005655Z" level=info msg="loading plugin "io.containerd.runtime.v2.task"..." type=io.containerd.runtime.v2 Mar 30 04:08:59 minikube dockerd[2201]: time="2020-03-30T04:08:59.286103843Z" level=info msg="loading plugin "io.containerd.monitor.v1.cgroups"..." type=io.containerd.monitor.v1 Mar 30 04:08:59 minikube dockerd[2201]: time="2020-03-30T04:08:59.286467987Z" level=info msg="loading plugin "io.containerd.service.v1.tasks-service"..." type=io.containerd.service.v1 Mar 30 04:08:59 minikube dockerd[2201]: time="2020-03-30T04:08:59.286548741Z" level=info msg="loading plugin "io.containerd.internal.v1.restart"..." type=io.containerd.internal.v1 Mar 30 04:08:59 minikube dockerd[2201]: time="2020-03-30T04:08:59.286626808Z" level=info msg="loading plugin "io.containerd.grpc.v1.containers"..." type=io.containerd.grpc.v1 Mar 30 04:08:59 minikube dockerd[2201]: time="2020-03-30T04:08:59.286681146Z" level=info msg="loading plugin "io.containerd.grpc.v1.content"..." type=io.containerd.grpc.v1 Mar 30 04:08:59 minikube dockerd[2201]: time="2020-03-30T04:08:59.286728293Z" level=info msg="loading plugin "io.containerd.grpc.v1.diff"..." type=io.containerd.grpc.v1 Mar 30 04:08:59 minikube dockerd[2201]: time="2020-03-30T04:08:59.286774396Z" level=info msg="loading plugin "io.containerd.grpc.v1.events"..." type=io.containerd.grpc.v1 Mar 30 04:08:59 minikube dockerd[2201]: time="2020-03-30T04:08:59.286819384Z" level=info msg="loading plugin "io.containerd.grpc.v1.healthcheck"..." type=io.containerd.grpc.v1 Mar 30 04:08:59 minikube dockerd[2201]: time="2020-03-30T04:08:59.286864991Z" level=info msg="loading plugin "io.containerd.grpc.v1.images"..." type=io.containerd.grpc.v1 Mar 30 04:08:59 minikube dockerd[2201]: time="2020-03-30T04:08:59.286914517Z" level=info msg="loading plugin "io.containerd.grpc.v1.leases"..." type=io.containerd.grpc.v1 Mar 30 04:08:59 minikube dockerd[2201]: time="2020-03-30T04:08:59.286960164Z" level=info msg="loading plugin "io.containerd.grpc.v1.namespaces"..." type=io.containerd.grpc.v1 Mar 30 04:08:59 minikube dockerd[2201]: time="2020-03-30T04:08:59.287026626Z" level=info msg="loading plugin "io.containerd.internal.v1.opt"..." type=io.containerd.internal.v1 Mar 30 04:08:59 minikube dockerd[2201]: time="2020-03-30T04:08:59.287120756Z" level=info msg="loading plugin "io.containerd.grpc.v1.snapshots"..." type=io.containerd.grpc.v1 Mar 30 04:08:59 minikube dockerd[2201]: time="2020-03-30T04:08:59.287174233Z" level=info msg="loading plugin "io.containerd.grpc.v1.tasks"..." type=io.containerd.grpc.v1 Mar 30 04:08:59 minikube dockerd[2201]: time="2020-03-30T04:08:59.287224418Z" level=info msg="loading plugin "io.containerd.grpc.v1.version"..." type=io.containerd.grpc.v1 Mar 30 04:08:59 minikube dockerd[2201]: time="2020-03-30T04:08:59.287270614Z" level=info msg="loading plugin "io.containerd.grpc.v1.introspection"..." type=io.containerd.grpc.v1 Mar 30 04:08:59 minikube dockerd[2201]: time="2020-03-30T04:08:59.287456832Z" level=info msg=serving... address="/var/run/docker/containerd/containerd-debug.sock" Mar 30 04:08:59 minikube dockerd[2201]: time="2020-03-30T04:08:59.287546076Z" level=info msg=serving... address="/var/run/docker/containerd/containerd.sock" Mar 30 04:08:59 minikube dockerd[2201]: time="2020-03-30T04:08:59.287595970Z" level=info msg="containerd successfully booted in 0.006026s" Mar 30 04:08:59 minikube dockerd[2201]: time="2020-03-30T04:08:59.293629654Z" level=info msg="parsed scheme: \"unix\"" module=grpc Mar 30 04:08:59 minikube dockerd[2201]: time="2020-03-30T04:08:59.293751896Z" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc Mar 30 04:08:59 minikube dockerd[2201]: time="2020-03-30T04:08:59.293860848Z" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///var/run/docker/containerd/containerd.sock 0 }] }" module=grpc Mar 30 04:08:59 minikube dockerd[2201]: time="2020-03-30T04:08:59.293929178Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc Mar 30 04:08:59 minikube dockerd[2201]: time="2020-03-30T04:08:59.294683992Z" level=info msg="parsed scheme: \"unix\"" module=grpc Mar 30 04:08:59 minikube dockerd[2201]: time="2020-03-30T04:08:59.294711568Z" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc Mar 30 04:08:59 minikube dockerd[2201]: time="2020-03-30T04:08:59.294724982Z" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///var/run/docker/containerd/containerd.sock 0 }] }" module=grpc Mar 30 04:08:59 minikube dockerd[2201]: time="2020-03-30T04:08:59.294733344Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc Mar 30 04:09:21 minikube dockerd[2201]: time="2020-03-30T04:09:21.116845551Z" level=warning msg="Your kernel does not support cgroup blkio weight" Mar 30 04:09:21 minikube dockerd[2201]: time="2020-03-30T04:09:21.119858722Z" level=warning msg="Your kernel does not support cgroup blkio weight_device" Mar 30 04:09:21 minikube dockerd[2201]: time="2020-03-30T04:09:21.120370676Z" level=warning msg="Your kernel does not support cgroup blkio throttle.read_bps_device" Mar 30 04:09:21 minikube dockerd[2201]: time="2020-03-30T04:09:21.120502168Z" level=warning msg="Your kernel does not support cgroup blkio throttle.write_bps_device" Mar 30 04:09:21 minikube dockerd[2201]: time="2020-03-30T04:09:21.120611926Z" level=warning msg="Your kernel does not support cgroup blkio throttle.read_iops_device" Mar 30 04:09:21 minikube dockerd[2201]: time="2020-03-30T04:09:21.120711770Z" level=warning msg="Your kernel does not support cgroup blkio throttle.write_iops_device" Mar 30 04:09:21 minikube dockerd[2201]: time="2020-03-30T04:09:21.129718552Z" level=info msg="Loading containers: start." Mar 30 04:09:21 minikube dockerd[2201]: time="2020-03-30T04:09:21.475790242Z" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address" Mar 30 04:09:21 minikube dockerd[2201]: time="2020-03-30T04:09:21.642019825Z" level=info msg="Loading containers: done." Mar 30 04:09:21 minikube dockerd[2201]: time="2020-03-30T04:09:21.762922666Z" level=info msg="Docker daemon" commit=afacb8b7f0 graphdriver(s)=overlay2 version=19.03.8 Mar 30 04:09:21 minikube dockerd[2201]: time="2020-03-30T04:09:21.763220462Z" level=info msg="Daemon has completed initialization" Mar 30 04:09:21 minikube systemd[1]: Started Docker Application Container Engine. Mar 30 04:09:21 minikube dockerd[2201]: time="2020-03-30T04:09:21.888298128Z" level=info msg="API listen on /var/run/docker.sock" Mar 30 04:09:21 minikube dockerd[2201]: time="2020-03-30T04:09:21.888413346Z" level=info msg="API listen on [::]:2376" Mar 30 04:09:37 minikube dockerd[2201]: time="2020-03-30T04:09:37.185961064Z" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/8442feb49ca57488d1b19738260e2dae706b18c9cdee5b10895ee514bc9b8aee/shim.sock" debug=false pid=3139 Mar 30 04:09:37 minikube dockerd[2201]: time="2020-03-30T04:09:37.689674469Z" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/d89bc732e6180a56eb4c8f6fc59cda5f487d420cffa2ee1b3a099de2ca7a0470/shim.sock" debug=false pid=3219 Mar 30 04:09:37 minikube dockerd[2201]: time="2020-03-30T04:09:37.829143523Z" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/33936602b40c39a15912f35170689d6a24db5f8eab958774a0ed51624d804638/shim.sock" debug=false pid=3245 Mar 30 04:09:38 minikube dockerd[2201]: time="2020-03-30T04:09:38.013579219Z" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/30d94a5432a2fcb8a18b22a6c5b4a6272c1a6ef66ee8b93513ee5d7b0c15df22/shim.sock" debug=false pid=3287 Mar 30 04:09:38 minikube dockerd[2201]: time="2020-03-30T04:09:38.114977072Z" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/c73429962fe3097431642bad07ed1fe4b44d3542f9cdcba809f4d6e5f57b4b82/shim.sock" debug=false pid=3313 Mar 30 04:09:38 minikube dockerd[2201]: time="2020-03-30T04:09:38.451909492Z" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/8542ffaea60463b66dd2518f324f743d8b841bc0992e7214d8771747fb4b6bfd/shim.sock" debug=false pid=3358 Mar 30 04:09:38 minikube dockerd[2201]: time="2020-03-30T04:09:38.618509732Z" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/46301bfe3009bd63fefb8fc9ccacdbe418e44df77bed0144ecebef386f2994d0/shim.sock" debug=false pid=3382 Mar 30 04:09:39 minikube dockerd[2201]: time="2020-03-30T04:09:39.314168549Z" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/c7d468df7e1ab103c8013fc67cf0302155409b4f478ef5dbef30f46d9ed89982/shim.sock" debug=false pid=3451 ==> container status <== CONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID c7d468df7e1ab 303ce5db0e90d 7 minutes ago Running etcd 0 c73429962fe30 46301bfe3009b a31f78c7c8ce1 7 minutes ago Running kube-scheduler 0 33936602b40c3 8542ffaea6046 d3e55153f52fb 7 minutes ago Running kube-controller-manager 0 d89bc732e6180 30d94a5432a2f 74060cea7f704 7 minutes ago Running kube-apiserver 0 8442feb49ca57 ==> describe nodes <== Name: minikube Roles: master Labels: beta.kubernetes.io/arch=amd64 beta.kubernetes.io/os=linux kubernetes.io/arch=amd64 kubernetes.io/hostname=minikube kubernetes.io/os=linux minikube.k8s.io/commit=48fefd43444d2f8852f527c78f0141b377b1e42a minikube.k8s.io/name=minikube minikube.k8s.io/updated_at=2020_03_30T09_39_54_0700 minikube.k8s.io/version=v1.9.0 node-role.kubernetes.io/master= Annotations: kubeadm.alpha.kubernetes.io/cri-socket: /var/run/dockershim.sock node.alpha.kubernetes.io/ttl: 0 volumes.kubernetes.io/controller-managed-attach-detach: true CreationTimestamp: Mon, 30 Mar 2020 04:09:44 +0000 Taints: Unschedulable: false Lease: HolderIdentity: minikube AcquireTime: RenewTime: Mon, 30 Mar 2020 04:16:37 +0000 Conditions: Type Status LastHeartbeatTime LastTransitionTime Reason Message ---- ------ ----------------- ------------------ ------ ------- MemoryPressure False Mon, 30 Mar 2020 04:15:09 +0000 Mon, 30 Mar 2020 04:09:39 +0000 KubeletHasSufficientMemory kubelet has sufficient memory available DiskPressure False Mon, 30 Mar 2020 04:15:09 +0000 Mon, 30 Mar 2020 04:09:39 +0000 KubeletHasNoDiskPressure kubelet has no disk pressure PIDPressure False Mon, 30 Mar 2020 04:15:09 +0000 Mon, 30 Mar 2020 04:09:39 +0000 KubeletHasSufficientPID kubelet has sufficient PID available Ready True Mon, 30 Mar 2020 04:15:09 +0000 Mon, 30 Mar 2020 04:09:56 +0000 KubeletReady kubelet is posting ready status Addresses: InternalIP: 192.168.39.24 Hostname: minikube Capacity: cpu: 2 ephemeral-storage: 16954224Ki hugepages-2Mi: 0 memory: 2855912Ki pods: 110 Allocatable: cpu: 2 ephemeral-storage: 15625012813 hugepages-2Mi: 0 memory: 2753512Ki pods: 110 System Info: Machine ID: 26fb9b85d84047409a85038cacf536f6 System UUID: 26fb9b85-d840-4740-9a85-038cacf536f6 Boot ID: 7675ed1c-e999-4234-9f7a-3fe2ca37612e Kernel Version: 4.19.107 OS Image: Buildroot 2019.02.10 Operating System: linux Architecture: amd64 Container Runtime Version: docker://19.3.8 Kubelet Version: v1.18.0 Kube-Proxy Version: v1.18.0 Non-terminated Pods: (0 in total) Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits AGE --------- ---- ------------ ---------- --------------- ------------- --- Allocated resources: (Total limits may be over 100 percent, i.e., overcommitted.) Resource Requests Limits -------- -------- ------ cpu 0 (0%) 0 (0%) memory 0 (0%) 0 (0%) ephemeral-storage 0 (0%) 0 (0%) hugepages-2Mi 0 (0%) 0 (0%) Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal NodeHasSufficientMemory 7m11s (x6 over 7m12s) kubelet, minikube Node minikube status is now: NodeHasSufficientMemory Normal NodeHasNoDiskPressure 7m11s (x6 over 7m12s) kubelet, minikube Node minikube status is now: NodeHasNoDiskPressure Normal NodeHasSufficientPID 7m11s (x5 over 7m12s) kubelet, minikube Node minikube status is now: NodeHasSufficientPID Normal Starting 6m51s kubelet, minikube Starting kubelet. Normal NodeHasSufficientMemory 6m51s kubelet, minikube Node minikube status is now: NodeHasSufficientMemory Normal NodeHasNoDiskPressure 6m51s kubelet, minikube Node minikube status is now: NodeHasNoDiskPressure Normal NodeHasSufficientPID 6m51s kubelet, minikube Node minikube status is now: NodeHasSufficientPID Normal NodeNotReady 6m51s kubelet, minikube Node minikube status is now: NodeNotReady Normal NodeAllocatableEnforced 6m51s kubelet, minikube Updated Node Allocatable limit across pods Normal NodeReady 6m51s kubelet, minikube Node minikube status is now: NodeReady ==> dmesg <== [Mar30 04:06] You have booted with nomodeset. This means your GPU drivers are DISABLED [ +0.000001] Any video related functionality will be severely degraded, and you may not even be able to suspend the system properly [ +0.000000] Unless you actually understand what nomodeset does, you should reboot without enabling it [ +0.078812] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ +11.845813] platform regulatory.0: Direct firmware load for regulatory.db failed with error -2 [ +13.738227] systemd[1]: Failed to bump fs.file-max, ignoring: Invalid argument [ +0.006827] systemd-fstab-generator[1150]: Ignoring "noauto" for root device [ +0.014088] systemd[1]: File /usr/lib/systemd/system/systemd-journald.service:12 configures an IP firewall (IPAddressDeny=any), but the local system does not support BPF/cgroup based firewalling. [ +0.000006] systemd[1]: Proceeding WITHOUT firewalling in effect! (This warning is only shown for the first loaded unit using IP firewalling.) [Mar30 04:07] vboxguest: loading out-of-tree module taints kernel. [ +0.201247] vboxguest: PCI device not found, probably running on physical hardware. [ +0.003316] NFSD: the nfsdcld client tracking upcall will be removed in 3.10. Please transition to using nfsdcltrack. [ +2.020366] systemd-fstab-generator[2004]: Ignoring "noauto" for root device [ +35.672394] hrtimer: interrupt took 13993663 ns [Mar30 04:09] NFSD: Unable to end grace period: -110 [ +16.869534] kauditd_printk_skb: 65 callbacks suppressed [ +1.897571] systemd-fstab-generator[2409]: Ignoring "noauto" for root device [ +2.382953] systemd-fstab-generator[2635]: Ignoring "noauto" for root device [ +10.206861] kauditd_printk_skb: 107 callbacks suppressed [ +13.642883] systemd-fstab-generator[3722]: Ignoring "noauto" for root device ==> etcd [c7d468df7e1a] <== 2020-03-30 04:14:55.750301 W | etcdserver: read-only range request "key:\"/registry/health\" " with result "range_response_count:0 size:5" took too long (115.637072ms) to execute 2020-03-30 04:15:00.235607 W | etcdserver: read-only range request "key:\"/registry/jobs/\" range_end:\"/registry/jobs0\" limit:500 " with result "range_response_count:0 size:5" took too long (299.853164ms) to execute 2020-03-30 04:15:00.248952 W | etcdserver: read-only range request "key:\"/registry/leases/kube-system/kube-scheduler\" " with result "range_response_count:1 size:481" took too long (323.775715ms) to execute 2020-03-30 04:15:00.268722 W | etcdserver: read-only range request "key:\"/registry/leases/kube-system/kube-controller-manager\" " with result "range_response_count:1 size:508" took too long (343.19809ms) to execute 2020-03-30 04:15:02.617579 W | etcdserver: read-only range request "key:\"/registry/leases/kube-system/kube-controller-manager\" " with result "range_response_count:1 size:508" took too long (298.975638ms) to execute 2020-03-30 04:15:02.618314 W | etcdserver: read-only range request "key:\"/registry/services/endpoints/kube-system/kube-scheduler\" " with result "range_response_count:1 size:576" took too long (292.869681ms) to execute 2020-03-30 04:15:06.997572 W | etcdserver: read-only range request "key:\"/registry/leases/kube-system/kube-controller-manager\" " with result "range_response_count:1 size:508" took too long (303.823698ms) to execute 2020-03-30 04:15:06.999274 W | etcdserver: read-only range request "key:\"/registry/services/endpoints/kube-system/kube-scheduler\" " with result "range_response_count:1 size:576" took too long (287.443883ms) to execute 2020-03-30 04:15:09.345400 W | etcdserver: read-only range request "key:\"/registry/leases/kube-system/kube-controller-manager\" " with result "range_response_count:1 size:508" took too long (288.685612ms) to execute 2020-03-30 04:15:09.348694 W | etcdserver: read-only range request "key:\"/registry/services/endpoints/kube-system/kube-scheduler\" " with result "range_response_count:1 size:576" took too long (285.263251ms) to execute 2020-03-30 04:15:15.826492 W | etcdserver: read-only range request "key:\"/registry/health\" " with result "range_response_count:0 size:5" took too long (221.864013ms) to execute 2020-03-30 04:15:15.827623 W | etcdserver: read-only range request "key:\"/registry/services/endpoints/kube-system/kube-scheduler\" " with result "range_response_count:1 size:576" took too long (342.627847ms) to execute 2020-03-30 04:15:15.828279 W | etcdserver: read-only range request "key:\"/registry/leases/kube-system/kube-controller-manager\" " with result "range_response_count:1 size:508" took too long (345.122323ms) to execute 2020-03-30 04:15:20.304772 W | etcdserver: read-only range request "key:\"/registry/leases/kube-system/kube-controller-manager\" " with result "range_response_count:1 size:508" took too long (279.772043ms) to execute 2020-03-30 04:15:22.667126 W | etcdserver: read-only range request "key:\"/registry/leases/kube-system/kube-controller-manager\" " with result "range_response_count:1 size:508" took too long (299.354589ms) to execute 2020-03-30 04:15:27.068382 W | etcdserver: read-only range request "key:\"/registry/csidrivers\" range_end:\"/registry/csidrivert\" count_only:true " with result "range_response_count:0 size:5" took too long (131.910825ms) to execute 2020-03-30 04:15:27.070193 W | etcdserver: read-only range request "key:\"/registry/leases/kube-system/kube-controller-manager\" " with result "range_response_count:1 size:508" took too long (295.195423ms) to execute 2020-03-30 04:15:27.070519 W | etcdserver: read-only range request "key:\"/registry/leases/kube-system/kube-scheduler\" " with result "range_response_count:1 size:481" took too long (299.541472ms) to execute 2020-03-30 04:15:29.420571 W | etcdserver: read-only range request "key:\"/registry/leases/kube-system/kube-controller-manager\" " with result "range_response_count:1 size:508" took too long (283.231891ms) to execute 2020-03-30 04:15:29.422945 W | etcdserver: read-only range request "key:\"/registry/leases/kube-system/kube-scheduler\" " with result "range_response_count:1 size:481" took too long (283.430775ms) to execute 2020-03-30 04:15:35.951527 W | etcdserver: read-only range request "key:\"/registry/leases/kube-system/kube-scheduler\" " with result "range_response_count:1 size:481" took too long (286.029422ms) to execute 2020-03-30 04:15:35.954357 W | etcdserver: read-only range request "key:\"/registry/leases/kube-system/kube-controller-manager\" " with result "range_response_count:1 size:508" took too long (288.400553ms) to execute 2020-03-30 04:15:40.410181 W | etcdserver: read-only range request "key:\"/registry/leases/kube-system/kube-scheduler\" " with result "range_response_count:1 size:480" took too long (306.588029ms) to execute 2020-03-30 04:15:40.410666 W | etcdserver: read-only range request "key:\"/registry/leases/kube-system/kube-controller-manager\" " with result "range_response_count:1 size:508" took too long (296.693662ms) to execute 2020-03-30 04:15:42.785300 W | etcdserver: read-only range request "key:\"/registry/leases/kube-system/kube-controller-manager\" " with result "range_response_count:1 size:507" took too long (328.798444ms) to execute 2020-03-30 04:15:42.786315 W | etcdserver: read-only range request "key:\"/registry/leases/kube-system/kube-scheduler\" " with result "range_response_count:1 size:480" took too long (333.411564ms) to execute 2020-03-30 04:15:49.337609 W | etcdserver: read-only range request "key:\"/registry/leases/kube-system/kube-scheduler\" " with result "range_response_count:1 size:481" took too long (324.338436ms) to execute 2020-03-30 04:15:49.338926 W | etcdserver: read-only range request "key:\"/registry/networkpolicies\" range_end:\"/registry/networkpoliciet\" count_only:true " with result "range_response_count:0 size:5" took too long (299.063128ms) to execute 2020-03-30 04:15:55.832087 W | etcdserver: read-only range request "key:\"/registry/leases/kube-system/kube-scheduler\" " with result "range_response_count:1 size:481" took too long (338.335749ms) to execute 2020-03-30 04:15:55.833636 W | etcdserver: read-only range request "key:\"/registry/services/endpoints/kube-system/kube-controller-manager\" " with result "range_response_count:1 size:603" took too long (300.716991ms) to execute 2020-03-30 04:15:55.835440 W | etcdserver: read-only range request "key:\"/registry/health\" " with result "range_response_count:0 size:5" took too long (228.600653ms) to execute 2020-03-30 04:16:00.344672 W | etcdserver: read-only range request "key:\"/registry/leases/kube-system/kube-scheduler\" " with result "range_response_count:1 size:481" took too long (309.3231ms) to execute 2020-03-30 04:16:00.346278 W | etcdserver: read-only range request "key:\"/registry/leases/kube-system/kube-controller-manager\" " with result "range_response_count:1 size:508" took too long (315.528941ms) to execute 2020-03-30 04:16:02.791614 W | etcdserver: read-only range request "key:\"/registry/leases/kube-system/kube-controller-manager\" " with result "range_response_count:1 size:507" took too long (343.906072ms) to execute 2020-03-30 04:16:02.792160 W | etcdserver: read-only range request "key:\"/registry/leases/kube-system/kube-scheduler\" " with result "range_response_count:1 size:481" took too long (330.195734ms) to execute 2020-03-30 04:16:05.308831 W | etcdserver: read-only range request "key:\"/registry/services/endpoints/kube-system/kube-controller-manager\" " with result "range_response_count:1 size:603" took too long (451.881698ms) to execute 2020-03-30 04:16:05.310341 W | etcdserver: read-only range request "key:\"/registry/leases/kube-system/kube-scheduler\" " with result "range_response_count:1 size:481" took too long (470.214515ms) to execute 2020-03-30 04:16:09.792120 W | etcdserver: read-only range request "key:\"/registry/services/endpoints/kube-system/kube-controller-manager\" " with result "range_response_count:1 size:603" took too long (340.274523ms) to execute 2020-03-30 04:16:09.797258 W | etcdserver: read-only range request "key:\"/registry/leases/kube-system/kube-scheduler\" " with result "range_response_count:1 size:481" took too long (364.202344ms) to execute 2020-03-30 04:16:12.201627 W | etcdserver: read-only range request "key:\"/registry/leases/kube-system/kube-scheduler\" " with result "range_response_count:1 size:481" took too long (319.123479ms) to execute 2020-03-30 04:16:12.203531 W | etcdserver: read-only range request "key:\"/registry/services/endpoints/kube-system/kube-controller-manager\" " with result "range_response_count:1 size:603" took too long (285.486986ms) to execute 2020-03-30 04:16:18.777441 W | etcdserver: read-only range request "key:\"/registry/leases/kube-system/kube-scheduler\" " with result "range_response_count:1 size:481" took too long (330.774657ms) to execute 2020-03-30 04:16:18.779921 W | etcdserver: read-only range request "key:\"/registry/services/endpoints/kube-system/kube-controller-manager\" " with result "range_response_count:1 size:603" took too long (308.080585ms) to execute 2020-03-30 04:16:22.908376 W | etcdserver: read-only range request "key:\"/registry/leases\" range_end:\"/registry/leaset\" count_only:true " with result "range_response_count:0 size:7" took too long (101.729989ms) to execute 2020-03-30 04:16:25.475493 W | etcdserver: read-only range request "key:\"/registry/leases/kube-system/kube-scheduler\" " with result "range_response_count:1 size:480" took too long (354.279491ms) to execute 2020-03-30 04:16:25.475714 W | etcdserver: read-only range request "key:\"/registry/services/endpoints/kube-system/kube-controller-manager\" " with result "range_response_count:1 size:603" took too long (306.559092ms) to execute 2020-03-30 04:16:30.053922 W | etcdserver: read-only range request "key:\"/registry/leases/kube-system/kube-scheduler\" " with result "range_response_count:1 size:481" took too long (293.043876ms) to execute 2020-03-30 04:16:30.054314 W | etcdserver: read-only range request "key:\"/registry/services/endpoints/kube-system/kube-controller-manager\" " with result "range_response_count:1 size:603" took too long (290.465277ms) to execute 2020-03-30 04:16:32.498689 W | etcdserver: read-only range request "key:\"/registry/leases/kube-system/kube-scheduler\" " with result "range_response_count:1 size:481" took too long (347.851089ms) to execute 2020-03-30 04:16:32.499357 W | etcdserver: read-only range request "key:\"/registry/services/endpoints/kube-system/kube-controller-manager\" " with result "range_response_count:1 size:603" took too long (348.300151ms) to execute 2020-03-30 04:16:37.065649 W | etcdserver: read-only range request "key:\"/registry/leases/kube-system/kube-scheduler\" " with result "range_response_count:1 size:481" took too long (351.629967ms) to execute 2020-03-30 04:16:37.068133 W | etcdserver: read-only range request "key:\"/registry/leases/kube-system/kube-controller-manager\" " with result "range_response_count:1 size:508" took too long (349.610118ms) to execute 2020-03-30 04:16:37.072596 W | etcdserver: read-only range request "key:\"/registry/apiregistration.k8s.io/apiservices\" range_end:\"/registry/apiregistration.k8s.io/apiservicet\" count_only:true " with result "range_response_count:0 size:7" took too long (131.611717ms) to execute 2020-03-30 04:16:39.533685 W | etcdserver: read-only range request "key:\"/registry/poddisruptionbudgets\" range_end:\"/registry/poddisruptionbudgett\" count_only:true " with result "range_response_count:0 size:5" took too long (103.381281ms) to execute 2020-03-30 04:16:39.534239 W | etcdserver: read-only range request "key:\"/registry/leases/kube-system/kube-controller-manager\" " with result "range_response_count:1 size:508" took too long (337.681365ms) to execute 2020-03-30 04:16:39.534680 W | etcdserver: read-only range request "key:\"/registry/services/endpoints/kube-system/kube-scheduler\" " with result "range_response_count:1 size:576" took too long (313.106721ms) to execute 2020-03-30 04:16:41.930268 W | etcdserver: read-only range request "key:\"/registry/leases/kube-system/kube-scheduler\" " with result "range_response_count:1 size:480" took too long (337.57257ms) to execute 2020-03-30 04:16:41.931448 W | etcdserver: read-only range request "key:\"/registry/leases/kube-system/kube-controller-manager\" " with result "range_response_count:1 size:507" took too long (328.274959ms) to execute 2020-03-30 04:16:46.453964 W | etcdserver: read-only range request "key:\"/registry/leases/kube-system/kube-scheduler\" " with result "range_response_count:1 size:481" took too long (321.024252ms) to execute 2020-03-30 04:16:46.455274 W | etcdserver: read-only range request "key:\"/registry/leases/kube-system/kube-controller-manager\" " with result "range_response_count:1 size:508" took too long (316.255271ms) to execute ==> kernel <== 04:16:47 up 10 min, 0 users, load average: 1.98, 1.61, 0.84 Linux minikube 4.19.107 #1 SMP Thu Mar 26 11:33:10 PDT 2020 x86_64 GNU/Linux PRETTY_NAME="Buildroot 2019.02.10" ==> kube-apiserver [30d94a5432a2] <== I0330 04:09:44.469372 1 cluster_authentication_trust_controller.go:440] Starting cluster_authentication_trust_controller controller I0330 04:09:44.469414 1 shared_informer.go:223] Waiting for caches to sync for cluster_authentication_trust_controller I0330 04:09:44.469442 1 autoregister_controller.go:141] Starting autoregister controller I0330 04:09:44.469447 1 cache.go:32] Waiting for caches to sync for autoregister controller I0330 04:09:44.479921 1 controller.go:86] Starting OpenAPI controller I0330 04:09:44.480153 1 customresource_discovery_controller.go:209] Starting DiscoveryController I0330 04:09:44.480335 1 naming_controller.go:291] Starting NamingConditionController I0330 04:09:44.480454 1 establishing_controller.go:76] Starting EstablishingController I0330 04:09:44.480546 1 nonstructuralschema_controller.go:186] Starting NonStructuralSchemaConditionController I0330 04:09:44.480636 1 apiapproval_controller.go:186] Starting KubernetesAPIApprovalPolicyConformantConditionController I0330 04:09:44.516043 1 dynamic_cafile_content.go:167] Starting client-ca-bundle::/var/lib/minikube/certs/ca.crt I0330 04:09:44.516098 1 dynamic_cafile_content.go:167] Starting request-header::/var/lib/minikube/certs/front-proxy-ca.crt I0330 04:09:44.516511 1 crdregistration_controller.go:111] Starting crd-autoregister controller I0330 04:09:44.516541 1 shared_informer.go:223] Waiting for caches to sync for crd-autoregister E0330 04:09:44.536855 1 controller.go:152] Unable to remove old endpoints from kubernetes service: StorageError: key not found, Code: 1, Key: /registry/masterleases/192.168.39.24, ResourceVersion: 0, AdditionalErrorMsg: I0330 04:09:44.619620 1 shared_informer.go:230] Caches are synced for crd-autoregister I0330 04:09:44.666225 1 cache.go:39] Caches are synced for AvailableConditionController controller I0330 04:09:44.667965 1 cache.go:39] Caches are synced for APIServiceRegistrationController controller I0330 04:09:44.669550 1 shared_informer.go:230] Caches are synced for cluster_authentication_trust_controller I0330 04:09:44.670063 1 cache.go:39] Caches are synced for autoregister controller I0330 04:09:45.144484 1 trace.go:116] Trace[419871089]: "Get" url:/api/v1/nodes/minikube,user-agent:kubelet/v1.18.0 (linux/amd64) kubernetes/9e99141,client:192.168.39.24 (started: 2020-03-30 04:09:44.642529132 +0000 UTC m=+6.475677548) (total time: 501.892769ms): Trace[419871089]: [500.781668ms] [500.717476ms] About to write a response I0330 04:09:45.146765 1 trace.go:116] Trace[1364245529]: "Create" url:/apis/certificates.k8s.io/v1beta1/certificatesigningrequests,user-agent:kubelet/v1.18.0 (linux/amd64) kubernetes/9e99141,client:192.168.39.24 (started: 2020-03-30 04:09:44.575680628 +0000 UTC m=+6.408829057) (total time: 571.031048ms): Trace[1364245529]: [570.922753ms] [570.832778ms] Object stored in database I0330 04:09:45.156355 1 trace.go:116] Trace[848244561]: "Get" url:/api/v1/namespaces/kube-system/pods/kube-apiserver-minikube,user-agent:kubelet/v1.18.0 (linux/amd64) kubernetes/9e99141,client:192.168.39.24 (started: 2020-03-30 04:09:44.656136593 +0000 UTC m=+6.489285004) (total time: 500.180103ms): Trace[848244561]: [500.180103ms] [500.111317ms] END I0330 04:09:45.156905 1 trace.go:116] Trace[2146107601]: "Get" url:/api/v1/nodes/minikube,user-agent:kubelet/v1.18.0 (linux/amd64) kubernetes/9e99141,client:192.168.39.24 (started: 2020-03-30 04:09:44.653120714 +0000 UTC m=+6.486269121) (total time: 503.749101ms): Trace[2146107601]: [502.773752ms] [502.631082ms] About to write a response I0330 04:09:45.164312 1 trace.go:116] Trace[1784091134]: "Create" url:/api/v1/namespaces/default/events,user-agent:kubelet/v1.18.0 (linux/amd64) kubernetes/9e99141,client:192.168.39.24 (started: 2020-03-30 04:09:44.508778756 +0000 UTC m=+6.341927167) (total time: 655.499849ms): Trace[1784091134]: [655.499849ms] [655.416255ms] END I0330 04:09:45.177440 1 trace.go:116] Trace[2104369566]: "GuaranteedUpdate etcd3" type:*core.RangeAllocation (started: 2020-03-30 04:09:44.652495418 +0000 UTC m=+6.485643854) (total time: 524.920271ms): Trace[2104369566]: [501.092582ms] [501.092582ms] initial value restored I0330 04:09:45.465560 1 controller.go:130] OpenAPI AggregationController: action for item : Nothing (removed from the queue). I0330 04:09:45.465663 1 controller.go:130] OpenAPI AggregationController: action for item k8s_internal_local_delegation_chain_0000000000: Nothing (removed from the queue). I0330 04:09:45.808462 1 trace.go:116] Trace[1140586222]: "List etcd3" key:/services/specs,resourceVersion:,limit:0,continue: (started: 2020-03-30 04:09:45.301644473 +0000 UTC m=+7.134792887) (total time: 506.721948ms): Trace[1140586222]: [506.721948ms] [506.721948ms] END I0330 04:09:45.809287 1 trace.go:116] Trace[1292623146]: "List" url:/api/v1/services,user-agent:kube-apiserver/v1.18.0 (linux/amd64) kubernetes/9e99141,client:127.0.0.1 (started: 2020-03-30 04:09:45.301632477 +0000 UTC m=+7.134780895) (total time: 507.551537ms): Trace[1292623146]: [506.909237ms] [506.902841ms] Listing from storage done I0330 04:09:45.810338 1 trace.go:116] Trace[56728193]: "Get" url:/api/v1/namespaces/default,user-agent:kube-apiserver/v1.18.0 (linux/amd64) kubernetes/9e99141,client:127.0.0.1 (started: 2020-03-30 04:09:45.301942891 +0000 UTC m=+7.135091301) (total time: 508.325156ms): Trace[56728193]: [508.325156ms] [508.318994ms] END I0330 04:09:45.812739 1 trace.go:116] Trace[714030423]: "Get" url:/api/v1/namespaces/kube-system,user-agent:kube-apiserver/v1.18.0 (linux/amd64) kubernetes/9e99141,client:127.0.0.1 (started: 2020-03-30 04:09:45.306244414 +0000 UTC m=+7.139392830) (total time: 506.43826ms): Trace[714030423]: [506.304125ms] [506.29542ms] About to write a response I0330 04:09:45.815427 1 trace.go:116] Trace[1415538191]: "GuaranteedUpdate etcd3" type:*core.RangeAllocation (started: 2020-03-30 04:09:45.300867318 +0000 UTC m=+7.134015729) (total time: 514.52877ms): Trace[1415538191]: [514.479604ms] [514.479604ms] initial value restored I0330 04:09:45.816600 1 trace.go:116] Trace[1515959019]: "Create" url:/api/v1/namespaces/default/events,user-agent:kubelet/v1.18.0 (linux/amd64) kubernetes/9e99141,client:192.168.39.24 (started: 2020-03-30 04:09:45.250885955 +0000 UTC m=+7.084034362) (total time: 565.666777ms): Trace[1515959019]: [565.666777ms] [565.632245ms] END I0330 04:09:45.851876 1 storage_scheduling.go:134] created PriorityClass system-node-critical with value 2000001000 I0330 04:09:45.865475 1 storage_scheduling.go:134] created PriorityClass system-cluster-critical with value 2000000000 I0330 04:09:45.865880 1 storage_scheduling.go:143] all system priority classes are created successfully or already exist. I0330 04:09:47.205196 1 controller.go:606] quota admission added evaluator for: roles.rbac.authorization.k8s.io I0330 04:09:47.332224 1 controller.go:606] quota admission added evaluator for: rolebindings.rbac.authorization.k8s.io I0330 04:09:47.646503 1 controller.go:606] quota admission added evaluator for: endpoints I0330 04:09:47.657362 1 controller.go:606] quota admission added evaluator for: leases.coordination.k8s.io W0330 04:09:47.696837 1 lease.go:224] Resetting endpoints for master service "kubernetes" to [192.168.39.24] I0330 04:09:47.749436 1 controller.go:606] quota admission added evaluator for: endpointslices.discovery.k8s.io I0330 04:09:48.003806 1 controller.go:606] quota admission added evaluator for: serviceaccounts I0330 04:09:49.583805 1 controller.go:606] quota admission added evaluator for: deployments.apps I0330 04:09:49.668827 1 controller.go:606] quota admission added evaluator for: daemonsets.apps I0330 04:09:54.683232 1 controller.go:606] quota admission added evaluator for: replicasets.apps I0330 04:09:55.164379 1 controller.go:606] quota admission added evaluator for: controllerrevisions.apps ==> kube-controller-manager [8542ffaea604] <== E0330 04:09:55.319033 1 daemon_controller.go:292] kube-system/kube-proxy failed with : pods "kube-proxy-" is forbidden: no providers available to validate pod request I0330 04:09:55.320250 1 event.go:278] Event(v1.ObjectReference{Kind:"DaemonSet", Namespace:"kube-system", Name:"kube-proxy", UID:"a0b4a679-1fb6-4e96-a3b8-0cc868cfa473", APIVersion:"apps/v1", ResourceVersion:"238", FieldPath:""}): type: 'Warning' reason: 'FailedCreate' Error creating: pods "kube-proxy-" is forbidden: no providers available to validate pod request I0330 04:09:55.351814 1 shared_informer.go:230] Caches are synced for service account I0330 04:09:55.354141 1 shared_informer.go:230] Caches are synced for garbage collector E0330 04:09:55.404425 1 daemon_controller.go:957] pods "kube-proxy-" is forbidden: no providers available to validate pod request E0330 04:09:55.404504 1 daemon_controller.go:292] kube-system/kube-proxy failed with : pods "kube-proxy-" is forbidden: no providers available to validate pod request I0330 04:09:55.404616 1 event.go:278] Event(v1.ObjectReference{Kind:"DaemonSet", Namespace:"kube-system", Name:"kube-proxy", UID:"a0b4a679-1fb6-4e96-a3b8-0cc868cfa473", APIVersion:"apps/v1", ResourceVersion:"238", FieldPath:""}): type: 'Warning' reason: 'FailedCreate' Error creating: pods "kube-proxy-" is forbidden: no providers available to validate pod request E0330 04:09:55.466633 1 replica_set.go:535] sync "kube-system/coredns-66bff467f8" failed with pods "coredns-66bff467f8-" is forbidden: no providers available to validate pod request I0330 04:09:55.467172 1 event.go:278] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"kube-system", Name:"coredns-66bff467f8", UID:"d6bcb9c5-9a42-4994-b34d-8d45f86626c9", APIVersion:"apps/v1", ResourceVersion:"324", FieldPath:""}): type: 'Warning' reason: 'FailedCreate' Error creating: pods "coredns-66bff467f8-" is forbidden: no providers available to validate pod request E0330 04:09:55.602503 1 daemon_controller.go:957] pods "kube-proxy-" is forbidden: no providers available to validate pod request E0330 04:09:55.602672 1 daemon_controller.go:292] kube-system/kube-proxy failed with : pods "kube-proxy-" is forbidden: no providers available to validate pod request I0330 04:09:55.602740 1 event.go:278] Event(v1.ObjectReference{Kind:"DaemonSet", Namespace:"kube-system", Name:"kube-proxy", UID:"a0b4a679-1fb6-4e96-a3b8-0cc868cfa473", APIVersion:"apps/v1", ResourceVersion:"238", FieldPath:""}): type: 'Warning' reason: 'FailedCreate' Error creating: pods "kube-proxy-" is forbidden: no providers available to validate pod request E0330 04:09:55.927744 1 daemon_controller.go:957] pods "kube-proxy-" is forbidden: no providers available to validate pod request E0330 04:09:55.927794 1 daemon_controller.go:292] kube-system/kube-proxy failed with : pods "kube-proxy-" is forbidden: no providers available to validate pod request I0330 04:09:55.928328 1 event.go:278] Event(v1.ObjectReference{Kind:"DaemonSet", Namespace:"kube-system", Name:"kube-proxy", UID:"a0b4a679-1fb6-4e96-a3b8-0cc868cfa473", APIVersion:"apps/v1", ResourceVersion:"238", FieldPath:""}): type: 'Warning' reason: 'FailedCreate' Error creating: pods "kube-proxy-" is forbidden: no providers available to validate pod request E0330 04:09:56.110620 1 replica_set.go:535] sync "kube-system/coredns-66bff467f8" failed with pods "coredns-66bff467f8-" is forbidden: no providers available to validate pod request I0330 04:09:56.110700 1 event.go:278] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"kube-system", Name:"coredns-66bff467f8", UID:"d6bcb9c5-9a42-4994-b34d-8d45f86626c9", APIVersion:"apps/v1", ResourceVersion:"324", FieldPath:""}): type: 'Warning' reason: 'FailedCreate' Error creating: pods "coredns-66bff467f8-" is forbidden: no providers available to validate pod request E0330 04:09:56.580589 1 daemon_controller.go:957] pods "kube-proxy-" is forbidden: no providers available to validate pod request E0330 04:09:56.580633 1 daemon_controller.go:292] kube-system/kube-proxy failed with : pods "kube-proxy-" is forbidden: no providers available to validate pod request I0330 04:09:56.581380 1 event.go:278] Event(v1.ObjectReference{Kind:"DaemonSet", Namespace:"kube-system", Name:"kube-proxy", UID:"a0b4a679-1fb6-4e96-a3b8-0cc868cfa473", APIVersion:"apps/v1", ResourceVersion:"238", FieldPath:""}): type: 'Warning' reason: 'FailedCreate' Error creating: pods "kube-proxy-" is forbidden: no providers available to validate pod request E0330 04:09:57.392930 1 replica_set.go:535] sync "kube-system/coredns-66bff467f8" failed with pods "coredns-66bff467f8-" is forbidden: no providers available to validate pod request I0330 04:09:57.393886 1 event.go:278] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"kube-system", Name:"coredns-66bff467f8", UID:"d6bcb9c5-9a42-4994-b34d-8d45f86626c9", APIVersion:"apps/v1", ResourceVersion:"324", FieldPath:""}): type: 'Warning' reason: 'FailedCreate' Error creating: pods "coredns-66bff467f8-" is forbidden: no providers available to validate pod request E0330 04:09:57.881472 1 daemon_controller.go:957] pods "kube-proxy-" is forbidden: no providers available to validate pod request E0330 04:09:57.881619 1 daemon_controller.go:292] kube-system/kube-proxy failed with : pods "kube-proxy-" is forbidden: no providers available to validate pod request I0330 04:09:57.883677 1 event.go:278] Event(v1.ObjectReference{Kind:"DaemonSet", Namespace:"kube-system", Name:"kube-proxy", UID:"a0b4a679-1fb6-4e96-a3b8-0cc868cfa473", APIVersion:"apps/v1", ResourceVersion:"238", FieldPath:""}): type: 'Warning' reason: 'FailedCreate' Error creating: pods "kube-proxy-" is forbidden: no providers available to validate pod request E0330 04:09:59.980851 1 replica_set.go:535] sync "kube-system/coredns-66bff467f8" failed with pods "coredns-66bff467f8-" is forbidden: no providers available to validate pod request I0330 04:09:59.982498 1 event.go:278] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"kube-system", Name:"coredns-66bff467f8", UID:"d6bcb9c5-9a42-4994-b34d-8d45f86626c9", APIVersion:"apps/v1", ResourceVersion:"324", FieldPath:""}): type: 'Warning' reason: 'FailedCreate' Error creating: pods "coredns-66bff467f8-" is forbidden: no providers available to validate pod request E0330 04:10:00.476709 1 daemon_controller.go:957] pods "kube-proxy-" is forbidden: no providers available to validate pod request E0330 04:10:00.487621 1 daemon_controller.go:292] kube-system/kube-proxy failed with : pods "kube-proxy-" is forbidden: no providers available to validate pod request I0330 04:10:00.489289 1 event.go:278] Event(v1.ObjectReference{Kind:"DaemonSet", Namespace:"kube-system", Name:"kube-proxy", UID:"a0b4a679-1fb6-4e96-a3b8-0cc868cfa473", APIVersion:"apps/v1", ResourceVersion:"238", FieldPath:""}): type: 'Warning' reason: 'FailedCreate' Error creating: pods "kube-proxy-" is forbidden: no providers available to validate pod request E0330 04:10:05.103399 1 replica_set.go:535] sync "kube-system/coredns-66bff467f8" failed with pods "coredns-66bff467f8-" is forbidden: no providers available to validate pod request I0330 04:10:05.104047 1 event.go:278] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"kube-system", Name:"coredns-66bff467f8", UID:"d6bcb9c5-9a42-4994-b34d-8d45f86626c9", APIVersion:"apps/v1", ResourceVersion:"324", FieldPath:""}): type: 'Warning' reason: 'FailedCreate' Error creating: pods "coredns-66bff467f8-" is forbidden: no providers available to validate pod request E0330 04:10:05.618692 1 daemon_controller.go:957] pods "kube-proxy-" is forbidden: no providers available to validate pod request E0330 04:10:05.618729 1 daemon_controller.go:292] kube-system/kube-proxy failed with : pods "kube-proxy-" is forbidden: no providers available to validate pod request I0330 04:10:05.619290 1 event.go:278] Event(v1.ObjectReference{Kind:"DaemonSet", Namespace:"kube-system", Name:"kube-proxy", UID:"a0b4a679-1fb6-4e96-a3b8-0cc868cfa473", APIVersion:"apps/v1", ResourceVersion:"238", FieldPath:""}): type: 'Warning' reason: 'FailedCreate' Error creating: pods "kube-proxy-" is forbidden: no providers available to validate pod request E0330 04:10:15.346058 1 replica_set.go:535] sync "kube-system/coredns-66bff467f8" failed with pods "coredns-66bff467f8-" is forbidden: no providers available to validate pod request I0330 04:10:15.346398 1 event.go:278] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"kube-system", Name:"coredns-66bff467f8", UID:"d6bcb9c5-9a42-4994-b34d-8d45f86626c9", APIVersion:"apps/v1", ResourceVersion:"324", FieldPath:""}): type: 'Warning' reason: 'FailedCreate' Error creating: pods "coredns-66bff467f8-" is forbidden: no providers available to validate pod request E0330 04:10:15.874562 1 daemon_controller.go:957] pods "kube-proxy-" is forbidden: no providers available to validate pod request E0330 04:10:15.874600 1 daemon_controller.go:292] kube-system/kube-proxy failed with : pods "kube-proxy-" is forbidden: no providers available to validate pod request I0330 04:10:15.874845 1 event.go:278] Event(v1.ObjectReference{Kind:"DaemonSet", Namespace:"kube-system", Name:"kube-proxy", UID:"a0b4a679-1fb6-4e96-a3b8-0cc868cfa473", APIVersion:"apps/v1", ResourceVersion:"238", FieldPath:""}): type: 'Warning' reason: 'FailedCreate' Error creating: pods "kube-proxy-" is forbidden: no providers available to validate pod request E0330 04:10:35.829441 1 replica_set.go:535] sync "kube-system/coredns-66bff467f8" failed with pods "coredns-66bff467f8-" is forbidden: no providers available to validate pod request I0330 04:10:35.830080 1 event.go:278] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"kube-system", Name:"coredns-66bff467f8", UID:"d6bcb9c5-9a42-4994-b34d-8d45f86626c9", APIVersion:"apps/v1", ResourceVersion:"324", FieldPath:""}): type: 'Warning' reason: 'FailedCreate' Error creating: pods "coredns-66bff467f8-" is forbidden: no providers available to validate pod request E0330 04:10:36.359855 1 daemon_controller.go:957] pods "kube-proxy-" is forbidden: no providers available to validate pod request E0330 04:10:36.360048 1 daemon_controller.go:292] kube-system/kube-proxy failed with : pods "kube-proxy-" is forbidden: no providers available to validate pod request I0330 04:10:36.361709 1 event.go:278] Event(v1.ObjectReference{Kind:"DaemonSet", Namespace:"kube-system", Name:"kube-proxy", UID:"a0b4a679-1fb6-4e96-a3b8-0cc868cfa473", APIVersion:"apps/v1", ResourceVersion:"238", FieldPath:""}): type: 'Warning' reason: 'FailedCreate' Error creating: pods "kube-proxy-" is forbidden: no providers available to validate pod request E0330 04:11:16.793272 1 replica_set.go:535] sync "kube-system/coredns-66bff467f8" failed with pods "coredns-66bff467f8-" is forbidden: no providers available to validate pod request I0330 04:11:16.793835 1 event.go:278] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"kube-system", Name:"coredns-66bff467f8", UID:"d6bcb9c5-9a42-4994-b34d-8d45f86626c9", APIVersion:"apps/v1", ResourceVersion:"324", FieldPath:""}): type: 'Warning' reason: 'FailedCreate' Error creating: pods "coredns-66bff467f8-" is forbidden: no providers available to validate pod request E0330 04:11:17.334639 1 daemon_controller.go:957] pods "kube-proxy-" is forbidden: no providers available to validate pod request E0330 04:11:17.334768 1 daemon_controller.go:292] kube-system/kube-proxy failed with : pods "kube-proxy-" is forbidden: no providers available to validate pod request I0330 04:11:17.336109 1 event.go:278] Event(v1.ObjectReference{Kind:"DaemonSet", Namespace:"kube-system", Name:"kube-proxy", UID:"a0b4a679-1fb6-4e96-a3b8-0cc868cfa473", APIVersion:"apps/v1", ResourceVersion:"238", FieldPath:""}): type: 'Warning' reason: 'FailedCreate' Error creating: pods "kube-proxy-" is forbidden: no providers available to validate pod request E0330 04:12:38.717517 1 replica_set.go:535] sync "kube-system/coredns-66bff467f8" failed with pods "coredns-66bff467f8-" is forbidden: no providers available to validate pod request I0330 04:12:38.718095 1 event.go:278] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"kube-system", Name:"coredns-66bff467f8", UID:"d6bcb9c5-9a42-4994-b34d-8d45f86626c9", APIVersion:"apps/v1", ResourceVersion:"324", FieldPath:""}): type: 'Warning' reason: 'FailedCreate' Error creating: pods "coredns-66bff467f8-" is forbidden: no providers available to validate pod request E0330 04:12:39.261473 1 daemon_controller.go:957] pods "kube-proxy-" is forbidden: no providers available to validate pod request I0330 04:12:39.261527 1 event.go:278] Event(v1.ObjectReference{Kind:"DaemonSet", Namespace:"kube-system", Name:"kube-proxy", UID:"a0b4a679-1fb6-4e96-a3b8-0cc868cfa473", APIVersion:"apps/v1", ResourceVersion:"238", FieldPath:""}): type: 'Warning' reason: 'FailedCreate' Error creating: pods "kube-proxy-" is forbidden: no providers available to validate pod request E0330 04:12:39.261977 1 daemon_controller.go:292] kube-system/kube-proxy failed with : pods "kube-proxy-" is forbidden: no providers available to validate pod request E0330 04:15:22.577727 1 replica_set.go:535] sync "kube-system/coredns-66bff467f8" failed with pods "coredns-66bff467f8-" is forbidden: no providers available to validate pod request I0330 04:15:22.586521 1 event.go:278] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"kube-system", Name:"coredns-66bff467f8", UID:"d6bcb9c5-9a42-4994-b34d-8d45f86626c9", APIVersion:"apps/v1", ResourceVersion:"324", FieldPath:""}): type: 'Warning' reason: 'FailedCreate' Error creating: pods "coredns-66bff467f8-" is forbidden: no providers available to validate pod request E0330 04:15:23.106801 1 daemon_controller.go:957] pods "kube-proxy-" is forbidden: no providers available to validate pod request E0330 04:15:23.106843 1 daemon_controller.go:292] kube-system/kube-proxy failed with : pods "kube-proxy-" is forbidden: no providers available to validate pod request I0330 04:15:23.107092 1 event.go:278] Event(v1.ObjectReference{Kind:"DaemonSet", Namespace:"kube-system", Name:"kube-proxy", UID:"a0b4a679-1fb6-4e96-a3b8-0cc868cfa473", APIVersion:"apps/v1", ResourceVersion:"238", FieldPath:""}): type: 'Warning' reason: 'FailedCreate' Error creating: pods "kube-proxy-" is forbidden: no providers available to validate pod request ==> kube-scheduler [46301bfe3009] <== I0330 04:09:38.960660 1 registry.go:150] Registering EvenPodsSpread predicate and priority function I0330 04:09:38.960768 1 registry.go:150] Registering EvenPodsSpread predicate and priority function W0330 04:09:38.965504 1 feature_gate.go:235] Setting GA feature gate BlockVolume=true. It will be removed in a future release. W0330 04:09:38.965546 1 feature_gate.go:235] Setting GA feature gate CSIBlockVolume=true. It will be removed in a future release. I0330 04:09:40.143087 1 serving.go:313] Generated self-signed cert in-memory W0330 04:09:44.554025 1 authentication.go:349] Unable to get configmap/extension-apiserver-authentication in kube-system. Usually fixed by 'kubectl create rolebinding -n kube-system ROLEBINDING_NAME --role=extension-apiserver-authentication-reader --serviceaccount=YOUR_NS:YOUR_SA' W0330 04:09:44.554073 1 authentication.go:297] Error looking up in-cluster authentication configuration: configmaps "extension-apiserver-authentication" is forbidden: User "system:kube-scheduler" cannot get resource "configmaps" in API group "" in the namespace "kube-system" W0330 04:09:44.554086 1 authentication.go:298] Continuing without authentication configuration. This may treat all requests as anonymous. W0330 04:09:44.554095 1 authentication.go:299] To require authentication configuration lookup to succeed, set --authentication-tolerate-lookup-failure=false I0330 04:09:44.600469 1 registry.go:150] Registering EvenPodsSpread predicate and priority function I0330 04:09:44.600531 1 registry.go:150] Registering EvenPodsSpread predicate and priority function W0330 04:09:44.602849 1 authorization.go:47] Authorization is disabled W0330 04:09:44.602913 1 authentication.go:40] Authentication is disabled I0330 04:09:44.602930 1 deprecated_insecure_serving.go:51] Serving healthz insecurely on [::]:10251 I0330 04:09:44.605539 1 secure_serving.go:178] Serving securely on 127.0.0.1:10259 I0330 04:09:44.605969 1 configmap_cafile_content.go:202] Starting client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0330 04:09:44.606055 1 shared_informer.go:223] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0330 04:09:44.606418 1 tlsconfig.go:240] Starting DynamicServingCertificateController E0330 04:09:44.618847 1 reflector.go:178] k8s.io/client-go/informers/factory.go:135: Failed to list *v1.PersistentVolume: persistentvolumes is forbidden: User "system:kube-scheduler" cannot list resource "persistentvolumes" in API group "" at the cluster scope E0330 04:09:44.629777 1 reflector.go:178] k8s.io/client-go/informers/factory.go:135: Failed to list *v1.CSINode: csinodes.storage.k8s.io is forbidden: User "system:kube-scheduler" cannot list resource "csinodes" in API group "storage.k8s.io" at the cluster scope E0330 04:09:44.630689 1 reflector.go:178] k8s.io/kubernetes/cmd/kube-scheduler/app/server.go:233: Failed to list *v1.Pod: pods is forbidden: User "system:kube-scheduler" cannot list resource "pods" in API group "" at the cluster scope E0330 04:09:44.633699 1 reflector.go:178] k8s.io/client-go/informers/factory.go:135: Failed to list *v1.Node: nodes is forbidden: User "system:kube-scheduler" cannot list resource "nodes" in API group "" at the cluster scope E0330 04:09:44.638872 1 reflector.go:178] k8s.io/client-go/informers/factory.go:135: Failed to list *v1.PersistentVolumeClaim: persistentvolumeclaims is forbidden: User "system:kube-scheduler" cannot list resource "persistentvolumeclaims" in API group "" at the cluster scope E0330 04:09:44.646463 1 reflector.go:178] k8s.io/apiserver/pkg/server/dynamiccertificates/configmap_cafile_content.go:206: Failed to list *v1.ConfigMap: configmaps "extension-apiserver-authentication" is forbidden: User "system:kube-scheduler" cannot list resource "configmaps" in API group "" in the namespace "kube-system" E0330 04:09:44.647145 1 reflector.go:178] k8s.io/kubernetes/cmd/kube-scheduler/app/server.go:233: Failed to list *v1.Pod: pods is forbidden: User "system:kube-scheduler" cannot list resource "pods" in API group "" at the cluster scope E0330 04:09:44.647639 1 reflector.go:178] k8s.io/client-go/informers/factory.go:135: Failed to list *v1.PersistentVolume: persistentvolumes is forbidden: User "system:kube-scheduler" cannot list resource "persistentvolumes" in API group "" at the cluster scope E0330 04:09:44.647853 1 reflector.go:178] k8s.io/client-go/informers/factory.go:135: Failed to list *v1.StorageClass: storageclasses.storage.k8s.io is forbidden: User "system:kube-scheduler" cannot list resource "storageclasses" in API group "storage.k8s.io" at the cluster scope E0330 04:09:44.648058 1 reflector.go:178] k8s.io/client-go/informers/factory.go:135: Failed to list *v1.Service: services is forbidden: User "system:kube-scheduler" cannot list resource "services" in API group "" at the cluster scope E0330 04:09:44.648250 1 reflector.go:178] k8s.io/client-go/informers/factory.go:135: Failed to list *v1.CSINode: csinodes.storage.k8s.io is forbidden: User "system:kube-scheduler" cannot list resource "csinodes" in API group "storage.k8s.io" at the cluster scope E0330 04:09:44.648337 1 reflector.go:178] k8s.io/client-go/informers/factory.go:135: Failed to list *v1beta1.PodDisruptionBudget: poddisruptionbudgets.policy is forbidden: User "system:kube-scheduler" cannot list resource "poddisruptionbudgets" in API group "policy" at the cluster scope E0330 04:09:44.649806 1 reflector.go:178] k8s.io/client-go/informers/factory.go:135: Failed to list *v1.Node: nodes is forbidden: User "system:kube-scheduler" cannot list resource "nodes" in API group "" at the cluster scope E0330 04:09:44.649816 1 reflector.go:178] k8s.io/apiserver/pkg/server/dynamiccertificates/configmap_cafile_content.go:206: Failed to list *v1.ConfigMap: configmaps "extension-apiserver-authentication" is forbidden: User "system:kube-scheduler" cannot list resource "configmaps" in API group "" in the namespace "kube-system" E0330 04:09:44.658467 1 reflector.go:178] k8s.io/client-go/informers/factory.go:135: Failed to list *v1.StorageClass: storageclasses.storage.k8s.io is forbidden: User "system:kube-scheduler" cannot list resource "storageclasses" in API group "storage.k8s.io" at the cluster scope E0330 04:09:44.658702 1 reflector.go:178] k8s.io/client-go/informers/factory.go:135: Failed to list *v1.Service: services is forbidden: User "system:kube-scheduler" cannot list resource "services" in API group "" at the cluster scope E0330 04:09:44.664599 1 reflector.go:178] k8s.io/client-go/informers/factory.go:135: Failed to list *v1.PersistentVolumeClaim: persistentvolumeclaims is forbidden: User "system:kube-scheduler" cannot list resource "persistentvolumeclaims" in API group "" at the cluster scope E0330 04:09:44.665079 1 reflector.go:178] k8s.io/client-go/informers/factory.go:135: Failed to list *v1beta1.PodDisruptionBudget: poddisruptionbudgets.policy is forbidden: User "system:kube-scheduler" cannot list resource "poddisruptionbudgets" in API group "policy" at the cluster scope E0330 04:09:46.323298 1 reflector.go:178] k8s.io/client-go/informers/factory.go:135: Failed to list *v1.CSINode: csinodes.storage.k8s.io is forbidden: User "system:kube-scheduler" cannot list resource "csinodes" in API group "storage.k8s.io" at the cluster scope E0330 04:09:46.328411 1 reflector.go:178] k8s.io/kubernetes/cmd/kube-scheduler/app/server.go:233: Failed to list *v1.Pod: pods is forbidden: User "system:kube-scheduler" cannot list resource "pods" in API group "" at the cluster scope E0330 04:09:46.452252 1 reflector.go:178] k8s.io/client-go/informers/factory.go:135: Failed to list *v1.PersistentVolumeClaim: persistentvolumeclaims is forbidden: User "system:kube-scheduler" cannot list resource "persistentvolumeclaims" in API group "" at the cluster scope E0330 04:09:47.083827 1 reflector.go:178] k8s.io/apiserver/pkg/server/dynamiccertificates/configmap_cafile_content.go:206: Failed to list *v1.ConfigMap: configmaps "extension-apiserver-authentication" is forbidden: User "system:kube-scheduler" cannot list resource "configmaps" in API group "" in the namespace "kube-system" I0330 04:09:50.106742 1 leaderelection.go:242] attempting to acquire leader lease kube-system/kube-scheduler... I0330 04:09:50.123352 1 leaderelection.go:252] successfully acquired lease kube-system/kube-scheduler I0330 04:09:51.006512 1 shared_informer.go:230] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file ==> kubelet <== -- Logs begin at Mon 2020-03-30 04:07:01 UTC, end at Mon 2020-03-30 04:16:47 UTC. -- Mar 30 04:09:56 minikube kubelet[3731]: I0330 04:09:56.251413 3731 kubelet.go:1821] Starting kubelet main sync loop. Mar 30 04:09:56 minikube kubelet[3731]: E0330 04:09:56.251483 3731 kubelet.go:1845] skipping pod synchronization - [container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful] Mar 30 04:09:56 minikube kubelet[3731]: I0330 04:09:56.287482 3731 kubelet_node_status.go:294] Setting node annotation to enable volume controller attach/detach Mar 30 04:09:56 minikube kubelet[3731]: E0330 04:09:56.356701 3731 kubelet.go:1845] skipping pod synchronization - container runtime status check may not have completed yet Mar 30 04:09:56 minikube kubelet[3731]: I0330 04:09:56.389678 3731 kubelet_node_status.go:70] Attempting to register node minikube Mar 30 04:09:56 minikube kubelet[3731]: I0330 04:09:56.444339 3731 kubelet_node_status.go:112] Node minikube was previously registered Mar 30 04:09:56 minikube kubelet[3731]: I0330 04:09:56.444499 3731 kubelet_node_status.go:73] Successfully registered node minikube Mar 30 04:09:56 minikube kubelet[3731]: I0330 04:09:56.538250 3731 setters.go:559] Node became not ready: {Type:Ready Status:False LastHeartbeatTime:2020-03-30 04:09:56.53818339 +0000 UTC m=+7.097890822 LastTransitionTime:2020-03-30 04:09:56.53818339 +0000 UTC m=+7.097890822 Reason:KubeletNotReady Message:container runtime status check may not have completed yet} Mar 30 04:09:56 minikube kubelet[3731]: E0330 04:09:56.593774 3731 kubelet.go:1845] skipping pod synchronization - container runtime status check may not have completed yet Mar 30 04:09:56 minikube kubelet[3731]: I0330 04:09:56.746844 3731 cpu_manager.go:184] [cpumanager] starting with none policy Mar 30 04:09:56 minikube kubelet[3731]: I0330 04:09:56.746882 3731 cpu_manager.go:185] [cpumanager] reconciling every 10s Mar 30 04:09:56 minikube kubelet[3731]: I0330 04:09:56.746907 3731 state_mem.go:36] [cpumanager] initializing new in-memory state store Mar 30 04:09:56 minikube kubelet[3731]: I0330 04:09:56.747154 3731 state_mem.go:88] [cpumanager] updated default cpuset: "" Mar 30 04:09:56 minikube kubelet[3731]: I0330 04:09:56.747166 3731 state_mem.go:96] [cpumanager] updated cpuset assignments: "map[]" Mar 30 04:09:56 minikube kubelet[3731]: I0330 04:09:56.747179 3731 policy_none.go:43] [cpumanager] none policy: Start Mar 30 04:09:56 minikube kubelet[3731]: I0330 04:09:56.749288 3731 plugin_manager.go:114] Starting Kubelet Plugin Manager Mar 30 04:09:56 minikube kubelet[3731]: I0330 04:09:56.994380 3731 topology_manager.go:233] [topologymanager] Topology Admit Handler Mar 30 04:09:57 minikube kubelet[3731]: I0330 04:09:57.005111 3731 topology_manager.go:233] [topologymanager] Topology Admit Handler Mar 30 04:09:57 minikube kubelet[3731]: E0330 04:09:57.010003 3731 kubelet.go:1663] Failed creating a mirror pod for "etcd-minikube_kube-system(367a34d4a17a33b67b2b866e44dc8cc1)": pods "etcd-minikube" is forbidden: no providers available to validate pod request Mar 30 04:09:57 minikube kubelet[3731]: I0330 04:09:57.019625 3731 topology_manager.go:233] [topologymanager] Topology Admit Handler Mar 30 04:09:57 minikube kubelet[3731]: E0330 04:09:57.024743 3731 kubelet.go:1663] Failed creating a mirror pod for "kube-apiserver-minikube_kube-system(261527358c97b97187fa626a08a809b6)": pods "kube-apiserver-minikube" is forbidden: no providers available to validate pod request Mar 30 04:09:57 minikube kubelet[3731]: I0330 04:09:57.041425 3731 topology_manager.go:233] [topologymanager] Topology Admit Handler Mar 30 04:09:57 minikube kubelet[3731]: E0330 04:09:57.047681 3731 kubelet.go:1663] Failed creating a mirror pod for "kube-controller-manager-minikube_kube-system(6b67ecbb7b08edb9395c116637cc448c)": pods "kube-controller-manager-minikube" is forbidden: no providers available to validate pod request Mar 30 04:09:57 minikube kubelet[3731]: E0330 04:09:57.066751 3731 kubelet.go:1663] Failed creating a mirror pod for "kube-scheduler-minikube_kube-system(820039297e3264fe6a059ad7eccb9d3f)": pods "kube-scheduler-minikube" is forbidden: no providers available to validate pod request Mar 30 04:09:57 minikube kubelet[3731]: I0330 04:09:57.097198 3731 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "etcd-certs" (UniqueName: "kubernetes.io/host-path/367a34d4a17a33b67b2b866e44dc8cc1-etcd-certs") pod "etcd-minikube" (UID: "367a34d4a17a33b67b2b866e44dc8cc1") Mar 30 04:09:57 minikube kubelet[3731]: I0330 04:09:57.097347 3731 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "kubeconfig" (UniqueName: "kubernetes.io/host-path/6b67ecbb7b08edb9395c116637cc448c-kubeconfig") pod "kube-controller-manager-minikube" (UID: "6b67ecbb7b08edb9395c116637cc448c") Mar 30 04:09:57 minikube kubelet[3731]: I0330 04:09:57.097436 3731 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "k8s-certs" (UniqueName: "kubernetes.io/host-path/6b67ecbb7b08edb9395c116637cc448c-k8s-certs") pod "kube-controller-manager-minikube" (UID: "6b67ecbb7b08edb9395c116637cc448c") Mar 30 04:09:57 minikube kubelet[3731]: I0330 04:09:57.097549 3731 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "usr-share-ca-certificates" (UniqueName: "kubernetes.io/host-path/6b67ecbb7b08edb9395c116637cc448c-usr-share-ca-certificates") pod "kube-controller-manager-minikube" (UID: "6b67ecbb7b08edb9395c116637cc448c") Mar 30 04:09:57 minikube kubelet[3731]: I0330 04:09:57.097690 3731 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "etcd-data" (UniqueName: "kubernetes.io/host-path/367a34d4a17a33b67b2b866e44dc8cc1-etcd-data") pod "etcd-minikube" (UID: "367a34d4a17a33b67b2b866e44dc8cc1") Mar 30 04:09:57 minikube kubelet[3731]: I0330 04:09:57.097792 3731 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "ca-certs" (UniqueName: "kubernetes.io/host-path/261527358c97b97187fa626a08a809b6-ca-certs") pod "kube-apiserver-minikube" (UID: "261527358c97b97187fa626a08a809b6") Mar 30 04:09:57 minikube kubelet[3731]: I0330 04:09:57.097908 3731 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "k8s-certs" (UniqueName: "kubernetes.io/host-path/261527358c97b97187fa626a08a809b6-k8s-certs") pod "kube-apiserver-minikube" (UID: "261527358c97b97187fa626a08a809b6") Mar 30 04:09:57 minikube kubelet[3731]: I0330 04:09:57.098111 3731 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "usr-share-ca-certificates" (UniqueName: "kubernetes.io/host-path/261527358c97b97187fa626a08a809b6-usr-share-ca-certificates") pod "kube-apiserver-minikube" (UID: "261527358c97b97187fa626a08a809b6") Mar 30 04:09:57 minikube kubelet[3731]: I0330 04:09:57.098209 3731 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "ca-certs" (UniqueName: "kubernetes.io/host-path/6b67ecbb7b08edb9395c116637cc448c-ca-certs") pod "kube-controller-manager-minikube" (UID: "6b67ecbb7b08edb9395c116637cc448c") Mar 30 04:09:57 minikube kubelet[3731]: I0330 04:09:57.098301 3731 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "flexvolume-dir" (UniqueName: "kubernetes.io/host-path/6b67ecbb7b08edb9395c116637cc448c-flexvolume-dir") pod "kube-controller-manager-minikube" (UID: "6b67ecbb7b08edb9395c116637cc448c") Mar 30 04:09:57 minikube kubelet[3731]: I0330 04:09:57.098414 3731 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "kubeconfig" (UniqueName: "kubernetes.io/host-path/820039297e3264fe6a059ad7eccb9d3f-kubeconfig") pod "kube-scheduler-minikube" (UID: "820039297e3264fe6a059ad7eccb9d3f") Mar 30 04:09:57 minikube kubelet[3731]: I0330 04:09:57.098492 3731 reconciler.go:157] Reconciler: start to sync state Mar 30 04:09:57 minikube kubelet[3731]: E0330 04:09:57.424752 3731 kubelet.go:1663] Failed creating a mirror pod for "kube-apiserver-minikube_kube-system(261527358c97b97187fa626a08a809b6)": pods "kube-apiserver-minikube" is forbidden: no providers available to validate pod request Mar 30 04:09:57 minikube kubelet[3731]: E0330 04:09:57.426406 3731 kubelet.go:1663] Failed creating a mirror pod for "kube-scheduler-minikube_kube-system(820039297e3264fe6a059ad7eccb9d3f)": pods "kube-scheduler-minikube" is forbidden: no providers available to validate pod request Mar 30 04:09:57 minikube kubelet[3731]: E0330 04:09:57.426957 3731 kubelet.go:1663] Failed creating a mirror pod for "etcd-minikube_kube-system(367a34d4a17a33b67b2b866e44dc8cc1)": pods "etcd-minikube" is forbidden: no providers available to validate pod request Mar 30 04:09:57 minikube kubelet[3731]: E0330 04:09:57.571092 3731 kubelet.go:1663] Failed creating a mirror pod for "kube-controller-manager-minikube_kube-system(6b67ecbb7b08edb9395c116637cc448c)": pods "kube-controller-manager-minikube" is forbidden: no providers available to validate pod request Mar 30 04:11:15 minikube kubelet[3731]: E0330 04:11:15.479619 3731 kubelet.go:1663] Failed creating a mirror pod for "kube-controller-manager-minikube_kube-system(6b67ecbb7b08edb9395c116637cc448c)": pods "kube-controller-manager-minikube" is forbidden: no providers available to validate pod request Mar 30 04:11:17 minikube kubelet[3731]: E0330 04:11:17.255439 3731 kubelet.go:1663] Failed creating a mirror pod for "kube-scheduler-minikube_kube-system(820039297e3264fe6a059ad7eccb9d3f)": pods "kube-scheduler-minikube" is forbidden: no providers available to validate pod request Mar 30 04:11:20 minikube kubelet[3731]: E0330 04:11:20.258653 3731 kubelet.go:1663] Failed creating a mirror pod for "kube-apiserver-minikube_kube-system(261527358c97b97187fa626a08a809b6)": pods "kube-apiserver-minikube" is forbidden: no providers available to validate pod request Mar 30 04:11:24 minikube kubelet[3731]: E0330 04:11:24.255030 3731 kubelet.go:1663] Failed creating a mirror pod for "etcd-minikube_kube-system(367a34d4a17a33b67b2b866e44dc8cc1)": pods "etcd-minikube" is forbidden: no providers available to validate pod request Mar 30 04:12:21 minikube kubelet[3731]: E0330 04:12:21.260307 3731 kubelet.go:1663] Failed creating a mirror pod for "kube-controller-manager-minikube_kube-system(6b67ecbb7b08edb9395c116637cc448c)": pods "kube-controller-manager-minikube" is forbidden: no providers available to validate pod request Mar 30 04:12:28 minikube kubelet[3731]: E0330 04:12:28.265149 3731 kubelet.go:1663] Failed creating a mirror pod for "kube-scheduler-minikube_kube-system(820039297e3264fe6a059ad7eccb9d3f)": pods "kube-scheduler-minikube" is forbidden: no providers available to validate pod request Mar 30 04:12:35 minikube kubelet[3731]: E0330 04:12:35.270384 3731 kubelet.go:1663] Failed creating a mirror pod for "kube-apiserver-minikube_kube-system(261527358c97b97187fa626a08a809b6)": pods "kube-apiserver-minikube" is forbidden: no providers available to validate pod request Mar 30 04:12:44 minikube kubelet[3731]: E0330 04:12:44.254587 3731 kubelet.go:1663] Failed creating a mirror pod for "etcd-minikube_kube-system(367a34d4a17a33b67b2b866e44dc8cc1)": pods "etcd-minikube" is forbidden: no providers available to validate pod request Mar 30 04:13:32 minikube kubelet[3731]: E0330 04:13:32.389443 3731 kubelet.go:1663] Failed creating a mirror pod for "kube-controller-manager-minikube_kube-system(6b67ecbb7b08edb9395c116637cc448c)": pods "kube-controller-manager-minikube" is forbidden: no providers available to validate pod request Mar 30 04:13:54 minikube kubelet[3731]: E0330 04:13:54.277780 3731 kubelet.go:1663] Failed creating a mirror pod for "kube-scheduler-minikube_kube-system(820039297e3264fe6a059ad7eccb9d3f)": pods "kube-scheduler-minikube" is forbidden: no providers available to validate pod request Mar 30 04:14:01 minikube kubelet[3731]: E0330 04:14:01.261594 3731 kubelet.go:1663] Failed creating a mirror pod for "etcd-minikube_kube-system(367a34d4a17a33b67b2b866e44dc8cc1)": pods "etcd-minikube" is forbidden: no providers available to validate pod request Mar 30 04:14:04 minikube kubelet[3731]: E0330 04:14:04.256619 3731 kubelet.go:1663] Failed creating a mirror pod for "kube-apiserver-minikube_kube-system(261527358c97b97187fa626a08a809b6)": pods "kube-apiserver-minikube" is forbidden: no providers available to validate pod request Mar 30 04:14:44 minikube kubelet[3731]: E0330 04:14:44.273846 3731 kubelet.go:1663] Failed creating a mirror pod for "kube-controller-manager-minikube_kube-system(6b67ecbb7b08edb9395c116637cc448c)": pods "kube-controller-manager-minikube" is forbidden: no providers available to validate pod request Mar 30 04:15:17 minikube kubelet[3731]: E0330 04:15:17.262693 3731 kubelet.go:1663] Failed creating a mirror pod for "kube-scheduler-minikube_kube-system(820039297e3264fe6a059ad7eccb9d3f)": pods "kube-scheduler-minikube" is forbidden: no providers available to validate pod request Mar 30 04:15:18 minikube kubelet[3731]: E0330 04:15:18.269672 3731 kubelet.go:1663] Failed creating a mirror pod for "kube-apiserver-minikube_kube-system(261527358c97b97187fa626a08a809b6)": pods "kube-apiserver-minikube" is forbidden: no providers available to validate pod request Mar 30 04:15:19 minikube kubelet[3731]: E0330 04:15:19.262039 3731 kubelet.go:1663] Failed creating a mirror pod for "etcd-minikube_kube-system(367a34d4a17a33b67b2b866e44dc8cc1)": pods "etcd-minikube" is forbidden: no providers available to validate pod request Mar 30 04:16:02 minikube kubelet[3731]: E0330 04:16:02.267110 3731 kubelet.go:1663] Failed creating a mirror pod for "kube-controller-manager-minikube_kube-system(6b67ecbb7b08edb9395c116637cc448c)": pods "kube-controller-manager-minikube" is forbidden: no providers available to validate pod request Mar 30 04:16:40 minikube kubelet[3731]: E0330 04:16:40.275015 3731 kubelet.go:1663] Failed creating a mirror pod for "kube-scheduler-minikube_kube-system(820039297e3264fe6a059ad7eccb9d3f)": pods "kube-scheduler-minikube" is forbidden: no providers available to validate pod request Mar 30 04:16:42 minikube kubelet[3731]: E0330 04:16:42.298794 3731 kubelet.go:1663] Failed creating a mirror pod for "kube-apiserver-minikube_kube-system(261527358c97b97187fa626a08a809b6)": pods "kube-apiserver-minikube" is forbidden: no providers available to validate pod request Mar 30 04:16:47 minikube kubelet[3731]: E0330 04:16:47.254804 3731 kubelet.go:1663] Failed creating a mirror pod for "etcd-minikube_kube-system(367a34d4a17a33b67b2b866e44dc8cc1)": pods "etcd-minikube" is forbidden: no providers available to validate pod request

[Madhu-1]

am seeing this issue when i enable --extra-config=apiserver.enable-admission-plugins=PodSecurityPolicy when starting minkube

[tstromberg]

Your root cause is here:

Mar 30 04:12:35 minikube kubelet[3731]: E0330 04:12:35.270384 3731 kubelet.go:1663] Failed creating a mirror pod for "kube-apiserver-minikube_kube-system(261527358c97b97187fa626a08a809b6)": pods "kube-apiserver-minikube" is forbidden: no providers available to validate pod request

I believe you will need to provide additional configuration to use PSP, otherwise the apiserver won't be allowed to start-up. Do you mind checking if the guidance at https://minikube.sigs.k8s.io/docs/tutorials/using_psp/ is helpful?

[Madhu-1]

@tstromberg we are using PSP with minikube 1.6.0 and all is working fine https://github.com/ceph/ceph-csi/blob/master/scripts/minikube.sh#L81-L97 this is failing on minikube 1.8.0 onwards

[medyagh]

@Madhu-1 here are couple things to try. and I am curious to see the result:

1- Does it work with latest minikube and setting kubernetes version to an older version ? (the one existed before 1.8.0) ?

minikube start --kubernetes-version=....

2- Does it with work latest minikube version and an old VM ISO? you can specify the iso_url

minikube start --kubernetes-version=.... iso_url=""

[Madhu-1]

@medyagh no its didn't work with any of the kubernetes version. Even I specified 1.6.0. ISO when starting the minikube it failed for me

minikube start --memory="${MEMORY}" --iso-url='https://storage.googleapis.com/minikube/iso/minikube-v1.6.0.iso' -b kubeadm --kubernetes-version="${KUBE_VERSION}" --vm-driver="${VM_DRIVER}" --feature-gates="${K8S_FEATURE_GATES}" ${EXTRA_CONFIG}

[tstromberg]

@Madhu-1 - When you mention that this works in minikube v1.6.0, what version of Kubernetes were you using?

[Madhu-1]

@Madhu-1 - When you mention that this works in minikube v1.6.0, what version of Kubernetes were you using?

we are using kubernetes 1.14.10, 1.15.6 and 1.17.0

[elluvium]

Same to me with minikube v1.9.2, k8s v1.18.0, docker 19.03.8

[GaryBissland]

Same setup as @elluvium + driver=docker , same issue.

Looking through the logs it appears this issue occurs because psp admission controller is enabled but the psp.yaml (as described in tutorial ) is not applied which blocks system pods from starting.

Manually running the psp.yaml using kubectl during the 'minikube start' in separate terminal after control plane is active gets round this issue.

[Madhu-1]

@tstromberg @medyagh any update on this one?

[tstromberg]

@GaryBissland has the right workaround. @Madhu-1 - you didn't mention anything about psp.yaml, did you place it in the appropriate place?

https://minikube.sigs.k8s.io/docs/tutorials/using_psp/

If so, I think we have a regression with supporting a local addons directory. Please include the output of minikube start --alsologtostderr

[Madhu-1]

yes am placing PSP at the same location specified in the tutorial ( things are working fine if I use minikube 1.6.2

minikube.log attaching the logs here

[Madhu-1]

@tstromberg any update on when this will be fixed?

[priyawadhwa]

I think to fix this issue, on minikube start we need to get all of the files in /etc/kubernetes/addons and apply all of them.

This could be added in this function:

https://github.com/kubernetes/minikube/blob/ea20609a3aad5113dbb0bd2e7db027e19b0346ee/pkg/addons/addons.go#L215

and we would add the additional files to this list:

https://github.com/kubernetes/minikube/blob/ea20609a3aad5113dbb0bd2e7db027e19b0346ee/pkg/addons/addons.go#L246

If anyone is interested in fixing this, feel free to assign yourself to the issue by commenting /assign