Closed kmayura1 closed 4 years ago
sharifelgamal
commented
5 years ago Hi there, could you try the start command, but with minikube start -v 8 --alsologtostderr? Those logs will be a lot more granular and hopefully more helpful for figuring out what's happening.
kmayura1
commented
5 years ago Hi Thank you fot the reply, i got these logs from above command:
I0621 06:29:49.893954 8194 utils.go:240] > Jun 21 05:23:29 minikube kubelet[3217]: I0621 05:23:29.601737 3217 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "storage-provisioner-token-wrx72" (UniqueName: "kubernetes.io/secret/b4135fb1-93e4-11e9-95b4-080027d4e105-storage-provisioner-token-wrx72") pod "storage-provisioner" (UID: "b4135fb1-93e4-11e9-95b4-080027d4e105")
I0621 06:29:49.893989 8194 utils.go:240] > Jun 21 05:23:29 minikube kubelet[3217]: I0621 05:23:29.602080 3217 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "tmp" (UniqueName: "kubernetes.io/host-path/b4135fb1-93e4-11e9-95b4-080027d4e105-tmp") pod "storage-provisioner" (UID: "b4135fb1-93e4-11e9-95b4-080027d4e105")
I0621 06:29:49.894019 8194 utils.go:240] > Jun 21 05:23:29 minikube kubelet[3217]: W0621 05:23:29.813314 3217 container.go:409] Failed to create summary reader for "/system.slice/run-r308ba0b90998406681f0af4efc27f199.scope": none of the resources are being tracked.
I0621 06:29:49.901227 8194 logs.go:76] Gathering logs for dmesg ...
I0621 06:29:49.901249 8194 ssh_runner.go:137] Run with output: sudo dmesg -PH -L=never --level warn,err,crit,alert,emerg | tail -n 200
I0621 06:29:49.908995 8194 utils.go:240] > [Jun21 05:21] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge.
I0621 06:29:49.909052 8194 utils.go:240] > [ +16.870238] platform regulatory.0: Direct firmware load for regulatory.db failed with error -2
I0621 06:29:49.909080 8194 utils.go:240] > [Jun21 05:22] systemd-fstab-generator[1291]: Ignoring "noauto" for root device
I0621 06:29:49.909112 8194 utils.go:240] > [ +0.002403] systemd[1]: File /usr/lib/systemd/system/systemd-journald.service:35 configures an IP firewall (IPAddressDeny=any), but the local system does not support BPF/cgroup based firewalling.
I0621 06:29:49.909136 8194 utils.go:240] > [ +0.000002] systemd[1]: Proceeding WITHOUT firewalling in effect! (This warning is only shown for the first loaded unit using IP firewalling.)
I0621 06:29:49.909164 8194 utils.go:240] > [ +0.301947] vboxguest: loading out-of-tree module taints kernel.
I0621 06:29:49.909182 8194 utils.go:240] > [ +0.018997] vgdrvHeartbeatInit: Setting up heartbeat to trigger every 2000 milliseconds
I0621 06:29:49.909202 8194 utils.go:240] > [ +0.000181] vboxguest: misc device minor 57, IRQ 20, I/O port d020, MMIO at 00000000f0000000 (size 0x400000)
I0621 06:29:49.909220 8194 utils.go:240] > [ +0.000164] vboxvideo: Unknown symbol ttm_bo_mmap (err 0)
I0621 06:29:49.909231 8194 utils.go:240] > [ +0.000012] vboxvideo: Unknown symbol ttm_bo_global_release (err 0)
I0621 06:29:49.909240 8194 utils.go:240] > [ +0.000005] vboxvideo: Unknown symbol ttm_pool_unpopulate (err 0)
I0621 06:29:49.909250 8194 utils.go:240] > [ +0.000004] vboxvideo: Unknown symbol ttm_bo_manager_func (err 0)
I0621 06:29:49.909259 8194 utils.go:240] > [ +0.000003] vboxvideo: Unknown symbol ttm_bo_global_init (err 0)
I0621 06:29:49.909268 8194 utils.go:240] > [ +0.000001] vboxvideo: Unknown symbol ttm_bo_default_io_mem_pfn (err 0)
I0621 06:29:49.909278 8194 utils.go:240] > [ +0.000006] vboxvideo: Unknown symbol ttm_bo_device_release (err 0)
I0621 06:29:49.909287 8194 utils.go:240] > [ +0.000013] vboxvideo: Unknown symbol ttm_bo_kunmap (err 0)
I0621 06:29:49.909298 8194 utils.go:240] > [ +0.000003] vboxvideo: Unknown symbol ttm_bo_del_sub_from_lru (err 0)
I0621 06:29:49.909385 8194 utils.go:240] > [ +0.000007] vboxvideo: Unknown symbol ttm_bo_device_init (err 0)
I0621 06:29:49.909401 8194 utils.go:240] > [ +0.000001] vboxvideo: Unknown symbol ttm_bo_init_mm (err 0)
I0621 06:29:49.909416 8194 utils.go:240] > [ +0.000001] vboxvideo: Unknown symbol ttm_bo_dma_acc_size (err 0)
I0621 06:29:49.909426 8194 utils.go:240] > [ +0.000004] vboxvideo: Unknown symbol ttm_tt_init (err 0)
I0621 06:29:49.909441 8194 utils.go:240] > [ +0.000002] vboxvideo: Unknown symbol ttm_bo_kmap (err 0)
I0621 06:29:49.909452 8194 utils.go:240] > [ +0.000007] vboxvideo: Unknown symbol ttm_bo_add_to_lru (err 0)
I0621 06:29:49.909462 8194 utils.go:240] > [ +0.000002] vboxvideo: Unknown symbol ttm_bo_unref (err 0)
I0621 06:29:49.909474 8194 utils.go:240] > [ +0.000002] vboxvideo: Unknown symbol ttm_mem_global_release (err 0)
I0621 06:29:49.909494 8194 utils.go:240] > [ +0.000002] vboxvideo: Unknown symbol ttm_mem_global_init (err 0)
I0621 06:29:49.909504 8194 utils.go:240] > [ +0.000010] vboxvideo: Unknown symbol ttm_bo_init (err 0)
I0621 06:29:49.909513 8194 utils.go:240] > [ +0.000002] vboxvideo: Unknown symbol ttm_bo_validate (err 0)
I0621 06:29:49.909522 8194 utils.go:240] > [ +0.000008] vboxvideo: Unknown symbol ttm_tt_fini (err 0)
I0621 06:29:49.909533 8194 utils.go:240] > [ +0.000016] vboxvideo: Unknown symbol ttm_bo_eviction_valuable (err 0)
I0621 06:29:49.909543 8194 utils.go:240] > [ +0.000002] vboxvideo: Unknown symbol ttm_pool_populate (err 0)
I0621 06:29:49.909551 8194 utils.go:240] > [ +0.170016] hpet1: lost 629 rtc interrupts
I0621 06:29:49.909565 8194 utils.go:240] > [ +0.011634] NFSD: the nfsdcld client tracking upcall will be removed in 3.10. Please transition to using nfsdcltrack.
I0621 06:29:49.909578 8194 utils.go:240] > [ +0.043257] VBoxService 5.1.38 r122592 (verbosity: 0) linux.amd64 (May 9 2018 12:22:30) release log
I0621 06:29:49.909589 8194 utils.go:240] > 00:00:00.008594 main Log opened 2019-06-21T05:22:12.774537000Z
I0621 06:29:49.909599 8194 utils.go:240] > [ +0.001315] 00:00:00.009908 main OS Product: Linux
I0621 06:29:49.909608 8194 utils.go:240] > [ +0.000155] 00:00:00.010100 main OS Release: 4.15.0
I0621 06:29:49.909619 8194 utils.go:240] > [ +0.000075] 00:00:00.010179 main OS Version: #1 SMP Thu Jun 6 15:07:18 PDT 2019
I0621 06:29:49.909630 8194 utils.go:240] > [ +0.000093] 00:00:00.010268 main Executable: /usr/sbin/VBoxService
I0621 06:29:49.909638 8194 utils.go:240] > 00:00:00.010268 main Process ID: 2057
I0621 06:29:49.909648 8194 utils.go:240] > 00:00:00.010269 main Package type: LINUX_64BITS_GENERIC
I0621 06:29:49.909659 8194 utils.go:240] > [ +0.000078] 00:00:00.010357 main 5.1.38 r122592 started. Verbose level = 0
I0621 06:29:49.909671 8194 utils.go:240] > [ +5.014631] hpet1: lost 282 rtc interrupts
I0621 06:29:49.909680 8194 utils.go:240] > [ +5.003479] hpet1: lost 318 rtc interrupts
I0621 06:29:49.909695 8194 utils.go:240] > [ +1.810172] systemd-fstab-generator[2284]: Ignoring "noauto" for root device
I0621 06:29:49.909705 8194 utils.go:240] > [ +3.230159] hpet_rtc_timer_reinit: 69 callbacks suppressed
I0621 06:29:49.909713 8194 utils.go:240] > [ +0.000001] hpet1: lost 320 rtc interrupts
I0621 06:29:49.909720 8194 utils.go:240] > [ +5.023070] hpet1: lost 320 rtc interrupts
I0621 06:29:49.909727 8194 utils.go:240] > [ +5.001568] hpet1: lost 318 rtc interrupts
I0621 06:29:49.909735 8194 utils.go:240] > [ +5.001352] hpet1: lost 318 rtc interrupts
I0621 06:29:49.909742 8194 utils.go:240] > [ +5.000639] hpet1: lost 319 rtc interrupts
I0621 06:29:49.909750 8194 utils.go:240] > [ +5.000681] hpet1: lost 318 rtc interrupts
I0621 06:29:49.909761 8194 utils.go:240] > [ +4.821036] systemd-fstab-generator[2970]: Ignoring "noauto" for root device
I0621 06:29:49.909768 8194 utils.go:240] > [ +0.180097] hpet1: lost 318 rtc interrupts
I0621 06:29:49.909776 8194 utils.go:240] > [ +5.001034] hpet1: lost 318 rtc interrupts
I0621 06:29:49.909786 8194 utils.go:240] > [Jun21 05:23] systemd-fstab-generator[3198]: Ignoring "noauto" for root device
I0621 06:29:49.909794 8194 utils.go:240] > [ +1.106947] hpet1: lost 319 rtc interrupts
I0621 06:29:49.909805 8194 utils.go:240] > [ +5.002379] hpet1: lost 319 rtc interrupts
I0621 06:29:49.909815 8194 utils.go:240] > [ +5.006720] hpet_rtc_timer_reinit: 21 callbacks suppressed
I0621 06:29:49.909823 8194 utils.go:240] > [ +0.000001] hpet1: lost 318 rtc interrupts
I0621 06:29:49.909831 8194 utils.go:240] > [ +5.001219] hpet1: lost 318 rtc interrupts
I0621 06:29:49.909838 8194 utils.go:240] > [ +5.001371] hpet1: lost 318 rtc interrupts
I0621 06:29:49.909846 8194 utils.go:240] > [ +5.000895] hpet1: lost 318 rtc interrupts
I0621 06:29:49.909855 8194 utils.go:240] > [ +10.001922] hpet_rtc_timer_reinit: 36 callbacks suppressed
I0621 06:29:49.909863 8194 utils.go:240] > [ +0.000010] hpet1: lost 318 rtc interrupts
I0621 06:29:49.909870 8194 utils.go:240] > [ +5.001404] hpet1: lost 319 rtc interrupts
I0621 06:29:49.909877 8194 utils.go:240] > [ +5.002824] hpet1: lost 318 rtc interrupts
I0621 06:29:49.909886 8194 utils.go:240] > [ +5.000417] hpet1: lost 318 rtc interrupts
I0621 06:29:49.909893 8194 utils.go:240] > [ +5.001004] hpet1: lost 318 rtc interrupts
I0621 06:29:49.909900 8194 utils.go:240] > [Jun21 05:24] hpet1: lost 318 rtc interrupts
I0621 06:29:49.909907 8194 utils.go:240] > [ +5.002121] hpet1: lost 318 rtc interrupts
I0621 06:29:49.909920 8194 utils.go:240] > [ +3.052271] NFSD: Unable to end grace period: -110
I0621 06:29:49.909928 8194 utils.go:240] > [ +1.952793] hpet1: lost 319 rtc interrupts
I0621 06:29:49.909935 8194 utils.go:240] > [ +5.011618] hpet1: lost 320 rtc interrupts
I0621 06:29:49.909943 8194 utils.go:240] > [ +5.004769] hpet1: lost 319 rtc interrupts
I0621 06:29:49.909951 8194 utils.go:240] > [ +5.007508] hpet1: lost 320 rtc interrupts
I0621 06:29:49.909958 8194 utils.go:240] > [ +5.004808] hpet1: lost 318 rtc interrupts
I0621 06:29:49.909965 8194 utils.go:240] > [ +5.000251] hpet1: lost 319 rtc interrupts
I0621 06:29:49.909973 8194 utils.go:240] > [ +5.001911] hpet1: lost 319 rtc interrupts
I0621 06:29:49.909981 8194 utils.go:240] > [ +5.004280] hpet1: lost 319 rtc interrupts
I0621 06:29:49.909994 8194 utils.go:240] > [ +5.000114] hpet1: lost 319 rtc interrupts
I0621 06:29:49.910002 8194 utils.go:240] > [ +5.002489] hpet1: lost 318 rtc interrupts
I0621 06:29:49.910009 8194 utils.go:240] > [Jun21 05:25] hpet1: lost 318 rtc interrupts
I0621 06:29:49.910016 8194 utils.go:240] > [ +5.002654] hpet1: lost 318 rtc interrupts
I0621 06:29:49.910023 8194 utils.go:240] > [ +5.002400] hpet1: lost 318 rtc interrupts
I0621 06:29:49.910030 8194 utils.go:240] > [ +5.002743] hpet1: lost 319 rtc interrupts
I0621 06:29:49.910040 8194 utils.go:240] > [ +5.002692] hpet1: lost 319 rtc interrupts
I0621 06:29:49.910047 8194 utils.go:240] > [ +5.002870] hpet1: lost 318 rtc interrupts
I0621 06:29:49.910054 8194 utils.go:240] > [ +5.002656] hpet1: lost 318 rtc interrupts
I0621 06:29:49.910062 8194 utils.go:240] > [ +5.006071] hpet1: lost 318 rtc interrupts
I0621 06:29:49.910069 8194 utils.go:240] > [ +5.008863] hpet1: lost 319 rtc interrupts
I0621 06:29:49.910423 8194 utils.go:240] > [ +5.004545] hpet1: lost 318 rtc interrupts
I0621 06:29:49.910436 8194 utils.go:240] > [ +5.006925] hpet1: lost 320 rtc interrupts
I0621 06:29:49.910443 8194 utils.go:240] > [ +5.008160] hpet1: lost 319 rtc interrupts
I0621 06:29:49.910451 8194 utils.go:240] > [Jun21 05:26] hpet1: lost 319 rtc interrupts
I0621 06:29:49.910458 8194 utils.go:240] > [ +5.007660] hpet1: lost 318 rtc interrupts
I0621 06:29:49.910466 8194 utils.go:240] > [ +5.006719] hpet1: lost 319 rtc interrupts
I0621 06:29:49.910473 8194 utils.go:240] > [ +5.003364] hpet1: lost 318 rtc interrupts
I0621 06:29:49.910479 8194 utils.go:240] > [ +5.006294] hpet1: lost 319 rtc interrupts
I0621 06:29:49.910487 8194 utils.go:240] > [ +5.012045] hpet1: lost 319 rtc interrupts
I0621 06:29:49.910493 8194 utils.go:240] > [ +5.006124] hpet1: lost 318 rtc interrupts
I0621 06:29:49.910502 8194 utils.go:240] > [ +5.001443] hpet1: lost 319 rtc interrupts
I0621 06:29:49.910509 8194 utils.go:240] > [ +5.000975] hpet1: lost 320 rtc interrupts
I0621 06:29:49.910516 8194 utils.go:240] > [ +5.005289] hpet1: lost 318 rtc interrupts
I0621 06:29:49.910524 8194 utils.go:240] > [ +5.002124] hpet1: lost 318 rtc interrupts
I0621 06:29:49.910530 8194 utils.go:240] > [ +5.003222] hpet1: lost 319 rtc interrupts
I0621 06:29:49.910546 8194 utils.go:240] > [Jun21 05:27] hpet1: lost 318 rtc interrupts
I0621 06:29:49.910553 8194 utils.go:240] > [ +5.004351] hpet1: lost 319 rtc interrupts
I0621 06:29:49.910562 8194 utils.go:240] > [ +5.002108] hpet1: lost 318 rtc interrupts
I0621 06:29:49.910579 8194 utils.go:240] > [ +5.002960] hpet1: lost 319 rtc interrupts
I0621 06:29:49.910587 8194 utils.go:240] > [ +5.000213] hpet1: lost 318 rtc interrupts
I0621 06:29:49.910620 8194 utils.go:240] > [ +5.001675] hpet1: lost 318 rtc interrupts
I0621 06:29:49.910630 8194 utils.go:240] > [ +5.003600] hpet1: lost 318 rtc interrupts
I0621 06:29:49.910647 8194 utils.go:240] > [ +5.004167] hpet1: lost 319 rtc interrupts
I0621 06:29:49.910654 8194 utils.go:240] > [ +5.002763] hpet1: lost 319 rtc interrupts
I0621 06:29:49.910661 8194 utils.go:240] > [ +5.002282] hpet1: lost 318 rtc interrupts
I0621 06:29:49.910674 8194 utils.go:240] > [ +5.001907] hpet1: lost 318 rtc interrupts
I0621 06:29:49.910687 8194 utils.go:240] > [ +5.004519] hpet1: lost 318 rtc interrupts
I0621 06:29:49.910720 8194 utils.go:240] > [Jun21 05:28] hpet1: lost 318 rtc interrupts
I0621 06:29:49.910733 8194 utils.go:240] > [ +5.002927] hpet1: lost 319 rtc interrupts
I0621 06:29:49.910742 8194 utils.go:240] > [ +5.001698] hpet1: lost 318 rtc interrupts
I0621 06:29:49.910749 8194 utils.go:240] > [ +5.001134] hpet1: lost 318 rtc interrupts
I0621 06:29:49.910756 8194 utils.go:240] > [ +5.005464] hpet1: lost 318 rtc interrupts
I0621 06:29:49.910764 8194 utils.go:240] > [ +4.999329] hpet1: lost 319 rtc interrupts
I0621 06:29:49.910772 8194 utils.go:240] > [ +5.000287] hpet1: lost 319 rtc interrupts
I0621 06:29:49.910778 8194 utils.go:240] > [ +5.000896] hpet1: lost 318 rtc interrupts
I0621 06:29:49.910787 8194 utils.go:240] > [ +5.002106] hpet1: lost 318 rtc interrupts
I0621 06:29:49.910794 8194 utils.go:240] > [ +5.000225] hpet1: lost 318 rtc interrupts
I0621 06:29:49.910804 8194 utils.go:240] > [ +5.000961] hpet1: lost 318 rtc interrupts
I0621 06:29:49.910811 8194 utils.go:240] > [ +5.000395] hpet1: lost 318 rtc interrupts
I0621 06:29:49.910819 8194 utils.go:240] > [Jun21 05:29] hpet1: lost 318 rtc interrupts
I0621 06:29:49.910831 8194 utils.go:240] > [ +5.001243] hpet1: lost 318 rtc interrupts
I0621 06:29:49.910838 8194 utils.go:240] > [ +5.002881] hpet1: lost 319 rtc interrupts
I0621 06:29:49.910846 8194 utils.go:240] > [ +5.004316] hpet1: lost 319 rtc interrupts
I0621 06:29:49.910854 8194 utils.go:240] > [ +5.001537] hpet1: lost 319 rtc interrupts
I0621 06:29:49.910861 8194 utils.go:240] > [ +5.003232] hpet1: lost 318 rtc interrupts
I0621 06:29:49.910869 8194 utils.go:240] > [ +5.003119] hpet1: lost 318 rtc interrupts
I0621 06:29:49.910876 8194 utils.go:240] > [ +5.001300] hpet1: lost 319 rtc interrupts
I0621 06:29:49.910883 8194 utils.go:240] > [ +5.002793] hpet1: lost 318 rtc interrupts
I0621 06:29:49.910890 8194 utils.go:240] > [ +5.003060] hpet1: lost 318 rtc interrupts
I0621 06:29:49.911859 8194 logs.go:76] Gathering logs for kube-apiserver ...
I0621 06:29:49.911875 8194 ssh_runner.go:137] Run with output: docker logs --tail 200 28ef728d43e4
I0621 06:29:49.945667 8194 utils.go:240] ! I0621 05:23:15.215749 1 client.go:352] parsed scheme: ""
I0621 06:29:49.945760 8194 utils.go:240] ! I0621 05:23:15.215774 1 client.go:352] scheme "" not registered, fallback to default scheme
I0621 06:29:49.945797 8194 utils.go:240] ! I0621 05:23:15.215803 1 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{127.0.0.1:2379 0 <nil>}]
I0621 06:29:49.945852 8194 utils.go:240] ! I0621 05:23:15.215840 1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{127.0.0.1:2379 <nil>}]
I0621 06:29:49.945973 8194 utils.go:240] ! I0621 05:23:15.216217 1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{127.0.0.1:2379 <nil>}]
I0621 06:29:49.946231 8194 utils.go:240] ! I0621 05:23:15.225586 1 client.go:352] parsed scheme: ""
I0621 06:29:49.946353 8194 utils.go:240] ! I0621 05:23:15.225647 1 client.go:352] scheme "" not registered, fallback to default scheme
I0621 06:29:49.946418 8194 utils.go:240] ! I0621 05:23:15.225682 1 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{127.0.0.1:2379 0 <nil>}]
I0621 06:29:49.946490 8194 utils.go:240] ! I0621 05:23:15.225716 1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{127.0.0.1:2379 <nil>}]
I0621 06:29:49.946696 8194 utils.go:240] ! I0621 05:23:15.226124 1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{127.0.0.1:2379 <nil>}]
I0621 06:29:49.946833 8194 utils.go:240] ! I0621 05:23:15.235464 1 client.go:352] parsed scheme: ""
I0621 06:29:49.947037 8194 utils.go:240] ! I0621 05:23:15.235491 1 client.go:352] scheme "" not registered, fallback to default scheme
I0621 06:29:49.947312 8194 utils.go:240] ! I0621 05:23:15.235545 1 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{127.0.0.1:2379 0 <nil>}]
I0621 06:29:49.947332 8194 utils.go:240] ! I0621 05:23:15.235583 1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{127.0.0.1:2379 <nil>}]
I0621 06:29:49.947346 8194 utils.go:240] ! I0621 05:23:15.235760 1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{127.0.0.1:2379 <nil>}]
I0621 06:29:49.947355 8194 utils.go:240] ! I0621 05:23:15.247610 1 client.go:352] parsed scheme: ""
I0621 06:29:49.947376 8194 utils.go:240] ! I0621 05:23:15.247640 1 client.go:352] scheme "" not registered, fallback to default scheme
I0621 06:29:49.947391 8194 utils.go:240] ! I0621 05:23:15.247706 1 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{127.0.0.1:2379 0 <nil>}]
I0621 06:29:49.947405 8194 utils.go:240] ! I0621 05:23:15.247973 1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{127.0.0.1:2379 <nil>}]
I0621 06:29:49.947419 8194 utils.go:240] ! I0621 05:23:15.248313 1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{127.0.0.1:2379 <nil>}]
I0621 06:29:49.947437 8194 utils.go:240] ! I0621 05:23:15.256853 1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{127.0.0.1:2379 <nil>}]
I0621 06:29:49.947458 8194 utils.go:240] ! I0621 05:23:15.257440 1 client.go:352] parsed scheme: ""
I0621 06:29:49.947489 8194 utils.go:240] ! I0621 05:23:15.257462 1 client.go:352] scheme "" not registered, fallback to default scheme
I0621 06:29:49.947512 8194 utils.go:240] ! I0621 05:23:15.257489 1 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{127.0.0.1:2379 0 <nil>}]
I0621 06:29:49.947538 8194 utils.go:240] ! I0621 05:23:15.257741 1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{127.0.0.1:2379 <nil>}]
I0621 06:29:49.947555 8194 utils.go:240] ! I0621 05:23:15.265690 1 client.go:352] parsed scheme: ""
I0621 06:29:49.947578 8194 utils.go:240] ! I0621 05:23:15.265746 1 client.go:352] scheme "" not registered, fallback to default scheme
I0621 06:29:49.947602 8194 utils.go:240] ! I0621 05:23:15.265770 1 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{127.0.0.1:2379 0 <nil>}]
I0621 06:29:49.947625 8194 utils.go:240] ! I0621 05:23:15.265862 1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{127.0.0.1:2379 <nil>}]
I0621 06:29:49.947648 8194 utils.go:240] ! I0621 05:23:15.266137 1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{127.0.0.1:2379 <nil>}]
I0621 06:29:49.947662 8194 utils.go:240] ! I0621 05:23:15.274281 1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{127.0.0.1:2379 <nil>}]
I0621 06:29:49.947672 8194 utils.go:240] ! I0621 05:23:15.274854 1 client.go:352] parsed scheme: ""
I0621 06:29:49.947684 8194 utils.go:240] ! I0621 05:23:15.274988 1 client.go:352] scheme "" not registered, fallback to default scheme
I0621 06:29:49.947699 8194 utils.go:240] ! I0621 05:23:15.275055 1 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{127.0.0.1:2379 0 <nil>}]
I0621 06:29:49.947713 8194 utils.go:240] ! I0621 05:23:15.275181 1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{127.0.0.1:2379 <nil>}]
I0621 06:29:49.947727 8194 utils.go:240] ! I0621 05:23:15.294333 1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{127.0.0.1:2379 <nil>}]
I0621 06:29:49.947758 8194 utils.go:240] ! W0621 05:23:15.390514 1 genericapiserver.go:344] Skipping API batch/v2alpha1 because it has no resources.
I0621 06:29:49.947778 8194 utils.go:240] ! W0621 05:23:15.395548 1 genericapiserver.go:344] Skipping API node.k8s.io/v1alpha1 because it has no resources.
I0621 06:29:49.947815 8194 utils.go:240] ! W0621 05:23:15.398682 1 genericapiserver.go:344] Skipping API rbac.authorization.k8s.io/v1alpha1 because it has no resources.
I0621 06:29:49.947841 8194 utils.go:240] ! W0621 05:23:15.399290 1 genericapiserver.go:344] Skipping API scheduling.k8s.io/v1alpha1 because it has no resources.
I0621 06:29:49.947860 8194 utils.go:240] ! W0621 05:23:15.401091 1 genericapiserver.go:344] Skipping API storage.k8s.io/v1alpha1 because it has no resources.
I0621 06:29:49.947878 8194 utils.go:240] ! E0621 05:23:16.035455 1 prometheus.go:138] failed to register depth metric admission_quota_controller: duplicate metrics collector registration attempted
I0621 06:29:49.947920 8194 utils.go:240] ! E0621 05:23:16.035632 1 prometheus.go:150] failed to register adds metric admission_quota_controller: duplicate metrics collector registration attempted
I0621 06:29:49.947943 8194 utils.go:240] ! E0621 05:23:16.035898 1 prometheus.go:162] failed to register latency metric admission_quota_controller: duplicate metrics collector registration attempted
I0621 06:29:49.947961 8194 utils.go:240] ! E0621 05:23:16.036117 1 prometheus.go:174] failed to register work_duration metric admission_quota_controller: duplicate metrics collector registration attempted
I0621 06:29:49.947980 8194 utils.go:240] ! E0621 05:23:16.036154 1 prometheus.go:189] failed to register unfinished_work_seconds metric admission_quota_controller: duplicate metrics collector registration attempted
I0621 06:29:49.948004 8194 utils.go:240] ! E0621 05:23:16.036167 1 prometheus.go:202] failed to register longest_running_processor_microseconds metric admission_quota_controller: duplicate metrics collector registration attempted
I0621 06:29:49.948034 8194 utils.go:240] ! I0621 05:23:16.036184 1 plugins.go:158] Loaded 9 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,MutatingAdmissionWebhook.
I0621 06:29:49.948060 8194 utils.go:240] ! I0621 05:23:16.036332 1 plugins.go:161] Loaded 6 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,ValidatingAdmissionWebhook,ResourceQuota.
I0621 06:29:49.948070 8194 utils.go:240] ! I0621 05:23:16.037773 1 client.go:352] parsed scheme: ""
I0621 06:29:49.948083 8194 utils.go:240] ! I0621 05:23:16.037818 1 client.go:352] scheme "" not registered, fallback to default scheme
I0621 06:29:49.948097 8194 utils.go:240] ! I0621 05:23:16.037959 1 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{127.0.0.1:2379 0 <nil>}]
I0621 06:29:49.948110 8194 utils.go:240] ! I0621 05:23:16.037988 1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{127.0.0.1:2379 <nil>}]
I0621 06:29:49.948119 8194 utils.go:240] ! I0621 05:23:16.046053 1 client.go:352] parsed scheme: ""
I0621 06:29:49.948131 8194 utils.go:240] ! I0621 05:23:16.046077 1 client.go:352] scheme "" not registered, fallback to default scheme
I0621 06:29:49.948145 8194 utils.go:240] ! I0621 05:23:16.046099 1 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{127.0.0.1:2379 0 <nil>}]
I0621 06:29:49.948159 8194 utils.go:240] ! I0621 05:23:16.046202 1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{127.0.0.1:2379 <nil>}]
I0621 06:29:49.948173 8194 utils.go:240] ! I0621 05:23:16.046491 1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{127.0.0.1:2379 <nil>}]
I0621 06:29:49.948187 8194 utils.go:240] ! I0621 05:23:16.059879 1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{127.0.0.1:2379 <nil>}]
I0621 06:29:49.948204 8194 utils.go:240] ! I0621 05:23:17.246614 1 secure_serving.go:116] Serving securely on [::]:8443
I0621 06:29:49.948214 8194 utils.go:240] ! I0621 05:23:17.246666 1 crd_finalizer.go:242] Starting CRDFinalizer
I0621 06:29:49.948226 8194 utils.go:240] ! I0621 05:23:17.247086 1 controller.go:81] Starting OpenAPI AggregationController
I0621 06:29:49.948238 8194 utils.go:240] ! I0621 05:23:17.247599 1 available_controller.go:320] Starting AvailableConditionController
I0621 06:29:49.948251 8194 utils.go:240] ! I0621 05:23:17.247622 1 cache.go:32] Waiting for caches to sync for AvailableConditionController controller
I0621 06:29:49.948264 8194 utils.go:240] ! I0621 05:23:17.247715 1 apiservice_controller.go:94] Starting APIServiceRegistrationController
I0621 06:29:49.948278 8194 utils.go:240] ! I0621 05:23:17.247911 1 cache.go:32] Waiting for caches to sync for APIServiceRegistrationController controller
I0621 06:29:49.948291 8194 utils.go:240] ! I0621 05:23:17.248374 1 autoregister_controller.go:139] Starting autoregister controller
I0621 06:29:49.948303 8194 utils.go:240] ! I0621 05:23:17.248493 1 cache.go:32] Waiting for caches to sync for autoregister controller
I0621 06:29:49.948317 8194 utils.go:240] ! I0621 05:23:17.248599 1 customresource_discovery_controller.go:208] Starting DiscoveryController
I0621 06:29:49.948328 8194 utils.go:240] ! I0621 05:23:17.248757 1 naming_controller.go:284] Starting NamingConditionController
I0621 06:29:49.948343 8194 utils.go:240] ! I0621 05:23:17.248905 1 establishing_controller.go:73] Starting EstablishingController
I0621 06:29:49.948375 8194 utils.go:240] ! I0621 05:23:17.302795 1 crdregistration_controller.go:112] Starting crd-autoregister controller
I0621 06:29:49.948418 8194 utils.go:240] ! I0621 05:23:17.302816 1 controller_utils.go:1027] Waiting for caches to sync for crd-autoregister controller
I0621 06:29:49.948453 8194 utils.go:240] ! E0621 05:23:17.303688 1 controller.go:148] Unable to remove old endpoints from kubernetes service: StorageError: key not found, Code: 1, Key: /registry/masterleases/192.168.99.164, ResourceVersion: 0, AdditionalErrorMsg:
I0621 06:29:49.948479 8194 utils.go:240] ! I0621 05:23:17.355534 1 cache.go:39] Caches are synced for APIServiceRegistrationController controller
I0621 06:29:49.948493 8194 utils.go:240] ! I0621 05:23:17.355561 1 cache.go:39] Caches are synced for autoregister controller
I0621 06:29:49.948506 8194 utils.go:240] ! I0621 05:23:17.361534 1 cache.go:39] Caches are synced for AvailableConditionController controller
I0621 06:29:49.948518 8194 utils.go:240] ! I0621 05:23:17.403813 1 controller_utils.go:1034] Caches are synced for crd-autoregister controller
I0621 06:29:49.948531 8194 utils.go:240] ! I0621 05:23:18.245424 1 controller.go:107] OpenAPI AggregationController: Processing item
I0621 06:29:49.948547 8194 utils.go:240] ! I0621 05:23:18.245619 1 controller.go:130] OpenAPI AggregationController: action for item : Nothing (removed from the queue).
I0621 06:29:49.948578 8194 utils.go:240] ! I0621 05:23:18.245852 1 controller.go:130] OpenAPI AggregationController: action for item k8s_internal_local_delegation_chain_0000000000: Nothing (removed from the queue).
I0621 06:29:49.948596 8194 utils.go:240] ! I0621 05:23:18.257623 1 storage_scheduling.go:113] created PriorityClass system-node-critical with value 2000001000
I0621 06:29:49.948611 8194 utils.go:240] ! I0621 05:23:18.269547 1 storage_scheduling.go:113] created PriorityClass system-cluster-critical with value 2000000000
I0621 06:29:49.948625 8194 utils.go:240] ! I0621 05:23:18.269759 1 storage_scheduling.go:122] all system priority classes are created successfully or already exist.
I0621 06:29:49.948638 8194 utils.go:240] ! I0621 05:23:18.276561 1 storage_rbac.go:195] created clusterrole.rbac.authorization.k8s.io/cluster-admin
I0621 06:29:49.948652 8194 utils.go:240] ! I0621 05:23:18.281155 1 storage_rbac.go:195] created clusterrole.rbac.authorization.k8s.io/system:discovery
I0621 06:29:49.948666 8194 utils.go:240] ! I0621 05:23:18.287096 1 storage_rbac.go:195] created clusterrole.rbac.authorization.k8s.io/system:basic-user
I0621 06:29:49.948680 8194 utils.go:240] ! I0621 05:23:18.291141 1 storage_rbac.go:195] created clusterrole.rbac.authorization.k8s.io/system:public-info-viewer
I0621 06:29:49.948693 8194 utils.go:240] ! I0621 05:23:18.294268 1 storage_rbac.go:195] created clusterrole.rbac.authorization.k8s.io/admin
I0621 06:29:49.948705 8194 utils.go:240] ! I0621 05:23:18.297598 1 storage_rbac.go:195] created clusterrole.rbac.authorization.k8s.io/edit
I0621 06:29:49.948718 8194 utils.go:240] ! I0621 05:23:18.302204 1 storage_rbac.go:195] created clusterrole.rbac.authorization.k8s.io/view
I0621 06:29:49.948732 8194 utils.go:240] ! I0621 05:23:18.305772 1 storage_rbac.go:195] created clusterrole.rbac.authorization.k8s.io/system:aggregate-to-admin
I0621 06:29:49.948746 8194 utils.go:240] ! I0621 05:23:18.310254 1 storage_rbac.go:195] created clusterrole.rbac.authorization.k8s.io/system:aggregate-to-edit
I0621 06:29:49.948957 8194 utils.go:240] ! I0621 05:23:18.314488 1 storage_rbac.go:195] created clusterrole.rbac.authorization.k8s.io/system:aggregate-to-view
I0621 06:29:49.948979 8194 utils.go:240] ! I0621 05:23:18.318435 1 storage_rbac.go:195] created clusterrole.rbac.authorization.k8s.io/system:heapster
I0621 06:29:49.948997 8194 utils.go:240] ! I0621 05:23:18.321773 1 storage_rbac.go:195] created clusterrole.rbac.authorization.k8s.io/system:node
I0621 06:29:49.949017 8194 utils.go:240] ! I0621 05:23:18.325479 1 storage_rbac.go:195] created clusterrole.rbac.authorization.k8s.io/system:node-problem-detector
I0621 06:29:49.949031 8194 utils.go:240] ! I0621 05:23:18.328510 1 storage_rbac.go:195] created clusterrole.rbac.authorization.k8s.io/system:node-proxier
I0621 06:29:49.949054 8194 utils.go:240] ! I0621 05:23:18.333980 1 storage_rbac.go:195] created clusterrole.rbac.authorization.k8s.io/system:kubelet-api-admin
I0621 06:29:49.949077 8194 utils.go:240] ! I0621 05:23:18.339230 1 storage_rbac.go:195] created clusterrole.rbac.authorization.k8s.io/system:node-bootstrapper
I0621 06:29:49.949102 8194 utils.go:240] ! I0621 05:23:18.343816 1 storage_rbac.go:195] created clusterrole.rbac.authorization.k8s.io/system:auth-delegator
I0621 06:29:49.949122 8194 utils.go:240] ! I0621 05:23:18.348081 1 storage_rbac.go:195] created clusterrole.rbac.authorization.k8s.io/system:kube-aggregator
I0621 06:29:49.949141 8194 utils.go:240] ! I0621 05:23:18.353979 1 storage_rbac.go:195] created clusterrole.rbac.authorization.k8s.io/system:kube-controller-manager
I0621 06:29:49.949157 8194 utils.go:240] ! I0621 05:23:18.361733 1 storage_rbac.go:195] created clusterrole.rbac.authorization.k8s.io/system:kube-scheduler
I0621 06:29:49.949174 8194 utils.go:240] ! I0621 05:23:18.366880 1 storage_rbac.go:195] created clusterrole.rbac.authorization.k8s.io/system:kube-dns
I0621 06:29:49.949199 8194 utils.go:240] ! I0621 05:23:18.372224 1 storage_rbac.go:195] created clusterrole.rbac.authorization.k8s.io/system:persistent-volume-provisioner
I0621 06:29:49.949218 8194 utils.go:240] ! I0621 05:23:18.375518 1 storage_rbac.go:195] created clusterrole.rbac.authorization.k8s.io/system:csi-external-attacher
I0621 06:29:49.949244 8194 utils.go:240] ! I0621 05:23:18.379884 1 storage_rbac.go:195] created clusterrole.rbac.authorization.k8s.io/system:aws-cloud-provider
I0621 06:29:49.949269 8194 utils.go:240] ! I0621 05:23:18.384125 1 storage_rbac.go:195] created clusterrole.rbac.authorization.k8s.io/system:certificates.k8s.io:certificatesigningrequests:nodeclient
I0621 06:29:49.949294 8194 utils.go:240] ! I0621 05:23:18.387508 1 storage_rbac.go:195] created clusterrole.rbac.authorization.k8s.io/system:certificates.k8s.io:certificatesigningrequests:selfnodeclient
I0621 06:29:49.949314 8194 utils.go:240] ! I0621 05:23:18.391479 1 storage_rbac.go:195] created clusterrole.rbac.authorization.k8s.io/system:volume-scheduler
I0621 06:29:49.949329 8194 utils.go:240] ! I0621 05:23:18.395215 1 storage_rbac.go:195] created clusterrole.rbac.authorization.k8s.io/system:csi-external-provisioner
I0621 06:29:49.949345 8194 utils.go:240] ! I0621 05:23:18.399124 1 storage_rbac.go:195] created clusterrole.rbac.authorization.k8s.io/system:controller:attachdetach-controller
I0621 06:29:49.949363 8194 utils.go:240] ! I0621 05:23:18.403905 1 storage_rbac.go:195] created clusterrole.rbac.authorization.k8s.io/system:controller:clusterrole-aggregation-controller
I0621 06:29:49.949378 8194 utils.go:240] ! I0621 05:23:18.408209 1 storage_rbac.go:195] created clusterrole.rbac.authorization.k8s.io/system:controller:cronjob-controller
I0621 06:29:49.949394 8194 utils.go:240] ! I0621 05:23:18.411916 1 storage_rbac.go:195] created clusterrole.rbac.authorization.k8s.io/system:controller:daemon-set-controller
I0621 06:29:49.949409 8194 utils.go:240] ! I0621 05:23:18.415568 1 storage_rbac.go:195] created clusterrole.rbac.authorization.k8s.io/system:controller:deployment-controller
I0621 06:29:49.949424 8194 utils.go:240] ! I0621 05:23:18.418989 1 storage_rbac.go:195] created clusterrole.rbac.authorization.k8s.io/system:controller:disruption-controller
I0621 06:29:49.949443 8194 utils.go:240] ! I0621 05:23:18.422968 1 storage_rbac.go:195] created clusterrole.rbac.authorization.k8s.io/system:controller:endpoint-controller
I0621 06:29:49.949468 8194 utils.go:240] ! I0621 05:23:18.427094 1 storage_rbac.go:195] created clusterrole.rbac.authorization.k8s.io/system:controller:expand-controller
I0621 06:29:49.949485 8194 utils.go:240] ! I0621 05:23:18.432513 1 storage_rbac.go:195] created clusterrole.rbac.authorization.k8s.io/system:controller:generic-garbage-collector
I0621 06:29:49.949500 8194 utils.go:240] ! I0621 05:23:18.438423 1 storage_rbac.go:195] created clusterrole.rbac.authorization.k8s.io/system:controller:horizontal-pod-autoscaler
I0621 06:29:49.949515 8194 utils.go:240] ! I0621 05:23:18.442798 1 storage_rbac.go:195] created clusterrole.rbac.authorization.k8s.io/system:controller:job-controller
I0621 06:29:49.949530 8194 utils.go:240] ! I0621 05:23:18.447155 1 storage_rbac.go:195] created clusterrole.rbac.authorization.k8s.io/system:controller:namespace-controller
I0621 06:29:49.949569 8194 utils.go:240] ! I0621 05:23:18.450715 1 storage_rbac.go:195] created clusterrole.rbac.authorization.k8s.io/system:controller:node-controller
I0621 06:29:49.949589 8194 utils.go:240] ! I0621 05:23:18.455940 1 storage_rbac.go:195] created clusterrole.rbac.authorization.k8s.io/system:controller:persistent-volume-binder
I0621 06:29:49.949606 8194 utils.go:240] ! I0621 05:23:18.460608 1 storage_rbac.go:195] created clusterrole.rbac.authorization.k8s.io/system:controller:pod-garbage-collector
I0621 06:29:49.949621 8194 utils.go:240] ! I0621 05:23:18.464841 1 storage_rbac.go:195] created clusterrole.rbac.authorization.k8s.io/system:controller:replicaset-controller
I0621 06:29:49.949638 8194 utils.go:240] ! I0621 05:23:18.468798 1 storage_rbac.go:195] created clusterrole.rbac.authorization.k8s.io/system:controller:replication-controller
I0621 06:29:49.949653 8194 utils.go:240] ! I0621 05:23:18.472371 1 storage_rbac.go:195] created clusterrole.rbac.authorization.k8s.io/system:controller:resourcequota-controller
I0621 06:29:49.949668 8194 utils.go:240] ! I0621 05:23:18.477090 1 storage_rbac.go:195] created clusterrole.rbac.authorization.k8s.io/system:controller:route-controller
I0621 06:29:49.949684 8194 utils.go:240] ! I0621 05:23:18.480719 1 storage_rbac.go:195] created clusterrole.rbac.authorization.k8s.io/system:controller:service-account-controller
I0621 06:29:49.949700 8194 utils.go:240] ! I0621 05:23:18.486091 1 storage_rbac.go:195] created clusterrole.rbac.authorization.k8s.io/system:controller:service-controller
I0621 06:29:49.949715 8194 utils.go:240] ! I0621 05:23:18.490488 1 storage_rbac.go:195] created clusterrole.rbac.authorization.k8s.io/system:controller:statefulset-controller
I0621 06:29:49.949730 8194 utils.go:240] ! I0621 05:23:18.497275 1 storage_rbac.go:195] created clusterrole.rbac.authorization.k8s.io/system:controller:ttl-controller
I0621 06:29:49.949745 8194 utils.go:240] ! I0621 05:23:18.510370 1 storage_rbac.go:195] created clusterrole.rbac.authorization.k8s.io/system:controller:certificate-controller
I0621 06:29:49.949761 8194 utils.go:240] ! I0621 05:23:18.551213 1 storage_rbac.go:195] created clusterrole.rbac.authorization.k8s.io/system:controller:pvc-protection-controller
I0621 06:29:49.949782 8194 utils.go:240] ! I0621 05:23:18.592272 1 storage_rbac.go:195] created clusterrole.rbac.authorization.k8s.io/system:controller:pv-protection-controller
I0621 06:29:49.949796 8194 utils.go:240] ! I0621 05:23:18.632447 1 storage_rbac.go:223] created clusterrolebinding.rbac.authorization.k8s.io/cluster-admin
I0621 06:29:49.949811 8194 utils.go:240] ! I0621 05:23:18.670649 1 storage_rbac.go:223] created clusterrolebinding.rbac.authorization.k8s.io/system:discovery
I0621 06:29:49.949825 8194 utils.go:240] ! I0621 05:23:18.710001 1 storage_rbac.go:223] created clusterrolebinding.rbac.authorization.k8s.io/system:basic-user
I0621 06:29:49.949843 8194 utils.go:240] ! I0621 05:23:18.750929 1 storage_rbac.go:223] created clusterrolebinding.rbac.authorization.k8s.io/system:public-info-viewer
I0621 06:29:49.949857 8194 utils.go:240] ! I0621 05:23:18.790446 1 storage_rbac.go:223] created clusterrolebinding.rbac.authorization.k8s.io/system:node-proxier
I0621 06:29:49.949873 8194 utils.go:240] ! I0621 05:23:18.831305 1 storage_rbac.go:223] created clusterrolebinding.rbac.authorization.k8s.io/system:kube-controller-manager
I0621 06:29:49.949887 8194 utils.go:240] ! I0621 05:23:18.871602 1 storage_rbac.go:223] created clusterrolebinding.rbac.authorization.k8s.io/system:kube-dns
I0621 06:29:49.949901 8194 utils.go:240] ! I0621 05:23:18.910479 1 storage_rbac.go:223] created clusterrolebinding.rbac.authorization.k8s.io/system:kube-scheduler
I0621 06:29:49.949917 8194 utils.go:240] ! I0621 05:23:18.950492 1 storage_rbac.go:223] created clusterrolebinding.rbac.authorization.k8s.io/system:aws-cloud-provider
I0621 06:29:49.949931 8194 utils.go:240] ! I0621 05:23:18.991081 1 storage_rbac.go:223] created clusterrolebinding.rbac.authorization.k8s.io/system:volume-scheduler
I0621 06:29:49.949944 8194 utils.go:240] ! I0621 05:23:19.031139 1 storage_rbac.go:223] created clusterrolebinding.rbac.authorization.k8s.io/system:node
I0621 06:29:49.949960 8194 utils.go:240] ! I0621 05:23:19.070477 1 storage_rbac.go:223] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:attachdetach-controller
I0621 06:29:49.949983 8194 utils.go:240] ! I0621 05:23:19.110979 1 storage_rbac.go:223] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:clusterrole-aggregation-controller
I0621 06:29:49.950000 8194 utils.go:240] ! I0621 05:23:19.115250 1 controller.go:606] quota admission added evaluator for: leases.coordination.k8s.io
I0621 06:29:49.950017 8194 utils.go:240] ! I0621 05:23:19.150672 1 storage_rbac.go:223] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:cronjob-controller
I0621 06:29:49.950039 8194 utils.go:240] ! I0621 05:23:19.192619 1 storage_rbac.go:223] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:daemon-set-controller
I0621 06:29:49.950069 8194 utils.go:240] ! I0621 05:23:19.232085 1 storage_rbac.go:223] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:deployment-controller
I0621 06:29:49.950109 8194 utils.go:240] ! I0621 05:23:19.270491 1 storage_rbac.go:223] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:disruption-controller
I0621 06:29:49.950134 8194 utils.go:240] ! I0621 05:23:19.310368 1 storage_rbac.go:223] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:endpoint-controller
I0621 06:29:49.950156 8194 utils.go:240] ! I0621 05:23:19.351540 1 storage_rbac.go:223] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:expand-controller
I0621 06:29:49.950174 8194 utils.go:240] ! I0621 05:23:19.390969 1 storage_rbac.go:223] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:generic-garbage-collector
I0621 06:29:49.950198 8194 utils.go:240] ! I0621 05:23:19.435440 1 storage_rbac.go:223] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:horizontal-pod-autoscaler
I0621 06:29:49.950219 8194 utils.go:240] ! I0621 05:23:19.471781 1 storage_rbac.go:223] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:job-controller
I0621 06:29:49.950237 8194 utils.go:240] ! I0621 05:23:19.510026 1 storage_rbac.go:223] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:namespace-controller
I0621 06:29:49.950253 8194 utils.go:240] ! I0621 05:23:19.550542 1 storage_rbac.go:223] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:node-controller
I0621 06:29:49.950269 8194 utils.go:240] ! I0621 05:23:19.590653 1 storage_rbac.go:223] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:persistent-volume-binder
I0621 06:29:49.950289 8194 utils.go:240] ! I0621 05:23:19.631419 1 storage_rbac.go:223] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:pod-garbage-collector
I0621 06:29:49.950317 8194 utils.go:240] ! I0621 05:23:19.672243 1 storage_rbac.go:223] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:replicaset-controller
I0621 06:29:49.950337 8194 utils.go:240] ! I0621 05:23:19.710441 1 storage_rbac.go:223] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:replication-controller
I0621 06:29:49.950354 8194 utils.go:240] ! I0621 05:23:19.761456 1 storage_rbac.go:223] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:resourcequota-controller
I0621 06:29:49.950370 8194 utils.go:240] ! I0621 05:23:19.792184 1 storage_rbac.go:223] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:route-controller
I0621 06:29:49.950387 8194 utils.go:240] ! I0621 05:23:19.833175 1 storage_rbac.go:223] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:service-account-controller
I0621 06:29:49.950402 8194 utils.go:240] ! I0621 05:23:19.870573 1 storage_rbac.go:223] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:service-controller
I0621 06:29:49.950419 8194 utils.go:240] ! I0621 05:23:19.910271 1 storage_rbac.go:223] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:statefulset-controller
I0621 06:29:49.950434 8194 utils.go:240] ! I0621 05:23:19.951626 1 storage_rbac.go:223] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:ttl-controller
I0621 06:29:49.950450 8194 utils.go:240] ! I0621 05:23:19.991151 1 storage_rbac.go:223] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:certificate-controller
I0621 06:29:49.950466 8194 utils.go:240] ! I0621 05:23:20.032423 1 storage_rbac.go:223] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:pvc-protection-controller
I0621 06:29:49.950482 8194 utils.go:240] ! I0621 05:23:20.070533 1 storage_rbac.go:223] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:pv-protection-controller
I0621 06:29:49.950497 8194 utils.go:240] ! I0621 05:23:20.108293 1 controller.go:606] quota admission added evaluator for: roles.rbac.authorization.k8s.io
I0621 06:29:49.950513 8194 utils.go:240] ! I0621 05:23:20.110894 1 storage_rbac.go:254] created role.rbac.authorization.k8s.io/extension-apiserver-authentication-reader in kube-system
I0621 06:29:49.950529 8194 utils.go:240] ! I0621 05:23:20.150690 1 storage_rbac.go:254] created role.rbac.authorization.k8s.io/system:controller:bootstrap-signer in kube-system
I0621 06:29:49.950544 8194 utils.go:240] ! I0621 05:23:20.191366 1 storage_rbac.go:254] created role.rbac.authorization.k8s.io/system:controller:cloud-provider in kube-system
I0621 06:29:49.950559 8194 utils.go:240] ! I0621 05:23:20.230895 1 storage_rbac.go:254] created role.rbac.authorization.k8s.io/system:controller:token-cleaner in kube-system
I0621 06:29:49.950576 8194 utils.go:240] ! I0621 05:23:20.270767 1 storage_rbac.go:254] created role.rbac.authorization.k8s.io/system::leader-locking-kube-controller-manager in kube-system
I0621 06:29:49.950592 8194 utils.go:240] ! I0621 05:23:20.311905 1 storage_rbac.go:254] created role.rbac.authorization.k8s.io/system::leader-locking-kube-scheduler in kube-system
I0621 06:29:49.950605 8194 utils.go:240] ! I0621 05:23:20.328788 1 controller.go:606] quota admission added evaluator for: endpoints
I0621 06:29:49.950632 8194 utils.go:240] ! I0621 05:23:20.352629 1 storage_rbac.go:254] created role.rbac.authorization.k8s.io/system:controller:bootstrap-signer in kube-public
I0621 06:29:49.950647 8194 utils.go:240] ! I0621 05:23:20.388926 1 controller.go:606] quota admission added evaluator for: rolebindings.rbac.authorization.k8s.io
I0621 06:29:49.950667 8194 utils.go:240] ! I0621 05:23:20.391752 1 storage_rbac.go:284] created rolebinding.rbac.authorization.k8s.io/system::extension-apiserver-authentication-reader in kube-system
I0621 06:29:49.950685 8194 utils.go:240] ! I0621 05:23:20.430280 1 storage_rbac.go:284] created rolebinding.rbac.authorization.k8s.io/system::leader-locking-kube-controller-manager in kube-system
I0621 06:29:49.950702 8194 utils.go:240] ! I0621 05:23:20.471146 1 storage_rbac.go:284] created rolebinding.rbac.authorization.k8s.io/system::leader-locking-kube-scheduler in kube-system
I0621 06:29:49.950718 8194 utils.go:240] ! I0621 05:23:20.510337 1 storage_rbac.go:284] created rolebinding.rbac.authorization.k8s.io/system:controller:bootstrap-signer in kube-system
I0621 06:29:49.950737 8194 utils.go:240] ! I0621 05:23:20.551471 1 storage_rbac.go:284] created rolebinding.rbac.authorization.k8s.io/system:controller:cloud-provider in kube-system
I0621 06:29:49.950753 8194 utils.go:240] ! I0621 05:23:20.590771 1 storage_rbac.go:284] created rolebinding.rbac.authorization.k8s.io/system:controller:token-cleaner in kube-system
I0621 06:29:49.950769 8194 utils.go:240] ! I0621 05:23:20.633037 1 storage_rbac.go:284] created rolebinding.rbac.authorization.k8s.io/system:controller:bootstrap-signer in kube-public
I0621 06:29:49.950786 8194 utils.go:240] ! W0621 05:23:20.814341 1 lease.go:222] Resetting endpoints for master service "kubernetes" to [192.168.99.164]
I0621 06:29:49.950799 8194 utils.go:240] ! I0621 05:23:21.199652 1 controller.go:606] quota admission added evaluator for: serviceaccounts
I0621 06:29:49.950812 8194 utils.go:240] ! I0621 05:23:21.906547 1 controller.go:606] quota admission added evaluator for: deployments.apps
I0621 06:29:49.950824 8194 utils.go:240] ! I0621 05:23:22.186836 1 controller.go:606] quota admission added evaluator for: daemonsets.apps
I0621 06:29:49.950836 8194 utils.go:240] ! I0621 05:23:27.725890 1 controller.go:606] quota admission added evaluator for: namespaces
I0621 06:29:49.950848 8194 utils.go:240] ! I0621 05:23:28.329173 1 controller.go:606] quota admission added evaluator for: replicasets.apps
I0621 06:29:49.950861 8194 utils.go:240] ! I0621 05:23:28.356565 1 controller.go:606] quota admission added evaluator for: controllerrevisions.apps
I0621 06:29:49.953501 8194 logs.go:76] Gathering logs for coredns ...
I0621 06:29:49.953510 8194 ssh_runner.go:137] Run with output: docker logs --tail 200 4bc600330725
I0621 06:29:49.994561 8194 utils.go:240] > .:53
I0621 06:29:49.994586 8194 utils.go:240] > 2019-06-21T05:23:50.493Z [INFO] CoreDNS-1.3.1
I0621 06:29:49.994597 8194 utils.go:240] > 2019-06-21T05:23:50.494Z [INFO] linux/amd64, go1.11.4, 6b56a9c
I0621 06:29:49.994603 8194 utils.go:240] > CoreDNS-1.3.1
I0621 06:29:49.994610 8194 utils.go:240] > linux/amd64, go1.11.4, 6b56a9c
I0621 06:29:49.994624 8194 utils.go:240] > 2019-06-21T05:23:50.494Z [INFO] plugin/reload: Running configuration MD5 = 599b9eb76b8c147408aed6a0bbe0f669
W0621 06:29:49.996499 8194 exit.go:100] Error starting cluster: timed out waiting to elevate kube-system RBAC privileges: Temporary Error: creating clusterrolebinding: Post https://192.168.99.164:8443/apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings?timeout=1m0s: dial tcp 192.168.99.164:8443: i/o timeout
Temporary Error: creating clusterrolebinding: Post https://192.168.99.164:8443/apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings?timeout=1m0s: dial tcp 192.168.99.164:8443: i/o timeout
Temporary Error: creating clusterrolebinding: Post https://192.168.99.164:8443/apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings?timeout=1m0s: dial tcp 192.168.99.164:8443: i/o timeout
Temporary Error: creating clusterrolebinding: Post https://192.168.99.164:8443/apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings?timeout=1m0s: dial tcp 192.168.99.164:8443: i/o timeout
Temporary Error: creating clusterrolebinding: Post https://192.168.99.164:8443/apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings?timeout=1m0s: dial tcp 192.168.99.164:8443: i/o timeout
Temporary Error: creating clusterrolebinding: Post https://192.168.99.164:8443/apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings?timeout=1m0s: dial tcp 192.168.99.164:8443: i/o timeout
creating clusterrolebinding: Post https://192.168.99.164:8443/apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings?timeout=1m0s: dial tcp 192.168.99.164:8443: connect: network is unreachable
š£ Error starting cluster: timed out waiting to elevate kube-system RBAC privileges: Temporary Error: creating clusterrolebinding: Post https://192.168.99.164:8443/apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings?timeout=1m0s: dial tcp 192.168.99.164:8443: i/o timeout
Temporary Error: creating clusterrolebinding: Post https://192.168.99.164:8443/apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings?timeout=1m0s: dial tcp 192.168.99.164:8443: i/o timeout
Temporary Error: creating clusterrolebinding: Post https://192.168.99.164:8443/apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings?timeout=1m0s: dial tcp 192.168.99.164:8443: i/o timeout
Temporary Error: creating clusterrolebinding: Post https://192.168.99.164:8443/apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings?timeout=1m0s: dial tcp 192.168.99.164:8443: i/o timeout
Temporary Error: creating clusterrolebinding: Post https://192.168.99.164:8443/apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings?timeout=1m0s: dial tcp 192.168.99.164:8443: i/o timeout
Temporary Error: creating clusterrolebinding: Post https://192.168.99.164:8443/apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings?timeout=1m0s: dial tcp 192.168.99.164:8443: i/o timeout
creating clusterrolebinding: Post https://192.168.99.164:8443/apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings?timeout=1m0s: dial tcp 192.168.99.164:8443: connect: network is unreachable
šæ Sorry that minikube crashed. If this was unexpected, we would love to hear from you:
š https://github.com/kubernetes/minikube/issues/newHi, any update please?
tigerpeng2001
commented
5 years ago I ran into similar issue today. Checked the log, it looked like some issues with the VirtualBox. So I tried uninstall/installed it, now I can start the Minikube. The VirtualBox is installed with dmg file.
kmayura1
commented
5 years ago sadly, these are the logs even after reinstalling virtualbox from scratch , š Launching Kubernetes ...
š£ Error starting cluster: timed out waiting to elevate kube-system RBAC privileges: Temporary Error: creating clusterrolebinding: Post https://192.168.99.165:8443/apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings?timeout=1m0s: dial tcp 192.168.99.165:8443: i/o timeout Temporary Error: creating clusterrolebinding: Post https://192.168.99.165:8443/apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings?timeout=1m0s: dial tcp 192.168.99.165:8443: i/o timeout Temporary Error: creating clusterrolebinding: Post https://192.168.99.165:8443/apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings?timeout=1m0s: dial tcp 192.168.99.165:8443: i/o timeout Temporary Error: creating clusterrolebinding: Post https://192.168.99.165:8443/apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings?timeout=1m0s: dial tcp 192.168.99.165:8443: i/o timeout Temporary Error: creating clusterrolebinding: Post https://192.168.99.165:8443/apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings?timeout=1m0s: dial tcp 192.168.99.165:8443: i/o timeout Temporary Error: creating clusterrolebinding: Post https://192.168.99.165:8443/apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings?timeout=1m0s: dial tcp 192.168.99.165:8443: i/o timeout Temporary Error: creating clusterrolebinding: Post https://192.168.99.165:8443/apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings?timeout=1m0s: dial tcp 192.168.99.165:8443: i/o timeout Temporary Error: creating clusterrolebinding: Post https://192.168.99.165:8443/apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings?timeout=1m0s: dial tcp 192.168.99.165:8443: i/o timeout creating clusterrolebinding: Post https://192.168.99.165:8443/apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings?timeout=1m0s: dial tcp 192.168.99.165:8443: connect: network is unreachable
šæ Sorry that minikube crashed. If this was unexpected, we would love to hear from you: š https://github.com/kubernetes/minikube/issues/new ā Problems detected in "kube-addon-manager": error: unable to recognize "STDIN": Get https://localhost:8443/api?timeout=32s: dial tcp 127.0.0.1:8443: connect: connection refused error: no objeWRN: == Error gettcts passed to apply
tstromberg
commented
5 years ago I'm not sure what's going on with the apiserver here, but it's mighty suspicious.
If you don't mind, I'm curious if minikube v1.2.0 / kubernetes 1.15 suffers this issue. Please run minikube delete first to clear up the old state.
kmayura1
commented
5 years ago logs.txt unfortuantely that didn't help, I tried to remove whole minikube and reinstalled, still same issue. š minikube v1.2.0 on darwin (amd64) š„ Creating virtualbox VM (CPUs=2, Memory=2048MB, Disk=20000MB) ... š³ Configuring environment for Kubernetes v1.15.0 on Docker 18.09.6 š Pulling images ... š Launching Kubernetes ...
attached minikube logs
bzied321
commented
5 years ago Hi, I experienced the same problem on macOS Mojave 10.14.5 and fixed it using suggestions from other issue threads. Reinstalling both VirtualBox and minikube did not fix anything - the source of my problem was being connected to Cisco AnyConnect VPN. Here are the steps I used:
$ minikube stop $ minikube delete $ brew cask uninstall minikube $ rm -rf ~/.minikube ~/.kube Go to https://www.virtualbox.org/wiki/Downloads, use VirtualBox_Uninstall.tool script provided in OS X host .dm file Disconnect from VPN Restart laptop, make sure that you are not reconnected to VPN Install VirtualBox using VirtualBox.pkg from the same .dmg file as the previous step. $ brew cask install minikube $ minikube start --alsologtostderr -v=9 Connect to VPN (if you wish)
Hope this helps. If anyone has suggestions for starting minikube while connected to a VPN like Cisco AnyConnect, please let me know! Thanks
kmayura1
commented
5 years ago Hi, I experienced the same problem on macOS Mojave 10.14.5 and fixed it using suggestions from other issue threads. Reinstalling both VirtualBox and minikube did not fix anything - the source of my problem was being connected to Cisco AnyConnect VPN. Here are the steps I used:
$ minikube stop $ minikube delete $ brew cask uninstall minikube $ rm -rf ~/.minikube ~/.kube Go to https://www.virtualbox.org/wiki/Downloads, use VirtualBox_Uninstall.tool script provided in OS X host .dm file Disconnect from VPN Restart laptop, make sure that you are not reconnected to VPN Install VirtualBox using VirtualBox.pkg from the same .dmg file as the previous step. $ brew cask install minikube $ minikube start --alsologtostderr -v=9 Connect to VPN (if you wish)
Hope this helps. If anyone has suggestions for starting minikube while connected to a VPN like Cisco AnyConnect, please let me know! Thanks
Thanks verymuch , you really solved my struggle working in vm inside Mac.. this really solved my issue. btw what is --alsologtostderr -v=9?
bzied321
commented
5 years ago @kmayura1 those flags do not fundamentally change the minikube start command.
--alsologtostderr : logs to standard error as well as files -v=9 : parses logs into more human-readable format
den-is
commented
5 years ago any updates still getting same with minikube 1.3.0 and not 1.3.1 macos 10.14.6 (18G87) virtualbox 6.0.10 r132072
I0817 20:52:38.462621 58542 kubeadm.go:241] Configuring cluster permissions ...
I0817 20:53:38.498071 58542 utils.go:130] error: Temporary Error: creating clusterrolebinding: Post https://192.168.99.119:8443/apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings?timeout=1m0s: dial tcp 192.168.99.119:8443: i/o timeout - sleeping 500ms
I0817 20:53:38.998176 58542 utils.go:119] retry loop 1and loop never exits same even after suggested above full reset of minikube on host... and using manually downloaded minikube binary release
philwhitwell
commented
5 years ago Hi, I experienced the same problem on macOS Mojave 10.14.5 and fixed it using suggestions from other issue threads. Reinstalling both VirtualBox and minikube did not fix anything - the source of my problem was being connected to Cisco AnyConnect VPN. Here are the steps I used:
$ minikube stop $ minikube delete $ brew cask uninstall minikube $ rm -rf ~/.minikube ~/.kube Go to https://www.virtualbox.org/wiki/Downloads, use VirtualBox_Uninstall.tool script provided in OS X host .dm file Disconnect from VPN Restart laptop, make sure that you are not reconnected to VPN Install VirtualBox using VirtualBox.pkg from the same .dmg file as the previous step. $ brew cask install minikube $ minikube start --alsologtostderr -v=9 Connect to VPN (if you wish)
Hope this helps. If anyone has suggestions for starting minikube while connected to a VPN like Cisco AnyConnect, please let me know! Thanks
This was helpful but just to save the pain. I did not need to uninstall VirtualBox and restart
tstromberg
commented
5 years ago Heads up: After refactoring code related to RBAC privilege elevation, I suspect that minikube v1.4.0b2 addresses this bug.
g-boros
commented
4 years ago @tstromberg it looks like it's still having issues with v1.4.0b2. will check if related to Cisco VPN.
Ā± % minikube version
minikube version: v1.4.0-beta.2
commit: d1e468085d9af12e5d130a6cb3b2186a5db87a0e
Ā± % minikube start -v 8 --alsologtostderr
I0919 16:18:40.381055 47875 notify.go:125] Checking for updates...
I0919 16:18:41.350194 47875 start.go:235] hostinfo: {"hostname":"HW15399.local","uptime":1990944,"bootTime":1566911777,"procs":526,"os":"darwin","platform":"darwin","platformFamily":"","platformVersion":"10.14","kernelVersion":"18.0.0","virtualizationSystem":"","virtualizationRole":"","hostid":"df83ad37-501e-3b4f-b1f0-04f3ac90fe35"}
W0919 16:18:41.350329 47875 start.go:243] gopshost.Virtualization returned error: not implemented yet
š minikube v1.4.0-beta.2 on Darwin 10.14
...
...
I0919 16:23:05.570938 47875 utils.go:167] > Sep 19 14:21:12 minikube dockerd[2390]: time="2019-09-19T14:21:12.058423937Z" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/688886c21d14f92cef52f6f1e181969a12fbc5c947794f73df895938dabd3073/shim.sock" debug=false pid=4758
I0919 16:23:05.570965 47875 utils.go:167] > Sep 19 14:21:12 minikube dockerd[2390]: time="2019-09-19T14:21:12.211119163Z" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/b0e7b774a511fc79518518d3593865dd9c337e70af97c98240cddc86324c72e3/shim.sock" debug=false pid=4823
I0919 16:23:05.574212 47875 logs.go:78] Gathering logs for container status ...
I0919 16:23:05.574223 47875 ssh_runner.go:138] Run with output: sudo crictl ps -a || sudo docker ps -a
I0919 16:23:05.590164 47875 utils.go:167] > CONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID
I0919 16:23:05.593224 47875 utils.go:167] > b0e7b774a511f 4689081edb103 About a minute ago Running storage-provisioner 0 688886c21d14f
I0919 16:23:05.593294 47875 utils.go:167] > a2c19233bb576 bf261d1579144 About a minute ago Running coredns 0 05761bd696d3a
I0919 16:23:05.593327 47875 utils.go:167] > 2fe41b6e549c9 bf261d1579144 About a minute ago Running coredns 0 8abe6a4444dce
I0919 16:23:05.593354 47875 utils.go:167] > d0a44b700f493 63b54e18db505 About a minute ago Running kube-proxy 0 5184a3279343f
I0919 16:23:05.593380 47875 utils.go:167] > 45de98519df8c 696ee147a625a 2 minutes ago Running kube-apiserver 0 a989c8c3776b4
I0919 16:23:05.593433 47875 utils.go:167] > 8978e237f08e8 1020041ba01cb 2 minutes ago Running kube-scheduler 0 13223da85098b
I0919 16:23:05.593710 47875 utils.go:167] > 4fe6e5ec1ccd3 b2756210eeabf 2 minutes ago Running etcd 0 b71eef9515abf
I0919 16:23:05.593761 47875 utils.go:167] > 74d24c3d2fc5c 163c0e28a1669 2 minutes ago Running kube-controller-manager 0 d447eb648d91f
I0919 16:23:05.593795 47875 utils.go:167] > 12bd7de77e461 119701e77cbc4 2 minutes ago Running kube-addon-manager 0 949a80c2bea70
W0919 16:23:05.595332 47875 exit.go:99] Error starting cluster: timed out waiting to elevate kube-system RBAC privileges: Temporary Error: creating clusterrolebinding: Post https://192.168.99.103:8443/apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings: dial tcp 192.168.99.103:8443: i/o timeout
š£ Error starting cluster: timed out waiting to elevate kube-system RBAC privileges: Temporary Error: creating clusterrolebinding: Post https://192.168.99.103:8443/apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings: dial tcp 192.168.99.103:8443: i/o timeout
šæ Sorry that minikube crashed. If this was unexpected, we would love to hear from you:
š https://github.com/kubernetes/minikube/issues/new/choose@kmayura1 I believe this issue is now addressed by minikube v1.4, as it reliably passes the correct context to the elevate function. If you still see this issue with minikube v1.4 or higher, please reopen this issue by commenting with /reopen
Thank you for reporting this issue!
holleyism
commented
4 years ago Just downloaded 1.4 and ran into the same error.
I0920 09:29:07.234201 57754 utils.go:167] ! I0920 14:26:58.213281 1 serving.go:319] Generated self-signed cert in-memory
I0920 09:29:07.234254 57754 utils.go:167] ! W0920 14:27:01.065843 1 authentication.go:262] Unable to get configmap/extension-apiserver-authentication in kube-system. Usually fixed by 'kubectl create rolebinding -n kube-system ROLEBINDING_NAME --role=extension-apiserver-authentication-reader --serviceaccount=YOUR_NS:YOUR_SA'
I0920 09:29:07.234283 57754 utils.go:167] ! W0920 14:27:01.066159 1 authentication.go:199] Error looking up in-cluster authentication configuration: configmaps "extension-apiserver-authentication" is forbidden: User "system:kube-scheduler" cannot get resource "configmaps" in API group "" in the namespace "kube-system"
I0920 09:29:07.234300 57754 utils.go:167] ! W0920 14:27:01.066312 1 authentication.go:200] Continuing without authentication configuration. This may treat all requests as anonymous.
I0920 09:29:07.234317 57754 utils.go:167] ! W0920 14:27:01.066360 1 authentication.go:201] To require authentication configuration lookup to succeed, set --authentication-tolerate-lookup-failure=false
I0920 09:29:07.234325 57754 utils.go:167] ! I0920 14:27:01.068591 1 server.go:143] Version: v1.16.0
I0920 09:29:07.234340 57754 utils.go:167] ! I0920 14:27:01.071885 1 defaults.go:91] TaintNodesByCondition is enabled, PodToleratesNodeTaints predicate is mandatory
I0920 09:29:07.234349 57754 utils.go:167] ! W0920 14:27:01.076001 1 authorization.go:47] Authorization is disabled
I0920 09:29:07.234359 57754 utils.go:167] ! W0920 14:27:01.076070 1 authentication.go:79] Authentication is disabled
I0920 09:29:07.234371 57754 utils.go:167] ! I0920 14:27:01.076120 1 deprecated_insecure_serving.go:51] Serving healthz insecurely on [::]:10251
I0920 09:29:07.234386 57754 utils.go:167] ! I0920 14:27:01.085625 1 secure_serving.go:123] Serving securely on 127.0.0.1:10259
I0920 09:29:07.234409 57754 utils.go:167] ! E0920 14:27:01.155629 1 reflector.go:123] k8s.io/client-go/informers/factory.go:134: Failed to list *v1.ReplicaSet: replicasets.apps is forbidden: User "system:kube-scheduler" cannot list resource "replicasets" in API group "apps" at the cluster scope
I0920 09:29:07.234438 57754 utils.go:167] ! E0920 14:27:01.167245 1 reflector.go:123] k8s.io/kubernetes/cmd/kube-scheduler/app/server.go:236: Failed to list *v1.Pod: pods is forbidden: User "system:kube-scheduler" cannot list resource "pods" in API group "" at the cluster scope
I0920 09:29:07.234464 57754 utils.go:167] ! E0920 14:27:01.167604 1 reflector.go:123] k8s.io/client-go/informers/factory.go:134: Failed to list *v1beta1.CSINode: csinodes.storage.k8s.io is forbidden: User "system:kube-scheduler" cannot list resource "csinodes" in API group "storage.k8s.io" at the cluster scope
I0920 09:29:07.234491 57754 utils.go:167] ! E0920 14:27:01.168007 1 reflector.go:123] k8s.io/client-go/informers/factory.go:134: Failed to list *v1.ReplicationController: replicationcontrollers is forbidden: User "system:kube-scheduler" cannot list resource "replicationcontrollers" in API group "" at the cluster
scope
I0920 09:29:07.234535 57754 utils.go:167] ! E0920 14:27:01.168116 1 reflector.go:123] k8s.io/client-go/informers/factory.go:134: Failed to list *v1beta1.PodDisruptionBudget: poddisruptionbudgets.policy is forbidden: User "system:kube-scheduler" cannot list resource "poddisruptionbudgets" in API group "policy" at
the cluster scope
I0920 09:29:07.234577 57754 utils.go:167] ! E0920 14:27:01.169817 1 reflector.go:123] k8s.io/client-go/informers/factory.go:134: Failed to list *v1.StatefulSet: statefulsets.apps is forbidden: User "system:kube-scheduler" cannot list resource "statefulsets" in API group "apps" at the cluster scope
I0920 09:29:07.234715 57754 utils.go:167] ! E0920 14:27:01.169886 1 reflector.go:123] k8s.io/client-go/informers/factory.go:134: Failed to list *v1.Node: nodes is forbidden: User "system:kube-scheduler" cannot list resource "nodes" in API group "" at the cluster scope
I0920 09:29:07.235219 57754 utils.go:167] ! E0920 14:27:01.169915 1 reflector.go:123] k8s.io/client-go/informers/factory.go:134: Failed to list *v1.StorageClass: storageclasses.storage.k8s.io is forbidden: User "system:kube-scheduler" cannot list resource "storageclasses" in API group "storage.k8s.io" at the cluster scope
I0920 09:29:07.235268 57754 utils.go:167] ! E0920 14:27:01.169958 1 reflector.go:123] k8s.io/client-go/informers/factory.go:134: Failed to list *v1.Service: services is forbidden: User "system:kube-scheduler" cannot list resource "services" in API group "" at the cluster scope
I0920 09:29:07.235303 57754 utils.go:167] ! E0920 14:27:01.175752 1 reflector.go:123] k8s.io/client-go/informers/factory.go:134: Failed to list *v1.PersistentVolume: persistentvolumes is forbidden: User "system:kube-scheduler" cannot list resource "persistentvolumes" in API group "" at the cluster scope
I0920 09:29:07.235863 57754 utils.go:167] ! E0920 14:27:01.176775 1 reflector.go:123] k8s.io/client-go/informers/factory.go:134: Failed to list *v1.PersistentVolumeClaim: persistentvolumeclaims is forbidden: User "system:kube-scheduler" cannot list resource "persistentvolumeclaims" in API group "" at the cluster
scope
I0920 09:29:07.235896 57754 utils.go:167] ! E0920 14:27:02.157606 1 reflector.go:123] k8s.io/client-go/informers/factory.go:134: Failed to list *v1.ReplicaSet: replicasets.apps is forbidden: User "system:kube-scheduler" cannot list resource "replicasets" in API group "apps" at the cluster scope
I0920 09:29:07.235924 57754 utils.go:167] ! E0920 14:27:02.169052 1 reflector.go:123] k8s.io/kubernetes/cmd/kube-scheduler/app/server.go:236: Failed to list *v1.Pod: pods is forbidden: User "system:kube-scheduler" cannot list resource "pods" in API group "" at the cluster scope
I0920 09:29:07.235958 57754 utils.go:167] ! E0920 14:27:02.170430 1 reflector.go:123] k8s.io/client-go/informers/factory.go:134: Failed to list *v1beta1.CSINode: csinodes.storage.k8s.io is forbidden: User "system:kube-scheduler" cannot list resource "csinodes" in API group "storage.k8s.io" at the cluster scope
I0920 09:29:07.235986 57754 utils.go:167] ! E0920 14:27:02.173179 1 reflector.go:123] k8s.io/client-go/informers/factory.go:134: Failed to list *v1.ReplicationController: replicationcontrollers is forbidden: User "system:kube-scheduler" cannot list resource "replicationcontrollers" in API group "" at the cluster
scope
I0920 09:29:07.236015 57754 utils.go:167] ! E0920 14:27:02.173245 1 reflector.go:123] k8s.io/client-go/informers/factory.go:134: Failed to list *v1beta1.PodDisruptionBudget: poddisruptionbudgets.policy is forbidden: User "system:kube-scheduler" cannot list resource "poddisruptionbudgets" in API group "policy" at
the cluster scope
I0920 09:29:07.236041 57754 utils.go:167] ! E0920 14:27:02.174140 1 reflector.go:123] k8s.io/client-go/informers/factory.go:134: Failed to list *v1.StatefulSet: statefulsets.apps is forbidden: User "system:kube-scheduler" cannot list resource "statefulsets" in API group "apps" at the cluster scope
I0920 09:29:07.236063 57754 utils.go:167] ! E0920 14:27:02.177499 1 reflector.go:123] k8s.io/client-go/informers/factory.go:134: Failed to list *v1.Node: nodes is forbidden: User "system:kube-scheduler" cannot list resource "nodes" in API group "" at the cluster scope
I0920 09:29:07.236789 57754 utils.go:167] ! E0920 14:27:02.180380 1 reflector.go:123] k8s.io/client-go/informers/factory.go:134: Failed to list *v1.StorageClass: storageclasses.storage.k8s.io is forbidden: User "system:kube-scheduler" cannot list resource "storageclasses" in API group "storage.k8s.io" at the cluster scope
I0920 09:29:07.236824 57754 utils.go:167] ! E0920 14:27:02.182731 1 reflector.go:123] k8s.io/client-go/informers/factory.go:134: Failed to list *v1.Service: services is forbidden: User "system:kube-scheduler" cannot list resource "services" in API group "" at the cluster scope
I0920 09:29:07.236869 57754 utils.go:167] ! E0920 14:27:02.186453 1 reflector.go:123] k8s.io/client-go/informers/factory.go:134: Failed to list *v1.PersistentVolume: persistentvolumes is forbidden: User "system:kube-scheduler" cannot list resource "persistentvolumes" in API group "" at the cluster scope
I0920 09:29:07.236910 57754 utils.go:167] ! E0920 14:27:02.188026 1 reflector.go:123] k8s.io/client-go/informers/factory.go:134: Failed to list *v1.PersistentVolumeClaim: persistentvolumeclaims is forbidden: User "system:kube-scheduler" cannot list resource "persistentvolumeclaims" in API group "" at the cluster
scope
I0920 09:29:07.236926 57754 utils.go:167] ! I0920 14:27:04.104163 1 leaderelection.go:241] attempting to acquire leader lease kube-system/kube-scheduler...
I0920 09:29:07.236938 57754 utils.go:167] ! I0920 14:27:04.133869 1 leaderelection.go:251] successfully acquired lease kube-system/kube-scheduler
I0920 09:29:07.238590 57754 logs.go:78] Gathering logs for storage-provisioner [b53dfd77ed27] ...
I0920 09:29:07.238604 57754 ssh_runner.go:138] Run with output: docker logs --tail 200 b53dfd77ed27
W0920 09:29:07.287585 57754 exit.go:101] Error starting cluster: timed out waiting to elevate kube-system RBAC privileges: Temporary Error: creating clusterrolebinding: Post https://192.168.99.111:8443/apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings: dial tcp 192.168.99.111:8443: i/o timeout
*
X Error starting cluster: timed out waiting to elevate kube-system RBAC privileges: Temporary Error: creating clusterrolebinding: Post https://192.168.99.111:8443/apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings: dial tcp 192.168.99.111:8443: i/o timeout
*
* Sorry that minikube crashed. If this was unexpected, we would love to hear from you:
- https://github.com/kubernetes/minikube/issues/new/choose
* Problems detected in kube-addon-manager [a7de3d41615c]:
- error: no objects passed tserviceaccount/storage-provisioner unchangededited: I went back and uninstalled/reinstalled VirtualBox and was able to start. I then deleted and started my VPN and ran minikube start and it worked as well.
k8s-ci-robot
commented
4 years ago @holleyism: You can't reopen an issue/PR unless you authored it or you are a collaborator.
den-is
commented
4 years ago same here.. damn it......... i had no time to restart reinstall and to do rain-dance but for now, simple AnyConnect disabling didn't help
macos 10.14.6; minikube 1.4; virtualbox 6.0.12 r133076
sharifelgamal
commented
4 years ago /reopen
k8s-ci-robot
commented
4 years ago @sharifelgamal: Reopened this issue.
tstromberg
commented
4 years ago If you see this message with minikube v1.4.0 (or newer), please attach the output of:
minikube start --alsologtostderr -v=3minikube statusminikube logs. Thank you!den-is
commented
4 years ago UPD: minikube initial setup worked after computer restart without components reinstall.
minikube on mac installed via brew
Then stop/start cycle worked fine with enabled AnyConnect
tstromberg
commented
4 years ago Based on other issue reports, this issue still can happen in minikube v1.4, and appears to be mostly constrained VirtualBox users. The gist of the issue seems to be that there is a connectivity issue between the host and the VirtualBox VM (ip:8443), probably due to firewall or VPN interference.
If you are experiencing this, I would love to see the output of:
minikube ssh sudo pgrep apiserver
minikube status
As a workaround, I highly recommend using the minikube hyperkit driver:
minikube start --vm-driver=hyperkit
For more information, see:
https://minikube.sigs.k8s.io/docs/reference/drivers/hyperkit/
SolrSeeker
commented
4 years ago minikube 1.4.0 kubectl 1.16.0 Virtualbox 6.0.12 r133076
minikube start --alsologtostderr -v=3
minikube status
minikube logs
SolrSeeker
commented
4 years ago minikube 1.4.0 kubectl 1.16.0 Virtualbox 6.0.12 r133076
minikube ssh sudo pgrep apiserver
minikube status
tstromberg
commented
4 years ago @SolrSeeker - Thank you for confirming my suspicion. Any chance you could also run these commands to help me sort out what kind of networking issue we are seeing here?
minikube ssh "ss -tlpn"for i in 22 111 8443 10250; do nc -zvvv $(minikube ip) $i; doneping -c1 $(minikube ip)rpcinfo $(minikube ip)route get $(minikube ip)ifconfig vboxnet0VBoxManage list hostonlyifsSolrSeeker
commented
4 years ago minikube ssh "ss -tlpn"
for i in 22 111 8443 10250; do nc -zvvv $(minikube ip) $i; done
ping -c1 $(minikube ip)
rpcinfo $(minikube ip)
route get $(minikube ip)
ifconfig vboxnet0
VBoxManage list hostonlyifs
tstromberg
commented
4 years ago As mentioned in #3022, some workarounds people have found:
sudo route -n add -net 192.168.99.1/24 -interface vboxnet0--vm-driver=hyperkit)Root cause remains unknown.
bauson-com
commented
4 years ago Hi, I experienced the same problem on macOS Mojave 10.14.5 and fixed it using suggestions from other issue threads. Reinstalling both VirtualBox and minikube did not fix anything - the source of my problem was being connected to Cisco AnyConnect VPN. Here are the steps I used:
$ minikube stop $ minikube delete $ brew cask uninstall minikube $ rm -rf ~/.minikube ~/.kube Go to https://www.virtualbox.org/wiki/Downloads, use VirtualBox_Uninstall.tool script provided in OS X host .dm file Disconnect from VPN Restart laptop, make sure that you are not reconnected to VPN Install VirtualBox using VirtualBox.pkg from the same .dmg file as the previous step. $ brew cask install minikube $ minikube start --alsologtostderr -v=9 Connect to VPN (if you wish)
Hope this helps. If anyone has suggestions for starting minikube while connected to a VPN like Cisco AnyConnect, please let me know! Thanks
Does it mean I need to be disconnected to VPN every time I start minikube? Or there is a work around to make it work regardless if connected or not connected to Cisco AnyConnect VPN?
Thanks btw
den-is
commented
4 years ago @bauson-com only disconnect will not help you will have to restart mac (or maybe there is some routes refresh commnd.. idk...)
So here are my observations, actually same one thing just from different points of view:
minikube-virtualbox will always work if you have never started AnyConnect VPN before minikube manipulations
You can enable Anyconnect VPN after minikube has finished bootstrapping and/or starting minikube - everything will work
You will not be able setup minikube or start minikube if you had anyconnect enabled in you current session, even if you quit it, you will have to restart mac.
I have my very own handmade kubernetes on virtualbox setup using Vagrant+Ansible. a.k.a pure environement for expermentations not biased by minikube. I observe absolutely same issue. Virtualbox can't add routes after anyconnect. But in addition I use custom host-only virtualbox subnet 10.22.0.0/24 for my kubernetes.
here are netstat -nr outputs
# First ViartualBox was up then enabled AnyConnect... all routes exist
Internet:
Destination Gateway Flags Refs Use Netif Expire
default 10.1.1.1 UGSc 127 0 en9
1.0.0.1/32 10.1.1.1 UGSc 0 0 en9
1.1.1.1/32 10.1.1.1 UGSc 0 0 en9
10.1.1/24 link#7 UCS 2 0 en9 !
10.1.1.1 0:c:42:b1:c8:19 UHLSr 55 0 en9
10.1.1.5 0:11:32:67:64:23 UHLWIi 2 16728 en9 1052
10.1.1.103 18:65:90:2b:4:1 UHLWIi 1 182 en9 1065
10.1.1.197/32 link#7 UCS 0 0 en9 !
10.22/24 link#22 UC 3 0 vboxnet !
10.22.0.2 8:0:27:e8:cc:fc UHLWI 0 0 vboxnet 500
10.22.0.21 8:0:27:0:57:e UHLWIi 2 4978 vboxnet 633
10.22.0.31 8:0:27:93:45:d UHLWI 0 0 vboxnet 588
10.66.100/24 10.55.78.22 UGSc 0 0 utun2
10.70/16 10.55.78.22 UGSc 0 0 utun2
10.55/16 10.55.78.22 UGSc 0 0 utun2
10.55.10.17/32 10.55.78.22 UGSc 1 0 utun2
10.55.10.18/32 10.55.78.22 UGSc 0 0 utun2
10.55.78.22/32 127.0.0.1 UGSc 7 0 lo0
10.201/16 10.55.78.22 UGSc 0 0 utun2
127 127.0.0.1 UCS 0 0 lo0
127.0.0.1 127.0.0.1 UH 49 127320 lo0
169.254 link#7 UCS 1 0 en9 !
169.254 link#19 UCSI 1 0 en8 !
169.254.169.254 link#7 UHLSW 0 24 en9 !
169.254.219.207 1a:65:90:2b:4:fc UHLSW 2 79 en8 109
169.254.233.102/32 link#19 UCS 0 0 en8 !
192.168.5 10.55.78.22 UGSc 0 0 utun2
208.122.223.158/32 10.1.1.1 UGSc 1 0 en9
224.0.0/4 link#7 UmCS 2 0 en9 !
224.0.0/4 link#19 UmCSI 0 0 en8 !
224.0.0.251 1:0:5e:0:0:fb UHmLWI 0 0 en9
239.255.255.250 1:0:5e:7f:ff:fa UHmLWI 0 28 en9
255.255.255.255/32 link#7 UCS 0 0 en9 !
255.255.255.255/32 link#19 UCSI 0 0 en8 !#VPN enabled before VirtualBox.. you can't see 10.22/24 anymore in bellow result
Internet:
Destination Gateway Flags Refs Use Netif Expire
default 10.1.1.1 UGSc 132 0 en9
1.0.0.1/32 10.1.1.1 UGSc 0 0 en9
1.1.1.1/32 10.1.1.1 UGSc 0 0 en9
10.1.1/24 link#7 UCS 3 0 en9 !
10.1.1.1 0:c:42:b1:c8:19 UHLSr 78 0 en9
10.1.1.5 0:11:32:67:64:23 UHLWIi 1 3264 en9 1127
10.1.1.103 18:65:90:2b:4:1 UHLWIi 2 11 en9 541
10.1.1.197/32 link#7 UCS 0 0 en9 !
10.66.100/24 10.55.78.39 UGSc 0 0 utun1
10.70/16 10.55.78.39 UGSc 1 0 utun1
10.55/16 10.55.78.39 UGSc 0 0 utun1
10.55.10.17/32 10.55.78.39 UGSc 1 0 utun1
10.55.10.18/32 10.55.78.39 UGSc 0 0 utun1
10.55.78.39/32 127.0.0.1 UGSc 7 0 lo0
10.201/16 10.55.78.39 UGSc 0 0 utun1
127 127.0.0.1 UCS 0 0 lo0
127.0.0.1 127.0.0.1 UH 39 99594 lo0
169.254 link#7 UCS 2 0 en9 !
169.254.23.138/32 link#16 UCS 1 0 en8 !
169.254.169.254 link#7 UHLSW 1 34 en9 !
169.254.219.207 1a:65:90:2b:4:fc UHLSW 0 1 en8 533
192.168.5 10.55.78.39 UGSc 0 0 utun1
208.122.223.158/32 10.1.1.1 UGSc 1 0 en9
224.0.0/4 link#7 UmCS 2 0 en9 !
224.0.0/4 link#16 UmCSI 0 0 en8 !
224.0.0.251 1:0:5e:0:0:fb UHmLWI 0 0 en9
239.255.255.250 1:0:5e:7f:ff:fa UHmLWI 0 32 en9
255.255.255.255/32 link#7 UCS 0 0 en9 !
255.255.255.255/32 link#16 UCSI 0 0 en8 !similar part is missing:
10.22/24 link#22 UC 3 0 vboxnet !
10.22.0.2 8:0:27:e8:cc:fc UHLWI 0 0 vboxnet 500
10.22.0.21 8:0:27:0:57:e UHLWIi 2 4978 vboxnet 633
10.22.0.31 8:0:27:93:45:d UHLWI 0 0 vboxnet 588I have created issue on virtualbox forums https://forums.virtualbox.org/viewtopic.php?f=8&t=95310
bzied321
commented
4 years ago @bauson-com FYI, I did not run into the same problem again (with or without my VPN connected) when starting minikube after the steps I recorded
tstromberg
commented
4 years ago NOTE: minikube v1.5 detects this issue, and provides an early exit if a network interface misconfiguration is detected:
minikube is unable to connect to the VM: dial tcp 192.168.99.108:22: connect: operation timed out
This is likely due to one of two reasons:
- VPN or firewall interference
- virtualbox network configuration issue
Suggested workarounds:
- Disable your local VPN or firewall software
- Configure your local VPN or firewall to allow access to 192.168.99.108
- Restart or reinstall virtualbox
- Use an alternative --vm-driver
medyagh
commented
4 years ago @kmayura1 do you mind tryin with the latest version of minikube to see if the problem is detected by minikube ? (also there is a newer verison of virtualbox that I recommend upgrading to)
r0bnet
commented
4 years ago Had the same problem with Cisco AnyConnect. Had to reinstall VirtualBox and then it worked. I guess restarting machine or re-configure some routes should also do the job but I didn't try. minikube version: 1.9.0 VirtualBox version: 6.1.4
mtmk
commented
3 years ago Hi, I experienced the same problem on macOS Mojave 10.14.5 and fixed it using suggestions from other issue threads. Reinstalling both VirtualBox and minikube did not fix anything - the source of my problem was being connected to Cisco AnyConnect VPN. Here are the steps I used:
$ minikube stop $ minikube delete $ brew cask uninstall minikube $ rm -rf ~/.minikube ~/.kube Go to https://www.virtualbox.org/wiki/Downloads, use VirtualBox_Uninstall.tool script provided in OS X host .dm file Disconnect from VPN Restart laptop, make sure that you are not reconnected to VPN Install VirtualBox using VirtualBox.pkg from the same .dmg file as the previous step. $ brew cask install minikube $ minikube start --alsologtostderr -v=9 Connect to VPN (if you wish)
Hope this helps. If anyone has suggestions for starting minikube while connected to a VPN like Cisco AnyConnect, please let me know! Thanks
Thanks. I think the one fixed it for me was the rm -rf ~/.minikube ~/.kube line. I had a Minikube install from ages ago, the old config from that might've been interfering.
I restarted my macOS Majove, Had reinstalled docker after couple of minikube start hangings, followed few steps everytime starting minikube but still issue, tried with other versions too. rm -rf ~/.minikube rm -rf ~/.kube brew cask reinstall virtualbox brew reinstall kubernetes-cli curl -Lo minikube https://storage.googleapis.com/minikube/releases/v1.1.1/minikube-darwin-amd64 && chmod +x minikube && sudo cp minikube /usr/local/bin/ && rm minikube
minikube start š minikube v1.1.1 on darwin (amd64) š„ Creating virtualbox VM (CPUs=2, Memory=2048MB, Disk=20000MB) ... š³ Configuring environment for Kubernetes v1.14.3 on Docker 18.09.6 š¾ Downloading kubeadm v1.14.3 š¾ Downloading kubelet v1.14.3 š Pulling images ... š Launching Kubernetes ...
Minikube logs ::::
I0620 12:52:39.138524 1 leaderelection.go:217] attempting to acquire leader lease kube-system/kube-scheduler... I0620 12:52:39.146724 1 leaderelection.go:227] successfully acquired lease kube-system/kube-scheduler
==> kubelet <== -- Logs begin at Thu 2019-06-20 12:50:53 UTC, end at Thu 2019-06-20 12:55:34 UTC. -- Jun 20 12:52:33 minikube kubelet[3230]: E0620 12:52:33.146561 3230 kubelet.go:2244] node "minikube" not found Jun 20 12:52:33 minikube kubelet[3230]: E0620 12:52:33.247011 3230 kubelet.go:2244] node "minikube" not found Jun 20 12:52:33 minikube kubelet[3230]: E0620 12:52:33.348321 3230 kubelet.go:2244] node "minikube" not found Jun 20 12:52:33 minikube kubelet[3230]: E0620 12:52:33.448682 3230 kubelet.go:2244] node "minikube" not found Jun 20 12:52:33 minikube kubelet[3230]: E0620 12:52:33.548808 3230 kubelet.go:2244] node "minikube" not found Jun 20 12:52:33 minikube kubelet[3230]: E0620 12:52:33.649491 3230 kubelet.go:2244] node "minikube" not found Jun 20 12:52:33 minikube kubelet[3230]: E0620 12:52:33.749727 3230 kubelet.go:2244] node "minikube" not found Jun 20 12:52:33 minikube kubelet[3230]: E0620 12:52:33.850310 3230 kubelet.go:2244] node "minikube" not found Jun 20 12:52:33 minikube kubelet[3230]: E0620 12:52:33.950992 3230 kubelet.go:2244] node "minikube" not found Jun 20 12:52:33 minikube kubelet[3230]: I0620 12:52:33.996984 3230 kubelet_node_status.go:283] Setting node annotation to enable volume controller attach/detach Jun 20 12:52:33 minikube kubelet[3230]: I0620 12:52:33.997464 3230 kubelet_node_status.go:283] Setting node annotation to enable volume controller attach/detach Jun 20 12:52:33 minikube kubelet[3230]: I0620 12:52:33.997814 3230 kubelet_node_status.go:283] Setting node annotation to enable volume controller attach/detach Jun 20 12:52:34 minikube kubelet[3230]: I0620 12:52:34.004385 3230 kubelet_node_status.go:283] Setting node annotation to enable volume controller attach/detach Jun 20 12:52:34 minikube kubelet[3230]: E0620 12:52:34.051474 3230 kubelet.go:2244] node "minikube" not found Jun 20 12:52:34 minikube kubelet[3230]: E0620 12:52:34.152419 3230 kubelet.go:2244] node "minikube" not found Jun 20 12:52:34 minikube kubelet[3230]: E0620 12:52:34.253045 3230 kubelet.go:2244] node "minikube" not found Jun 20 12:52:34 minikube kubelet[3230]: E0620 12:52:34.353602 3230 kubelet.go:2244] node "minikube" not found Jun 20 12:52:34 minikube kubelet[3230]: E0620 12:52:34.456007 3230 kubelet.go:2244] node "minikube" not found Jun 20 12:52:34 minikube kubelet[3230]: E0620 12:52:34.557146 3230 kubelet.go:2244] node "minikube" not found Jun 20 12:52:34 minikube kubelet[3230]: E0620 12:52:34.657687 3230 kubelet.go:2244] node "minikube" not found Jun 20 12:52:34 minikube kubelet[3230]: E0620 12:52:34.758119 3230 kubelet.go:2244] node "minikube" not found Jun 20 12:52:34 minikube kubelet[3230]: E0620 12:52:34.858423 3230 kubelet.go:2244] node "minikube" not found Jun 20 12:52:34 minikube kubelet[3230]: E0620 12:52:34.958695 3230 kubelet.go:2244] node "minikube" not found Jun 20 12:52:35 minikube kubelet[3230]: E0620 12:52:35.059076 3230 kubelet.go:2244] node "minikube" not found Jun 20 12:52:35 minikube kubelet[3230]: E0620 12:52:35.159999 3230 kubelet.go:2244] node "minikube" not found Jun 20 12:52:35 minikube kubelet[3230]: E0620 12:52:35.261426 3230 kubelet.go:2244] node "minikube" not found Jun 20 12:52:35 minikube kubelet[3230]: E0620 12:52:35.362034 3230 kubelet.go:2244] node "minikube" not found Jun 20 12:52:35 minikube kubelet[3230]: E0620 12:52:35.463033 3230 kubelet.go:2244] node "minikube" not found Jun 20 12:52:35 minikube kubelet[3230]: E0620 12:52:35.563441 3230 kubelet.go:2244] node "minikube" not found Jun 20 12:52:35 minikube kubelet[3230]: E0620 12:52:35.663668 3230 kubelet.go:2244] node "minikube" not found Jun 20 12:52:35 minikube kubelet[3230]: E0620 12:52:35.764994 3230 kubelet.go:2244] node "minikube" not found Jun 20 12:52:35 minikube kubelet[3230]: E0620 12:52:35.866000 3230 kubelet.go:2244] node "minikube" not found Jun 20 12:52:35 minikube kubelet[3230]: E0620 12:52:35.966534 3230 kubelet.go:2244] node "minikube" not found Jun 20 12:52:36 minikube kubelet[3230]: E0620 12:52:36.067169 3230 kubelet.go:2244] node "minikube" not found Jun 20 12:52:36 minikube kubelet[3230]: E0620 12:52:36.167937 3230 kubelet.go:2244] node "minikube" not found Jun 20 12:52:36 minikube kubelet[3230]: E0620 12:52:36.264483 3230 controller.go:194] failed to get node "minikube" when trying to set owner ref to the node lease: nodes "minikube" not found Jun 20 12:52:36 minikube kubelet[3230]: I0620 12:52:36.266752 3230 reconciler.go:154] Reconciler: start to sync state Jun 20 12:52:36 minikube kubelet[3230]: E0620 12:52:36.268165 3230 kubelet.go:2244] node "minikube" not found Jun 20 12:52:36 minikube kubelet[3230]: I0620 12:52:36.287922 3230 kubelet_node_status.go:75] Successfully registered node minikube Jun 20 12:52:36 minikube kubelet[3230]: E0620 12:52:36.345166 3230 controller.go:115] failed to ensure node lease exists, will retry in 1.6s, error: namespaces "kube-node-lease" not found Jun 20 12:52:47 minikube kubelet[3230]: I0620 12:52:47.112171 3230 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "lib-modules" (UniqueName: "kubernetes.io/host-path/4d96ec6e-935a-11e9-b941-080027ad4350-lib-modules") pod "kube-proxy-j5gx8" (UID: "4d96ec6e-935a-11e9-b941-080027ad4350") Jun 20 12:52:47 minikube kubelet[3230]: I0620 12:52:47.112583 3230 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "kube-proxy-token-4rtnv" (UniqueName: "kubernetes.io/secret/4d96ec6e-935a-11e9-b941-080027ad4350-kube-proxy-token-4rtnv") pod "kube-proxy-j5gx8" (UID: "4d96ec6e-935a-11e9-b941-080027ad4350") Jun 20 12:52:47 minikube kubelet[3230]: I0620 12:52:47.112759 3230 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "kube-proxy" (UniqueName: "kubernetes.io/configmap/4d96ec6e-935a-11e9-b941-080027ad4350-kube-proxy") pod "kube-proxy-j5gx8" (UID: "4d96ec6e-935a-11e9-b941-080027ad4350") Jun 20 12:52:47 minikube kubelet[3230]: I0620 12:52:47.112811 3230 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "xtables-lock" (UniqueName: "kubernetes.io/host-path/4d96ec6e-935a-11e9-b941-080027ad4350-xtables-lock") pod "kube-proxy-j5gx8" (UID: "4d96ec6e-935a-11e9-b941-080027ad4350") Jun 20 12:52:47 minikube kubelet[3230]: I0620 12:52:47.919513 3230 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "tmp" (UniqueName: "kubernetes.io/host-path/4e1a7293-935a-11e9-b941-080027ad4350-tmp") pod "storage-provisioner" (UID: "4e1a7293-935a-11e9-b941-080027ad4350") Jun 20 12:52:47 minikube kubelet[3230]: I0620 12:52:47.919638 3230 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "storage-provisioner-token-zrl9x" (UniqueName: "kubernetes.io/secret/4e1a7293-935a-11e9-b941-080027ad4350-storage-provisioner-token-zrl9x") pod "storage-provisioner" (UID: "4e1a7293-935a-11e9-b941-080027ad4350") Jun 20 12:52:48 minikube kubelet[3230]: I0620 12:52:48.020147 3230 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "config-volume" (UniqueName: "kubernetes.io/configmap/4d8460c8-935a-11e9-b941-080027ad4350-config-volume") pod "coredns-fb8b8dccf-sgz9v" (UID: "4d8460c8-935a-11e9-b941-080027ad4350") Jun 20 12:52:48 minikube kubelet[3230]: I0620 12:52:48.020191 3230 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "config-volume" (UniqueName: "kubernetes.io/configmap/4d81a4f1-935a-11e9-b941-080027ad4350-config-volume") pod "coredns-fb8b8dccf-jf5wq" (UID: "4d81a4f1-935a-11e9-b941-080027ad4350") Jun 20 12:52:48 minikube kubelet[3230]: I0620 12:52:48.020211 3230 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "coredns-token-2ggqz" (UniqueName: "kubernetes.io/secret/4d81a4f1-935a-11e9-b941-080027ad4350-coredns-token-2ggqz") pod "coredns-fb8b8dccf-jf5wq" (UID: "4d81a4f1-935a-11e9-b941-080027ad4350") Jun 20 12:52:48 minikube kubelet[3230]: I0620 12:52:48.020280 3230 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "coredns-token-2ggqz" (UniqueName: "kubernetes.io/secret/4d8460c8-935a-11e9-b941-080027ad4350-coredns-token-2ggqz") pod "coredns-fb8b8dccf-sgz9v" (UID: "4d8460c8-935a-11e9-b941-080027ad4350")