search

kubernetes / minikube

Run Kubernetes locally
https://minikube.sigs.k8s.io/
Apache License 2.0
28.77k stars 4.81k forks source link

minikube cluster fails to start #6429

Closed smitna closed 4 years ago

smitna commented 4 years ago

Transcript of session ending at command that failed:

$ egrep -o 'systemd.unified_cgroup_hierarchy=[[:digit:]]+' /proc/cmdline systemd.unified_cgroup_hierarchy=0 $ getenforce Permissive $ virt-host-validate | egrep -v ': PASS$' || echo 'All virt-host-validate tests passed.' All virt-host-validate tests passed. $ minikube version minikube version: v1.7.0-beta.1 commit: edd3f066a2017ed512b7d02f37d8c3c0d74e60cc $ minikube delete --all=true 🔥 Deleting "minikube" in kvm2 ... 💔 The "minikube" cluster has been deleted. 🔥 Successfully deleted all profiles $ minikube config view

  • memory: 4096 $ minikube start 😄 minikube v1.7.0-beta.1 on Fedora 31 ✨ Automatically selected the 'kvm2' driver (alternates: [none]) 🔥 Creating kvm2 VM (CPUs=2, Memory=4096MB, Disk=20000MB) ... 🐳 Preparing Kubernetes v1.17.2 on Docker '19.03.5' ... ❌ Unable to load cached images: loading cached images: Docker load /var/lib/minikube/images/pause_3.1: loadimage docker.: docker load -i /var/lib/minikube/images/pause_3.1: wait: remote command exited without exit status or exit signal stdout:

stderr:

💣 Failed to update cluster: downloading binaries: NewSession: read tcp 192.168.39.1:49984->192.168.39.132:22: read: connection reset by peer

😿 minikube is exiting due to an error. If the above message is not useful, open an issue: 👉 https://github.com/kubernetes/minikube/issues/new/choose

The output of the minikube logs command:

  • ==> Docker <==
  • -- Logs begin at Thu 2020-01-30 04:08:43 UTC, end at Thu 2020-01-30 04:11:40 UTC. --
  • Jan 30 04:08:51 minikube dockerd[2023]: time="2020-01-30T04:08:51.279395914Z" level=info msg="loading plugin "io.containerd.snapshotter.v1.native"..." type=io.containerd.snapshotter.v1
  • Jan 30 04:08:51 minikube dockerd[2023]: time="2020-01-30T04:08:51.279575079Z" level=info msg="loading plugin "io.containerd.snapshotter.v1.overlayfs"..." type=io.containerd.snapshotter.v1
  • Jan 30 04:08:51 minikube dockerd[2023]: time="2020-01-30T04:08:51.279961834Z" level=info msg="loading plugin "io.containerd.snapshotter.v1.zfs"..." type=io.containerd.snapshotter.v1
  • Jan 30 04:08:51 minikube dockerd[2023]: time="2020-01-30T04:08:51.280280034Z" level=info msg="skip loading plugin "io.containerd.snapshotter.v1.zfs"..." type=io.containerd.snapshotter.v1
  • Jan 30 04:08:51 minikube dockerd[2023]: time="2020-01-30T04:08:51.280308172Z" level=info msg="loading plugin "io.containerd.metadata.v1.bolt"..." type=io.containerd.metadata.v1
  • Jan 30 04:08:51 minikube dockerd[2023]: time="2020-01-30T04:08:51.280396437Z" level=warning msg="could not use snapshotter zfs in metadata plugin" error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.zfs must be a zfs filesystem to be used with the zfs snapshotter: skip plugin"
  • Jan 30 04:08:51 minikube dockerd[2023]: time="2020-01-30T04:08:51.280419438Z" level=warning msg="could not use snapshotter btrfs in metadata plugin" error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.btrfs must be a btrfs filesystem to be used with the btrfs snapshotter"
  • Jan 30 04:08:51 minikube dockerd[2023]: time="2020-01-30T04:08:51.280434604Z" level=warning msg="could not use snapshotter aufs in metadata plugin" error="modprobe aufs failed: "modprobe: FATAL: Module aufs not found in directory /lib/modules/4.19.88\n": exit status 1"
  • Jan 30 04:08:52 minikube dockerd[2023]: time="2020-01-30T04:08:52.026991967Z" level=info msg="loading plugin "io.containerd.differ.v1.walking"..." type=io.containerd.differ.v1
  • Jan 30 04:08:52 minikube dockerd[2023]: time="2020-01-30T04:08:52.027153493Z" level=info msg="loading plugin "io.containerd.gc.v1.scheduler"..." type=io.containerd.gc.v1
  • Jan 30 04:08:52 minikube dockerd[2023]: time="2020-01-30T04:08:52.027342605Z" level=info msg="loading plugin "io.containerd.service.v1.containers-service"..." type=io.containerd.service.v1
  • Jan 30 04:08:52 minikube dockerd[2023]: time="2020-01-30T04:08:52.027428511Z" level=info msg="loading plugin "io.containerd.service.v1.content-service"..." type=io.containerd.service.v1
  • Jan 30 04:08:52 minikube dockerd[2023]: time="2020-01-30T04:08:52.027494816Z" level=info msg="loading plugin "io.containerd.service.v1.diff-service"..." type=io.containerd.service.v1
  • Jan 30 04:08:52 minikube dockerd[2023]: time="2020-01-30T04:08:52.027592216Z" level=info msg="loading plugin "io.containerd.service.v1.images-service"..." type=io.containerd.service.v1
  • Jan 30 04:08:52 minikube dockerd[2023]: time="2020-01-30T04:08:52.027715509Z" level=info msg="loading plugin "io.containerd.service.v1.leases-service"..." type=io.containerd.service.v1
  • Jan 30 04:08:52 minikube dockerd[2023]: time="2020-01-30T04:08:52.027785126Z" level=info msg="loading plugin "io.containerd.service.v1.namespaces-service"..." type=io.containerd.service.v1
  • Jan 30 04:08:52 minikube dockerd[2023]: time="2020-01-30T04:08:52.027861893Z" level=info msg="loading plugin "io.containerd.service.v1.snapshots-service"..." type=io.containerd.service.v1
  • Jan 30 04:08:52 minikube dockerd[2023]: time="2020-01-30T04:08:52.027927352Z" level=info msg="loading plugin "io.containerd.runtime.v1.linux"..." type=io.containerd.runtime.v1
  • Jan 30 04:08:52 minikube dockerd[2023]: time="2020-01-30T04:08:52.028651572Z" level=info msg="loading plugin "io.containerd.runtime.v2.task"..." type=io.containerd.runtime.v2
  • Jan 30 04:08:52 minikube dockerd[2023]: time="2020-01-30T04:08:52.028878460Z" level=info msg="loading plugin "io.containerd.monitor.v1.cgroups"..." type=io.containerd.monitor.v1
  • Jan 30 04:08:52 minikube dockerd[2023]: time="2020-01-30T04:08:52.029450658Z" level=info msg="loading plugin "io.containerd.service.v1.tasks-service"..." type=io.containerd.service.v1
  • Jan 30 04:08:52 minikube dockerd[2023]: time="2020-01-30T04:08:52.029519482Z" level=info msg="loading plugin "io.containerd.internal.v1.restart"..." type=io.containerd.internal.v1
  • Jan 30 04:08:52 minikube dockerd[2023]: time="2020-01-30T04:08:52.029653664Z" level=info msg="loading plugin "io.containerd.grpc.v1.containers"..." type=io.containerd.grpc.v1
  • Jan 30 04:08:52 minikube dockerd[2023]: time="2020-01-30T04:08:52.029686646Z" level=info msg="loading plugin "io.containerd.grpc.v1.content"..." type=io.containerd.grpc.v1
  • Jan 30 04:08:52 minikube dockerd[2023]: time="2020-01-30T04:08:52.029716932Z" level=info msg="loading plugin "io.containerd.grpc.v1.diff"..." type=io.containerd.grpc.v1
  • Jan 30 04:08:52 minikube dockerd[2023]: time="2020-01-30T04:08:52.029744052Z" level=info msg="loading plugin "io.containerd.grpc.v1.events"..." type=io.containerd.grpc.v1
  • Jan 30 04:08:52 minikube dockerd[2023]: time="2020-01-30T04:08:52.029772186Z" level=info msg="loading plugin "io.containerd.grpc.v1.healthcheck"..." type=io.containerd.grpc.v1
  • Jan 30 04:08:52 minikube dockerd[2023]: time="2020-01-30T04:08:52.029803345Z" level=info msg="loading plugin "io.containerd.grpc.v1.images"..." type=io.containerd.grpc.v1
  • Jan 30 04:08:52 minikube dockerd[2023]: time="2020-01-30T04:08:52.029831122Z" level=info msg="loading plugin "io.containerd.grpc.v1.leases"..." type=io.containerd.grpc.v1
  • Jan 30 04:08:52 minikube dockerd[2023]: time="2020-01-30T04:08:52.029859379Z" level=info msg="loading plugin "io.containerd.grpc.v1.namespaces"..." type=io.containerd.grpc.v1
  • Jan 30 04:08:52 minikube dockerd[2023]: time="2020-01-30T04:08:52.029890141Z" level=info msg="loading plugin "io.containerd.internal.v1.opt"..." type=io.containerd.internal.v1
  • Jan 30 04:08:52 minikube dockerd[2023]: time="2020-01-30T04:08:52.030024778Z" level=info msg="loading plugin "io.containerd.grpc.v1.snapshots"..." type=io.containerd.grpc.v1
  • Jan 30 04:08:52 minikube dockerd[2023]: time="2020-01-30T04:08:52.030066212Z" level=info msg="loading plugin "io.containerd.grpc.v1.tasks"..." type=io.containerd.grpc.v1
  • Jan 30 04:08:52 minikube dockerd[2023]: time="2020-01-30T04:08:52.030098519Z" level=info msg="loading plugin "io.containerd.grpc.v1.version"..." type=io.containerd.grpc.v1
  • Jan 30 04:08:52 minikube dockerd[2023]: time="2020-01-30T04:08:52.030126760Z" level=info msg="loading plugin "io.containerd.grpc.v1.introspection"..." type=io.containerd.grpc.v1
  • Jan 30 04:08:52 minikube dockerd[2023]: time="2020-01-30T04:08:52.030376696Z" level=info msg=serving... address="/var/run/docker/containerd/containerd-debug.sock"
  • Jan 30 04:08:52 minikube dockerd[2023]: time="2020-01-30T04:08:52.030477188Z" level=info msg=serving... address="/var/run/docker/containerd/containerd.sock"
  • Jan 30 04:08:52 minikube dockerd[2023]: time="2020-01-30T04:08:52.030509542Z" level=info msg="containerd successfully booted in 0.755056s"
  • Jan 30 04:08:52 minikube dockerd[2023]: time="2020-01-30T04:08:52.043707475Z" level=info msg="parsed scheme: \"unix\"" module=grpc
  • Jan 30 04:08:52 minikube dockerd[2023]: time="2020-01-30T04:08:52.043896685Z" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
  • Jan 30 04:08:52 minikube dockerd[2023]: time="2020-01-30T04:08:52.044009515Z" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///var/run/docker/containerd/containerd.sock 0 }] }" module=grpc
  • Jan 30 04:08:52 minikube dockerd[2023]: time="2020-01-30T04:08:52.044099289Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
  • Jan 30 04:08:52 minikube dockerd[2023]: time="2020-01-30T04:08:52.045366158Z" level=info msg="parsed scheme: \"unix\"" module=grpc
  • Jan 30 04:08:52 minikube dockerd[2023]: time="2020-01-30T04:08:52.045463859Z" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
  • Jan 30 04:08:52 minikube dockerd[2023]: time="2020-01-30T04:08:52.045552969Z" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///var/run/docker/containerd/containerd.sock 0 }] }" module=grpc
  • Jan 30 04:08:52 minikube dockerd[2023]: time="2020-01-30T04:08:52.045714249Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
  • Jan 30 04:08:52 minikube dockerd[2023]: time="2020-01-30T04:08:52.288366606Z" level=warning msg="Your kernel does not support cgroup blkio weight"
  • Jan 30 04:08:52 minikube dockerd[2023]: time="2020-01-30T04:08:52.290233588Z" level=warning msg="Your kernel does not support cgroup blkio weight_device"
  • Jan 30 04:08:52 minikube dockerd[2023]: time="2020-01-30T04:08:52.290470492Z" level=warning msg="Your kernel does not support cgroup blkio throttle.read_bps_device"
  • Jan 30 04:08:52 minikube dockerd[2023]: time="2020-01-30T04:08:52.290719101Z" level=warning msg="Your kernel does not support cgroup blkio throttle.write_bps_device"
  • Jan 30 04:08:52 minikube dockerd[2023]: time="2020-01-30T04:08:52.290926565Z" level=warning msg="Your kernel does not support cgroup blkio throttle.read_iops_device"
  • Jan 30 04:08:52 minikube dockerd[2023]: time="2020-01-30T04:08:52.291167907Z" level=warning msg="Your kernel does not support cgroup blkio throttle.write_iops_device"
  • Jan 30 04:08:52 minikube dockerd[2023]: time="2020-01-30T04:08:52.291935107Z" level=info msg="Loading containers: start."
  • Jan 30 04:08:52 minikube dockerd[2023]: time="2020-01-30T04:08:52.619032751Z" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address"
  • Jan 30 04:08:52 minikube dockerd[2023]: time="2020-01-30T04:08:52.793384167Z" level=info msg="Loading containers: done."
  • Jan 30 04:08:52 minikube dockerd[2023]: time="2020-01-30T04:08:52.932269586Z" level=info msg="Docker daemon" commit=633a0ea838 graphdriver(s)=overlay2 version=19.03.5
  • Jan 30 04:08:52 minikube dockerd[2023]: time="2020-01-30T04:08:52.933104827Z" level=info msg="Daemon has completed initialization"
  • Jan 30 04:08:53 minikube dockerd[2023]: time="2020-01-30T04:08:53.566367036Z" level=info msg="API listen on /var/run/docker.sock"
  • Jan 30 04:08:53 minikube systemd[1]: Started Docker Application Container Engine.
  • Jan 30 04:08:53 minikube dockerd[2023]: time="2020-01-30T04:08:53.566406201Z" level=info msg="API listen on [::]:2376"
  • ==> container status <==
  • time="2020-01-30T04:11:42Z" level=fatal msg="failed to connect: failed to connect, make sure you are running as root and the runtime has been started: context deadline exceeded"
  • CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
  • ==> dmesg <==
  • [Jan30 04:08] You have booted with nomodeset. This means your GPU drivers are DISABLED
  • [ +0.000001] Any video related functionality will be severely degraded, and you may not even be able to suspend the system properly
  • [ +0.000000] Unless you actually understand what nomodeset does, you should reboot without enabling it
  • [ +0.039712] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge.
  • [ +2.342371] platform regulatory.0: Direct firmware load for regulatory.db failed with error -2
  • [ +0.962092] systemd[1]: Failed to bump fs.file-max, ignoring: Invalid argument
  • [ +0.010015] systemd-fstab-generator[1143]: Ignoring "noauto" for root device
  • [ +0.002076] systemd[1]: File /usr/lib/systemd/system/systemd-journald.service:12 configures an IP firewall (IPAddressDeny=any), but the local system does not support BPF/cgroup based firewalling.
  • [ +0.000002] systemd[1]: Proceeding WITHOUT firewalling in effect! (This warning is only shown for the first loaded unit using IP firewalling.)
  • [ +1.188450] NFSD: the nfsdcld client tracking upcall will be removed in 3.10. Please transition to using nfsdcltrack.
  • [ +2.029430] vboxguest: loading out-of-tree module taints kernel.
  • [ +0.003338] vboxguest: PCI device not found, probably running on physical hardware.
  • [ +4.218016] systemd-fstab-generator[1991]: Ignoring "noauto" for root device
  • [ +0.295751] systemd-fstab-generator[2007]: Ignoring "noauto" for root device
  • [Jan30 04:10] NFSD: Unable to end grace period: -110
  • ==> kernel <==
  • 04:11:42 up 3 min, 0 users, load average: 0.18, 0.12, 0.04
  • Linux minikube 4.19.88 #1 SMP Wed Jan 22 13:21:15 PST 2020 x86_64 GNU/Linux
  • PRETTY_NAME="Buildroot 2019.02.8"
  • ==> kubelet <==
  • -- Logs begin at Thu 2020-01-30 04:08:43 UTC, end at Thu 2020-01-30 04:11:42 UTC. --
  • -- No entries --

The operating system version: cpe: cpe:/o:fedoraproject:fedora:31 arch: x86_64

Additional notes: 1) Although the above output claims only one alternate vm-driver, previously I did install (and later remove) virtualbox but minikube aborted in a similar fashion. 2) I also tried 1.6.2 (latest stable release) but a similar error occurred.

smitna commented 4 years ago

After a few more experiments and log reviews, I suspected that the underlying issue was linked to network i/o. Indeed, the problem seems to have been Broadcom WiFi (either hardware or kernel module). On either a wired link or Atheros WiFi, I was able to start a cluster successfully 10 times in a row (i.e. 20 total).