search

kubernetes / minikube

Run Kubernetes locally
https://minikube.sigs.k8s.io/
Apache License 2.0
29.49k stars 4.89k forks source link

Cannot start minikube: "cannot stat '/var/run/cri-dockerd.sock'" #14411

Closed EHJ-52n closed 2 years ago

EHJ-52n commented 2 years ago

What Happened?

~ minikube start --kubernetes-version=stable
😄  minikube v1.26.0 on Ubuntu 21.10
✨  Using the docker driver based on existing profile
👍  Starting control plane node minikube in cluster minikube
🚜  Pulling base image ...
    > gcr.io/k8s-minikube/kicbase: 378.98 MiB / 378.98 MiB  100.00% 5.16 MiB p/
    > gcr.io/k8s-minikube/kicbase: 0 B [_________________________] ?% ? p/s 48s
🤷  docker "minikube" container is missing, will recreate.
🔥  Creating docker container (CPUs=2, Memory=7900MB) ...

❌  Exiting due to RUNTIME_ENABLE: stat /var/run/cri-dockerd.sock: Process exited with status 1
stdout:

stderr:
stat: cannot stat '/var/run/cri-dockerd.sock': No such file or directory

╭───────────────────────────────────────────────────────────────────────────────────────────╮
│                                                                                           │
│    😿  If the above advice does not help, please let us know:                             │
│    👉  https://github.com/kubernetes/minikube/issues/new/choose                           │
│                                                                                           │
│    Please run `minikube logs --file=logs.txt` and attach logs.txt to the GitHub issue.    │
│                                                                                           │
╰───────────────────────────────────────────────────────────────────────────────────────────╯

~ minikube start --kubernetes-version=stable --alsologtostderr
I0624 10:37:32.526852   39866 out.go:296] Setting OutFile to fd 1 ...
I0624 10:37:32.526931   39866 out.go:348] isatty.IsTerminal(1) = true
I0624 10:37:32.526935   39866 out.go:309] Setting ErrFile to fd 2...
I0624 10:37:32.526940   39866 out.go:348] isatty.IsTerminal(2) = true
I0624 10:37:32.527140   39866 root.go:329] Updating PATH: /home/eike/.minikube/bin
I0624 10:37:32.527286   39866 out.go:303] Setting JSON to false
I0624 10:37:32.549626   39866 start.go:115] hostinfo: {"hostname":"brandur","uptime":5516,"bootTime":1656054336,"procs":429,"os":"linux","platform":"ubuntu","platformFamily":"debian","platformVersion":"21.10","kernelVersion":"5.13.0-52-generic","kernelArch":"x86_64","virtualizationSystem":"kvm","virtualizationRole":"host","hostId":"18a889f2-32b8-4e3f-a3c7-2c08ad7f66d8"}
I0624 10:37:32.549729   39866 start.go:125] virtualization: kvm host
I0624 10:37:32.560766   39866 out.go:177] 😄  minikube v1.26.0 on Ubuntu 21.10
😄  minikube v1.26.0 on Ubuntu 21.10
I0624 10:37:32.563621   39866 notify.go:193] Checking for updates...
I0624 10:37:32.563863   39866 config.go:178] Loaded profile config "minikube": Driver=docker, ContainerRuntime=docker, KubernetesVersion=v1.24.1
I0624 10:37:32.563905   39866 driver.go:360] Setting default libvirt URI to qemu:///system
I0624 10:37:32.586166   39866 docker.go:137] docker version: linux-20.10.17
I0624 10:37:32.586239   39866 cli_runner.go:164] Run: docker system info --format "{{json .}}"
I0624 10:37:32.647442   39866 info.go:265] docker info: {ID:CWRE:ADLY:Q7N2:FN7R:ITKC:F3IW:WJON:GFLS:MEHJ:ECHV:5DCV:NNGG Containers:4 ContainersRunning:1 ContainersPaused:0 ContainersStopped:3 Images:4 Driver:overlay2 DriverStatus:[[Backing Filesystem extfs] [Supports d_type true] [Native Overlay Diff true] [userxattr false]] SystemStatus:<nil> Plugins:{Volume:[local] Network:[bridge host ipvlan macvlan null overlay] Authorization:<nil> Log:[awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog]} MemoryLimit:true SwapLimit:true KernelMemory:false KernelMemoryTCP:false CPUCfsPeriod:true CPUCfsQuota:true CPUShares:true CPUSet:true PidsLimit:true IPv4Forwarding:true BridgeNfIptables:true BridgeNfIP6Tables:true Debug:false NFd:32 OomKillDisable:false NGoroutines:39 SystemTime:2022-06-24 10:37:32.602284542 +0200 CEST LoggingDriver:json-file CgroupDriver:systemd NEventsListener:0 KernelVersion:5.13.0-52-generic OperatingSystem:Ubuntu 21.10 OSType:linux Architecture:x86_64 IndexServerAddress:https://index.docker.io/v1/ RegistryConfig:{AllowNondistributableArtifactsCIDRs:[] AllowNondistributableArtifactsHostnames:[] InsecureRegistryCIDRs:[127.0.0.0/8] IndexConfigs:{DockerIo:{Name:docker.io Mirrors:[] Secure:true Official:true}} Mirrors:[]} NCPU:8 MemTotal:33371557888 GenericResources:<nil> DockerRootDir:/var/lib/docker HTTPProxy: HTTPSProxy: NoProxy: Name:brandur Labels:[] ExperimentalBuild:true ServerVersion:20.10.17 ClusterStore: ClusterAdvertise: Runtimes:{Runc:{Path:runc}} DefaultRuntime:runc Swarm:{NodeID: NodeAddr: LocalNodeState:inactive ControlAvailable:false Error: RemoteManagers:<nil>} LiveRestoreEnabled:false Isolation: InitBinary:docker-init ContainerdCommit:{ID:10c12954828e7c7c9b6e0ea9b0c02b01407d3ae1 Expected:10c12954828e7c7c9b6e0ea9b0c02b01407d3ae1} RuncCommit:{ID:v1.1.2-0-ga916309 Expected:v1.1.2-0-ga916309} InitCommit:{ID:de40ad0 Expected:de40ad0} SecurityOptions:[name=apparmor name=seccomp,profile=default name=cgroupns] ProductLicense: Warnings:<nil> ServerErrors:[] ClientInfo:{Debug:false Plugins:[map[Experimental:true Name:app Path:/usr/libexec/docker/cli-plugins/docker-app SchemaVersion:0.1.0 ShortDescription:Docker App Vendor:Docker Inc. Version:v0.9.1-beta3] map[Name:buildx Path:/usr/libexec/docker/cli-plugins/docker-buildx SchemaVersion:0.1.0 ShortDescription:Docker Buildx Vendor:Docker Inc. Version:v0.8.2-docker] map[Name:compose Path:/home/eike/.docker/cli-plugins/docker-compose SchemaVersion:0.1.0 ShortDescription:Docker Compose Vendor:Docker Inc. Version:v2.6.1] map[Name:scan Path:/usr/libexec/docker/cli-plugins/docker-scan SchemaVersion:0.1.0 ShortDescription:Docker Scan Vendor:Docker Inc. Version:v0.17.0]] Warnings:<nil>}}
I0624 10:37:32.647516   39866 docker.go:254] overlay module found
I0624 10:37:32.656764   39866 out.go:177] ✨  Using the docker driver based on existing profile
✨  Using the docker driver based on existing profile
I0624 10:37:32.659335   39866 start.go:284] selected driver: docker
I0624 10:37:32.659343   39866 start.go:805] validating driver "docker" against &{Name:minikube KeepContext:false EmbedCerts:false MinikubeISO: KicBaseImage:gcr.io/k8s-minikube/kicbase:v0.0.29@sha256:be897edc9ed473a9678010f390a0092f488f6a1c30865f571c3b6388f9f56f9b Memory:7900 CPUs:2 DiskSize:20000 VMDriver: Driver:docker HyperkitVpnKitSock: HyperkitVSockPorts:[] DockerEnv:[] ContainerVolumeMounts:[] InsecureRegistry:[] RegistryMirror:[] HostOnlyCIDR:192.168.59.1/24 HypervVirtualSwitch: HypervUseExternalSwitch:false HypervExternalAdapter: KVMNetwork:default KVMQemuURI:qemu:///system KVMGPU:false KVMHidden:false KVMNUMACount:1 APIServerPort:0 DockerOpt:[] DisableDriverMounts:false NFSShare:[] NFSSharesRoot:/nfsshares UUID: NoVTXCheck:false DNSProxy:false HostDNSResolver:true HostOnlyNicType:virtio NatNicType:virtio SSHIPAddress: SSHUser:root SSHKey: SSHPort:22 KubernetesConfig:{KubernetesVersion:v1.24.1 ClusterName:minikube Namespace:default APIServerName:minikubeCA APIServerNames:[] APIServerIPs:[] DNSDomain:cluster.local ContainerRuntime:docker CRISocket: NetworkPlugin: FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: RegistryAliases: ExtraOptions:[{Component:kubelet Key:housekeeping-interval Value:5m}] ShouldLoadCachedImages:true EnableDefaultCNI:false CNI: NodeIP: NodePort:8443 NodeName:} Nodes:[{Name: IP:192.168.49.2 Port:8443 KubernetesVersion:v1.24.1 ContainerRuntime:docker ControlPlane:true Worker:true}] Addons:map[ambassador:false auto-pause:false csi-hostpath-driver:false dashboard:false default-storageclass:true efk:false freshpod:false gcp-auth:false gvisor:false helm-tiller:false ingress:false ingress-dns:false istio:false istio-provisioner:false kong:false kubevirt:false logviewer:false metallb:false metrics-server:false nvidia-driver-installer:false nvidia-gpu-device-plugin:false olm:false pod-security-policy:false portainer:false registry:true registry-aliases:false registry-creds:false storage-provisioner:true storage-provisioner-gluster:false volumesnapshots:false] CustomAddonImages:map[KubeRegistryProxy:google_containers/kube-registry-proxy:0.4@sha256:1040f25a5273de0d72c54865a8efd47e3292de9fb8e5353e3fa76736b854f2da Registry:registry:2.7.1@sha256:d5459fcb27aecc752520df4b492b08358a1912fcdfa454f7d2101d4b09991daa] CustomAddonRegistries:map[] VerifyComponents:map[apiserver:true system_pods:true] StartHostTimeout:6m0s ScheduledStop:<nil> ExposedPorts:[] ListenAddress: Network: Subnet: MultiNodeRequested:false ExtraDisks:0 CertExpiration:26280h0m0s Mount:false MountString:/home/eike:/minikube-host Mount9PVersion:9p2000.L MountGID:docker MountIP: MountMSize:262144 MountOptions:[] MountPort:0 MountType:9p MountUID:docker BinaryMirror: DisableOptimizations:false DisableMetrics:false CustomQemuFirmwarePath:}
I0624 10:37:32.659430   39866 start.go:816] status for docker: {Installed:true Healthy:true Running:false NeedsImprovement:false Error:<nil> Reason: Fix: Doc: Version:}
I0624 10:37:32.659481   39866 cli_runner.go:164] Run: docker system info --format "{{json .}}"
I0624 10:37:32.719093   39866 info.go:265] docker info: {ID:CWRE:ADLY:Q7N2:FN7R:ITKC:F3IW:WJON:GFLS:MEHJ:ECHV:5DCV:NNGG Containers:4 ContainersRunning:1 ContainersPaused:0 ContainersStopped:3 Images:4 Driver:overlay2 DriverStatus:[[Backing Filesystem extfs] [Supports d_type true] [Native Overlay Diff true] [userxattr false]] SystemStatus:<nil> Plugins:{Volume:[local] Network:[bridge host ipvlan macvlan null overlay] Authorization:<nil> Log:[awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog]} MemoryLimit:true SwapLimit:true KernelMemory:false KernelMemoryTCP:false CPUCfsPeriod:true CPUCfsQuota:true CPUShares:true CPUSet:true PidsLimit:true IPv4Forwarding:true BridgeNfIptables:true BridgeNfIP6Tables:true Debug:false NFd:32 OomKillDisable:false NGoroutines:39 SystemTime:2022-06-24 10:37:32.676308714 +0200 CEST LoggingDriver:json-file CgroupDriver:systemd NEventsListener:0 KernelVersion:5.13.0-52-generic OperatingSystem:Ubuntu 21.10 OSType:linux Architecture:x86_64 IndexServerAddress:https://index.docker.io/v1/ RegistryConfig:{AllowNondistributableArtifactsCIDRs:[] AllowNondistributableArtifactsHostnames:[] InsecureRegistryCIDRs:[127.0.0.0/8] IndexConfigs:{DockerIo:{Name:docker.io Mirrors:[] Secure:true Official:true}} Mirrors:[]} NCPU:8 MemTotal:33371557888 GenericResources:<nil> DockerRootDir:/var/lib/docker HTTPProxy: HTTPSProxy: NoProxy: Name:brandur Labels:[] ExperimentalBuild:true ServerVersion:20.10.17 ClusterStore: ClusterAdvertise: Runtimes:{Runc:{Path:runc}} DefaultRuntime:runc Swarm:{NodeID: NodeAddr: LocalNodeState:inactive ControlAvailable:false Error: RemoteManagers:<nil>} LiveRestoreEnabled:false Isolation: InitBinary:docker-init ContainerdCommit:{ID:10c12954828e7c7c9b6e0ea9b0c02b01407d3ae1 Expected:10c12954828e7c7c9b6e0ea9b0c02b01407d3ae1} RuncCommit:{ID:v1.1.2-0-ga916309 Expected:v1.1.2-0-ga916309} InitCommit:{ID:de40ad0 Expected:de40ad0} SecurityOptions:[name=apparmor name=seccomp,profile=default name=cgroupns] ProductLicense: Warnings:<nil> ServerErrors:[] ClientInfo:{Debug:false Plugins:[map[Experimental:true Name:app Path:/usr/libexec/docker/cli-plugins/docker-app SchemaVersion:0.1.0 ShortDescription:Docker App Vendor:Docker Inc. Version:v0.9.1-beta3] map[Name:buildx Path:/usr/libexec/docker/cli-plugins/docker-buildx SchemaVersion:0.1.0 ShortDescription:Docker Buildx Vendor:Docker Inc. Version:v0.8.2-docker] map[Name:compose Path:/home/eike/.docker/cli-plugins/docker-compose SchemaVersion:0.1.0 ShortDescription:Docker Compose Vendor:Docker Inc. Version:v2.6.1] map[Name:scan Path:/usr/libexec/docker/cli-plugins/docker-scan SchemaVersion:0.1.0 ShortDescription:Docker Scan Vendor:Docker Inc. Version:v0.17.0]] Warnings:<nil>}}
I0624 10:37:32.736571   39866 cni.go:95] Creating CNI manager for ""
I0624 10:37:32.736587   39866 cni.go:169] CNI unnecessary in this configuration, recommending no CNI
I0624 10:37:32.736595   39866 start_flags.go:310] config:
{Name:minikube KeepContext:false EmbedCerts:false MinikubeISO: KicBaseImage:gcr.io/k8s-minikube/kicbase:v0.0.29@sha256:be897edc9ed473a9678010f390a0092f488f6a1c30865f571c3b6388f9f56f9b Memory:7900 CPUs:2 DiskSize:20000 VMDriver: Driver:docker HyperkitVpnKitSock: HyperkitVSockPorts:[] DockerEnv:[] ContainerVolumeMounts:[] InsecureRegistry:[] RegistryMirror:[] HostOnlyCIDR:192.168.59.1/24 HypervVirtualSwitch: HypervUseExternalSwitch:false HypervExternalAdapter: KVMNetwork:default KVMQemuURI:qemu:///system KVMGPU:false KVMHidden:false KVMNUMACount:1 APIServerPort:0 DockerOpt:[] DisableDriverMounts:false NFSShare:[] NFSSharesRoot:/nfsshares UUID: NoVTXCheck:false DNSProxy:false HostDNSResolver:true HostOnlyNicType:virtio NatNicType:virtio SSHIPAddress: SSHUser:root SSHKey: SSHPort:22 KubernetesConfig:{KubernetesVersion:v1.24.1 ClusterName:minikube Namespace:default APIServerName:minikubeCA APIServerNames:[] APIServerIPs:[] DNSDomain:cluster.local ContainerRuntime:docker CRISocket: NetworkPlugin: FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: RegistryAliases: ExtraOptions:[{Component:kubelet Key:housekeeping-interval Value:5m}] ShouldLoadCachedImages:true EnableDefaultCNI:false CNI: NodeIP: NodePort:8443 NodeName:} Nodes:[{Name: IP:192.168.49.2 Port:8443 KubernetesVersion:v1.24.1 ContainerRuntime:docker ControlPlane:true Worker:true}] Addons:map[ambassador:false auto-pause:false csi-hostpath-driver:false dashboard:false default-storageclass:true efk:false freshpod:false gcp-auth:false gvisor:false helm-tiller:false ingress:false ingress-dns:false istio:false istio-provisioner:false kong:false kubevirt:false logviewer:false metallb:false metrics-server:false nvidia-driver-installer:false nvidia-gpu-device-plugin:false olm:false pod-security-policy:false portainer:false registry:true registry-aliases:false registry-creds:false storage-provisioner:true storage-provisioner-gluster:false volumesnapshots:false] CustomAddonImages:map[KubeRegistryProxy:google_containers/kube-registry-proxy:0.4@sha256:1040f25a5273de0d72c54865a8efd47e3292de9fb8e5353e3fa76736b854f2da Registry:registry:2.7.1@sha256:d5459fcb27aecc752520df4b492b08358a1912fcdfa454f7d2101d4b09991daa] CustomAddonRegistries:map[] VerifyComponents:map[apiserver:true system_pods:true] StartHostTimeout:6m0s ScheduledStop:<nil> ExposedPorts:[] ListenAddress: Network: Subnet: MultiNodeRequested:false ExtraDisks:0 CertExpiration:26280h0m0s Mount:false MountString:/home/eike:/minikube-host Mount9PVersion:9p2000.L MountGID:docker MountIP: MountMSize:262144 MountOptions:[] MountPort:0 MountType:9p MountUID:docker BinaryMirror: DisableOptimizations:false DisableMetrics:false CustomQemuFirmwarePath:}
I0624 10:37:32.743262   39866 out.go:177] 👍  Starting control plane node minikube in cluster minikube
👍  Starting control plane node minikube in cluster minikube
I0624 10:37:32.746130   39866 cache.go:120] Beginning downloading kic base image for docker with docker
I0624 10:37:32.748925   39866 out.go:177] 🚜  Pulling base image ...
🚜  Pulling base image ...
I0624 10:37:32.754397   39866 preload.go:132] Checking if preload exists for k8s version v1.24.1 and runtime docker
I0624 10:37:32.754444   39866 preload.go:148] Found local preload: /home/eike/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v18-v1.24.1-docker-overlay2-amd64.tar.lz4
I0624 10:37:32.754454   39866 cache.go:57] Caching tarball of preloaded images
I0624 10:37:32.754447   39866 image.go:75] Checking for gcr.io/k8s-minikube/kicbase:v0.0.29@sha256:be897edc9ed473a9678010f390a0092f488f6a1c30865f571c3b6388f9f56f9b in local docker daemon
I0624 10:37:32.754604   39866 preload.go:174] Found /home/eike/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v18-v1.24.1-docker-overlay2-amd64.tar.lz4 in cache, skipping download
I0624 10:37:32.754613   39866 cache.go:60] Finished verifying existence of preloaded tar for  v1.24.1 on docker
I0624 10:37:32.754709   39866 profile.go:148] Saving config to /home/eike/.minikube/profiles/minikube/config.json ...
I0624 10:37:32.773413   39866 image.go:79] Found gcr.io/k8s-minikube/kicbase:v0.0.29@sha256:be897edc9ed473a9678010f390a0092f488f6a1c30865f571c3b6388f9f56f9b in local docker daemon, skipping pull
I0624 10:37:32.773431   39866 cache.go:142] gcr.io/k8s-minikube/kicbase:v0.0.29@sha256:be897edc9ed473a9678010f390a0092f488f6a1c30865f571c3b6388f9f56f9b exists in daemon, skipping load
I0624 10:37:32.773441   39866 cache.go:208] Successfully downloaded all kic artifacts
I0624 10:37:32.773472   39866 start.go:352] acquiring machines lock for minikube: {Name:mkb0b325578739f33981247292911294e08eae5b Clock:{} Delay:500ms Timeout:10m0s Cancel:<nil>}
I0624 10:37:32.773523   39866 start.go:356] acquired machines lock for "minikube" in 37.177µs
I0624 10:37:32.773534   39866 start.go:94] Skipping create...Using existing machine configuration
I0624 10:37:32.773539   39866 fix.go:55] fixHost starting: 
I0624 10:37:32.773674   39866 cli_runner.go:164] Run: docker container inspect minikube --format={{.State.Status}}
I0624 10:37:32.792178   39866 fix.go:103] recreateIfNeeded on minikube: state=Running err=<nil>
W0624 10:37:32.792203   39866 fix.go:129] unexpected machine state, will restart: <nil>
I0624 10:37:32.795683   39866 out.go:177] 🏃  Updating the running docker "minikube" container ...
🏃  Updating the running docker "minikube" container ...
I0624 10:37:32.798493   39866 machine.go:88] provisioning docker machine ...
I0624 10:37:32.798511   39866 ubuntu.go:169] provisioning hostname "minikube"
I0624 10:37:32.798557   39866 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
I0624 10:37:32.817986   39866 main.go:134] libmachine: Using SSH client type: native
I0624 10:37:32.818097   39866 main.go:134] libmachine: &{{{<nil> 0 [] [] []} docker [0x7dae00] 0x7dde60 <nil>  [] 0s} 127.0.0.1 49172 <nil> <nil>}
I0624 10:37:32.818106   39866 main.go:134] libmachine: About to run SSH command:
sudo hostname minikube && echo "minikube" | sudo tee /etc/hostname
I0624 10:37:32.947280   39866 main.go:134] libmachine: SSH cmd err, output: <nil>: minikube

I0624 10:37:32.947362   39866 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
I0624 10:37:32.966323   39866 main.go:134] libmachine: Using SSH client type: native
I0624 10:37:32.966422   39866 main.go:134] libmachine: &{{{<nil> 0 [] [] []} docker [0x7dae00] 0x7dde60 <nil>  [] 0s} 127.0.0.1 49172 <nil> <nil>}
I0624 10:37:32.966434   39866 main.go:134] libmachine: About to run SSH command:

        if ! grep -xq '.*\sminikube' /etc/hosts; then
            if grep -xq '127.0.1.1\s.*' /etc/hosts; then
                sudo sed -i 's/^127.0.1.1\s.*/127.0.1.1 minikube/g' /etc/hosts;
            else 
                echo '127.0.1.1 minikube' | sudo tee -a /etc/hosts; 
            fi
        fi
I0624 10:37:33.090638   39866 main.go:134] libmachine: SSH cmd err, output: <nil>: 
I0624 10:37:33.090658   39866 ubuntu.go:175] set auth options {CertDir:/home/eike/.minikube CaCertPath:/home/eike/.minikube/certs/ca.pem CaPrivateKeyPath:/home/eike/.minikube/certs/ca-key.pem CaCertRemotePath:/etc/docker/ca.pem ServerCertPath:/home/eike/.minikube/machines/server.pem ServerKeyPath:/home/eike/.minikube/machines/server-key.pem ClientKeyPath:/home/eike/.minikube/certs/key.pem ServerCertRemotePath:/etc/docker/server.pem ServerKeyRemotePath:/etc/docker/server-key.pem ClientCertPath:/home/eike/.minikube/certs/cert.pem ServerCertSANs:[] StorePath:/home/eike/.minikube}
I0624 10:37:33.090672   39866 ubuntu.go:177] setting up certificates
I0624 10:37:33.090681   39866 provision.go:83] configureAuth start
I0624 10:37:33.090720   39866 cli_runner.go:164] Run: docker container inspect -f "{{range .NetworkSettings.Networks}}{{.IPAddress}},{{.GlobalIPv6Address}}{{end}}" minikube
I0624 10:37:33.108804   39866 provision.go:138] copyHostCerts
I0624 10:37:33.108849   39866 exec_runner.go:144] found /home/eike/.minikube/key.pem, removing ...
I0624 10:37:33.108855   39866 exec_runner.go:207] rm: /home/eike/.minikube/key.pem
I0624 10:37:33.108899   39866 exec_runner.go:151] cp: /home/eike/.minikube/certs/key.pem --> /home/eike/.minikube/key.pem (1679 bytes)
I0624 10:37:33.108946   39866 exec_runner.go:144] found /home/eike/.minikube/ca.pem, removing ...
I0624 10:37:33.108952   39866 exec_runner.go:207] rm: /home/eike/.minikube/ca.pem
I0624 10:37:33.108970   39866 exec_runner.go:151] cp: /home/eike/.minikube/certs/ca.pem --> /home/eike/.minikube/ca.pem (1070 bytes)
I0624 10:37:33.108996   39866 exec_runner.go:144] found /home/eike/.minikube/cert.pem, removing ...
I0624 10:37:33.109001   39866 exec_runner.go:207] rm: /home/eike/.minikube/cert.pem
I0624 10:37:33.109015   39866 exec_runner.go:151] cp: /home/eike/.minikube/certs/cert.pem --> /home/eike/.minikube/cert.pem (1115 bytes)
I0624 10:37:33.109040   39866 provision.go:112] generating server cert: /home/eike/.minikube/machines/server.pem ca-key=/home/eike/.minikube/certs/ca.pem private-key=/home/eike/.minikube/certs/ca-key.pem org=eike.minikube san=[192.168.49.2 127.0.0.1 localhost 127.0.0.1 minikube minikube]
I0624 10:37:33.140929   39866 provision.go:172] copyRemoteCerts
I0624 10:37:33.140972   39866 ssh_runner.go:195] Run: sudo mkdir -p /etc/docker /etc/docker /etc/docker
I0624 10:37:33.141000   39866 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
I0624 10:37:33.160385   39866 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:49172 SSHKeyPath:/home/eike/.minikube/machines/minikube/id_rsa Username:docker}
I0624 10:37:33.244390   39866 ssh_runner.go:362] scp /home/eike/.minikube/certs/ca.pem --> /etc/docker/ca.pem (1070 bytes)
I0624 10:37:33.253926   39866 ssh_runner.go:362] scp /home/eike/.minikube/machines/server.pem --> /etc/docker/server.pem (1196 bytes)
I0624 10:37:33.263904   39866 ssh_runner.go:362] scp /home/eike/.minikube/machines/server-key.pem --> /etc/docker/server-key.pem (1675 bytes)
I0624 10:37:33.273367   39866 provision.go:86] duration metric: configureAuth took 182.676441ms
I0624 10:37:33.273388   39866 ubuntu.go:193] setting minikube options for container-runtime
I0624 10:37:33.273509   39866 config.go:178] Loaded profile config "minikube": Driver=docker, ContainerRuntime=docker, KubernetesVersion=v1.24.1
I0624 10:37:33.273547   39866 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
I0624 10:37:33.292437   39866 main.go:134] libmachine: Using SSH client type: native
I0624 10:37:33.292529   39866 main.go:134] libmachine: &{{{<nil> 0 [] [] []} docker [0x7dae00] 0x7dde60 <nil>  [] 0s} 127.0.0.1 49172 <nil> <nil>}
I0624 10:37:33.292537   39866 main.go:134] libmachine: About to run SSH command:
df --output=fstype / | tail -n 1
I0624 10:37:33.418506   39866 main.go:134] libmachine: SSH cmd err, output: <nil>: overlay

I0624 10:37:33.418526   39866 ubuntu.go:71] root file system type: overlay
I0624 10:37:33.418636   39866 provision.go:309] Updating docker unit: /lib/systemd/system/docker.service ...
I0624 10:37:33.418683   39866 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
I0624 10:37:33.436946   39866 main.go:134] libmachine: Using SSH client type: native
I0624 10:37:33.437039   39866 main.go:134] libmachine: &{{{<nil> 0 [] [] []} docker [0x7dae00] 0x7dde60 <nil>  [] 0s} 127.0.0.1 49172 <nil> <nil>}
I0624 10:37:33.437080   39866 main.go:134] libmachine: About to run SSH command:
sudo mkdir -p /lib/systemd/system && printf %s "[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
BindsTo=containerd.service
After=network-online.target firewalld.service containerd.service
Wants=network-online.target
Requires=docker.socket
StartLimitBurst=3
StartLimitIntervalSec=60

[Service]
Type=notify
Restart=on-failure

# This file is a systemd drop-in unit that inherits from the base dockerd configuration.
# The base configuration already specifies an 'ExecStart=...' command. The first directive
# here is to clear out that command inherited from the base configuration. Without this,
# the command from the base configuration and the command specified here are treated as
# a sequence of commands, which is not the desired behavior, nor is it valid -- systemd
# will catch this invalid input and refuse to start the service with an error like:
#  Service has more than one ExecStart= setting, which is only allowed for Type=oneshot services.

# NOTE: default-ulimit=nofile is set to an arbitrary number for consistency with other
# container runtimes. If left unlimited, it may result in OOM issues with MySQL.
ExecStart=
ExecStart=/usr/bin/dockerd -H tcp://0.0.0.0:2376 -H unix:///var/run/docker.sock --default-ulimit=nofile=1048576:1048576 --tlsverify --tlscacert /etc/docker/ca.pem --tlscert /etc/docker/server.pem --tlskey /etc/docker/server-key.pem --label provider=docker --insecure-registry 10.96.0.0/12 
ExecReload=/bin/kill -s HUP \$MAINPID

# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity

# Uncomment TasksMax if your systemd version supports it.
# Only systemd 226 and above support this version.
TasksMax=infinity
TimeoutStartSec=0

# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes

# kill only the docker process, not all processes in the cgroup
KillMode=process

[Install]
WantedBy=multi-user.target
" | sudo tee /lib/systemd/system/docker.service.new
I0624 10:37:33.563292   39866 main.go:134] libmachine: SSH cmd err, output: <nil>: [Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
BindsTo=containerd.service
After=network-online.target firewalld.service containerd.service
Wants=network-online.target
Requires=docker.socket
StartLimitBurst=3
StartLimitIntervalSec=60

[Service]
Type=notify
Restart=on-failure

# This file is a systemd drop-in unit that inherits from the base dockerd configuration.
# The base configuration already specifies an 'ExecStart=...' command. The first directive
# here is to clear out that command inherited from the base configuration. Without this,
# the command from the base configuration and the command specified here are treated as
# a sequence of commands, which is not the desired behavior, nor is it valid -- systemd
# will catch this invalid input and refuse to start the service with an error like:
#  Service has more than one ExecStart= setting, which is only allowed for Type=oneshot services.

# NOTE: default-ulimit=nofile is set to an arbitrary number for consistency with other
# container runtimes. If left unlimited, it may result in OOM issues with MySQL.
ExecStart=
ExecStart=/usr/bin/dockerd -H tcp://0.0.0.0:2376 -H unix:///var/run/docker.sock --default-ulimit=nofile=1048576:1048576 --tlsverify --tlscacert /etc/docker/ca.pem --tlscert /etc/docker/server.pem --tlskey /etc/docker/server-key.pem --label provider=docker --insecure-registry 10.96.0.0/12 
ExecReload=/bin/kill -s HUP $MAINPID

# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity

# Uncomment TasksMax if your systemd version supports it.
# Only systemd 226 and above support this version.
TasksMax=infinity
TimeoutStartSec=0

# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes

# kill only the docker process, not all processes in the cgroup
KillMode=process

[Install]
WantedBy=multi-user.target

I0624 10:37:33.563355   39866 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
I0624 10:37:33.582403   39866 main.go:134] libmachine: Using SSH client type: native
I0624 10:37:33.582546   39866 main.go:134] libmachine: &{{{<nil> 0 [] [] []} docker [0x7dae00] 0x7dde60 <nil>  [] 0s} 127.0.0.1 49172 <nil> <nil>}
I0624 10:37:33.582559   39866 main.go:134] libmachine: About to run SSH command:
sudo diff -u /lib/systemd/system/docker.service /lib/systemd/system/docker.service.new || { sudo mv /lib/systemd/system/docker.service.new /lib/systemd/system/docker.service; sudo systemctl -f daemon-reload && sudo systemctl -f enable docker && sudo systemctl -f restart docker; }
I0624 10:37:33.704320   39866 main.go:134] libmachine: SSH cmd err, output: <nil>: 
I0624 10:37:33.704345   39866 machine.go:91] provisioned docker machine in 905.844134ms
I0624 10:37:33.704358   39866 start.go:306] post-start starting for "minikube" (driver="docker")
I0624 10:37:33.704366   39866 start.go:316] creating required directories: [/etc/kubernetes/addons /etc/kubernetes/manifests /var/tmp/minikube /var/lib/minikube /var/lib/minikube/certs /var/lib/minikube/images /var/lib/minikube/binaries /tmp/gvisor /usr/share/ca-certificates /etc/ssl/certs]
I0624 10:37:33.704418   39866 ssh_runner.go:195] Run: sudo mkdir -p /etc/kubernetes/addons /etc/kubernetes/manifests /var/tmp/minikube /var/lib/minikube /var/lib/minikube/certs /var/lib/minikube/images /var/lib/minikube/binaries /tmp/gvisor /usr/share/ca-certificates /etc/ssl/certs
I0624 10:37:33.704454   39866 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
I0624 10:37:33.722996   39866 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:49172 SSHKeyPath:/home/eike/.minikube/machines/minikube/id_rsa Username:docker}
I0624 10:37:33.812766   39866 ssh_runner.go:195] Run: cat /etc/os-release
I0624 10:37:33.814242   39866 main.go:134] libmachine: Couldn't set key PRIVACY_POLICY_URL, no corresponding struct field found
I0624 10:37:33.814256   39866 main.go:134] libmachine: Couldn't set key VERSION_CODENAME, no corresponding struct field found
I0624 10:37:33.814264   39866 main.go:134] libmachine: Couldn't set key UBUNTU_CODENAME, no corresponding struct field found
I0624 10:37:33.814271   39866 info.go:137] Remote host: Ubuntu 20.04.2 LTS
I0624 10:37:33.814280   39866 filesync.go:126] Scanning /home/eike/.minikube/addons for local assets ...
I0624 10:37:33.814310   39866 filesync.go:126] Scanning /home/eike/.minikube/files for local assets ...
I0624 10:37:33.814322   39866 start.go:309] post-start completed in 109.957398ms
I0624 10:37:33.814355   39866 ssh_runner.go:195] Run: sh -c "df -h /var | awk 'NR==2{print $5}'"
I0624 10:37:33.814379   39866 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
I0624 10:37:33.833925   39866 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:49172 SSHKeyPath:/home/eike/.minikube/machines/minikube/id_rsa Username:docker}
I0624 10:37:33.914618   39866 ssh_runner.go:195] Run: sh -c "df -BG /var | awk 'NR==2{print $4}'"
I0624 10:37:33.916609   39866 fix.go:57] fixHost completed within 1.143067133s
I0624 10:37:33.916621   39866 start.go:81] releasing machines lock for "minikube", held for 1.143092527s
I0624 10:37:33.916672   39866 cli_runner.go:164] Run: docker container inspect -f "{{range .NetworkSettings.Networks}}{{.IPAddress}},{{.GlobalIPv6Address}}{{end}}" minikube
I0624 10:37:33.935459   39866 ssh_runner.go:195] Run: systemctl --version
I0624 10:37:33.935494   39866 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
I0624 10:37:33.935496   39866 ssh_runner.go:195] Run: curl -sS -m 2 https://k8s.gcr.io/
I0624 10:37:33.935539   39866 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
I0624 10:37:33.958217   39866 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:49172 SSHKeyPath:/home/eike/.minikube/machines/minikube/id_rsa Username:docker}
I0624 10:37:33.958315   39866 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:49172 SSHKeyPath:/home/eike/.minikube/machines/minikube/id_rsa Username:docker}
I0624 10:37:34.095419   39866 ssh_runner.go:195] Run: sudo systemctl cat docker.service
I0624 10:37:34.100899   39866 cruntime.go:273] skipping containerd shutdown because we are bound to it
I0624 10:37:34.100943   39866 ssh_runner.go:195] Run: sudo systemctl is-active --quiet service crio
I0624 10:37:34.106177   39866 ssh_runner.go:195] Run: /bin/bash -c "sudo mkdir -p /etc && printf %s "runtime-endpoint: unix:///var/run/cri-dockerd.sock
image-endpoint: unix:///var/run/cri-dockerd.sock
" | sudo tee /etc/crictl.yaml"
I0624 10:37:34.114516   39866 ssh_runner.go:195] Run: sudo systemctl unmask docker.service
I0624 10:37:34.196531   39866 ssh_runner.go:195] Run: sudo systemctl enable docker.socket
I0624 10:37:34.243835   39866 ssh_runner.go:195] Run: sudo systemctl daemon-reload
I0624 10:37:34.312412   39866 ssh_runner.go:195] Run: sudo systemctl restart docker
I0624 10:37:34.582389   39866 ssh_runner.go:195] Run: sudo systemctl enable cri-docker.socket
I0624 10:37:34.656270   39866 ssh_runner.go:195] Run: sudo systemctl daemon-reload
I0624 10:37:34.711986   39866 ssh_runner.go:195] Run: sudo systemctl start cri-docker.socket
I0624 10:37:34.718007   39866 start.go:447] Will wait 60s for socket path /var/run/cri-dockerd.sock
I0624 10:37:34.718053   39866 ssh_runner.go:195] Run: stat /var/run/cri-dockerd.sock
I0624 10:37:34.719635   39866 retry.go:31] will retry after 1.104660288s: stat /var/run/cri-dockerd.sock: Process exited with status 1
stdout:

stderr:
stat: cannot stat '/var/run/cri-dockerd.sock': No such file or directory
I0624 10:37:35.825380   39866 ssh_runner.go:195] Run: stat /var/run/cri-dockerd.sock
I0624 10:37:35.827370   39866 retry.go:31] will retry after 2.160763633s: stat /var/run/cri-dockerd.sock: Process exited with status 1
stdout:

stderr:
stat: cannot stat '/var/run/cri-dockerd.sock': No such file or directory
I0624 10:37:37.988272   39866 ssh_runner.go:195] Run: stat /var/run/cri-dockerd.sock
I0624 10:37:37.990049   39866 retry.go:31] will retry after 2.62026012s: stat /var/run/cri-dockerd.sock: Process exited with status 1
stdout:

stderr:
stat: cannot stat '/var/run/cri-dockerd.sock': No such file or directory
I0624 10:37:40.612631   39866 ssh_runner.go:195] Run: stat /var/run/cri-dockerd.sock
I0624 10:37:40.614516   39866 retry.go:31] will retry after 3.164785382s: stat /var/run/cri-dockerd.sock: Process exited with status 1
stdout:

stderr:
stat: cannot stat '/var/run/cri-dockerd.sock': No such file or directory
I0624 10:37:43.780186   39866 ssh_runner.go:195] Run: stat /var/run/cri-dockerd.sock
I0624 10:37:43.782267   39866 retry.go:31] will retry after 4.680977329s: stat /var/run/cri-dockerd.sock: Process exited with status 1
stdout:

stderr:
stat: cannot stat '/var/run/cri-dockerd.sock': No such file or directory
I0624 10:37:48.464189   39866 ssh_runner.go:195] Run: stat /var/run/cri-dockerd.sock
I0624 10:37:48.466137   39866 retry.go:31] will retry after 9.01243771s: stat /var/run/cri-dockerd.sock: Process exited with status 1
stdout:

stderr:
stat: cannot stat '/var/run/cri-dockerd.sock': No such file or directory
I0624 10:37:57.480155   39866 ssh_runner.go:195] Run: stat /var/run/cri-dockerd.sock
I0624 10:37:57.482515   39866 retry.go:31] will retry after 6.442959172s: stat /var/run/cri-dockerd.sock: Process exited with status 1
stdout:

stderr:
stat: cannot stat '/var/run/cri-dockerd.sock': No such file or directory
I0624 10:38:03.926007   39866 ssh_runner.go:195] Run: stat /var/run/cri-dockerd.sock
I0624 10:38:03.928002   39866 retry.go:31] will retry after 11.217246954s: stat /var/run/cri-dockerd.sock: Process exited with status 1
stdout:

stderr:
stat: cannot stat '/var/run/cri-dockerd.sock': No such file or directory
I0624 10:38:15.146381   39866 ssh_runner.go:195] Run: stat /var/run/cri-dockerd.sock
I0624 10:38:15.148536   39866 retry.go:31] will retry after 15.299675834s: stat /var/run/cri-dockerd.sock: Process exited with status 1
stdout:

stderr:
stat: cannot stat '/var/run/cri-dockerd.sock': No such file or directory
^C
~

Attach the log file

~ minikube logs --file=log.txt
E0624 10:40:01.313130   40783 logs.go:192] command /bin/bash -c "sudo /var/lib/minikube/binaries/v1.24.1/kubectl describe nodes --kubeconfig=/var/lib/minikube/kubeconfig" failed with error: /bin/bash -c "sudo /var/lib/minikube/binaries/v1.24.1/kubectl describe nodes --kubeconfig=/var/lib/minikube/kubeconfig": Process exited with status 1
stdout:

stderr:
error: stat /var/lib/minikube/kubeconfig: no such file or directory
 output: "\n** stderr ** \nerror: stat /var/lib/minikube/kubeconfig: no such file or directory\n\n** /stderr **"

❗  unable to fetch logs for: describe nodes
~

Operating System

Ubuntu

Driver

Docker

medyagh commented 2 years ago

I suspect you had an existing Minikube cluster, using old image that didnt have cri dockerd

does deleting minikube and starting a new one fix the problem?

spowelljr commented 2 years ago

Hi @EHJ-52n, following up with what @medyagh said. minikube v1.26.0 introduced cri-dockerd however, since you're using the updated binary with a cluster created with a previous version it's trying to start cri-dockerd but your cluster doesn't actually have that service installed.

Try running minikube delete --all and then try starting minikube again and your issue should be resolved, thanks!

EHJ-52n commented 2 years ago

@spowelljr Your piece of advice was working as you can see in the following:

~ minikube status
minikube
type: Control Plane
host: Stopped
kubelet: Stopped
apiserver: Stopped
kubeconfig: Stopped

~ minikube start --kubernetes-version=stable
😄  minikube v1.26.0 on Ubuntu 21.10
✨  Using the docker driver based on existing profile
👍  Starting control plane node minikube in cluster minikube
🚜  Pulling base image ...
🔄  Restarting existing docker container for "minikube" ...

❌  Exiting due to RUNTIME_ENABLE: stat /var/run/cri-dockerd.sock: Process exited with status 1
stdout:

stderr:
stat: cannot stat '/var/run/cri-dockerd.sock': No such file or directory

╭───────────────────────────────────────────────────────────────────────────────────────────╮
│                                                                                           │
│    😿  If the above advice does not help, please let us know:                             │
│    👉  https://github.com/kubernetes/minikube/issues/new/choose                           │
│                                                                                           │
│    Please run `minikube logs --file=logs.txt` and attach logs.txt to the GitHub issue.    │
│                                                                                           │
╰───────────────────────────────────────────────────────────────────────────────────────────╯

~ minikube delete --all
🔥  Deleting "minikube" in docker ...
🔥  Removing /home/eike/.minikube/machines/minikube ...
💀  Removed all traces of the "minikube" cluster.
🔥  Successfully deleted all profiles
~ minikube start --kubernetes-version=stable
😄  minikube v1.26.0 on Ubuntu 21.10
✨  Automatically selected the docker driver. Other choices: ssh, qemu2 (experimental)
📌  Using Docker driver with root privileges
👍  Starting control plane node minikube in cluster minikube
🚜  Pulling base image ...
    > gcr.io/k8s-minikube/kicbase: 385.99 MiB / 386.00 MiB  100.00% 8.41 MiB p/
    > gcr.io/k8s-minikube/kicbase: 0 B [_________________________] ?% ? p/s 40s
🔥  Creating docker container (CPUs=2, Memory=7900MB) ...
🐳  Preparing Kubernetes v1.24.1 on Docker 20.10.17 ...
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: storage-provisioner, default-storageclass
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default
~ minikube status                           
minikube
type: Control Plane
host: Running
kubelet: Running
apiserver: Running
kubeconfig: Configured

~ minikube version
minikube version: v1.26.0
commit: f4b412861bb746be73053c9f6d2895f12cf78565
spowelljr commented 2 years ago

Glad that resolved your issue @EHJ-52n!