search

kubernetes / minikube

Run Kubernetes locally
https://minikube.sigs.k8s.io/
Apache License 2.0
29.41k stars 4.88k forks source link

None driver and containerd runtime #19879

Open kristi-balla opened 1 day ago

kristi-balla commented 1 day ago

What Happened?

I was setting up a minikube cluster on AWS via minikube start --cni=calico --driver=none --container-runtime=containerd and keep getting this error: validate service connection: validate CRI v1 runtime API for endpoint \"unix:///run/containerd/containerd.sock\": rpc error: code = Unimplemented desc = unknown service runtime.v1.RuntimeService

I am completely baffled as to why this happened.

With --alsologtostderr: 

I1029 16:43:42.827445   42408 out.go:345] Setting OutFile to fd 1 ...
I1029 16:43:42.827736   42408 out.go:392] TERM=xterm,COLORTERM=, which probably does not support color
I1029 16:43:42.827752   42408 out.go:358] Setting ErrFile to fd 2...
I1029 16:43:42.827765   42408 out.go:392] TERM=xterm,COLORTERM=, which probably does not support color
I1029 16:43:42.828075   42408 root.go:338] Updating PATH: /home/consec/.minikube/bin
I1029 16:43:42.828579   42408 out.go:352] Setting JSON to false
I1029 16:43:42.839253   42408 start.go:129] hostinfo: {"hostname":"ip-X-X-X-X","uptime":3991,"bootTime":1730216232,"procs":79,"os":"linux","platform":"debian","platformFamily":"debian","platformVersion":"12.7","kernelVersion":"6.1.0-26-cloud-amd64","kernelArch":"x86_64","virtualizationSystem":"","virtualizationRole":"","hostId":"ec20eca7-bd33-fc41-d33d-fab89e7088e9"}
I1029 16:43:42.839687   42408 start.go:139] virtualization:  
I1029 16:43:42.843786   42408 out.go:177] * minikube v1.34.0 on Debian 12.7
* minikube v1.34.0 on Debian 12.7
I1029 16:43:42.850106   42408 notify.go:220] Checking for updates...
I1029 16:43:42.851328   42408 config.go:182] Loaded profile config "minikube": Driver=none, ContainerRuntime=containerd, KubernetesVersion=v1.31.0
I1029 16:43:42.852003   42408 exec_runner.go:51] Run: systemctl --version
I1029 16:43:42.858655   42408 driver.go:394] Setting default libvirt URI to qemu:///system
I1029 16:43:42.876369   42408 out.go:177] * Using the none driver based on existing profile
* Using the none driver based on existing profile
I1029 16:43:42.879831   42408 start.go:297] selected driver: none
I1029 16:43:42.879876   42408 start.go:901] validating driver "none" against &{Name:minikube KeepContext:false EmbedCerts:false MinikubeISO: KicBaseImage:gcr.io/k8s-minikube/kicbase:v0.0.45@sha256:81df288595202a317b1a4dc2506ca2e4ed5f22373c19a441b88cfbf4b9867c85 Memory:2200 CPUs:2 DiskSize:20000 Driver:none HyperkitVpnKitSock: HyperkitVSockPorts:[] DockerEnv:[] ContainerVolumeMounts:[] InsecureRegistry:[] RegistryMirror:[] HostOnlyCIDR:192.168.59.1/24 HypervVirtualSwitch: HypervUseExternalSwitch:false HypervExternalAdapter: KVMNetwork:default KVMQemuURI:qemu:///system KVMGPU:false KVMHidden:false KVMNUMACount:1 APIServerPort:8443 DockerOpt:[] DisableDriverMounts:false NFSShare:[] NFSSharesRoot:/nfsshares UUID: NoVTXCheck:false DNSProxy:false HostDNSResolver:true HostOnlyNicType:virtio NatNicType:virtio SSHIPAddress: SSHUser:root SSHKey: SSHPort:22 KubernetesConfig:{KubernetesVersion:v1.31.0 ClusterName:minikube Namespace:default APIServerHAVIP: APIServerName:minikubeCA APIServerNames:[] APIServerIPs:[] DNSDomain:cluster.local ContainerRuntime:containerd CRISocket: NetworkPlugin:cni FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: RegistryAliases: ExtraOptions:[{Component:kubelet Key:resolv-conf Value:/run/systemd/resolve/resolv.conf}] ShouldLoadCachedImages:false EnableDefaultCNI:false CNI:calico} Nodes:[{Name: IP:X.X.X.X Port:8443 KubernetesVersion:v1.31.0 ContainerRuntime:containerd ControlPlane:true Worker:true}] Addons:map[] CustomAddonImages:map[] CustomAddonRegistries:map[] VerifyComponents:map[apiserver:true system_pods:true] StartHostTimeout:6m0s ScheduledStop:<nil> ExposedPorts:[] ListenAddress: Network: Subnet: MultiNodeRequested:false ExtraDisks:0 CertExpiration:26280h0m0s Mount:false MountString:/home/consec:/minikube-host Mount9PVersion:9p2000.L MountGID:docker MountIP: MountMSize:262144 MountOptions:[] MountPort:0 MountType:9p MountUID:docker BinaryMirror: DisableOptimizations:false DisableMetrics:false CustomQemuFirmwarePath: SocketVMnetClientPath: SocketVMnetPath: StaticIP: SSHAuthSock: SSHAgentPID:0 GPUs: AutoPauseInterval:1m0s}
I1029 16:43:42.880502   42408 start.go:912] status for none: {Installed:true Healthy:true Running:false NeedsImprovement:false Error:<nil> Reason: Fix: Doc: Version:}
I1029 16:43:42.880575   42408 start.go:1730] auto setting extra-config to "kubelet.resolv-conf=/run/systemd/resolve/resolv.conf".
W1029 16:43:42.881229   42408 out.go:270] ! Using the 'containerd' runtime with the 'none' driver is an untested configuration!
! Using the 'containerd' runtime with the 'none' driver is an untested configuration!
W1029 16:43:42.882480   42408 out.go:270] ! Using the 'containerd' runtime with the 'none' driver is an untested configuration!
! Using the 'containerd' runtime with the 'none' driver is an untested configuration!
I1029 16:43:42.883712   42408 cni.go:84] Creating CNI manager for "calico"
I1029 16:43:42.884109   42408 start.go:340] cluster config:
{Name:minikube KeepContext:false EmbedCerts:false MinikubeISO: KicBaseImage:gcr.io/k8s-minikube/kicbase:v0.0.45@sha256:81df288595202a317b1a4dc2506ca2e4ed5f22373c19a441b88cfbf4b9867c85 Memory:2200 CPUs:2 DiskSize:20000 Driver:none HyperkitVpnKitSock: HyperkitVSockPorts:[] DockerEnv:[] ContainerVolumeMounts:[] InsecureRegistry:[] RegistryMirror:[] HostOnlyCIDR:192.168.59.1/24 HypervVirtualSwitch: HypervUseExternalSwitch:false HypervExternalAdapter: KVMNetwork:default KVMQemuURI:qemu:///system KVMGPU:false KVMHidden:false KVMNUMACount:1 APIServerPort:8443 DockerOpt:[] DisableDriverMounts:false NFSShare:[] NFSSharesRoot:/nfsshares UUID: NoVTXCheck:false DNSProxy:false HostDNSResolver:true HostOnlyNicType:virtio NatNicType:virtio SSHIPAddress: SSHUser:root SSHKey: SSHPort:22 KubernetesConfig:{KubernetesVersion:v1.31.0 ClusterName:minikube Namespace:default APIServerHAVIP: APIServerName:minikubeCA APIServerNames:[] APIServerIPs:[] DNSDomain:cluster.local ContainerRuntime:containerd CRISocket: NetworkPlugin:cni FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: RegistryAliases: ExtraOptions:[{Component:kubelet Key:resolv-conf Value:/run/systemd/resolve/resolv.conf}] ShouldLoadCachedImages:false EnableDefaultCNI:false CNI:calico} Nodes:[{Name: IP:X.X.X.X Port:8443 KubernetesVersion:v1.31.0 ContainerRuntime:containerd ControlPlane:true Worker:true}] Addons:map[] CustomAddonImages:map[] CustomAddonRegistries:map[] VerifyComponents:map[apiserver:true system_pods:true] StartHostTimeout:6m0s ScheduledStop:<nil> ExposedPorts:[] ListenAddress: Network: Subnet: MultiNodeRequested:false ExtraDisks:0 CertExpiration:26280h0m0s Mount:false MountString:/home/consec:/minikube-host Mount9PVersion:9p2000.L MountGID:docker MountIP: MountMSize:262144 MountOptions:[] MountPort:0 MountType:9p MountUID:docker BinaryMirror: DisableOptimizations:false DisableMetrics:false CustomQemuFirmwarePath: SocketVMnetClientPath: SocketVMnetPath: StaticIP: SSHAuthSock: SSHAgentPID:0 GPUs: AutoPauseInterval:1m0s}
I1029 16:43:42.893366   42408 out.go:177] * Starting "minikube" primary control-plane node in "minikube" cluster
* Starting "minikube" primary control-plane node in "minikube" cluster
I1029 16:43:42.897021   42408 profile.go:143] Saving config to /home/consec/.minikube/profiles/minikube/config.json ...
I1029 16:43:42.897838   42408 start.go:360] acquireMachinesLock for minikube: {Name:mk96712a414e582a3aa4b0148a12ab3915cef34b Clock:{} Delay:500ms Timeout:13m0s Cancel:<nil>}
I1029 16:43:42.897958   42408 start.go:364] duration metric: took 32.928µs to acquireMachinesLock for "minikube"
I1029 16:43:42.898031   42408 start.go:96] Skipping create...Using existing machine configuration
I1029 16:43:42.898095   42408 fix.go:54] fixHost starting: 
W1029 16:43:42.899824   42408 none.go:130] unable to get port: "minikube" does not appear in /home/consec/.kube/config
I1029 16:43:42.901267   42408 api_server.go:166] Checking apiserver status ...
I1029 16:43:42.901429   42408 exec_runner.go:51] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
W1029 16:43:42.942945   42408 api_server.go:170] stopped: unable to get apiserver pid: sudo pgrep -xnf kube-apiserver.*minikube.*: exit status 1
stdout:

stderr:
I1029 16:43:42.944248   42408 exec_runner.go:51] Run: sudo systemctl is-active --quiet service kubelet
I1029 16:43:42.976801   42408 fix.go:112] recreateIfNeeded on minikube: state=Stopped err=<nil>
W1029 16:43:42.976832   42408 fix.go:138] unexpected machine state, will restart: <nil>
I1029 16:43:42.981309   42408 out.go:177] * Restarting existing none bare metal machine for "minikube" ...
* Restarting existing none bare metal machine for "minikube" ...
I1029 16:43:42.986829   42408 profile.go:143] Saving config to /home/consec/.minikube/profiles/minikube/config.json ...
I1029 16:43:42.986981   42408 start.go:293] postStartSetup for "minikube" (driver="none")
I1029 16:43:42.987008   42408 start.go:322] creating required directories: [/etc/kubernetes/addons /etc/kubernetes/manifests /var/tmp/minikube /var/lib/minikube /var/lib/minikube/certs /var/lib/minikube/images /var/lib/minikube/binaries /tmp/gvisor /usr/share/ca-certificates /etc/ssl/certs]
I1029 16:43:42.987059   42408 exec_runner.go:51] Run: sudo mkdir -p /etc/kubernetes/addons /etc/kubernetes/manifests /var/tmp/minikube /var/lib/minikube /var/lib/minikube/certs /var/lib/minikube/images /var/lib/minikube/binaries /tmp/gvisor /usr/share/ca-certificates /etc/ssl/certs
I1029 16:43:42.996676   42408 main.go:141] libmachine: Couldn't set key VERSION_CODENAME, no corresponding struct field found
I1029 16:43:43.000891   42408 out.go:177] * OS release is Debian GNU/Linux 12 (bookworm)
* OS release is Debian GNU/Linux 12 (bookworm)
I1029 16:43:43.004135   42408 filesync.go:126] Scanning /home/consec/.minikube/addons for local assets ...
I1029 16:43:43.004216   42408 filesync.go:126] Scanning /home/consec/.minikube/files for local assets ...
I1029 16:43:43.004324   42408 start.go:296] duration metric: took 17.26217ms for postStartSetup
I1029 16:43:43.004351   42408 fix.go:56] duration metric: took 106.257571ms for fixHost
I1029 16:43:43.004364   42408 start.go:83] releasing machines lock for "minikube", held for 106.338781ms
I1029 16:43:43.004949   42408 exec_runner.go:51] Run: curl -sS -m 2 https://registry.k8s.io/
I1029 16:43:43.005041   42408 exec_runner.go:51] Run: sh -c "stat /etc/cni/net.d/*loopback.conf*"
W1029 16:43:43.009392   42408 cni.go:209] loopback cni configuration skipped: "/etc/cni/net.d/*loopback.conf*" not found
I1029 16:43:43.009466   42408 exec_runner.go:51] Run: sudo find /etc/cni/net.d -maxdepth 1 -type f ( ( -name *bridge* -or -name *podman* ) -and -not -name *.mk_disabled ) -printf "%p, " -exec sh -c "sudo mv {} {}.mk_disabled" ;
I1029 16:43:43.024644   42408 cni.go:259] no active bridge cni configs found in "/etc/cni/net.d" - nothing to disable
I1029 16:43:43.024674   42408 start.go:495] detecting cgroup driver to use...
I1029 16:43:43.025054   42408 detect.go:190] detected "systemd" cgroup driver on host os
I1029 16:43:43.025194   42408 exec_runner.go:51] Run: /bin/bash -c "sudo mkdir -p /etc && printf %s "runtime-endpoint: unix:///run/containerd/containerd.sock
" | sudo tee /etc/crictl.yaml"
I1029 16:43:43.054591   42408 exec_runner.go:51] Run: sh -c "sudo sed -i -r 's|^( *)sandbox_image = .*$|\1sandbox_image = "registry.k8s.io/pause:3.10"|' /etc/containerd/config.toml"
I1029 16:43:43.075196   42408 exec_runner.go:51] Run: sh -c "sudo sed -i -r 's|^( *)restrict_oom_score_adj = .*$|\1restrict_oom_score_adj = false|' /etc/containerd/config.toml"
I1029 16:43:43.092707   42408 containerd.go:146] configuring containerd to use "systemd" as cgroup driver...
I1029 16:43:43.092950   42408 exec_runner.go:51] Run: sh -c "sudo sed -i -r 's|^( *)SystemdCgroup = .*$|\1SystemdCgroup = true|g' /etc/containerd/config.toml"
I1029 16:43:43.110638   42408 exec_runner.go:51] Run: sh -c "sudo sed -i 's|"io.containerd.runtime.v1.linux"|"io.containerd.runc.v2"|g' /etc/containerd/config.toml"
I1029 16:43:43.132128   42408 exec_runner.go:51] Run: sh -c "sudo sed -i '/systemd_cgroup/d' /etc/containerd/config.toml"
I1029 16:43:43.144637   42408 exec_runner.go:51] Run: sh -c "sudo sed -i 's|"io.containerd.runc.v1"|"io.containerd.runc.v2"|g' /etc/containerd/config.toml"
I1029 16:43:43.156805   42408 exec_runner.go:51] Run: sh -c "sudo rm -rf /etc/cni/net.mk"
I1029 16:43:43.169323   42408 exec_runner.go:51] Run: sh -c "sudo sed -i -r 's|^( *)conf_dir = .*$|\1conf_dir = "/etc/cni/net.d"|g' /etc/containerd/config.toml"
I1029 16:43:43.180650   42408 exec_runner.go:51] Run: sh -c "sudo sed -i '/^ *enable_unprivileged_ports = .*/d' /etc/containerd/config.toml"
I1029 16:43:43.191154   42408 exec_runner.go:51] Run: sh -c "sudo sed -i -r 's|^( *)\[plugins."io.containerd.grpc.v1.cri"\]|&\n\1  enable_unprivileged_ports = true|' /etc/containerd/config.toml"
I1029 16:43:43.200796   42408 exec_runner.go:51] Run: sudo sysctl net.bridge.bridge-nf-call-iptables
I1029 16:43:43.209979   42408 exec_runner.go:51] Run: sudo sh -c "echo 1 > /proc/sys/net/ipv4/ip_forward"
I1029 16:43:43.225480   42408 exec_runner.go:51] Run: sudo systemctl daemon-reload
I1029 16:43:43.820399   42408 exec_runner.go:51] Run: sudo systemctl restart containerd
I1029 16:43:43.967238   42408 start.go:542] Will wait 60s for socket path /run/containerd/containerd.sock
I1029 16:43:43.967325   42408 exec_runner.go:51] Run: stat /run/containerd/containerd.sock
I1029 16:43:43.972737   42408 start.go:563] Will wait 60s for crictl version
I1029 16:43:43.972809   42408 exec_runner.go:51] Run: which crictl
I1029 16:43:43.976912   42408 exec_runner.go:51] Run: sudo /usr/bin/crictl version
I1029 16:43:44.009272   42408 retry.go:31] will retry after 7.090115557s: Temporary Error: sudo /usr/bin/crictl version: exit status 1
stdout:

stderr:
time="2024-10-29T16:43:44Z" level=fatal msg="validate service connection: validate CRI v1 runtime API for endpoint \"unix:///run/containerd/containerd.sock\": rpc error: code = Unimplemented desc = unknown service runtime.v1.RuntimeService"
I1029 16:43:51.102827   42408 exec_runner.go:51] Run: sudo /usr/bin/crictl version
I1029 16:43:51.140467   42408 retry.go:31] will retry after 7.579262981s: Temporary Error: sudo /usr/bin/crictl version: exit status 1
stdout:

stderr:
time="2024-10-29T16:43:51Z" level=fatal msg="validate service connection: validate CRI v1 runtime API for endpoint \"unix:///run/containerd/containerd.sock\": rpc error: code = Unimplemented desc = unknown service runtime.v1.RuntimeService"
I1029 16:43:58.720148   42408 exec_runner.go:51] Run: sudo /usr/bin/crictl version
I1029 16:43:58.766886   42408 retry.go:31] will retry after 30.637014163s: Temporary Error: sudo /usr/bin/crictl version: exit status 1
stdout:

stderr:
time="2024-10-29T16:43:58Z" level=fatal msg="validate service connection: validate CRI v1 runtime API for endpoint \"unix:///run/containerd/containerd.sock\": rpc error: code = Unimplemented desc = unknown service runtime.v1.RuntimeService"
I1029 16:44:29.404822   42408 exec_runner.go:51] Run: sudo /usr/bin/crictl version
I1029 16:44:29.433470   42408 out.go:201] 

W1029 16:44:29.436975   42408 out.go:270] X Exiting due to RUNTIME_ENABLE: Failed to start container runtime: Temporary Error: sudo /usr/bin/crictl version: exit status 1
stdout:

stderr:
time="2024-10-29T16:44:29Z" level=fatal msg="validate service connection: validate CRI v1 runtime API for endpoint \"unix:///run/containerd/containerd.sock\": rpc error: code = Unimplemented desc = unknown service runtime.v1.RuntimeService"

X Exiting due to RUNTIME_ENABLE: Failed to start container runtime: Temporary Error: sudo /usr/bin/crictl version: exit status 1
stdout:

stderr:
time="2024-10-29T16:44:29Z" level=fatal msg="validate service connection: validate CRI v1 runtime API for endpoint \"unix:///run/containerd/containerd.sock\": rpc error: code = Unimplemented desc = unknown service runtime.v1.RuntimeService"

I need calico to define some network policies in my cluster. Thus far, I've tried the following:

I am running the latest versions of docker and kubernetes on a debian bookworm machine.

Attach the log file

Operating System

Other

Driver

None (Baremetal)

afbjorklund commented 1 day ago

If you install containerd from Docker, it includes configuration to disable the CRI plugin:

/etc/containerd/config.toml

#   Copyright 2018-2020 Docker Inc.

#   Licensed under the Apache License, Version 2.0 (the "License");
#   you may not use this file except in compliance with the License.
#   You may obtain a copy of the License at

#       http://www.apache.org/licenses/LICENSE-2.0

#   Unless required by applicable law or agreed to in writing, software
#   distributed under the License is distributed on an "AS IS" BASIS,
#   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
#   See the License for the specific language governing permissions and
#   limitations under the License.

disabled_plugins = ["cri"]

It does not happen with the default containerd configuration, from https://containerd.io

https://kubernetes.io/docs/setup/production-environment/container-runtimes/#containerd

Another alternative is to use Docker as the container runtime, with the cri-dockerd daemon.

kristi-balla commented 1 day ago

@afbjorklund I did configure containerd according to these and these instructions, but the error still persists.

The configuration of docker as the runtime with the cri-dockerd daemon does work for me though. I was just wondering why the other one didn't

afbjorklund commented 1 day ago

You can use crictl (and ctr) for your troubleshooting, installing Kubernetes assumes that it works OK.

containerd config default > /etc/containerd/config.toml systemctl restart containerd crictl version

Version:  0.1.0
RuntimeName:  containerd
RuntimeVersion:  1.6.20~ds1
RuntimeApiVersion:  v1