search

kubernetes / minikube

Run Kubernetes locally
https://minikube.sigs.k8s.io/
Apache License 2.0
29.41k stars 4.88k forks source link

cri-dockerd not configured correctly causes minikube to blow up without a good solution message #15897

Open sktechnologiesadl opened 1 year ago

sktechnologiesadl commented 1 year ago

What Happened?

root@ip-172-31-5-188:~# sudo -E minikube start --driver=none 😄 minikube v1.29.0 on Ubuntu 22.04 (xen/amd64) ✨ Using the none driver based on existing profile 👍 Starting control plane node minikube in cluster minikube 🔄 Restarting existing none bare metal machine for "minikube" ... ℹ️ OS release is Ubuntu 22.04.2 LTS

❌ Exiting due to RUNTIME_ENABLE: Temporary Error: sudo /usr/bin/crictl version: exit status 1 stdout:

stderr: E0221 07:32:28.999758 10664 remote_runtime.go:145] "Version from runtime service failed" err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial unix /var/run/cri-dockerd.sock: connect: connection refused\"" time="2023-02-21T07:32:28Z" level=fatal msg="getting the runtime version: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial unix /var/run/cri-dockerd.sock: connect: connection refused\""

╭───────────────────────────────────────────────────────────────────────────────────────────╮ │ │ │ 😿 If the above advice does not help, please let us know: │ │ 👉 https://github.com/kubernetes/minikube/issues/new/choose │ │ │ │ Please run minikube logs --file=logs.txt and attach logs.txt to the GitHub issue. │ │ │ ╰───────────────────────────────────────────────────────────────────────────────────────────╯

root@ip-172-31-5-188:~#

Attach the log file

getting the above issue

Operating System

Ubuntu

Driver

Docker

sktechnologiesadl commented 1 year ago

find the logs below

sktechnologiesadl commented 1 year ago

root@ip-172-31-5-188:~# cat logs.txt *

stderr: I0221 07:43:43.731636 1735 exec_runner.go:51] Run: sudo systemctl is-active --quiet service kubelet I0221 07:43:43.744994 1735 fix.go:103] recreateIfNeeded on minikube: state=Stopped err= W0221 07:43:43.745014 1735 fix.go:129] unexpected machine state, will restart: I0221 07:43:43.747827 1735 out.go:177] 🔄 Restarting existing none bare metal machine for "minikube" ... I0221 07:43:43.751261 1735 profile.go:148] Saving config to /root/.minikube/profiles/minikube/config.json ... I0221 07:43:43.751404 1735 start.go:300] post-start starting for "minikube" (driver="none") I0221 07:43:43.751442 1735 start.go:328] creating required directories: [/etc/kubernetes/addons /etc/kubernetes/manifests /var/tmp/minikube /var/lib/minikube /var/lib/minikube/certs /var/lib/minikube/images /var/lib/minikube/binaries /tmp/gvisor /usr/share/ca-certificates /etc/ssl/certs] I0221 07:43:43.751472 1735 exec_runner.go:51] Run: sudo mkdir -p /etc/kubernetes/addons /etc/kubernetes/manifests /var/tmp/minikube /var/lib/minikube /var/lib/minikube/certs /var/lib/minikube/images /var/lib/minikube/binaries /tmp/gvisor /usr/share/ca-certificates /etc/ssl/certs I0221 07:43:43.760926 1735 main.go:141] libmachine: Couldn't set key VERSION_CODENAME, no corresponding struct field found I0221 07:43:43.760955 1735 main.go:141] libmachine: Couldn't set key PRIVACY_POLICY_URL, no corresponding struct field found I0221 07:43:43.760966 1735 main.go:141] libmachine: Couldn't set key UBUNTU_CODENAME, no corresponding struct field found I0221 07:43:43.763715 1735 out.go:177] ℹ️ OS release is Ubuntu 22.04.2 LTS I0221 07:43:43.765836 1735 filesync.go:126] Scanning /root/.minikube/addons for local assets ... I0221 07:43:43.765873 1735 filesync.go:126] Scanning /root/.minikube/files for local assets ... I0221 07:43:43.765893 1735 start.go:303] post-start completed in 14.478694ms I0221 07:43:43.765918 1735 fix.go:57] fixHost completed within 51.082966ms I0221 07:43:43.765925 1735 start.go:83] releasing machines lock for "minikube", held for 51.117855ms I0221 07:43:43.766242 1735 exec_runner.go:51] Run: cat /version.json I0221 07:43:43.766360 1735 exec_runner.go:51] Run: curl -sS -m 2 https://registry.k8s.io/ W0221 07:43:43.767219 1735 start.go:396] Unable to open version.json: cat /version.json: exit status 1 stdout:

stderr: cat: /version.json: No such file or directory I0221 07:43:43.767268 1735 exec_runner.go:51] Run: sh -c "stat /etc/cni/net.d/loopback.conf" W0221 07:43:43.768969 1735 cni.go:208] loopback cni configuration skipped: "/etc/cni/net.d/loopback.conf" not found I0221 07:43:43.769067 1735 exec_runner.go:51] Run: sudo mkdir -p /etc/systemd/system/cri-docker.service.d I0221 07:43:43.783997 1735 exec_runner.go:144] found /etc/systemd/system/cri-docker.service.d/10-cni.conf, removing ... I0221 07:43:43.784008 1735 exec_runner.go:207] rm: /etc/systemd/system/cri-docker.service.d/10-cni.conf I0221 07:43:43.784064 1735 exec_runner.go:151] cp: memory --> /etc/systemd/system/cri-docker.service.d/10-cni.conf (135 bytes) I0221 07:43:43.784164 1735 exec_runner.go:51] Run: sudo cp -a /tmp/minikube3287675091 /etc/systemd/system/cri-docker.service.d/10-cni.conf I0221 07:43:43.811830 1735 exec_runner.go:51] Run: sudo find /etc/cni/net.d -maxdepth 1 -type f ( ( -name bridge -or -name podman ) -and -not -name .mk_disabled ) -printf "%!p(MISSING), " -exec sh -c "sudo mv {} {}.mk_disabled" ; I0221 07:43:43.821176 1735 cni.go:258] no active bridge cni configs found in "/etc/cni/net.d" - nothing to disable I0221 07:43:43.821204 1735 start.go:483] detecting cgroup driver to use... I0221 07:43:43.821225 1735 detect.go:199] detected "systemd" cgroup driver on host os I0221 07:43:43.821360 1735 exec_runner.go:51] Run: /bin/bash -c "sudo mkdir -p /etc && printf %!s(MISSING) "runtime-endpoint: unix:///run/containerd/containerd.sock image-endpoint: unix:///run/containerd/containerd.sock " | sudo tee /etc/crictl.yaml" I0221 07:43:43.840621 1735 exec_runner.go:51] Run: sh -c "sudo sed -i -r 's|^( )sandbox_image = .$|\1sandbox_image = "registry.k8s.io/pause:3.9"|' /etc/containerd/config.toml" W0221 07:43:43.851653 1735 start.go:450] cannot ensure containerd is configured properly and reloaded for docker - cluster might be unstable: update sandbox_image: sh -c "sudo sed -i -r 's|^( )sandbox_image = .*$|\1sandbox_image = "registry.k8s.io/pause:3.9"|' /etc/containerd/config.toml": exit status 2 stdout:

stderr: sed: can't read /etc/containerd/config.toml: No such file or directory I0221 07:43:43.851673 1735 start.go:483] detecting cgroup driver to use... I0221 07:43:43.851694 1735 detect.go:199] detected "systemd" cgroup driver on host os I0221 07:43:43.851797 1735 exec_runner.go:51] Run: /bin/bash -c "sudo mkdir -p /etc && printf %!s(MISSING) "runtime-endpoint: unix:///var/run/cri-dockerd.sock image-endpoint: unix:///var/run/cri-dockerd.sock " | sudo tee /etc/crictl.yaml" I0221 07:43:43.872997 1735 exec_runner.go:51] Run: sudo systemctl unmask docker.service I0221 07:43:44.127696 1735 exec_runner.go:51] Run: sudo systemctl enable docker.socket I0221 07:43:44.379412 1735 docker.go:529] configuring docker to use "systemd" as cgroup driver... I0221 07:43:44.379437 1735 exec_runner.go:144] found /etc/docker/daemon.json, removing ... I0221 07:43:44.379444 1735 exec_runner.go:207] rm: /etc/docker/daemon.json I0221 07:43:44.379524 1735 exec_runner.go:151] cp: memory --> /etc/docker/daemon.json (143 bytes) I0221 07:43:44.379636 1735 exec_runner.go:51] Run: sudo cp -a /tmp/minikube1086914626 /etc/docker/daemon.json I0221 07:43:44.391790 1735 exec_runner.go:51] Run: sudo systemctl daemon-reload I0221 07:43:44.684332 1735 exec_runner.go:51] Run: sudo systemctl restart docker I0221 07:43:45.215272 1735 exec_runner.go:51] Run: sudo systemctl enable cri-docker.socket I0221 07:43:45.475338 1735 exec_runner.go:51] Run: sudo systemctl unmask cri-docker.socket I0221 07:43:45.713907 1735 exec_runner.go:51] Run: sudo systemctl enable cri-docker.socket I0221 07:43:45.951605 1735 exec_runner.go:51] Run: sudo systemctl daemon-reload I0221 07:43:46.197407 1735 exec_runner.go:51] Run: sudo systemctl restart cri-docker.socket I0221 07:43:46.230144 1735 start.go:530] Will wait 60s for socket path /var/run/cri-dockerd.sock I0221 07:43:46.230209 1735 exec_runner.go:51] Run: stat /var/run/cri-dockerd.sock I0221 07:43:46.231498 1735 start.go:551] Will wait 60s for crictl version I0221 07:43:46.231532 1735 exec_runner.go:51] Run: which crictl I0221 07:43:46.232396 1735 exec_runner.go:51] Run: sudo /usr/bin/crictl version I0221 07:43:48.502977 1735 exec_runner.go:84] Completed: sudo /usr/bin/crictl version: (2.270554923s) I0221 07:43:48.503007 1735 retry.go:31] will retry after 11.04660288s: Temporary Error: sudo /usr/bin/crictl version: exit status 1 stdout:

stderr: E0221 07:43:48.500684 2136 remote_runtime.go:145] "Version from runtime service failed" err="rpc error: code = Unavailable desc = connection closed before server preface received" time="2023-02-21T07:43:48Z" level=fatal msg="getting the runtime version: rpc error: code = Unavailable desc = connection closed before server preface received" I0221 07:43:59.549883 1735 exec_runner.go:51] Run: sudo /usr/bin/crictl version I0221 07:43:59.583151 1735 retry.go:31] will retry after 21.607636321s: Temporary Error: sudo /usr/bin/crictl version: exit status 1 stdout:

stderr: E0221 07:43:59.580723 2150 remote_runtime.go:145] "Version from runtime service failed" err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial unix /var/run/cri-dockerd.sock: connect: connection refused\"" time="2023-02-21T07:43:59Z" level=fatal msg="getting the runtime version: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial unix /var/run/cri-dockerd.sock: connect: connection refused\"" I0221 07:44:21.192504 1735 exec_runner.go:51] Run: sudo /usr/bin/crictl version I0221 07:44:21.232003 1735 out.go:177] W0221 07:44:21.233990 1735 out.go:239] ❌ Exiting due to RUNTIME_ENABLE: Temporary Error: sudo /usr/bin/crictl version: exit status 1 stdout:

stderr: E0221 07:44:21.226531 2157 remote_runtime.go:145] "Version from runtime service failed" err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial unix /var/run/cri-dockerd.sock: connect: connection refused\"" time="2023-02-21T07:44:21Z" level=fatal msg="getting the runtime version: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial unix /var/run/cri-dockerd.sock: connect: connection refused\""

W0221 07:44:21.234020 1735 out.go:239] W0221 07:44:21.235152 1735 out.go:239] ╭───────────────────────────────────────────────────────────────────────────────────────────╮ │ │ │ 😿 If the above advice does not help, please let us know: │ │ 👉 https://github.com/kubernetes/minikube/issues/new/choose │ │ │ │ Please run minikube logs --file=logs.txt and attach logs.txt to the GitHub issue. │ │ │ ╰───────────────────────────────────────────────────────────────────────────────────────────╯ I0221 07:44:21.238106 1735 out.go:177]

* root@ip-172-31-5-188:~# cat logs.txt

afbjorklund commented 1 year ago

Sounds like an issue with cri-dockerd, check the service logs in systemd ?

medyagh commented 1 year ago

seems like minikube should do more detection on None Driver to check if docker-dockerd is healthy

medyagh commented 1 year ago

we could add a health check in None driver here https://github.com/medyagh/minikube/blob/41f8a181f3e464de26a7fa4b4b487487a5202464/pkg/minikube/registry/drvs/none/none.go

similiar to other checks we do in other drivers https://github.com/medyagh/minikube/blob/41f8a181f3e464de26a7fa4b4b487487a5202464/pkg/minikube/registry/drvs/docker/docker.go#L261

medyagh commented 1 year ago

@x7upLime

x7upLime commented 1 year ago

could this specific error be caused by W0221 07:43:43.851653 1735 start.go:450] cannot ensure containerd is configured properly and reloaded for docker - cluster might be unstable: update sandbox_image: sh -c "sudo sed -i -r 's|^( )sandbox_image = .$|\1sandbox_image = "registry.k8s.io/pause:3.9"|' /etc/containerd/config.toml": exit status 2 ?

will investigate further..

afbjorklund commented 1 year ago

The config file is optional, so it is possible that it is missing (or only contains a few lines), needs to be tested for

For the currently resulting config, see containerd config dump (for the built-in values, containerd config default)

k8s-triage-robot commented 1 year ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot commented 9 months ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten