search

kubernetes / minikube

Run Kubernetes locally
https://minikube.sigs.k8s.io/
Apache License 2.0
29.46k stars 4.89k forks source link

cri-docker cannot started after stop and start minikube again #15096

Closed marverlous811 closed 1 year ago

marverlous811 commented 2 years ago

What Happened?

● cri-docker.service - CRI Interface for Docker Application Container Engine Loaded: loaded (/etc/systemd/system/cri-docker.service; enabled; vendor preset: enabled) Drop-In: /etc/systemd/system/cri-docker.service.d └─10-cni.conf Active: failed (Result: exit-code) since Mon 2022-10-10 11:52:07 UTC; 34s ago Docs: https://docs.mirantis.com Process: 1837 ExecStart=/usr/bin/cri-dockerd --container-runtime-endpoint fd:// --network-plugin=cni --cni-bin-dir=/opt/cni/bin --cni-cache-dir=/var/lib/cn Main PID: 1837 (code=exited, status=203/EXEC)

Oct 10 11:52:07 ubuntu-18 systemd[1]: cri-docker.service: Service hold-off time over, scheduling restart. Oct 10 11:52:07 ubuntu-18 systemd[1]: cri-docker.service: Scheduled restart job, restart counter is at 3. Oct 10 11:52:07 ubuntu-18 systemd[1]: Stopped CRI Interface for Docker Application Container Engine. Oct 10 11:52:07 ubuntu-18 systemd[1]: cri-docker.service: Start request repeated too quickly. Oct 10 11:52:07 ubuntu-18 systemd[1]: cri-docker.service: Failed with result 'exit-code'. Oct 10 11:52:07 ubuntu-18 systemd[1]: Failed to start CRI Interface for Docker Application Container Engine.

Attach the log file

stderr: I1010 11:53:11.290883 2431 exec_runner.go:51] Run: sudo systemctl is-active --quiet service kubelet I1010 11:53:11.298170 2431 fix.go:103] recreateIfNeeded on minikube: state=Stopped err= W1010 11:53:11.298182 2431 fix.go:129] unexpected machine state, will restart: I1010 11:53:11.299934 2431 out.go:177] 🔄 Restarting existing none bare metal machine for "minikube" ... I1010 11:53:11.311082 2431 profile.go:148] Saving config to /root/.minikube/profiles/minikube/config.json ... I1010 11:53:11.311199 2431 start.go:300] post-start starting for "minikube" (driver="none") I1010 11:53:11.311261 2431 start.go:328] creating required directories: [/etc/kubernetes/addons /etc/kubernetes/manifests /var/tmp/minikube /var/lib/minikube /var/lib/minikube/certs /var/lib/minikube/images /var/lib/minikube/binaries /tmp/gvisor /usr/share/ca-certificates /etc/ssl/certs] I1010 11:53:11.311296 2431 exec_runner.go:51] Run: sudo mkdir -p /etc/kubernetes/addons /etc/kubernetes/manifests /var/tmp/minikube /var/lib/minikube /var/lib/minikube/certs /var/lib/minikube/images /var/lib/minikube/binaries /tmp/gvisor /usr/share/ca-certificates /etc/ssl/certs I1010 11:53:11.330038 2431 main.go:134] libmachine: Couldn't set key PRIVACY_POLICY_URL, no corresponding struct field found I1010 11:53:11.330054 2431 main.go:134] libmachine: Couldn't set key VERSION_CODENAME, no corresponding struct field found I1010 11:53:11.330063 2431 main.go:134] libmachine: Couldn't set key UBUNTU_CODENAME, no corresponding struct field found I1010 11:53:11.332206 2431 out.go:177] ℹ️ OS release is Ubuntu 18.04 LTS I1010 11:53:11.341360 2431 filesync.go:126] Scanning /root/.minikube/addons for local assets ... I1010 11:53:11.341414 2431 filesync.go:126] Scanning /root/.minikube/files for local assets ... I1010 11:53:11.341441 2431 start.go:303] post-start completed in 30.23523ms I1010 11:53:11.341448 2431 fix.go:57] fixHost completed within 87.581715ms I1010 11:53:11.341451 2431 start.go:83] releasing machines lock for "minikube", held for 87.596023ms I1010 11:53:11.341955 2431 exec_runner.go:51] Run: sudo mkdir -p /etc/systemd/system/cri-docker.service.d I1010 11:53:11.350114 2431 exec_runner.go:51] Run: curl -sS -m 2 https://registry.k8s.io/ I1010 11:53:11.350236 2431 exec_runner.go:144] found /etc/systemd/system/cri-docker.service.d/10-cni.conf, removing ... I1010 11:53:11.350247 2431 exec_runner.go:207] rm: /etc/systemd/system/cri-docker.service.d/10-cni.conf I1010 11:53:11.350704 2431 exec_runner.go:151] cp: memory --> /etc/systemd/system/cri-docker.service.d/10-cni.conf (233 bytes) I1010 11:53:11.350816 2431 exec_runner.go:51] Run: sudo cp -a /tmp/minikube3496247821 /etc/systemd/system/cri-docker.service.d/10-cni.conf I1010 11:53:11.375199 2431 exec_runner.go:51] Run: sudo systemctl daemon-reload I1010 11:53:11.609006 2431 exec_runner.go:51] Run: sudo systemctl restart cri-docker I1010 11:53:11.625502 2431 out.go:177] W1010 11:53:11.634273 2431 out.go:239] ❌ Exiting due to RUNTIME_ENABLE: sudo systemctl restart cri-docker: exit status 1 stdout:

stderr: Job for cri-docker.service failed because the control process exited with error code. See "systemctl status cri-docker.service" and "journalctl -xe" for details.

W1010 11:53:11.634300 2431 out.go:239] W1010 11:53:11.635051 2431 out.go:239] ╭───────────────────────────────────────────────────────────────────────────────────────────╮ │ │ │ 😿 If the above advice does not help, please let us know: │ │ 👉 https://github.com/kubernetes/minikube/issues/new/choose │ │ │ │ Please run minikube logs --file=logs.txt and attach logs.txt to the GitHub issue. │ │ │ ╰───────────────────────────────────────────────────────────────────────────────────────────╯ I1010 11:53:11.643298 2431 out.go:177]

Operating System

Ubuntu

Driver

No response

afbjorklund commented 2 years ago

Weirdly, it seems to be missing an "i" in cni ?

--cni-bin-dir=/opt/cni/bin --cni-cache-dir=/var/lib/cn

Looks like the override file got truncated

marverlous811 commented 2 years ago

Weirdly, it seems to be missing an "i" in cni ?

--cni-bin-dir=/opt/cni/bin --cni-cache-dir=/var/lib/cn

Looks like the override file got truncated

I don't think that is the problem, because of i already have started it without any error yet. But after I stoped minikube or restarted the server, I always got this error. For some reason, my system can't start the cri-dockered again

afbjorklund commented 2 years ago

OK, maybe you pasted half the line or something. Can you run the command directly (as root), from the ExecStart ?

It might give some better output, especially if you also add the --log-level debug options to the end of cri-dockerd

/usr/bin/cri-dockerd --log-level debug --network-plugin=cni ... (i.e. all the other options, except for the one with fd://)

k8s-triage-robot commented 1 year ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

tuanaac commented 1 year ago

I'm facing the same issue with minikube version 1.28.0. Any updates for this issue? Thank you.

tuanaac commented 1 year ago

@marverlous811 I think I found the root cause. When I start minikube with minikube start --driver=none --network-plugin=cni --cni=bridge, then minikube creates the folder /etc/systemd/system/cri-docker.service.d containing a cni-config file (if it does not exist). This file contains the path /usr/bin/cri-dockerd used by systemd to start the service.

imageimage

And I assume that you also followed the official guide to install cri-dockerd, where the executable file cri-dockerd is actually moved to /usr/local/bin (https://github.com/Mirantis/cri-dockerd#build-and-install). That is the reason why cri-dockerd.service cannot be started. image

To resolve this issue, I copied the executable file cri-dockerd to /usr/bin: sudo cp /usr/local/bin/cri-dockerd /usr/bin/ After that minikube can start normally. I hope this answer helps you.

@afbjorklund I would like to ask you a question. Should minikube takes the path to cri-dockerd from /etc/systemd/system/cri-docker.service?

image