search

kubernetes / minikube

Run Kubernetes locally
https://minikube.sigs.k8s.io/
Apache License 2.0
29.45k stars 4.88k forks source link

sudo service cri-docker.socket start: Process exited with status 5 #15413

Open hualongfeng opened 1 year ago

hualongfeng commented 1 year ago

What Happened?

 minikube start --driver=ssh --ssh-ip-address=10.239.241.111 --ssh-user=ssp -v=4 --alsologtostderr --ssh-key='/home/fhl/.ssh/id_rsa'

I1128 23:37:37.108706 3909210 ssh_runner.go:195] Run: sudo service cri-docker.socket status
I1128 23:37:37.147498 3909210 ssh_runner.go:195] Run: sudo service cri-docker.socket start
I1128 23:37:37.963215 3909210 out.go:177]

W1128 23:37:37.963625 3909210 out.go:239] X Exiting due to RUNTIME_ENABLE: sudo service cri-docker.socket start: Process exited with status 5
stdout:

stderr:
Failed to start cri-docker.socket.service: Unit cri-docker.socket.service not found.

X Exiting due to RUNTIME_ENABLE: sudo service cri-docker.socket start: Process exited with status 5
stdout:

stderr:
Failed to start cri-docker.socket.service: Unit cri-docker.socket.service not found.

W1128 23:37:37.963651 3909210 out.go:239] *
*
W1128 23:37:37.964530 3909210 out.go:239] ╭─────────────────────────────────────────────────────────────────────────────────────────────╮
│                                                                                             │
│    * If the above advice does not help, please let us know:                                 │
│      https://github.com/kubernetes/minikube/issues/new/choose                               │
│                                                                                             │
│    * Please run `minikube logs --file=logs.txt` and attach logs.txt to the GitHub issue.    │
│                                                                                             │
╰─────────────────────────────────────────────────────────────────────────────────────────────╯
╭─────────────────────────────────────────────────────────────────────────────────────────────╮
│                                                                                             │
│    * If the above advice does not help, please let us know:                                 │
│      https://github.com/kubernetes/minikube/issues/new/choose                               │
│                                                                                             │
│    * Please run `minikube logs --file=logs.txt` and attach logs.txt to the GitHub issue.    │
│                                                                                             │
╰─────────────────────────────────────────────────────────────────────────────────────────────╯
I1128 23:37:37.964931 3909210 out.go:177]

I install cri-dockerd in target machine(target machine is real machine). And 

ssp@ceph-server5:~$ sudo service cri-docker status
● cri-docker.service - CRI Interface for Docker Application Container Engine
     Loaded: loaded (/etc/systemd/system/cri-docker.service; enabled; vendor preset: enabled)
     Active: active (running) since Mon 2022-11-28 11:56:08 UTC; 3h 47min ago
TriggeredBy: ● cri-docker.socket
       Docs: https://docs.mirantis.com
   Main PID: 28534 (cri-dockerd)
      Tasks: 35
     Memory: 18.9M
        CPU: 8.843s
     CGroup: /system.slice/cri-docker.service
             └─28534 /usr/local/bin/cri-dockerd --container-runtime-endpoint fd://

Nov 28 11:56:08 ceph-server5 cri-dockerd[28534]: time="2022-11-28T11:56:08Z" level=info msg="The binary conntrack is not installed, this can cause failures in network connection cleanup."
Nov 28 11:56:08 ceph-server5 cri-dockerd[28534]: time="2022-11-28T11:56:08Z" level=info msg="The binary conntrack is not installed, this can cause failures in network connection cleanup."
Nov 28 11:56:08 ceph-server5 cri-dockerd[28534]: time="2022-11-28T11:56:08Z" level=info msg="Loaded network plugin cni"
Nov 28 11:56:08 ceph-server5 cri-dockerd[28534]: time="2022-11-28T11:56:08Z" level=info msg="Docker cri networking managed by network plugin cni"
Nov 28 11:56:08 ceph-server5 cri-dockerd[28534]: time="2022-11-28T11:56:08Z" level=info msg="Docker Info: &{ID:E7DJ:WRPA:4GTU:ZTHK:6PBZ:SFWI:2GHL:4NTI:PVQ2:EKY6:SXFZ:GD2K Containers:0 ContainersRunning:0 Conta>
Nov 28 11:56:08 ceph-server5 cri-dockerd[28534]: time="2022-11-28T11:56:08Z" level=info msg="Setting cgroupDriver systemd"
Nov 28 11:56:08 ceph-server5 cri-dockerd[28534]: time="2022-11-28T11:56:08Z" level=info msg="Docker cri received runtime config &RuntimeConfig{NetworkConfig:&NetworkConfig{PodCidr:,},}"
Nov 28 11:56:08 ceph-server5 cri-dockerd[28534]: time="2022-11-28T11:56:08Z" level=info msg="Starting the GRPC backend for the Docker CRI interface."
Nov 28 11:56:08 ceph-server5 cri-dockerd[28534]: time="2022-11-28T11:56:08Z" level=info msg="Start cri-dockerd grpc backend"
Nov 28 11:56:08 ceph-server5 systemd[1]: Started CRI Interface for Docker Application Container Engine.

But 

ssp@ceph-server5:~$ sudo service cri-docker.socket status
Unit cri-docker.socket.service could not be found.

Attach the log file

I1128 23:37:33.971076 3909210 preload.go:132] Checking if preload exists for k8s version v1.25.3 and runtime docker I1128 23:37:33.971098 3909210 ssh_runner.go:195] Run: curl -x http://child-prc.intel.com:913 -sS -m 2 https://registry.k8s.io/ I1128 23:37:33.971120 3909210 ssh_runner.go:195] Run: sudo systemctl cat docker.service I1128 23:37:33.971132 3909210 sshutil.go:53] new ssh client: &{IP:10.239.241.111 Port:22 SSHKeyPath:/home/vagrant/.minikube/machines/minikube/id_rsa Username:ssp} I1128 23:37:33.971134 3909210 sshutil.go:53] new ssh client: &{IP:10.239.241.111 Port:22 SSHKeyPath:/home/vagrant/.minikube/machines/minikube/id_rsa Username:ssp} I1128 23:37:34.246717 3909210 ssh_runner.go:195] Run: sudo service containerd status I1128 23:37:35.056487 3909210 ssh_runner.go:235] Completed: curl -x http://child-prc.intel.com:913 -sS -m 2 https://registry.k8s.io/: (1.085344076s) I1128 23:37:35.057026 3909210 ssh_runner.go:195] Run: sudo service containerd stop W1128 23:37:35.888338 3909210 cruntime.go:284] disable failed: sudo service containerd stop: Process exited with status 1 stdout:

stderr: Job for containerd.service canceled. I1128 23:37:35.888442 3909210 ssh_runner.go:195] Run: sudo service containerd status W1128 23:37:35.938559 3909210 docker.go:136] disableOthers: containerd is still active I1128 23:37:35.938784 3909210 ssh_runner.go:195] Run: /bin/bash -c "sudo mkdir -p /etc && printf %s "runtime-endpoint: unix:///var/run/cri-dockerd.sock image-endpoint: unix:///var/run/cri-dockerd.sock " | sudo tee /etc/crictl.yaml" I1128 23:37:35.986154 3909210 ssh_runner.go:195] Run: sudo service docker restart I1128 23:37:37.108576 3909210 ssh_runner.go:235] Completed: sudo service docker restart: (1.122366052s) I1128 23:37:37.108630 3909210 openrc.go:158] restart output: I1128 23:37:37.108706 3909210 ssh_runner.go:195] Run: sudo service cri-docker.socket status I1128 23:37:37.147498 3909210 ssh_runner.go:195] Run: sudo service cri-docker.socket start I1128 23:37:37.963215 3909210 out.go:177]

W1128 23:37:37.963625 3909210 out.go:239] X Exiting due to RUNTIME_ENABLE: sudo service cri-docker.socket start: Process exited with status 5 stdout:

stderr: Failed to start cri-docker.socket.service: Unit cri-docker.socket.service not found.

X Exiting due to RUNTIME_ENABLE: sudo service cri-docker.socket start: Process exited with status 5 stdout:

stderr: Failed to start cri-docker.socket.service: Unit cri-docker.socket.service not found.

W1128 23:37:37.963651 3909210 out.go:239] W1128 23:37:37.964530 3909210 out.go:239] ╭─────────────────────────────────────────────────────────────────────────────────────────────╮ │ │ │ If the above advice does not help, please let us know: │ │ https://github.com/kubernetes/minikube/issues/new/choose │ │ │ │ Please run minikube logs --file=logs.txt and attach logs.txt to the GitHub issue. │ │ │ ╰─────────────────────────────────────────────────────────────────────────────────────────────╯ ╭─────────────────────────────────────────────────────────────────────────────────────────────╮ │ │ │ If the above advice does not help, please let us know: │ │ https://github.com/kubernetes/minikube/issues/new/choose │ │ │ │ Please run minikube logs --file=logs.txt and attach logs.txt to the GitHub issue. │ │ │ ╰─────────────────────────────────────────────────────────────────────────────────────────────╯

Operating System

Ubuntu

Driver

SSH

afbjorklund commented 1 year ago

Did you install cri-dockerd ?

The requirements for "ssh" is the same as for "none"

hualongfeng commented 1 year ago

Did you install cri-dockerd ?

The requirements for "ssh" is the same as for "none"

yes, I success in service cri-docker status. But error on service cri-docker.socket start.

hualongfeng commented 1 year ago

service cri-docker status. But error on service cri-docker.socket start

And the command output is successful.

root@minikube:~/cri-dockerd# systemctl status cri-docker.socket
● cri-docker.socket - CRI Docker Socket for the API
     Loaded: loaded (/etc/systemd/system/cri-docker.socket; enabled; vendor preset: enabled)
     Active: active (running) since Mon 2022-11-28 11:51:08 UTC; 4h 23min ago
   Triggers: ● cri-docker.service
     Listen: /run/cri-dockerd.sock (Stream)
      Tasks: 0 (limit: 230378)
     Memory: 0B
        CPU: 2ms
     CGroup: /system.slice/cri-docker.socket

Nov 28 11:51:08 ceph-server5 systemd[1]: Starting CRI Docker Socket for the API...
Nov 28 11:51:08 ceph-server5 systemd[1]: Listening on CRI Docker Socket for the API.
afbjorklund commented 1 year ago

The "service" command doesn't know about systemd units, so it tries to append .service

Unit cri-docker.socket.service not found.

hualongfeng commented 1 year ago

The "service" command doesn't know about systemd units, so it tries to append .service

Unit cri-docker.socket.service not found.

root@minikube:~/cri-dockerd# cp /etc/systemd/system/cri-docker.socket /etc/systemd/system/cri-docker.socket.service
root@minikube:~/cri-dockerd# systemctl enable --now cri-docker.socket
root@minikube:~/cri-dockerd# service cri-docker.socket status
○ cri-docker.socket.service - CRI Docker Socket for the API
     Loaded: bad-setting (Reason: Unit cri-docker.socket.service has a bad unit file setting.)
     Active: inactive (dead)

Nov 28 15:52:06 minikube systemd[1]: /etc/systemd/system/cri-docker.socket.service:5: Unknown section 'Socket'. Ignoring.
Nov 28 15:52:06 minikube systemd[1]: cri-docker.socket.service: Service has no ExecStart=, ExecStop=, or SuccessAction=. Refusing.
Nov 28 15:52:08 minikube systemd[1]: /etc/systemd/system/cri-docker.socket.service:5: Unknown section 'Socket'. Ignoring.
Nov 28 15:52:08 minikube systemd[1]: cri-docker.socket.service: Service has no ExecStart=, ExecStop=, or SuccessAction=. Refusing.
Nov 28 15:52:15 minikube systemd[1]: /etc/systemd/system/cri-docker.socket.service:5: Unknown section 'Socket'. Ignoring.
Nov 28 15:52:15 minikube systemd[1]: cri-docker.socket.service: Service has no ExecStart=, ExecStop=, or SuccessAction=. Refusing.
Nov 28 15:54:36 minikube systemd[1]: /etc/systemd/system/cri-docker.socket.service:5: Unknown section 'Socket'. Ignoring.
Nov 28 15:54:36 minikube systemd[1]: cri-docker.socket.service: Service has no ExecStart=, ExecStop=, or SuccessAction=. Refusing.
Nov 28 16:23:28 minikube systemd[1]: /etc/systemd/system/cri-docker.socket.service:5: Unknown section 'Socket'. Ignoring.
Nov 28 16:23:28 minikube systemd[1]: cri-docker.socket.service: Service has no ExecStart=, ExecStop=, or SuccessAction=. Refusing.
root@minikube:~/cri-dockerd# service cri-docker.socket start
Failed to start cri-docker.socket.service: Unit cri-docker.socket.service has a bad unit file setting.
See system logs and 'systemctl status cri-docker.socket.service' for details.
root@minikube:~/cri-dockerd# systemctl daemon-reload
root@minikube:~/cri-dockerd# systemctl enable --now cri-docker.socket
root@minikube:~/cri-dockerd# service cri-docker.socket status
○ cri-docker.socket.service - CRI Docker Socket for the API
     Loaded: bad-setting (Reason: Unit cri-docker.socket.service has a bad unit file setting.)
     Active: inactive (dead)

Nov 28 15:52:15 minikube systemd[1]: /etc/systemd/system/cri-docker.socket.service:5: Unknown section 'Socket'. Ignoring.
Nov 28 15:52:15 minikube systemd[1]: cri-docker.socket.service: Service has no ExecStart=, ExecStop=, or SuccessAction=. Refusing.
Nov 28 15:54:36 minikube systemd[1]: /etc/systemd/system/cri-docker.socket.service:5: Unknown section 'Socket'. Ignoring.
Nov 28 15:54:36 minikube systemd[1]: cri-docker.socket.service: Service has no ExecStart=, ExecStop=, or SuccessAction=. Refusing.
Nov 28 16:23:28 minikube systemd[1]: /etc/systemd/system/cri-docker.socket.service:5: Unknown section 'Socket'. Ignoring.
Nov 28 16:23:28 minikube systemd[1]: cri-docker.socket.service: Service has no ExecStart=, ExecStop=, or SuccessAction=. Refusing.
Nov 28 16:23:37 minikube systemd[1]: /etc/systemd/system/cri-docker.socket.service:5: Unknown section 'Socket'. Ignoring.
Nov 28 16:23:37 minikube systemd[1]: cri-docker.socket.service: Service has no ExecStart=, ExecStop=, or SuccessAction=. Refusing.
Nov 28 16:24:15 minikube systemd[1]: /etc/systemd/system/cri-docker.socket.service:5: Unknown section 'Socket'. Ignoring.
Nov 28 16:24:15 minikube systemd[1]: cri-docker.socket.service: Service has no ExecStart=, ExecStop=, or SuccessAction=. Refusing.
root@minikube:~/cri-dockerd# service cri-docker.socket start
Failed to start cri-docker.socket.service: Unit cri-docker.socket.service has a bad unit file setting.
See system logs and 'systemctl status cri-docker.socket.service' for details.
afbjorklund commented 1 year ago

For some reason the systemd is not found, so it is trying to use the OpenRC code. It should be systemctl.

i.e. service cri-docker.socket status should be systemctl status cri-docker.socket

It is perfectly reasonable to expect cri-dockerd to run without systemd, but it has not been implemented...

There should be some clues in the log, as to why running systemctl --version failed ?

hualongfeng commented 1 year ago

For some reason the systemd is not found, so it is trying to use the OpenRC code. It should be systemctl.

i.e. service cri-docker.socket status should be systemctl status cri-docker.socket

It is perfectly reasonable to expect cri-dockerd to run without systemd, but it has not been implemented...

There should be some clues in the log, as to why running systemctl --version failed ?

root@minikube:~/cri-dockerd# systemctl --version
systemd 249 (249.11-0ubuntu3)
+PAM +AUDIT +SELINUX +APPARMOR +IMA +SMACK +SECCOMP +GCRYPT +GNUTLS -OPENSSL +ACL +BLKID +CURL +ELFUTILS -FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP -LIBFDISK +PCRE2 -PWQUALITY -P11KIT -QRENCODE +BZIP2 +LZ4 +XZ +ZLIB +ZSTD -XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified
hualongfeng commented 1 year ago

Is there any method to solve the issue?

afbjorklund commented 1 year ago

It needs the log file, to troubleshoot.

hualongfeng commented 1 year ago

logs.txt

afbjorklund commented 1 year ago

Yeah, that looks like a bug.

I1129 00:33:23.366905 3924022 ssh_runner.go:195] Run: systemctl --version
I1129 00:33:23.366965 3924022 retry.go:31] will retry after 276.165072ms: new client: new client: Error creating new ssh host from driver: Error getting ssh host name for driver: IP address is not set
I1129 00:33:23.643348 3924022 retry.go:31] will retry after 540.190908ms: new client: new client: Error creating new ssh host from driver: Error getting ssh host name for driver: IP address is not set
I1129 00:33:24.184521 3924022 retry.go:31] will retry after 655.06503ms: new client: new client: Error creating new ssh host from driver: Error getting ssh host name for driver: IP address is not set

Probably sysinit is called "too early", before the driver knows the IP (it seems to be set OK, in the config)

... SSHIPAddress:10.239.241.111 SSHUser:ssp SSHKey:/home/vagrant/.ssh/id_rsa SSHPort:22 ...

hualongfeng commented 1 year ago

https://github.com/Mirantis/cri-dockerd/issues/133#issuecomment-1329607004 I asked cri-docker for the command service cri-docker.socket status service cri-docker.socket start is meaningless as the service command is hardcoded to use a .service unit suffix.

So why minikube use the command service cri-docker.socket status not systemctl status cri-docker.socket or service cri-docker status?

logopk commented 1 year ago

Let me add a Me2!

afbjorklund commented 1 year ago

This is probably a bug in the ssh driver, not detecting systemd on the remote system properly.

The amount of users that actually want to use OpenRC is probably diminishingly small.

Simplest fix is just removing the .socket

Then minikube can start the service directly

k8s-triage-robot commented 1 year ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

logopk commented 1 year ago

/remove-lifecycle stale

f0rkz commented 1 year ago

2023 - this is still an issue. 8|

k8s-triage-robot commented 9 months ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

logopk commented 9 months ago

/remove-lifecycle stale

k8s-triage-robot commented 6 months ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot commented 5 months ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

logopk commented 5 months ago

/remove-lifecycle rotten

k8s-triage-robot commented 2 months ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

logopk commented 2 months ago

/remove-lifecycle stale