search

Mirantis / virtlet

Kubernetes CRI implementation for running VM workloads
Apache License 2.0
743 stars 128 forks source link

Workload VM in 'RunContainerError' state after Kubernetes cluster reboot #783

Closed ramukima closed 6 years ago

ramukima commented 6 years ago

I am running a kubernetes cluster (single node) with virtlet running on a VM (say A). I am able to run VMs and virtlet example for ubuntu-vm (say B).

However, when I reboot my VM (A), all kubernetes services recover and become available. However, the VM (B) I had deployed, remains in the status 'RunContainerError'. Here is a sample describe pod output when this happens -

  Warning  Failed   19s               kubelet, kubemaster  Error: "/run/virtlet.sock": rpc error: code = 2 desc = failed to create domain "b3c4e0f9-1184-5394-6dee-fb381ca95c45": virError(Code=1, Domain=10, Message='internal error: process exited while connecting to monitor: I1016 15:42:14.853945    6474 vmwrapper.go:66] Obtaining PID of the VM container process...
E1016 15:42:14.854254    6474 vmwrapper.go:89] Failed to obtain tap fds for key "b8b9cbfe-d153-11e8-870e-0050563d373c": server returned error: bad fd key: "b8b9cbfe-d153-11e8-870e-0050563d373c"')
  Normal  Pulled  6s (x9 over 1m)  kubelet, kubemaster  Container image "virtlet.cloud/ubuntu" already present on machine
jiadexin commented 6 years ago

I have the same problem https://github.com/Mirantis/virtlet/issues/786.

I deployed virtlet on a single-node kubenertes cluster. After creating a vm in k8s,it works fine , and the instance file is in the directory /var/lib/virtlet:

ll /var/lib/virtlet/volumes/ -hs 4.0K drwxr-xr-x 2 root root 4.0K October 17 16:08 ./ 4.0K drwxr-xr-x 5 root root 4.0K October 17 16:13 ../ 20M -rw------- 1 64055 syslog 20M October 17 16:17 virtlet_root_7bfdd7f7-347b-5ed1-60b3-81cfc72ab1ba

but when i reboot the k8s node ,the vm files in the /var/lib/virtlet/volumes/ directory are missing:

ll /var/lib/virtlet/volumes/ -hs 4.0K drwxr-xr-x 2 root root 4.0K 10月 17 16:19 ./ 4.0K drwxr-xr-x 5 root root 4.0K 10月 17 16:18 ../