search

kata-containers / packaging

Kata Containers version 1.x packaging (for version 2.x see https://github.com/kata-containers/kata-containers).
https://katacontainers.io/
Apache License 2.0
119 stars 92 forks source link

Firecracker doesn't seem to work with k3s #1193

Closed stevefan1999-personal closed 3 years ago

stevefan1999-personal commented 3 years ago

Description of problem

In k3s you cannot launch firecracker VM I ran exactly the same steps in kata-deploy

Expected result

It works

Actual result

It doesn't work

Mar 03 08:18:42 k3s-home kata[387254]: time="2021-03-03T08:18:42.052944094Z" level=error msg="getting vm status failed" ID=5787c958de218600a2109c5c894f724cb7abef487a2901909d58decba1600db5 error="Get \"http://localhost/\": dial unix /run/vc/firecracker/5787c958de218600a2109c5c894f724c/root/run/firecracker.socket: connect: no such file or directory" source=virtcontainers subsystem=firecracker
Mar 03 08:18:42 k3s-home kata[387254]: time="2021-03-03T08:18:42.063430272Z" level=error msg="getting vm status failed" ID=5787c958de218600a2109c5c894f724cb7abef487a2901909d58decba1600db5 error="Get \"http://localhost/\": dial unix /run/vc/firecracker/5787c958de218600a2109c5c894f724c/root/run/firecracker.socket: connect: no such file or directory" source=virtcontainers subsystem=firecracker
Mar 03 08:18:42 k3s-home kata[387254]: time="2021-03-03T08:18:42.073955039Z" level=error msg="getting vm status failed" ID=5787c958de218600a2109c5c894f724cb7abef487a2901909d58decba1600db5 error="Get \"http://localhost/\": dial unix /run/vc/firecracker/5787c958de218600a2109c5c894f724c/root/run/firecracker.socket: connect: no such file or directory" source=virtcontainers subsystem=firecracker
Mar 03 08:18:42 k3s-home kata[387254]: time="2021-03-03T08:18:42.084493047Z" level=error msg="getting vm status failed" ID=5787c958de218600a2109c5c894f724cb7abef487a2901909d58decba1600db5 error="Get \"http://localhost/\": dial unix /run/vc/firecracker/5787c958de218600a2109c5c894f724c/root/run/firecracker.socket: connect: no such file or directory" source=virtcontainers subsystem=firecracker
Mar 03 08:18:42 k3s-home kata[387254]: time="2021-03-03T08:18:42.094936075Z" level=error msg="getting vm status failed" ID=5787c958de218600a2109c5c894f724cb7abef487a2901909d58decba1600db5 error="Get \"http://localhost/\": dial unix /run/vc/firecracker/5787c958de218600a2109c5c894f724c/root/run/firecracker.socket: connect: no such file or directory" source=virtcontainers subsystem=firecracker
Mar 03 08:18:42 k3s-home kata[387254]: time="2021-03-03T08:18:42.105563553Z" level=error msg="getting vm status failed" ID=5787c958de218600a2109c5c894f724cb7abef487a2901909d58decba1600db5 error="Get \"http://localhost/\": dial unix /run/vc/firecracker/5787c958de218600a2109c5c894f724c/root/run/firecracker.socket: connect: no such file or directory" source=virtcontainers subsystem=firecracker
Mar 03 08:18:42 k3s-home kata[387254]: time="2021-03-03T08:18:42.11608251Z" level=error msg="getting vm status failed" ID=5787c958de218600a2109c5c894f724cb7abef487a2901909d58decba1600db5 error="Get \"http://localhost/\": dial unix /run/vc/firecracker/5787c958de218600a2109c5c894f724c/root/run/firecracker.socket: connect: no such file or directory" source=virtcontainers subsystem=firecracker
Mar 03 08:18:42 k3s-home kata[387254]: time="2021-03-03T08:18:42.126590538Z" level=error msg="getting vm status failed" ID=5787c958de218600a2109c5c894f724cb7abef487a2901909d58decba1600db5 error="Get \"http://localhost/\": dial unix /run/vc/firecracker/5787c958de218600a2109c5c894f724c/root/run/firecracker.socket: connect: no such file or directory" source=virtcontainers subsystem=firecracker
Mar 03 08:18:42 k3s-home kata[387254]: time="2021-03-03T08:18:42.137126626Z" level=error msg="getting vm status failed" ID=5787c958de218600a2109c5c894f724cb7abef487a2901909d58decba1600db5 error="Get \"http://localhost/\": dial unix /run/vc/firecracker/5787c958de218600a2109c5c894f724c/root/run/firecracker.socket: connect: no such file or directory" source=virtcontainers subsystem=firecracker
Mar 03 08:18:42 k3s-home kata[387254]: time="2021-03-03T08:18:42.147679954Z" level=error msg="getting vm status failed" ID=5787c958de218600a2109c5c894f724cb7abef487a2901909d58decba1600db5 error="Get \"http://localhost/\": dial unix /run/vc/firecracker/5787c958de218600a2109c5c894f724c/root/run/firecracker.socket: connect: no such file or directory" source=virtcontainers subsystem=firecracker
Mar 03 08:18:42 k3s-home kata[387254]: time="2021-03-03T08:18:42.158316291Z" level=error msg="getting vm status failed" ID=5787c958de218600a2109c5c894f724cb7abef487a2901909d58decba1600db5 error="Get \"http://localhost/\": dial unix /run/vc/firecracker/5787c958de218600a2109c5c894f724c/root/run/firecracker.socket: connect: no such file or directory" source=virtcontainers subsystem=firecracker
Mar 03 08:18:42 k3s-home kata[387254]: time="2021-03-03T08:18:42.168969109Z" level=error msg="getting vm status failed" ID=5787c958de218600a2109c5c894f724cb7abef487a2901909d58decba1600db5 error="Get \"http://localhost/\": dial unix /run/vc/firecracker/5787c958de218600a2109c5c894f724c/root/run/firecracker.socket: connect: no such file or directory" source=virtcontainers subsystem=firecracker
Mar 03 08:18:42 k3s-home kata[387254]: time="2021-03-03T08:18:42.179572107Z" level=error msg="getting vm status failed" ID=5787c958de218600a2109c5c894f724cb7abef487a2901909d58decba1600db5 error="Get \"http://localhost/\": dial unix /run/vc/firecracker/5787c958de218600a2109c5c894f724c/root/run/firecracker.socket: connect: no such file or directory" source=virtcontainers subsystem=firecracker
Mar 03 08:18:42 k3s-home kata[387254]: time="2021-03-03T08:18:42.190039705Z" level=error msg="getting vm status failed" ID=5787c958de218600a2109c5c894f724cb7abef487a2901909d58decba1600db5 error="Get \"http://localhost/\": dial unix /run/vc/firecracker/5787c958de218600a2109c5c894f724c/root/run/firecracker.socket: connect: no such file or directory" source=virtcontainers subsystem=firecracker
Mar 03 08:18:42 k3s-home kata[387254]: time="2021-03-03T08:18:42.200692402Z" level=error msg="getting vm status failed" ID=5787c958de218600a2109c5c894f724cb7abef487a2901909d58decba1600db5 error="Get \"http://localhost/\": dial unix /run/vc/firecracker/5787c958de218600a2109c5c894f724c/root/run/firecracker.socket: connect: no such file or directory" source=virtcontainers subsystem=firecracker
Mar 03 08:18:42 k3s-home kata[387254]: time="2021-03-03T08:18:42.21124529Z" level=error msg="getting vm status failed" ID=5787c958de218600a2109c5c894f724cb7abef487a2901909d58decba1600db5 error="Get \"http://localhost/\": dial unix /run/vc/firecracker/5787c958de218600a2109c5c894f724c/root/run/firecracker.socket: connect: no such file or directory" source=virtcontainers subsystem=firecracker
Mar 03 08:18:42 k3s-home kata[387254]: time="2021-03-03T08:18:42.221838308Z" level=error msg="getting vm status failed" ID=5787c958de218600a2109c5c894f724cb7abef487a2901909d58decba1600db5 error="Get \"http://localhost/\": dial unix /run/vc/firecracker/5787c958de218600a2109c5c894f724c/root/run/firecracker.socket: connect: no such file or directory" source=virtcontainers subsystem=firecracker
Mar 03 08:18:42 k3s-home kata[387254]: time="2021-03-03T08:18:42.232487535Z" level=error msg="getting vm status failed" ID=5787c958de218600a2109c5c894f724cb7abef487a2901909d58decba1600db5 error="Get \"http://localhost/\": dial unix /run/vc/firecracker/5787c958de218600a2109c5c894f724c/root/run/firecracker.socket: connect: no such file or directory" source=virtcontainers subsystem=firecracker
Mar 03 08:18:42 k3s-home kata[387254]: time="2021-03-03T08:18:42.242972923Z" level=error msg="getting vm status failed" ID=5787c958de218600a2109c5c894f724cb7abef487a2901909d58decba1600db5 error="Get \"http://localhost/\": dial unix /run/vc/firecracker/5787c958de218600a2109c5c894f724c/root/run/firecracker.socket: connect: no such file or directory" source=virtcontainers subsystem=firecracker
Mar 03 08:18:42 k3s-home kata[387254]: time="2021-03-03T08:18:42.253421691Z" level=error msg="getting vm status failed" ID=5787c958de218600a2109c5c894f724cb7abef487a2901909d58decba1600db5 error="Get \"http://localhost/\": dial unix /run/vc/firecracker/5787c958de218600a2109c5c894f724c/root/run/firecracker.socket: connect: no such file or directory" source=virtcontainers subsystem=firecracker
Mar 03 08:18:42 k3s-home kata[387254]: time="2021-03-03T08:18:42.263881619Z" level=error msg="getting vm status failed" ID=5787c958de218600a2109c5c894f724cb7abef487a2901909d58decba1600db5 error="Get \"http://localhost/\": dial unix /run/vc/firecracker/5787c958de218600a2109c5c894f724c/root/run/firecracker.socket: connect: no such file or directory" source=virtcontainers subsystem=firecracker
Mar 03 08:18:42 k3s-home kata[387254]: time="2021-03-03T08:18:42.274587416Z" level=error msg="getting vm status failed" ID=5787c958de218600a2109c5c894f724cb7abef487a2901909d58decba1600db5 error="Get \"http://localhost/\": dial unix /run/vc/firecracker/5787c958de218600a2109c5c894f724c/root/run/firecracker.socket: connect: no such file or directory" source=virtcontainers subsystem=firecracker
Mar 03 08:18:42 k3s-home kata[387254]: time="2021-03-03T08:18:42.285146474Z" level=error msg="getting vm status failed" ID=5787c958de218600a2109c5c894f724cb7abef487a2901909d58decba1600db5 error="Get \"http://localhost/\": dial unix /run/vc/firecracker/5787c958de218600a2109c5c894f724c/root/run/firecracker.socket: connect: no such file or directory" source=virtcontainers subsystem=firecracker
Mar 03 08:18:42 k3s-home kata[387254]: time="2021-03-03T08:18:42.295969302Z" level=error msg="getting vm status failed" ID=5787c958de218600a2109c5c894f724cb7abef487a2901909d58decba1600db5 error="Get \"http://localhost/\": dial unix /run/vc/firecracker/5787c958de218600a2109c5c894f724c/root/run/firecracker.socket: connect: no such file or directory" source=virtcontainers subsystem=firecracker
Mar 03 08:18:42 k3s-home kata[387254]: time="2021-03-03T08:18:42.306911079Z" level=error msg="getting vm status failed" ID=5787c958de218600a2109c5c894f724cb7abef487a2901909d58decba1600db5 error="Get \"http://localhost/\": dial unix /run/vc/firecracker/5787c958de218600a2109c5c894f724c/root/run/firecracker.socket: connect: no such file or directory" source=virtcontainers subsystem=firecracker

Further information

I ran Kata in a...quite special environment. This k3s server is running in a KVM instance with nested VM support under Proxmox.

I'm running in AMD Ryzen CPU and I have confirmed SVM support in /proc/cpuinfo in the instance, and I can ensure the existence of /dev/kvm too. So Firecracker should have had started nicely and I'm not sure why the firecracker UDS is not found.

My NIC and block device are both running in VirtIO mode for near-native performance, I'm not sure if they mattered.

I also have enabled memory ballooning support though this is supposed to be transparent.

I'll try to run Kata in Docker to see what will happen.

stevefan1999-personal commented 3 years ago

In Docker it doesn't seem to work either

root@k3s-home:/opt/kata/bin# docker run --runtime=kata-qemu -itd alpine
Unable to find image 'alpine:latest' locally
latest: Pulling from library/alpine
ba3557a56b15: Pull complete
Digest: sha256:a75afd8b57e7f34e4dad8d65e2c7ba2e1975c795ce1ee22fa34f8cf46f96a3be
Status: Downloaded newer image for alpine:latest
d9da27ce10f8dfd39072752c23b84a6e72a79182af54e77640f32fb40f9cf07f

ls
docker: Error response from daemon: OCI runtime create failed: Failed to check if grpc server is working: rpc error: code = Unavailable desc = transport is closing: unknown.

Should I escalate this to the main Kata repo?

c3d commented 3 years ago

Does firecracker work by itself? I'm asking because it seems to be an issue with connecting through the firecracker socket, which might indicate firecracker just exited.

chavafg commented 3 years ago

Could you also mention which version of Kata you are using? Running kata-collect-data.sh could also provide more information. Also, did you configured docker and k3s (containerd?) to use devicemapper?

stevefan1999-personal commented 3 years ago

@chavafg the kata data collection is on the escalated issue, this is a global issue because I can't even start one in Docker as well, so let alone of whether k8s/runC side will run it.

I just think nested VM probably not supported. I really do hope it is the case and not because of AMD.

fidencio commented 3 years ago

This issue is being automatically closed as Kata Containers 1.x has now reached EOL (End of Life). This means it is no longer being maintained.

Important:

All users should switch to the latest Kata Containers 2.x release to ensure they are using a maintained release that contains the latest security fixes, performance improvements and new features.

This decision was discussed by the @kata-containers/architecture-committee and has been announced via the Kata Containers mailing list:

If you believe this issue still applies to Kata Containers 2.x, please open an issue against the Kata Containers 2.x repository, pointing to this one, providing details to allow us to migrate it.