Simplify firectl invoke

[arun-gupta]

firectl invocation can be simplified with the following defaults:

• pick up the latest firecracker binary
• pick up the default uncompressed Linux kernel image
• pick up the default root filesystem image

[nmeyerhans]

Current versions pick firecracker from PATH, unless overridden on the commandline.

We could come up with default values for kernel and rootfs locations and hardcode them, leaving the commandline options available as overrides. Or we could allow commonly repeated settings like these to live in a config file.

There are pros and cons of either option. I'd be interested in seeing more concrete proposals, or will come up with some myself at some point if nobody beats me to it.

[simonranger]

How about an ini style config file per vm?

The choice of firecracker binary could be a default or just use whatever's in PATH, but it would be great to be able to start a machine from a config file. Perhaps something like the following:

[cpu]
count = 1
disable_ht = false
#template = t2

[memory]
size = 128

[kernel]
image = vmlinuz
#options =

[drives]
root_image = image.ext4
#additional = []

[network]
#devices = []
#sockets = []

[nmeyerhans]

A per-VM config file would be useful. We could also define a global config file containing overridable defaults (e.g. for options that are often, but not always, identical between MicroVMs, e.g. kernel image, CPU template, hyperthreading configuration, etc)

I've got a small project that I've used for experimentation, etc, that places firectl inside a container and programmatically figures out some of the firectl command line based on the container's configuration. That allows things like the network configuration to be left out entirely, since the container runtime (e.g. Docker) will set this up. Other options, such as kernel image, can be built in to the container image, so they basically become build-time inputs rather than runtime inputs. I haven't published this yet, but I hope to do so soon as I think it could be an interesting context in which to experiment with different configuration interfaces to firecracker.

[yihuaf]

With regard to storing the kernel and rootfs (VM image), would it make sense to do something similar to docker where we store the image under something like /var/lib/firecracker/<name>. Then we can use an import command to "import" kernel and rootfs from arbitrary locations. We can also add an manifest perhaps per VM image?

[nmeyerhans]

@yihuaf That's a super interesting idea. I think it speaks to the larger question of the scope of the firectl project. Initially, it was written to provide a simple interface to Firecracker and to provide an example client for the Firecracker Go SDK. However, it has obviously proven useful to people, and there's no reason it needs to stop where it is.

To help us come up with a more complete roadmap, it'd be interesting to hear about use cases for firectl that aren't covered by other Firecracker integration work such as firecracker-containerd or kata containers

[yihuaf]

From a usability perspective, image management would be important. To me, firecracker at the moment is similar to runc to containers, instead of docker to containers. If I'm understanding it correctly, we want firectl to become more docker like in terms of user friendliness and scope?

Also a side note, there is nothing wrong in the future we can use docker containers to distribute the rootfs used by firecracker. Most people use firecracker the same way as containers, with only applications. So most people would not care about the kernel version. From a quick look at the containerd integration and kata container, these projects aims to run multiple containers inside the microVM. Based on my understanding, microVM has the ability to replace containers on a per container basis. If the launch time is 125 ms, that's faster or on par with docker run. My thinking is can we take dockerfiles and produce microVM images. Then we run microVM instead of containers, but using a docker-like interface.

[nmeyerhans]

If I'm understanding it correctly, we want firectl to become more docker like in terms of user friendliness and scope?

Not necessarily. The goal of firecracker-containerd is to allow the use of container images and other related tools (CNI plugins, etc) with Firecracker, so firectl is free to become something else. Eventually, when issues like moby #38043 are resolved, Docker should be able to directly run run containers in Firecracker VMs using firecracker-containerd.

The way things got started with firectl, I'd say it is closer in spirit to the qemu commands used to launch KVM virtual machines on Linux, e.g. qemu-system-x86_64 For example, the following qemu and firectl invocations generate roughly the same VM (there are a couple of notable differences, but they're not relevant to this discussion):

$ qemu-system-x86_64 -machine pc,accel=kvm -smp cpus=4 -m 1G -nographic  \
-drive file=image,if=virtio,format=raw  -kernel bzImage \
-append "rw console=ttyS0 noapic reboot=k panic=1 nomodules root=/dev/vda"

$ firectl --ncpus 4 --memory 1024 --kernel vmlinux --root-drive image \
--kernel-opts "rw console=ttyS0 noapic reboot=k panic=1 pci=off nomodules root=/dev/vda"

As I noted earlier, though, we don't have to stick with this kind of approach if we don't want to. Firectl's future isn't really tied to any specific use case or needs, and if there are features that will make it useful we should definitely consider them.

[yihuaf]

The containerd integration is about running containers inside firecracker VM, conceptually close to what Kata container propose. I agree that firectl is different and it controls the VM directly. Part of this PR is about the management of --kernel and --root-drive. Today, there is no uniform way to distribute the kernel and root-drive, like how we distribute Docker containers in docker hub. So instead of a default kernel and root-drive, is it more user friendly for user to have firectl to manage kernel and root-drive by names like how we manage the Docker image. With that being said, if the goal is to be more like qemu command line, then perhaps this feature is not that much useful.

For example, docker has the following: docker import rootfs.tgz <IMG NAME> docker run --name <CONTAINER NAME> <IMG NAME> You can reference the image by name when launching a container, and the image is essentially a rootfs.

For firectl, we can potentially have something like: firectl import root root-drive.tgz <NAME> firectl import kernel vmlinux <KERNEL NAME> firectl launch --ncpus 4 --memory 1024 <KERNEL NAME> <NAME>

This is just some rough idea. The actual command and options will depend on how people would use it and who is the intended audience of firectl. I think people comes from the Docker world may want something similar to this feature. Hope that helps :)

[kzys]

FYI https://www.weave.works/blog/fire-up-your-vms-with-weave-ignite looks closer to some of the stuff we have discussed here.

[yihuaf]

To me, ignite from weave work wraps the firecracker under docker image. It does achieve some of the functionalities we discussed here. With that being said, I feel that wrapping firecracker under docker diminish the start-up time claim of the firecracker. Docker in general launches around 500 ms to 1 seconds range, not around 100-200 ms range, which is a big selling point of firecracker. So if you only want the security aspect of the firecracker, then yes. For most of the use cases, it should be OK, but I'm not sure if it's acceptable for some of the use cases that's more sensitive to start up time.

[taqtiqa-mark]

To me, ignite from weave work wraps the firecracker under docker image.

@yihuaf I believe Ignite starts the VM as fast as you'd expect firecracker to, and then runs /sbin/init which is a requirement for any container you point Ignite to.

Just to correct myself - it does appear there is a container/vm shuffle right at the outset:

INFO[0005] Created VM with ID "09f2494e3242296a" and name "podman"                                                            
INFO[0007] Networking is handled by "cni"                                                                                     
INFO[0007] Started Firecracker VM "09f2494e3242296a" in a container with ID "ignite-09f2494e3242296a"                         
09f2494e3242296a                                                                                                              
INFO[0001] Moving IP address 10.61.0.28 (255.255.0.0) from container to VM

I believe /sbin/init is where things slow down. By how much is a function of what the container starts up as part of its init processes.

It seems firectl's use case is closer to that of libvirt, which is off topic so I've opened a separate issue.