discussion: since we researching moving to micro vms for containers, should we use disks to replace containers volumes

[muhamadazmy]

Currently containers (using containerd) can mount extra volumes that are pre reserved. These are created as a btrfs sub-volumes. We already faced some issues related to btrfs quotas.

Since we are moving to cloud-hypervisor containers (basically running containers in virtual machines) wouldn't it make more sense to create those volumes as raw disks instead. This brings the following benefits:

• Much easier quota and size calculations since the disk is allocated with full required size on disk immediately on creation.
• Unification with full fledged virtual machines so less code and unified interface
• No need to run virtiofsd per each volume (we already have to run for the container root)

Drawbacks:

• The "volume" will now have the performance of a "virtual disk" so 2 layers of file systems on top of each others. This is not that bad knowing that we already turn of "COW" for virtual disks

[delandtj]

since we're moving is a big word.... I don't think that is a decision that has already been taken. We need to look into the granularity of such a thing. Starting a container as opposed to starting a container/application in a vm is not a panacea. The main problem with vms (not counting that it's more expensive in terms of cpu/performance), is that we need to be very avaricious about how we allocate memory for an application.

So MicroVMs is definitely where we want to go. The term for these beasts is definitely in serverless, where the get spun up for a single application, mostly a function, and dies after that.

For running containers in VMs, it's about security mostly, where we separate real users (ssh access, for instance) from the real host. but running a VM per container will eat away on very scarce memory, as suddenly, you need a WHOLE OS to just run some python code.

I don't think we can use a VM for each workload separately.... Baidu does it, but they have their own HW to alleviate loss of mem due to VM overhead. The more, they really define their cloud more a a serverles/function cloud, and not, as we do, an all-purpose one

nevertheless, running a VM for containers is still a better idea than say rootless containers in terms of security, but we still need to see the repercussions of doing so before starting to think it's the warm water reinvented

[delandtj]

again, Microvms are defined for functions, not for applications and then I can imagine these thingies... but for running a VM for each application: how do you set the requirements? even with KSM enabled, the overhead would be disastrous

[muhamadazmy]

we did some benchmarking and we have the following results https://github.com/threefoldtech/zos/issues/1284#issuecomment-859393767 which seems very promising

So we can start discussing disks now