A potential security threaten in Kata-containers

[XDTG]

Kata container contains potential risks of DoS attack caused by dirty_pages. A malicious Kata container can make the host dirty pages reach a threshold and downgrade the co-resident victim Kata container IO performance greatly.

Principle: The microVM isolated Kata container calls the write system call in the guest kernel, triggering the fuse_send_write function in the guest kernel. In this function, the guest write system call’s metadata are intercepted by the virtiofs driver and transferred to the virtiofs daemon process in the host user space via virtio. While the virtiofs daemon on the host receives this request, it triggers the host system call write and writes the host dirty pages. When the host dirty pages reach the dirty_background_ratio threshold, the kernel wakes up background threads to sync the dirty memory to the disk, generating many IO workloads.

Experiment: We measure the victim container’s IO performance before and after the attack on our local test bed. In a malicious container, we leverage the dd command to write files. As we discussed above, write system call in the guest kernel is forwarded to the virtiofs daemon in the host, and triggers the host system call write to write the host’s dirty pages. On our local test bed, the performance of command dd if=/dev/zero of=/mnt/test bs=1M count=1024 in the victim container drops from 1.5GB/s to 147.8MB/s due to the attack, resulting in 93.4% slow down. In the Alibaba Cloud environment, the machine has 192GB memory, and the dirty_background_ratio is 10. When malicious Kata Containers write dirty pages of more than 19.2GB, the host kernel wakes up the kernel thread to sync dirty memory to the disk. We use two malicious Kata Containers to write dirty pages, the performance of command dd if=/dev/zero of=/mnt/test bs=1M count=1024 in the victim container drops from 850MB/s to 46MB/s, resulting in 94.6% slow down.

[fengwang666]

This sounds like an issue of virtiofs? I assume it can also happen if the write is legitimate.

[c3d]

@XDTG Thanks a lot for the report. I assume this is without any I/O throttling? See old discussion about I/O throttling that may apply in your case.

[c3d]

@XDTG Also, it looks like this is a host performance impact rather than a real security risk. I am not even sure you could reach a point where you have a real DOS that way, only a serious reduction in the performance of other containers. Is my understanding correct?

[rhvgoyal]

I think one approach to manage this would be by using cgroups. May be put kata container + virtiofsd in a cgroup and put resource limtis so that you have better isolation between containers w.r.t resource usage (something similar to native containers).

[rhvgoyal]

And this is not a security issue as such. This is more about better isolation between container issue. (noisy neighbor)