[Devel] Firecracker full-host performance tests

[raduweiss]

Feature Request

We currently measure and guard against single-dimension performance regressions (e.g., network throughput). However, we don't do the same for performance in full-host at-load multi-dimension scenarios. We should!

Describe the desired solution

TBA, but what we need is to have a way to configure test that fill up a host with microVMs, and run some specific mix of while measuring more performance dimensions. We also need to have more such configurations.

Describe possible alternatives

TBA

Additional context

We're just starting here. Will update as we go, but ideas are welcome.

Checks

• [x] Have you searched the Firecracker Issues database for similar requests?
• [x] Have you read all the existing relevant Firecracker documentation?
• [x] Have you read and understood Firecracker's core tenets?

[ustiugov]

We are highly interested in such tests as well in the context of vHive. In my opinion, there are a couple of questions regarding the full-host performance tests that need to be discussed first:

1. The suite of representative serverless functions or containers. We have already dockerized a wide range of Python functions plus few others, written in Java SpringBoot and Golang.
2. Which orchestrator framework to use because the infrastructure -- not just the hypervisor but shims, agents, etc. -- has implications on the host performance.

vHive allows to rapidly bootstrap and load many VMs with gRPC traffic, using our Firecracker-containerd fork, by booting or loading from snapshots. However, booting VMs with it is much slower than from a bare rootfs (e.g., 1 second vs 100ms).

Please let me know what you think.

[xmarcalx]

In the past year we have have worked on this series of issues which implemented:

1. a statistics module to collect data during testing (#2051#2424)

2. additional performance tests for networking and block devices (#2121 #2226 #2227 #2283 #2338 #2441)

Closing this issue and moving to shipped.