search

firecracker-microvm / firecracker

Secure and fast microVMs for serverless computing.
http://firecracker-microvm.io
Apache License 2.0
25.08k stars 1.75k forks source link

[Perf] Implement async IO for the block device using io_uring #1600

Open ioanachirca opened 4 years ago

ioanachirca commented 4 years ago

After some more debugging there is no doubt that there are 2 major issues which bring down the overall performance of our block device emulation:

Our block device emulation works by sequentially processing guest I/O requests. It will also serialize the latency of each request and this results in very low CPU (host/guest) usage when running fio benchmarks. The Firecracker emulation thread is mostly sleeping/waiting for I/O to complete and fio maxes out 4-5K IOPS. To improve on this we will need to add block async I/O support in Firecracker.

The increased guest interrupt overhead becomes significant only when the Firecracker block device is doing a lot of IOPS (70-80k). That happens only if it is backed by a RAMdisk to minimize latencies. As an improvement we will need to implement virtio driver and device event suppression / interrupt mitigation. I've experimented a bit with this and got a nice performance boost.

The current plan is to use io_uring to parallelize the block I/O operations. The io_uring interface was added in kernel 5.1 so this will be available only when running on host kernels that support io_uring, otherwise the emulation will work serially as before.

raduweiss commented 3 years ago

We're starting to prototype this out now (we'll post some details here once we have some outcomes).

alindima commented 2 years ago

This was released as dev preview in Firecracker v1.0.0 More info in the docs: https://github.com/firecracker-microvm/firecracker/blob/main/docs/api_requests/block-io-engine.md

We'll close this issue once we make the Async block io_engine GA.

serban300 commented 2 years ago

I think we can close the issue even if it was released as dev preview.

xmarcalx commented 2 years ago

Moved to shipped because released with Firecracker v1.0.0 and not further development is required.

xmarcalx commented 2 years ago

Sorry for moving the issue again, moving it back to dev preview because the condition to make such feature GA listed here are not met yet.

JonathanWoollett-Light commented 9 months ago

Still in developer preview, re-opening to indicate this.

kanpov commented 3 weeks ago

Still in developer preview, re-opening to indicate this.

Would support for 6.1 guest kernel (6.1 host is already supported) move this out of developer preview?