Ratelimiter: Very high CPU usage when ratelimiter is throttling guest RX.

firecracker-microvm / firecracker

Secure and fast microVMs for serverless computing.

http://firecracker-microvm.io

Apache License 2.0

25.83k stars 1.79k forks source link

Ratelimiter: Very high CPU usage when ratelimiter is throttling guest RX. #1439

Closed sandreim closed 4 years ago

sandreim commented 4 years ago

When rate limiting is enabled and the guest is receiving a lot of traffic that triggers throttling the emulation thread will use 100% CPU.

There are 2 possible approaches here:

quick fix: keep the current code structure and configure edge triggered epoll for the tap fd.
proper fix: rework the current state machine and use edge triggered epoll

pyrito commented 4 years ago

Hi @sandreim could you give some more context to this problem?

sandreim commented 4 years ago

Hi @pyrito,

The issue happens when the net device does not read all the the packets from the TAP device waiting for the limiter to replenish it's budget. When that happens the epoll loop will continuously fire the RX_TAP_EVENT until all packets will be read. There is already some code that unregisters the epoll event when there are no more free RX buffers but that does not cover the case when we stop processing the tap RX due to limiter budget.

pyrito commented 4 years ago

Hi @sandreim , I think @mi-yu and I can help take a stab at this problem. Just for reference, we're two students at UT Austin who are looking to make a couple of open-source contributions for our final Virtualization project.

pyrito commented 4 years ago

Hi @sandreim

I was wondering if you could explain what you meant by two of your suggested fixes?

"configure edge triggered epoll for the tap fd", we looked through the code and found the part of the code that does not unregister the epoll event when the tap RX is being blocked due to reaching its budget. Could you clarify what you meant by configuring the edge triggered epoll?
"rework the current state machine and use edge triggered epoll" could you give some general context towards how to make this "proper" fix.

Thank you for your time!

andreeaflorescu commented 4 years ago

Hey @pyrito. @sandreim already has a fix for this and he will probably post it early next week. Do you need help in finding another issue?

pyrito commented 4 years ago

@andreeaflorescu , yes that would be greatly appreciated! If there are any issues of easy difficulty that we could tackle within a week or so (with a proof of concept) that would be much appreciated. Thanks!