axboe / liburing

Library providing helpers for the Linux kernel io_uring support
MIT License
2.85k stars 402 forks source link

SQPOLL: io_sq_thread taking all the CPU #997

Closed jmillan closed 11 months ago

jmillan commented 11 months ago

uname -r: 6.1.0-13-amd64 liburing: 2.4

I'm successfully using liburing to send out UDP and TCP packets using io_uring_prep_sendto and io_uring_prep_write respectively.

I want to test the SQPOLL, so I initialize the ring with the given flag:

 unsigned int flags = IORING_SETUP_SINGLE_ISSUER | IORING_SETUP_SQPOLL;

  // Initialize io_uring.
  auto err = io_uring_queue_init(DepLibUring::QueueDepth, std::addressof(this->ring), flags);

Once after calling io_uring_prep_sendto() or io_uring_prep_write(), if IORING_SQ_NEED_WAKEUP flag is set, I call io_uring_submit():

if (IO_URING_READ_ONCE(*this->ring.sq.kflags) & IORING_SQ_NEED_WAKEUP)
{
    auto err = io_uring_submit(std::addressof(this->ring));
}

This is working properly in the sense that packets are sent as expected, but an io_sq_thread is taking all the CPU.

When the thread is idle there is no CPU usage, but once it's awaken with io_uring_submit, it takes all the CPU again. Am I using it incorrectly?

My use case is a media server relaying network packets from one endpoint to one or many other endpoints.

Screenshot 2023-11-20 at 16 45 04
jmillan commented 11 months ago

Maybe this happens for this scenario (and services alike) where the cadence of packets being sent does not allow the io_sq_thread sleep and keeps it awaken all the time. Maybe the scenario does not fit SQPOLL design. I just wonder if it is me misusing the API.

axboe commented 11 months ago

This is how polling works, it remains active until the desired idle time. It's impossible to poll for new SQ events without actually busy looping!

jmillan commented 11 months ago

It's impossible to poll for new SQ events without actually busy looping!

Yes, I understand it. I think SQ polling does definitely not suit my use case.

Thanks @axboe!