EAGAINs impacting the performance of io_uring

[vikramv3]

When testing an io_uring implementation using a file read test that performs random reads of size 1 MB each (on a multi-node cluster), I'm running into high perf variability to due a large number of requests failing with EAGAIN. The implementation retries the requests that fail with EAGAIN. The nodes on which the number of EAGAINs are high tend to have a lower read throughput. Sometimes the number of EAGAINs on a node is very large, and the resulting throughput is lower than an implementation that uses just synchronous read system calls. All the nodes have the same configuration.

I have ensured that the number of in-flight requests submitted doesn't exceed the submission queue depth, and also ensured that the number of in-flight requests are lesser than the number of io_uring's async bounded workers. However, I'm still seeing a large percentage of requests (upto 50% on some nodes) failing with EAGAINs. Are there some other sources of EAGAINs that I haven’t considered? Is there a way through which I can identify the root cause of the EAGAINs?

I am using liburing 2.4 (iopolling enabled, SQ depth = 128, CQ depth = 256). Each node has 48 CPUs, Linux 5.10, ext4 file system and 4 NVMe SSDs with RAID. The file reads are evenly distributed across the 4 disks of each node.

@axboe, @isilence - I would really appreciate your guidance on this. Thanks a lot for your time.

[axboe]

Use a 6.x based kernel.

[vikramv3]

Thanks @axboe! I'll test this out. Have some modifications gone into io_uring in 6.x with regard to EAGAINs?

[axboe]

Yes, otherwise I would not be suggesting that.