(very) slow writes compared to syscall writes

[markpapadakis]

I've been trying to troubleshoot, or rather understand, an issue specific to disk I/O (writes), where writes are just significantly slower when scheduled via IORING_OP_WRITEV SQE ops, compared to straight writev syscalls, roughly proportionally to the size(in bytes) of the data to write.

I tested this on 5.4, and on 6.8 and, if anything, it is slightly worse on 6.8. I am testing this on an ext4 volume, on an SSD, but I doubt it would make a difference if I was testing this on XFS or on HDD.

The file is open()-ed with (O_RDWR | O_LARGEFILE | O_CREAT | O_NOATIME).

io_uring_prep_writev(sqe, fd, iov, std::size(iov), real_cur_filesize /* append */);
io_uring_submit_and_wait(&ring);

over

const auto res = writev((fd, iov, std::size(iov));

The more data that needs to be written, the slower io-uring gets over writev(). e.g for 2.6k or so, the difference is barely noticeable. For 71k, its over x2 slower. For 140x it's almost x4 slower.

I may be missing something, but I don't know what that could be.

[markpapadakis]

UPDATE:

Issue manifests on XFS as well, so it doesn't appear to be filesystem specific.

[isilence]

Can you get a perf profile? It should give a hint what's happening?

perf record -g --inherit -- ./your_app
perf report --children > profile.txt

Or attach by pid / tid.