Operation with `IOSQE_IO_LINK | IOSQE_CQE_SKIP_SUCCESS` does not complete on ring with `IORING_SETUP_DEFER_TASKRUN`

Cydox commented 2 months ago

If I do two send operations on a socket followed by a close on the file descriptor linked by IOSQE_IO_LINK the close operation never happens. All SQEs have IOSQE_CQE_SKIP_SUCCESS set. The ring is created with IORING_SETUP_DEFER_TASKRUN.

Already did a git bisect on the kernel and this is introduced in commit 846072f16eed3b3fb4e59b677f3ed8afb8509b89 https://github.com/torvalds/linux/commit/846072f16eed3b3fb4e59b677f3ed8afb8509b89

Is it better to report this on the kernel mailing list, as it's not really an issue with liburing? I'm fine with using either the mailing list or github.

Interesting observations:

If I only do one send followed by the close, it works.
If I remove either IOSQE_IO_LINK or IOSQE_CQE_SKIP_SUCCESS on the close, it works.
If I use IORING_SETUP_COOP_TASKRUN instead of defer, it works.
If I add a third send operation that message get's sent (close still doesn't work).

Reproducer written in Zig (let me know if you want a reproducer in C):

const std = @import("std");

const linux = std.os.linux;
const posix = std.posix;
const sqe_t = std.os.linux.io_uring_sqe;
const cqe_t = std.os.linux.io_uring_cqe;

const event_type = enum {
    ACCEPT,
    READ,
    WRITE,
    CLOSE,
};

pub fn main() !void {
    const stdout = std.io.getStdOut();
    _ = stdout;
    std.debug.print("hello from normal syscall\n", .{});

    const peer = try std.net.Address.parseIp4("0.0.0.0", 9999);
    const sock = try std.posix.socket(posix.AF.INET, posix.SOCK.STREAM, 0);
    try std.posix.setsockopt(sock, posix.SOL.SOCKET, posix.SO.REUSEADDR, &std.mem.toBytes(@as(c_int, 1)));
    try std.posix.setsockopt(sock, 6, posix.TCP.NODELAY, &std.mem.toBytes(@as(c_int, 1)));

    try std.posix.bind(sock, &(peer.any), peer.getOsSockLen());
    defer std.posix.close(sock);
    try std.posix.listen(sock, 1);

    const ring_buf_size = @as(u64, 1) <<| 12;

    var io_uring = try std.os.linux.IoUring.init(
        @intCast(ring_buf_size),
        0 |
            linux.IORING_SETUP_DEFER_TASKRUN |
            linux.IORING_SETUP_SINGLE_ISSUER,
    );
    defer io_uring.deinit();

    _ = try io_uring.accept_multishot(@intFromEnum(event_type.ACCEPT), sock, null, null, 0);

    while (true) {
        _ = try io_uring.submit_and_wait(1);

        var cqes: [ring_buf_size]cqe_t = undefined;
        const n = try io_uring.copy_cqes(&cqes, 0);
        //std.debug.print("n: {}\n", .{n});

        for (cqes[0..n]) |c| {
            switch (@as(event_type, @enumFromInt(c.user_data))) {
                event_type.ACCEPT => {
                    var sqe = try io_uring.send(@intFromEnum(event_type.WRITE), c.res, "hello from server 1\n", 0);
                    sqe.flags |= linux.IOSQE_CQE_SKIP_SUCCESS;

                    sqe = try io_uring.send(@intFromEnum(event_type.WRITE), c.res, "hello from server 2\n", 0);
                    sqe.flags |= linux.IOSQE_IO_LINK;
                    sqe.flags |= linux.IOSQE_CQE_SKIP_SUCCESS;

                    sqe = try io_uring.close(@intFromEnum(event_type.CLOSE), c.res);
                    sqe.flags |= linux.IOSQE_IO_LINK;
                    sqe.flags |= linux.IOSQE_CQE_SKIP_SUCCESS;
                },
                event_type.READ => {},
                event_type.WRITE => {},
                event_type.CLOSE => {},
            }
        }
    }
}

Can be built using zig 0.13:

zig build-exe main.zig

To test simply connect using telnet:

$ telnet 127.0.0.1 9999

Expected behavior is to get two messages from the server and then be disconnected. Observed behavior is getting the two messages and the connection not being closed.