Buffers are not being reused: "bufs in automatic buffer selection empty, this should not happen..."

[mdecimus]

Hi,

Buffers are not being reused after calling io_uring_prep_provide_buffers(). To reproduce this error change:

#define MAX_CONNECTIONS 4096

to

#define MAX_CONNECTIONS 2

Then telnet to the echo server port, close the connection and repeat three times. The first time all works well as the first two buffers are used but on the third connection attempt io_uring runs out of buffers and returns "No buffer space available". I did some debugging and io_uring starts using buf id 1, then id 0 but on the third attempt rather than reusing any of the two available buffers it fails.

It might be a io_uring bug as the code looks fine. I'm using the 5.11 kernel.

Thanks.

[mdecimus]

Found the bug, buffers were not being re-added when the connection closes. Here is the fix:

      } else if (type == READ) {
            int bytes_read = cqe->res;
            int bid = cqe->flags >> 16;
            if (cqe->res <= 0) {
                // read failed, re-add the buffer
                add_provide_buf(&ring, bid, group_id);
                // connection closed or error
                shutdown(conn_i.fd, SHUT_RDWR);
            } else {
                // bytes have been read into bufs, now add write to socket sqe
                add_socket_write(&ring, conn_i.fd, bid, bytes_read, 0);
            }

[frevib]

Thanks for finding this. Could you create a PR for this?

[mdecimus]

Done!