Closed YoSTEALTH closed 5 months ago
ammarfaizi2
commented
5 months ago Can you create a reproducer in C?
I can't debug that Python code. Also, it is unclear what io_uring_sqe(num) and io_uring_put_sqe(ring, sqe) actually do as you customized it yourself, liburing does not have them.
YoSTEALTH
commented
5 months ago I am not really a C coder, not sure if I can do justice.
io_uring_sqe you can think of it as a temp single/multiple sqe holder that you later copy into io_uring_get_sqe using memcpy
Edit: ran liburing tests, currently on linux 6.6.25, same issue on 6.7+
$ make runtests --jobs=16
make[1]: Entering directory '/tmp/liburing/src'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/tmp/liburing/src'
make[1]: Entering directory '/tmp/liburing/test'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/tmp/liburing/test'
make[1]: Entering directory '/tmp/liburing/examples'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/tmp/liburing/examples'
make[1]: Entering directory '/tmp/liburing/test'
Running test 232c93d07b74.t 4 sec
Running test 35fa71a030ca.t 5 sec
Running test 500f9fbadef8.t Polling not supported in current dir, test skipped
Skipped
Running test 7ad0e4b2f83c.t 1 sec
Running test 8a9973408177.t 0 sec
Running test 917257daa0fe.t 0 sec
Running test a0908ae19763.t 0 sec
Running test a4c0b3decb33.t 1 sec
Running test accept.t 1 sec
Running test accept-link.t 0 sec
Running test accept-reuse.t 0 sec
Running test accept-test.t 0 sec
Running test across-fork.t 0 sec
Running test b19062a56726.t 0 sec
Running test b5837bd5311d.t 0 sec
Running test buf-ring.t 0 sec
Running test buf-ring-nommap.t 0 sec
Running test buf-ring-put.t 1 sec
Running test ce593a6c480a.t 1 sec
Running test close-opath.t 0 sec
Running test connect.t 0 sec
Running test connect-rep.t 0 sec
Running test coredump.t 0 sec
Running test cq-full.t 0 sec
Running test cq-overflow.t 10 sec
Running test cq-peek-batch.t 0 sec
Running test cq-ready.t 0 sec
Running test cq-size.t 0 sec
Running test d4ae271dfaae.t 0 sec
Running test d77a67ed5f27.t 0 sec
Running test defer.t 3 sec
Running test defer-taskrun.t 0 sec
Running test defer-tw-timeout.t 2 sec
Running test double-poll-crash.t 0 sec
Running test drop-submit.t 0 sec
Running test eeed8b54e0df.t Skipped
Running test empty-eownerdead.t 0 sec
Running test eploop.t 0 sec
Running test eventfd.t 0 sec
Running test eventfd-disable.t 0 sec
Running test eventfd-reg.t 0 sec
Running test eventfd-ring.t 0 sec
Running test evloop.t 0 sec
Running test exec-target.t 0 sec
Running test exit-no-cleanup.t 0 sec
Running test fadvise.t 0 sec
Running test fallocate.t 0 sec
Running test fc2a85cb02ef.t Test needs failslab/fail_futex/fail_page_alloc enabled, skipped
Skipped
Running test fd-install.t Skipped
Running test fd-pass.t 0 sec
Running test file-register.t 4 sec
Running test files-exit-hang-poll.t 1 sec
Running test files-exit-hang-timeout.t 1 sec
Running test file-update.t 0 sec
Running test file-verify.t 0 sec
Running test fixed-buf-iter.t 0 sec
Running test fixed-buf-merge.t 0 sec
Running test fixed-link.t 0 sec
Running test fixed-reuse.t 0 sec
Running test fpos.t 0 sec
Running test fsnotify.t Skipped
Running test fsync.t 0 sec
Running test futex.t Skipped
Running test hardlink.t not root, skipping AT_EMPTY_PATH test
0 sec
Running test io-cancel.t 3 sec
Running test iopoll.t File/device/fs doesn't support polled IO
0 sec
Running test iopoll-leak.t 0 sec
Running test iopoll-overflow.t 1 sec
Running test io_uring_enter.t 0 sec
Running test io_uring_passthrough.t Skipped
Running test io_uring_register.t Unable to map a huge page. Try increasing /proc/sys/vm/nr_hugepages by at least 1.
Skipping the hugepage test
expected Unknown error -14, but call succeeded
FAIL
Test io_uring_register.t failed with ret 1
Running test io_uring_setup.t 0 sec
Running test lfs-openat.t 0 sec
Running test lfs-openat-write.t 0 sec
Running test link.t 0 sec
Running test link_drain.t 0 sec
Running test link-timeout.t 2 sec
Running test madvise.t 0 sec
Running test mkdir.t 0 sec
Running test msg-ring.t 0 sec
Running test msg-ring-fd.t 0 sec
Running test msg-ring-flags.t 0 sec
Running test msg-ring-overflow.t 0 sec
Running test multicqes_drain.t 20 sec
Running test no-mmap-inval.t 0 sec
Running test nolibc.t 0 sec
Running test nop-all-sizes.t 0 sec
Running test nop.t 0 sec
Running test openat2.t 0 sec
Running test open-close.t 0 sec
Running test open-direct-link.t 0 sec
Running test open-direct-pick.t 0 sec
Running test personality.t Not root, skipping
0 sec
Running test pipe-bug.t 1 sec
Running test pipe-eof.t 0 sec
Running test pipe-reuse.t 0 sec
Running test poll.t 0 sec
Running test poll-cancel.t 0 sec
Running test poll-cancel-all.t 0 sec
Running test poll-cancel-ton.t 0 sec
Running test poll-link.t 0 sec
Running test poll-many.t 2 sec
Running test poll-mshot-overflow.t 0 sec
Running test poll-mshot-update.t 2 sec
Running test poll-race.t 1 sec
Running test poll-race-mshot.t 0 sec
Running test poll-ring.t 0 sec
Running test poll-v-poll.t 1 sec
Running test probe.t 0 sec
Running test read-before-exit.t 0 sec
Running test read-mshot.t Skipped
Running test read-mshot-empty.t Skipped
Running test read-write.t Not root, skipping test_write_efbig
1 sec
Running test recv-msgall.t 0 sec
Running test recv-msgall-stream.t 0 sec
Running test recv-multishot.t 0 sec
Running test reg-fd-only.t Enable huge pages to test big rings
Skipped
Running test reg-hint.t 0 sec
Running test reg-reg-ring.t 0 sec
Running test regbuf-merge.t 0 sec
Running test register-restrictions.t 0 sec
Running test rename.t 0 sec
Running test ringbuf-read.t 0 sec
Running test ringbuf-status.t Skipped
Running test ring-leak2.t 1 sec
Running test ring-leak.t Skipped
Running test rsrc_tags.t 0 sec
Running test rw_merge_test.t 0 sec
Running test self.t 0 sec
Running test send_recv.t 0 sec
Running test send_recvmsg.t 0 sec
Running test send-zerocopy.t too large non-root buffer registration, skip
can't register bufs, skip
too large non-root buffer registration, skip
can't register bufs, skip
1 sec
Running test shared-wq.t 0 sec
Running test short-read.t 0 sec
Running test shutdown.t 0 sec
Running test sigfd-deadlock.t 0 sec
Running test single-issuer.t 0 sec
Running test skip-cqe.t 0 sec
Running test socket.t 0 sec
Running test socket-io-cmd.t Not able to create a raw socket: Operation not permitted
Skipped
Running test socket-getsetsock-cmd.t Testing getsockopt SO_PEERNAME
Skipping tests.
Skipped
Running test socket-rw.t 0 sec
Running test socket-rw-eagain.t 0 sec
Running test socket-rw-offset.t 0 sec
Running test splice.t 0 sec
Running test sq-full.t 0 sec
Running test sq-full-cpp.t 0 sec
Running test sqpoll-cancel-hang.t 1 sec
Running test sqpoll-disable-exit.t 1 sec
Running test sq-poll-dup.t 0 sec
Running test sqpoll-exit-hang.t 1 sec
Running test sq-poll-kthread.t 2 sec
Running test sq-poll-share.t 1 sec
Running test sqpoll-sleep.t 0 sec
Running test sq-space_left.t 0 sec
Running test stdout.t This is a pipe test
This is a fixed pipe test
0 sec
Running test submit-and-wait.t 1 sec
Running test submit-link-fail.t 0 sec
Running test submit-reuse.t 0 sec
Running test symlink.t 0 sec
Running test sync-cancel.t 0 sec
Running test teardowns.t 0 sec
Running test thread-exit.t 0 sec
Running test timeout.t 8 sec
Running test timeout-new.t 3 sec
Running test truncate.t Ftruncate not supported, skipping
0 sec
Running test tty-write-dpoll.t 0 sec
Running test unlink.t 0 sec
Running test version.t 0 sec
Running test waitid.t Skipped
Running test wakeup-hang.t 2 sec
Running test wq-aff.t 0 sec
Running test xattr.t 0 sec
Running test statx.t 0 sec
Running test sq-full-cpp.t 0 sec [0]
Tests failed (1): <io_uring_register.t>
make[1]: *** [Makefile:274: runtests] Error 1
make[1]: Leaving directory '/tmp/liburing/test'
make: *** [Makefile:18: runtests] Error 2Is there a change io_uring_cqe pointer not being freed after io_uring_queue_exit is called? If io_uring_cq_advance or io_uring_cqe_seen isn't called before exiting?
isilence
commented
5 months ago What was the problem? library specific?
Is there a change io_uring_cqe pointer not being freed after io_uring_queue_exit is called? If io_uring_cq_advance or io_uring_cqe_seen isn't called before exiting?
No, CQE pointers are io_uring rings, they are always destroyed on the ring's exit
YoSTEALTH
commented
5 months ago When you call io_uring_queue_exit(ring) does it free the ring memory?
axboe
commented
5 months ago Yes, as Pavel mentioned, when you exit the ring, it is unmapped as well. That includes the SQE and CQE memory.
YoSTEALTH
commented
5 months ago How silly, I figured since I was assigning ring memory, I should be responsible to clearing the memory as well!
YoSTEALTH
commented
5 months ago btw there is no clear mention in man io_uring_queue_init that ring gets assigned memory by io_uring_queue_init itself. For you guys it might be obvious but for people that just read the man page and code (without reading C source) its missing the obvious.
ammarfaizi2
commented
5 months ago How silly, I figured since I was assigning ring memory, I should be responsible to clearing the memory as well!
What do you mean by "responsible to clearing memory"?
If what you mean is calling io_uring_queue_exit, then yes, you're responsible
for calling that once the ring is no longer used.
btw there is no clear mention in man io_uring_queue_init that ring gets assigned memory by io_uring_queue_init itself. For you guys it might be obvious but for people that just read the man page and code (without reading C source) its missing the obvious.
Can you give a sentence you think should be added to the man page?
I think the man 3 io_uring_queue_init already explains that clearly:
The io_uring_queue_init(3) function executes the io_uring_setup(2)
system call to initialize the submission and completion queues in
the kernel with at least entries entries in the submission queue
and then maps the resulting file descriptor to memory shared
between the application and the kernel.
[...]
On success, the resources held by ring should be released via a
corresponding call to io_uring_queue_exit(3).IOW, the initialization and destruction are already clear.
io_uring_queue_init to init the ring.io_uring_queue_exit to free the ring.Let's not drag the Python party into liburing problem. If you use liburing, you will also need to embrace the discipline of C programming.
YoSTEALTH
commented
5 months ago To me man io_uring_queue_init is too vague, doesn't mention anything about where the resources is coming from, or if user should provide it.
On the other hand man io_uring_queue_exit does a really good job stating
io_uring_queue_exit(3) will release all resources acquired and initialized by io_uring_queue_init(3). It first unmaps the memory shared between the application and the kernel and then closes the io_uring file descriptor.People using any language out there should be able to use liburing/io_uring.
edit: maybe if the function was like so, its clear that you don't have to pass ring into it and it will be created and returned!
struct io_uring* io_uring_queue_init(unsigned int entries, unsigned flags)
On GitHub this code seems to work fine: https://github.com/YoSTEALTH/Liburing/blob/master/test/helper/io_uring_put_sqe_test.py#L11-L23 On my local system it runs out of
sqeif the value is higher than_loop(1024, 255)This is where the error is raised https://github.com/YoSTEALTH/Liburing/blob/master/src/liburing/queue.pyx#L93-L108
Not sure whats causing it!