ming1
commented
8 months ago ublk needs IORING_SETUP_SQE128 which is supported since liburing 2.2, can you check what your liburing version is?
Thanks,
Closed kyleshu closed 8 months ago
ming1
commented
8 months ago ublk needs IORING_SETUP_SQE128 which is supported since liburing 2.2, can you check what your liburing version is?
Thanks,
ming1
commented
8 months ago Another reason could be that io_uring doesn't work in your system, and you can check if that is the case by:
cd liburing ./configure make
./test/nop
If ./test/nop doesn't complain, io_uring should work in your system.
kyleshu
commented
8 months ago Thanks for your replies.
ublk needs IORING_SETUP_SQE128 which is supported since liburing 2.2, can you check what your liburing version is?
I used to use the master branch of liburing, but now I switched to 2.5 release
Another reason could be that io_uring doesn't work in your system, and you can check if that is the case by:
cd liburing ./configure make
./test/nop
If ./test/nop doesn't complain, io_uring should work in your system.
nop.t does not complain
ubuntu@ip-172-31-94-195:~/liburing-liburing-2.5$ ./test/nop.t The exact steps I followed are listed below:
wget https://github.com/axboe/liburing/archive/refs/tags/liburing-2.5.tar.gz
tar xvf liburing-2.5.tar.gz
cd liburing-liburing-2.5
./configure
make
./test/nop.t
sudo make install
cd ..
git clone https://github.com/ming1/ubdsrv.git
cd ubdsrv
autoreconf -i
./configure
make
sudo make install
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib
sudo ldconfig
sudo modprobe ublk_drv
sudo ublk featuresIt still shows me can't get sqe ret -22
kyleshu
commented
8 months ago I tried to run the full test suite of liburing, and noticed a few failures, could this be related?
ubuntu@ip-172-31-94-195:~/liburing-liburing-2.5$ sudo make runtests
make[1]: Entering directory '/home/ubuntu/liburing-liburing-2.5/src'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/home/ubuntu/liburing-liburing-2.5/src'
make[1]: Entering directory '/home/ubuntu/liburing-liburing-2.5/test'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/home/ubuntu/liburing-liburing-2.5/test'
make[1]: Entering directory '/home/ubuntu/liburing-liburing-2.5/examples'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/home/ubuntu/liburing-liburing-2.5/examples'
make[1]: Entering directory '/home/ubuntu/liburing-liburing-2.5/test'
Running test 232c93d07b74.t 4 sec [4]
Running test 35fa71a030ca.t 5 sec [5]
Running test 500f9fbadef8.t 3 sec [3]
Running test 7ad0e4b2f83c.t 1 sec [1]
Running test 8a9973408177.t 0 sec [0]
Running test 917257daa0fe.t 0 sec [0]
Running test a0908ae19763.t 0 sec [0]
Running test a4c0b3decb33.t 6 sec [6]
Running test accept.t 1 sec [1]
Running test accept-link.t 0 sec [0]
Running test accept-reuse.t 0 sec [0]
Running test accept-test.t 0 sec [0]
Running test across-fork.t 0 sec [0]
Running test b19062a56726.t 0 sec [0]
Running test b5837bd5311d.t 0 sec [0]
Running test buf-ring.t 0 sec [0]
Running test ce593a6c480a.t 1 sec [1]
Running test close-opath.t 0 sec [0]
Running test connect.t 0 sec [0]
Running test connect-rep.t 0 sec [0]
Running test coredump.t 0 sec [0]
Running test cq-full.t 0 sec [0]
Running test cq-overflow.t 11 sec [11]
Running test cq-peek-batch.t 0 sec [0]
Running test cq-ready.t 0 sec [0]
Running test cq-size.t 0 sec [0]
Running test d4ae271dfaae.t 0 sec [0]
Running test d77a67ed5f27.t 0 sec [0]
Running test defer.t 3 sec [3]
Running test defer-taskrun.t 0 sec [0]
Running test double-poll-crash.t 0 sec [0]
Running test drop-submit.t 0 sec [0]
Running test eeed8b54e0df.t 0 sec [0]
Running test empty-eownerdead.t 0 sec [0]
Running test eploop.t 0 sec [0]
Running test eventfd.t 0 sec [0]
Running test eventfd-disable.t 0 sec [0]
Running test eventfd-reg.t 0 sec [0]
Running test eventfd-ring.t 0 sec [0]
Running test evloop.t 0 sec [0]
Running test exec-target.t 0 sec [0]
Running test exit-no-cleanup.t 0 sec [0]
Running test fadvise.t 0 sec [0]
Running test fallocate.t 0 sec [0]
Running test fc2a85cb02ef.t Test needs failslab/fail_futex/fail_page_alloc enabled, skipped
Skipped
Running test fd-pass.t 0 sec [0]
Running test file-register.t 3 sec [3]
Running test files-exit-hang-poll.t 1 sec [1]
Running test files-exit-hang-timeout.t 1 sec [1]
Running test file-update.t 0 sec [0]
Running test file-verify.t 7 sec [7]
Running test fixed-buf-iter.t 0 sec [0]
Running test fixed-link.t 0 sec [0]
Running test fixed-reuse.t 0 sec [0]
Running test fpos.t 0 sec [1]
Running test fsnotify.t 0 sec
Running test fsync.t 0 sec [0]
Running test hardlink.t 0 sec [0]
Running test io-cancel.t 2 sec [2]
Running test iopoll.t 0 sec [0]
Running test iopoll-leak.t 0 sec [0]
Running test iopoll-overflow.t 1 sec [1]
Running test io_uring_enter.t 0 sec [0]
Running test io_uring_passthrough.t Skipped
Running test io_uring_register.t 0 sec [0]
Running test io_uring_setup.t 0 sec [0]
Running test lfs-openat.t 0 sec [0]
Running test lfs-openat-write.t 0 sec [0]
Running test link.t 0 sec [0]
Running test link_drain.t 1 sec [0]
Running test link-timeout.t 1 sec [2]
Running test madvise.t 0 sec [0]
Running test mkdir.t 0 sec [0]
Running test msg-ring.t 0 sec [0]
Running test msg-ring-flags.t 0 sec [0]
Running test msg-ring-overflow.t 0 sec [0]
Running test multicqes_drain.t 21 sec [20]
Running test no-mmap-inval.t Got 0, wanted -EFAULT
Test no-mmap-inval.t failed with ret 1
Running test nolibc.t 0 sec [0]
Running test nop-all-sizes.t 0 sec [0]
Running test nop.t 0 sec [1]
Running test openat2.t 0 sec [0]
Running test open-close.t 0 sec [0]
Running test open-direct-link.t 0 sec [0]
Running test open-direct-pick.t 0 sec [0]
Running test personality.t 0 sec [0]
Running test pipe-bug.t 0 sec [0]
Running test pipe-eof.t 0 sec [0]
Running test pipe-reuse.t 0 sec [0]
Running test poll.t 0 sec [0]
Running test poll-cancel.t 0 sec [0]
Running test poll-cancel-all.t 0 sec [0]
Running test poll-cancel-ton.t 0 sec [0]
Running test poll-link.t 1 sec [0]
Running test poll-many.t 1 sec [2]
Running test poll-mshot-overflow.t 0 sec [0]
Running test poll-mshot-update.t 3 sec [2]
Running test poll-race.t 0 sec [0]
Running test poll-race-mshot.t 0 sec [1]
Running test poll-ring.t 0 sec [0]
Running test poll-v-poll.t 0 sec [0]
Running test pollfree.t Test pollfree.t timed out (may not be a failure)
Running test probe.t 0 sec [0]
Running test read-before-exit.t 1 sec [1]
Running test read-write.t 1 sec [1]
Running test recv-msgall.t 0 sec [0]
Running test recv-msgall-stream.t 0 sec [0]
Running test recv-multishot.t 0 sec [0]
Running test reg-fd-only.t 0 sec [0]
Running test reg-hint.t 0 sec [0]
Running test reg-reg-ring.t 0 sec [0]
Running test regbuf-merge.t 0 sec [0]
Running test register-restrictions.t 0 sec [0]
Running test rename.t 0 sec [0]
Running test ringbuf-read.t 0 sec [0]
Running test ring-leak2.t 1 sec [1]
Running test ring-leak.t sendmsg: Invalid argument
sendmsg: Invalid argument
sendmsg: Invalid argument
0 sec [0]
Running test rsrc_tags.t 0 sec [0]
Running test rw_merge_test.t 0 sec [0]
Running test self.t 0 sec [0]
Running test send_recv.t 0 sec [0]
Running test send_recvmsg.t 0 sec [0]
Running test send-zerocopy.t 15 sec [14]
Running test shared-wq.t 0 sec [0]
Running test short-read.t 0 sec [0]
Running test shutdown.t 0 sec [0]
Running test sigfd-deadlock.t 0 sec [0]
Running test single-issuer.t 0 sec [0]
Running test skip-cqe.t 0 sec [0]
Running test socket.t 0 sec [0]
Running test socket-io-cmd.t 0 sec
Running test socket-rw.t 0 sec [0]
Running test socket-rw-eagain.t 0 sec [0]
Running test socket-rw-offset.t 0 sec [0]
Running test splice.t 0 sec [0]
Running test sq-full.t 0 sec [0]
Running test sq-full-cpp.t 0 sec [0]
Running test sqpoll-cancel-hang.t 1 sec [1]
Running test sqpoll-disable-exit.t 0 sec [0]
Running test sq-poll-dup.t 1 sec [1]
Running test sqpoll-exit-hang.t 1 sec [1]
Running test sq-poll-kthread.t 2 sec [2]
Running test sq-poll-share.t 4 sec [4]
Running test sqpoll-sleep.t 0 sec [0]
Running test sq-space_left.t 0 sec [0]
Running test stdout.t This is a pipe test
This is a fixed pipe test
0 sec [0]
Running test submit-and-wait.t 1 sec [1]
Running test submit-link-fail.t 0 sec [0]
Running test submit-reuse.t 3 sec [3]
Running test symlink.t 0 sec [0]
Running test sync-cancel.t 0 sec [0]
Running test teardowns.t 0 sec [0]
Running test thread-exit.t 0 sec [0]
Running test timeout.t 8 sec [8]
Running test timeout-new.t 3 sec [3]
Running test tty-write-dpoll.t 0 sec [0]
Running test unlink.t 0 sec [0]
Running test version.t 0 sec [0]
Running test wakeup-hang.t 2 sec [2]
Running test wq-aff.t Skipped
Running test xattr.t 0 sec [0]
Running test statx.t 0 sec [0]
Running test sq-full-cpp.t 0 sec [0]
Tests timed out (1): <pollfree.t>
Tests failed (1): <no-mmap-inval.t>
make[1]: *** [Makefile:259: runtests] Error 1
make[1]: Leaving directory '/home/ubuntu/liburing-liburing-2.5/test'
make: *** [Makefile:21: runtests] Error 2no-mmap-inval.t does not give desired output, and pollfree.t consistently times out.
ming1
commented
8 months ago Looks io_uring is working.
Can you try the following patch and see if it can make a difference?
diff --git a/lib/ublksrv_cmd.c b/lib/ublksrv_cmd.c
index 5730e9a..8a44745 100644
--- a/lib/ublksrv_cmd.c
+++ b/lib/ublksrv_cmd.c
@@ -141,6 +141,7 @@ void ublksrv_ctrl_deinit(struct ublksrv_ctrl_dev *dev)
struct ublksrv_ctrl_dev *ublksrv_ctrl_init(struct ublksrv_dev_data *data)
{
+ struct io_uring_params p;
struct ublksrv_ctrl_dev *dev = (struct ublksrv_ctrl_dev *)calloc(1,
sizeof(*dev));
struct ublksrv_ctrl_dev_info *info = &dev->dev_info;
@@ -166,8 +167,9 @@ struct ublksrv_ctrl_dev *ublksrv_ctrl_init(struct ublksrv_dev_data *data)
dev->tgt_argc = data->tgt_argc;
dev->tgt_argv = data->tgt_argv;
- /* 32 is enough to send ctrl commands */
- ret = ublksrv_setup_ring(&dev->ring, 32, 32, IORING_SETUP_SQE128);
+ memset(&p, 0, sizeof(p));
+ p.flags = IORING_SETUP_SQE128;
+ ret = io_uring_queue_init_params(64, &dev->ring, &p);
if (ret < 0) {
fprintf(stderr, "queue_init: %s\n", strerror(-ret));
free(dev);It still looks one environment issue, can you run the following test?
wget https://github.com/ublk-org/ublksrv/blob/master/utils/nop.c
gcc -O2 -o nop nop.c -luring
./nop
And see if it can be reproduced on the above steps.
If yes, I guess you may still link to one old liburing.
Also you can check the liburing version on ubuntu by:
dpkg -l | grep liburing
If it is < liburing 2.2, you can upgrade to 2.2+ and see if there is any difference.
If liburing-devel 2.2+ isn't available, you can build from source, but please use the following script to build ublksrv:
https://github.com/ublk-org/ublksrv/blob/master/build_with_liburing_src
Meantime change the script and make sure that LIBURING_DIR points to the real liburing path in your system.
Thanks,
kyleshu
commented
8 months ago It still looks one environment issue, can you run the following test?
- wget https://github.com/ublk-org/ublksrv/blob/master/utils/nop.c
- gcc -O2 -o nop nop.c -luring
- ./nop
And see if it can be reproduced on the above steps.
If yes, I guess you may still link to one old liburing.
Also you can check the liburing version on ubuntu by:
dpkg -l | grep liburing
If it is < liburing 2.2, you can upgrade to 2.2+ and see if there is any difference.
If liburing-devel 2.2+ isn't available, you can build from source, but please use the following script to build ublksrv:
https://github.com/ublk-org/ublksrv/blob/master/build_with_liburing_src
Meantime change the script and make sure that LIBURING_DIR points to the real liburing path in your system.
Thanks,
Your guess is right. Many thanks for helping with that.
I am trying to use ublk on an AWS EC2 instance. The OS image I use is Ubuntu 22.04 LTS with the kernel version=6.7.6 (ubuntu image provided by AWS does not have ublk_drv module, so I updated the kernel version).
I installed liburing and ublksrv from source and did modprobe for ublk_drv.
However, it seems ublk cannot get sqe from the control ring.
Any ideas on what could possibly go wrong?