fio --enghelp: infinitely says 'nfs'

vt-alt commented 11 months ago

Please acknowledge the following before creating a ticket

[x] I have read the GitHub issues section of REPORTING-BUGS.

Description of the bug: fio -i infinitely says nfs.

Environment: ALT Linux Sisyphus

fio version: fio-3.36 compiled with with --dynamic-libengines. Without --dynamic-libengines there is no repeating.

Reproduction steps

Install fio and libaio with any other ext engine, tested with fio-engine-http and fio-engine-nbd, then run fio -i:

$ fio --enghelp
Available IO engines:
    io_uring
    io_uring_cmd
    sg
    mtd
    gfapi_async
    gfapi
    splice
    e4defrag
    falloc
    posixaio
    nfs
    nfs
    nfs
    nfs
    nfs
    nfs
    nfs
    nfs
    nfs
    nfs
    nfs
...

fio --enghelp=http and fio --enghelp=nbd still work.

If additional engines are not installed nfs printed once and internal engine list is correct.

If only fio-libaio.so is installed nfs is (incorrectly) listed as the last engine (i.e. printing stops after nfs).

GDB backtrace after ^C while nfs is printing:

(gdb) bt
#0  0x00007ffff7d39610 in __GI___libc_write (fd=1, buf=0x555555763500, nbytes=5) at ../sysdeps/unix/sysv/linux/write.c:26
#1  0x00007ffff7cbbf85 in _IO_new_file_write (f=0x7ffff7e125c0 <_IO_2_1_stdout_>, data=0x555555763500, n=5) at fileops.c:1181
#2  0x00007ffff7cba2b4 in new_do_write (fp=0x7ffff7e125c0 <_IO_2_1_stdout_>,
    data=0x555555763500 "\tnfs\n\nelete\n engines:\n:\tfffff\nCpus_allowed_list:\t0-19\nMems_allowed:\t00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,000000"..., to_do=to_do@entry=5)
    at /usr/src/debug/glibc-2.38.0.27.750a45a783-alt1/libio/libioP.h:1030
#3  0x00007ffff7cbb079 in _IO_new_do_write (fp=<optimized out>, data=<optimized out>, to_do=5) at fileops.c:426
#4  0x00007ffff7cbc0f7 in _IO_new_file_xsputn (n=5, data=<optimized out>, f=0x7ffff7e125c0 <_IO_2_1_stdout_>)
    at /usr/src/debug/glibc-2.38.0.27.750a45a783-alt1/libio/libioP.h:1030
#5  _IO_new_file_xsputn (f=0x7ffff7e125c0 <_IO_2_1_stdout_>, data=<optimized out>, n=5) at fileops.c:1197
#6  0x00007ffff7cb002a in __GI__IO_fwrite (buf=0x555555772560, size=5, count=1, fp=0x7ffff7e125c0 <_IO_2_1_stdout_>)
    at /usr/src/debug/glibc-2.38.0.27.750a45a783-alt1/libio/libioP.h:1030
#7  0x0000555555592333 in log_valist (args=0x7ffffffcd640, fmt=0x5555555ee9a6 "\t%s\n") at log.c:40
#8  log_info (format=format@entry=0x5555555ee9a6 "\t%s\n") at log.c:78
#9  0x000055555557f3ae in fio_show_ioengine_help (engine=<optimized out>) at ioengines.c:704
#10 0x0000555555584102 in parse_cmd_line (argc=argc@entry=2, argv=argv@entry=0x7fffffffe4a8, client_type=client_type@entry=1) at init.c:2626
#11 0x0000555555584d62 in parse_options (argc=argc@entry=2, argv=argv@entry=0x7fffffffe4a8) at init.c:3056
#12 0x0000555555570ab4 in main (argc=2, argv=0x7fffffffe4a8, envp=<optimized out>) at fio.c:42

vt-alt commented 11 months ago

Looking at the code, it seems, that flist is unsafe from adding the same element twice.

Debugging further shows when libaio engine is registered it calls to register nfs engine (which is built-in and already registered previously).

fio_libaio_register calls register_ioengine(&ioengine) which is defined as

FIO_STATIC struct ioengine_ops ioengine = {

where FIO_STATIC is curiously defined as: non-static if CONFIG_DYNAMIC_ENGINES. Thus ioengine from nfs winning ioengine value from libaio.

$ gdb -q ./fio
Reading symbols from ./fio...
(gdb) p ioengine
$1 = {list = {next = 0x0, prev = 0x0}, name = 0x9f7f8 "nfs", version = 33...

Perhaps, engine loading code is slightly overengineered. It tries to get ioengine address from symbol or from get_ioengine() (only used for cpp_null), and also __attribute__((constructor)) functions are automatically called by dlopen, but this last call is ignored if engine is loaded directly by a name, but used for listing with -i.

vt-alt commented 11 months ago

Possible workaround:

diff --git a/engines/nfs.c b/engines/nfs.c
index 970962a3..ce748d14 100644
--- a/engines/nfs.c
+++ b/engines/nfs.c
@@ -308,7 +308,7 @@ static int fio_libnfs_close(struct thread_data *td, struct fio_file *f)
         return ret;
 }

-struct ioengine_ops ioengine = {
+static struct ioengine_ops ioengine = {
         .name                = "nfs",
         .version        = FIO_IOOPS_VERSION,
         .setup                = fio_libnfs_setup,

Non-external engines' ioengine should be always static to not override its values from external engines.

axboe / fio

fio --enghelp: infinitely says 'nfs' #1655