~Hard to figure out the bug, will try just using tokio executor, maybe that will work.~
Nope, managed to get the stack trace from the core dump:
#0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
#1 0x00007f5f0b3df801 in __GI_abort () at abort.c:79
#2 0x00007f5f0b428897 in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0x7f5f0b555b9a "%s\n")
at ../sysdeps/posix/libc_fatal.c:181
#3 0x00007f5f0b42f90a in malloc_printerr (str=str@entry=0x7f5f0b557be8 "malloc(): smallbin double linked list corrupted")
at malloc.c:5350
#4 0x00007f5f0b433ce4 in _int_malloc (av=av@entry=0x7f5f04000020, bytes=bytes@entry=40) at malloc.c:3648
#5 0x00007f5f0b4390b1 in __libc_calloc (n=n@entry=1, elem_size=elem_size@entry=40) at malloc.c:3436
#6 0x00007f5f0c064d92 in spdk_for_each_thread (fn=0x7f5f0c02be10 <nvmf_tgt_destroy_poll_group>, ctx=0x0,
cpl=0x7f5f0c02bdb0 <nvmf_tgt_destroy_poll_group_done>) at thread.c:299
#7 0x00007f5f0c02bc9f in nvmf_tgt_advance_state () at nvmf_tgt.c:308
#8 0x00007f5f0c061508 in subsystem_state_change_done (i=<optimised out>, status=0) at subsystem.c:420
#9 0x00007f5f0c064842 in _call_completion (ctx=0x7f5f04425020) at thread.c:592
#10 0x00007f5f0c0588e2 in _spdk_event_queue_run_batch (reactor=0x7f5f0400d940, reactor=0x7f5f0400d940) at reactor.c:207
#11 _spdk_reactor_run (arg=0x7f5f0400d940) at reactor.c:506
#12 0x00007f5f0c058f78 in spdk_reactors_start () at reactor.c:692
#13 0x00007f5f0c057be9 in spdk_app_start (opts=0x7f5f0a520340,
start_fn=0x55c54f4190e0 <spdk_sys::event::AppOpts::start::start_wrapper>, arg1=0x7f5f0a5201a8, arg2=0x0) at app.c:576
#14 0x000055c54f419067 in spdk_sys::event::AppOpts::start (self=..., f=...) at spdk-sys/src/event.rs:61
#15 0x000055c54f4050f1 in spdk_sys::ete_test::ete_test () at spdk-sys/src/lib.rs:47
#16 0x000055c54f42145a in spdk_sys::ete_test::ete_test::{{closure}} () at spdk-sys/src/lib.rs:40
#17 0x000055c54f41b7de in core::ops::function::FnOnce::call_once ()
at /rustc/8a58268b5ad9c4a240be349a633069d48991eb0c/src/libcore/ops/function.rs:235
#18 0x000055c54f42e3af in <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once ()
at /rustc/8a58268b5ad9c4a240be349a633069d48991eb0c/src/liballoc/boxed.rs:770
#19 0x000055c54f483efa in __rust_maybe_catch_panic () at src/libpanic_unwind/lib.rs:80
#20 0x000055c54f4488be in std::panicking::try () at /rustc/8a58268b5ad9c4a240be349a633069d48991eb0c/src/libstd/panicking.rs:275
#21 std::panic::catch_unwind () at /rustc/8a58268b5ad9c4a240be349a633069d48991eb0c/src/libstd/panic.rs:394
#22 test::run_test::run_test_inner::{{closure}} () at src/libtest/lib.rs:1471
#23 0x000055c54f423c55 in std::sys_common::backtrace::__rust_begin_short_backtrace ()
at /rustc/8a58268b5ad9c4a240be349a633069d48991eb0c/src/libstd/sys_common/backtrace.rs:77
#24 0x000055c54f427df5 in std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}} ()
at /rustc/8a58268b5ad9c4a240be349a633069d48991eb0c/src/libstd/thread/mod.rs:470
#25 <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once ()
at /rustc/8a58268b5ad9c4a240be349a633069d48991eb0c/src/libstd/panic.rs:315
#26 std::panicking::try::do_call () at /rustc/8a58268b5ad9c4a240be349a633069d48991eb0c/src/libstd/panicking.rs:296
#27 0x000055c54f483efa in __rust_maybe_catch_panic () at src/libpanic_unwind/lib.rs:80
---Type <return> to continue, or q <return> to quit---
#28 0x000055c54f428482 in std::panicking::try () at /rustc/8a58268b5ad9c4a240be349a633069d48991eb0c/src/libstd/panicking.rs:275
#29 std::panic::catch_unwind () at /rustc/8a58268b5ad9c4a240be349a633069d48991eb0c/src/libstd/panic.rs:394
#30 std::thread::Builder::spawn_unchecked::{{closure}} ()
at /rustc/8a58268b5ad9c4a240be349a633069d48991eb0c/src/libstd/thread/mod.rs:469
#31 core::ops::function::FnOnce::call_once{{vtable-shim}} ()
at /rustc/8a58268b5ad9c4a240be349a633069d48991eb0c/src/libcore/ops/function.rs:235
#32 0x000055c54f4769df in <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once ()
at /rustc/8a58268b5ad9c4a240be349a633069d48991eb0c/src/liballoc/boxed.rs:770
#33 0x000055c54f483630 in <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once ()
at /rustc/8a58268b5ad9c4a240be349a633069d48991eb0c/src/liballoc/boxed.rs:770
#34 std::sys_common::thread::start_thread () at src/libstd/sys_common/thread.rs:13
#35 std::sys::unix::thread::Thread::new::thread_start () at src/libstd/sys/unix/thread.rs:79
#36 0x00007f5f0b9af6db in start_thread (arg=0x7f5f0a521700) at pthread_create.c:463
#37 0x00007f5f0b4c088f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
And the tests actually runs to completion, it's just that it fails to close spdk.
Getting some weird double frees, could either be my shitty executor code or environmental maybe? Trying to rule out environmental first:
~https://github.com/jkozlowski/starfish/pull/13~ Nope doesn't look like it's environmental, time to check the code.
~Hard to figure out the bug, will try just using tokio executor, maybe that will work.~
Nope, managed to get the stack trace from the core dump:
And the tests actually runs to completion, it's just that it fails to close spdk.