Assertion failures and ASAN violations when SIGINT-ing redpanda during archival

[VladLazar]

Version & Environment

Debug build of redpanda from tip of dev at the time of writing (0608811e9042e42feff1ccd8e2ff80c981717272). A topic with remote write enabled is present, but no data is being written to it. In my test env, I've set up the AWS bucket to reject all reads, but I don't think that's relevant here.

What went wrong?

Sending a SIGINT signal to redpanda running locally in the scenarion caused above leads to assertion failures or ASAN violations fairly regularly.

Assertion and decoded trace

seastar::output_stream<char>::~output_stream() [CharType = char]: Assertion `!_in_batch && "Was this stream properly closed?"' failed.

/work/dev/vbuild/llvm/src/compiler-rt/lib/asan/../sanitizer_common/sanitizer_common_interceptors.inc:4277
/work/dev/vbuild/debug/clang/v_deps_build/seastar-prefix/src/seastar/include/seastar/util/backtrace.hh:59
/work/dev/vbuild/debug/clang/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:754
/work/dev/vbuild/debug/clang/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:784
/work/dev/vbuild/debug/clang/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:796
/work/dev/vbuild/debug/clang/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:3662
/work/dev/vbuild/debug/clang/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:3638
/work/dev/vbuild/debug/clang/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:3634
/lib64/libc.so.6+0x3ea2f
/lib64/libc.so.6+0x8ec0b
/lib64/libc.so.6+0x3e985
/lib64/libc.so.6+0x287f3
/lib64/libc.so.6+0x2871a
/lib64/libc.so.6+0x37535
/work/dev/vbuild/debug/clang/rp_deps_install/include/seastar/core/iostream.hh:393
/work/dev/vbuild/llvm/install/bin/../include/c++/v1/tuple:210
/work/dev/vbuild/llvm/install/bin/../include/c++/v1/tuple:384
/work/dev/vbuild/llvm/install/bin/../include/c++/v1/tuple:470
/work/dev/vbuild/debug/clang/rp_deps_install/include/seastar/core/do_with.hh:37
/work/dev/vbuild/debug/clang/rp_deps_install/include/seastar/core/do_with.hh:45
/work/dev/vbuild/debug/clang/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:2329
/work/dev/vbuild/debug/clang/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:2736
/work/dev/vbuild/debug/clang/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:2905
/work/dev/vbuild/debug/clang/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:4139
/work/dev/vbuild/llvm/install/bin/../include/c++/v1/type_traits:3640
/work/dev/vbuild/llvm/install/bin/../include/c++/v1/__functional/invoke.h:61
/work/dev/vbuild/llvm/install/bin/../include/c++/v1/__functional/function.h:180
/work/dev/vbuild/llvm/install/bin/../include/c++/v1/__functional/function.h:354
/work/dev/vbuild/llvm/install/bin/../include/c++/v1/__functional/function.h:507
/work/dev/vbuild/llvm/install/bin/../include/c++/v1/__functional/function.h:1184
/work/dev/vbuild/debug/clang/v_deps_build/seastar-prefix/src/seastar/src/core/posix.cc:73
/lib64/libc.so.6+0x8cdec
/lib64/libc.so.6+0x11236f

ASAN Violations The output of ASAN is fairly verbose so I've attached it as a file. The redpanda logs precede the ASAN output, so you'll have to scroll to the bottom.

How to reproduce the issue?

I expect that this should reproduce fairly easily with a recent redpanda build with cloud storage enabled.

JIRA Link: CORE-1083

[VladLazar]

From the stack traces, I suspect that the recent change in shutdown order might be the cause, but it needs further investigation.

[VladLazar]

Evgeny mentioned that the ASAN problem is a known issue with the seastar TLS implementation.

[dotnwat]

sev/low, very old, but worth a quick reproducer test.