OOM and possible leak under load

[travisdowns]

Version & Environment

Redpanda version: 23.1.1

What went wrong?

During a load test, a redpanda node crashed with an out of memory error. The memory diagnostics indicate that the system was truly out of memory (it was not fragmentation) and failed while allocating a 128K buffer for logfile readahead. Most of the memory is used by small objects of 64-256 bytes.

Prior to the crash, shard 9 seems to go haywire, repeatedly issuing the following log lines (> 10,000 times) in rapid succession:

2023-03-17T04:22:40.526327646Z stderr F INFO  2023-03-17 04:22:40,526 [shard  9] storage - log_reader.cc:315 - stopped reading stream: parser_errc::end_of_stream
2023-03-17T04:22:40.526812211Z stderr F INFO  2023-03-17 04:22:40,526 [shard  9] storage - log_reader.cc:315 - stopped reading stream: parser_errc::end_of_stream

Here's the memory dump:

2023-03-17T04:22:41.10775648Z stderr F ERROR 2023-03-17 04:22:41,101 [shard  9] seastar_memory - Dumping seastar memory diagnostics
2023-03-17T04:22:41.107811757Z stderr F Used memory:  4642M
2023-03-17T04:22:41.107814916Z stderr F Free memory:  248K
2023-03-17T04:22:41.107817255Z stderr F Total memory: 5G
2023-03-17T04:22:41.107818994Z stderr F 
2023-03-17T04:22:41.10782089Z stderr F Small pools:
2023-03-17T04:22:41.107823006Z stderr F objsz   spansz  usedobj memory  unused  wst%
2023-03-17T04:22:41.107824738Z stderr F 8   4K  951 44K 37K 83
2023-03-17T04:22:41.107826461Z stderr F 10  4K  63  8K  7K  92
2023-03-17T04:22:41.107828151Z stderr F 12  4K  53  32K 31K 98
2023-03-17T04:22:41.107829841Z stderr F 14  4K  89  8K  7K  84
2023-03-17T04:22:41.107831556Z stderr F 16  4K  2k  56K 30K 53
2023-03-17T04:22:41.107833172Z stderr F 32  4K  4k  204K    84K 41
2023-03-17T04:22:41.107834887Z stderr F 32  4K  2k  204K    136K    66
2023-03-17T04:22:41.107836569Z stderr F 32  4K  1k  72K 31K 43
2023-03-17T04:22:41.107838448Z stderr F 32  4K  2k  696K    636K    91
2023-03-17T04:22:41.107840139Z stderr F 48  4K  2k  144K    51K 35
2023-03-17T04:22:41.107841812Z stderr F 48  4K  3k  1M  941K    85
2023-03-17T04:22:41.10784347Z stderr F 64   4K  25m 1514M   4K  0
2023-03-17T04:22:41.107845193Z stderr F 64  4K  3597k   220M    3K  0
2023-03-17T04:22:41.107846891Z stderr F 80  4K  3559k   272M    8K  0
2023-03-17T04:22:41.107848606Z stderr F 96  4K  8k  864K    123K    14
2023-03-17T04:22:41.107861102Z stderr F 112 4K  12k 7M  5739K   81
2023-03-17T04:22:41.107863285Z stderr F 128 4K  11k 1M  34K 2
2023-03-17T04:22:41.107865082Z stderr F 160 4K  19k 3M  70K 2
2023-03-17T04:22:41.107866805Z stderr F 192 4K  2k  528K    86K 16
2023-03-17T04:22:41.107868544Z stderr F 224 4K  607 168K    35K 20
2023-03-17T04:22:41.107870226Z stderr F 256 4K  3543k   865M    3K  0
2023-03-17T04:22:41.107872032Z stderr F 320 8K  645 304K    102K    33
2023-03-17T04:22:41.107873771Z stderr F 384 8K  1k  432K    38K 8
2023-03-17T04:22:41.107875445Z stderr F 448 4K  3k  1M  66K 4
2023-03-17T04:22:41.10787711Z stderr F 512  4K  932 544K    78K 14
2023-03-17T04:22:41.107878842Z stderr F 640 16K 266 608K    441K    72
2023-03-17T04:22:41.107880516Z stderr F 768 16K 580 592K    157K    26
2023-03-17T04:22:41.107882247Z stderr F 896 8K  230 248K    46K 18
2023-03-17T04:22:41.107883945Z stderr F 1024    4K  132 208K    76K 36
2023-03-17T04:22:41.107885627Z stderr F 1280    32K 1k  2M  504K    23
2023-03-17T04:22:41.1078874Z stderr F 1536  32K 59  256K    167K    65
2023-03-17T04:22:41.107889131Z stderr F 1792    16K 26  144K    98K 68
2023-03-17T04:22:41.107891034Z stderr F 2048    8K  92  304K    120K    39
2023-03-17T04:22:41.107892741Z stderr F 2560    64K 50  640K    515K    80
2023-03-17T04:22:41.107894464Z stderr F 3072    64K 6   448K    429K    95
2023-03-17T04:22:41.107896195Z stderr F 3584    32K 13  288K    242K    83
2023-03-17T04:22:41.107897886Z stderr F 4096    16K 126 1M  792K    61
2023-03-17T04:22:41.107899543Z stderr F 5120    128K    10  2M  2M  96
2023-03-17T04:22:41.107901258Z stderr F 6144    128K    137 2M  1M  57
2023-03-17T04:22:41.107902956Z stderr F 7168    64K 5   576K    539K    93
2023-03-17T04:22:41.107904622Z stderr F 8192    32K 26k 205M    624K    0
2023-03-17T04:22:41.107906599Z stderr F 10240   64K 63  4M  3M  83
2023-03-17T04:22:41.107908462Z stderr F 12288   64K 11  960K    828K    86
2023-03-17T04:22:41.107910152Z stderr F 14336   128K    379 9M  4M  41
2023-03-17T04:22:41.107919038Z stderr F 16384   64K 30k 470M    5M  1
2023-03-17T04:22:41.107920934Z stderr F Page spans:
2023-03-17T04:22:41.107922616Z stderr F index   size    free    used    spans
2023-03-17T04:22:41.107924248Z stderr F 0   4K  0B  2887M   739k
2023-03-17T04:22:41.107925889Z stderr F 1   8K  8K  1M  162
2023-03-17T04:22:41.107927563Z stderr F 2   16K 16K 3M  166
2023-03-17T04:22:41.107929188Z stderr F 3   32K 32K 1103M   35k
2023-03-17T04:22:41.107930878Z stderr F 4   64K 192K    478M    8k
2023-03-17T04:22:41.107932552Z stderr F 5   128K    0B  47M 373
2023-03-17T04:22:41.107934242Z stderr F 6   256K    0B  1M  5
2023-03-17T04:22:41.107935924Z stderr F 7   512K    0B  0B  0
2023-03-17T04:22:41.107937606Z stderr F 8   1M  0B  0B  0
2023-03-17T04:22:41.107939428Z stderr F 9   2M  0B  2M  1
2023-03-17T04:22:41.107941167Z stderr F 10  4M  0B  8M  2
2023-03-17T04:22:41.107942808Z stderr F 11  8M  0B  0B  0
2023-03-17T04:22:41.107944482Z stderr F 12  16M 0B  48M 3
2023-03-17T04:22:41.107946156Z stderr F 13  32M 0B  0B  0
2023-03-17T04:22:41.107947912Z stderr F 14  64M 0B  64M 1
2023-03-17T04:22:41.107949545Z stderr F 15  128M    0B  0B  0
2023-03-17T04:22:41.107951169Z stderr F 16  256M    0B  0B  0
2023-03-17T04:22:41.107952851Z stderr F 17  512M    0B  0B  0
2023-03-17T04:22:41.107954566Z stderr F 18  1G  0B  0B  0
2023-03-17T04:22:41.107956207Z stderr F 19  2G  0B  0B  0
2023-03-17T04:22:41.107957881Z stderr F 20  4G  0B  0B  0
2023-03-17T04:22:41.10795953Z stderr F 21   8G  0B  0B  0
2023-03-17T04:22:41.107961294Z stderr F 22  16G 0B  0B  0
2023-03-17T04:22:41.107962927Z stderr F 23  32G 0B  0B  0
2023-03-17T04:22:41.107964666Z stderr F 24  64G 0B  0B  0
2023-03-17T04:22:41.107966299Z stderr F 25  128G    0B  0B  0
2023-03-17T04:22:41.107967998Z stderr F 26  256G    0B  0B  0
2023-03-17T04:22:41.107969655Z stderr F 27  512G    0B  0B  0
2023-03-17T04:22:41.107971288Z stderr F 28  1T  0B  0B  0
2023-03-17T04:22:41.107972954Z stderr F 29  2T  0B  0B  0
2023-03-17T04:22:41.107974693Z stderr F 30  4T  0B  0B  0
2023-03-17T04:22:41.107976326Z stderr F 31  8T  0B  0B  0
2023-03-17T04:22:41.107977844Z stderr F 
2023-03-17T04:22:41.10798225Z stderr F ERROR 2023-03-17 04:22:41,107 [shard  9] seastar - Failed to allocate 131072 bytes
2023-03-17T04:22:41.107990209Z stderr F Aborting on shard 9.

Here's the decoded backtrace at the crash (which occurs on shard 9 in the same code that issues the log message above):

2023-03-17T04:22:41.107992301Z stderr F Backtrace:
[Backtrace #0]
{23.1.1-arm64/libexec/redpanda} 0x506864b: void seastar::backtrace<seastar::backtrace_buffer::append_backtrace()::'lambda'(seastar::frame)>(seastar::backtrace_buffer::append_backtrace()::'lambda'(seastar::frame)&&) at /v/build/v_deps_build/seastar-prefix/src/seastar/include/seastar/util/backtrace.hh:59
 (inlined by) seastar::backtrace_buffer::append_backtrace() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:754
 (inlined by) seastar::print_with_backtrace(seastar::backtrace_buffer&, bool) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:784
{23.1.1-arm64/libexec/redpanda} 0x50bbecb: seastar::print_with_backtrace(char const*, bool) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:796
 (inlined by) seastar::sigabrt_action() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:3663
 (inlined by) void seastar::install_oneshot_signal_handler<6, (void (*)())(&(seastar::sigabrt_action()))>()::'lambda'(int, siginfo_t*, void*)::operator()(int, siginfo_t*, void*) const at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:3639
 (inlined by) void seastar::install_oneshot_signal_handler<6, (void (*)())(&(seastar::sigabrt_action()))>()::'lambda'(int, siginfo_t*, void*)::__invoke(int, siginfo_t*, void*) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:3635
LLVMSymbolizer: error reading file: No such file or directory
{linux-vdso.so.1} 0x85f: ?? at ??:0
LLVMSymbolizer: error reading file: No such file or directory
{/opt/redpanda/lib/libc.so.6} 0x86067: ?? at ??:0
{/opt/redpanda/lib/libc.so.6} 0x3e87f: ?? at ??:0
{/opt/redpanda/lib/libc.so.6} 0x2aef7: ?? at ??:0
{23.1.1-arm64/libexec/redpanda} 0x4fe620f: seastar::memory::on_allocation_failure(unsigned long) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:1821
 (inlined by) seastar::memory::allocate_aligned(unsigned long, unsigned long) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:1443
{23.1.1-arm64/libexec/redpanda} 0x4fdc0f7: __libc_posix_memalign at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:1982
{23.1.1-arm64/libexec/redpanda} 0x4f99847: seastar::temporary_buffer<unsigned char>::aligned(unsigned long, unsigned long) at /v/build/v_deps_build/seastar-prefix/src/seastar/include/seastar/core/temporary_buffer.hh:209
 (inlined by) seastar::internal::file_read_state<unsigned char>::file_read_state(unsigned long, unsigned long, unsigned long, unsigned long, unsigned long, seastar::io_intent*) at /v/build/v_deps_build/seastar-prefix/src/seastar/include/seastar/core/internal/read_state.hh:33
 (inlined by) seastar::lw_shared_ptr_no_esft<seastar::internal::file_read_state<unsigned char> >::lw_shared_ptr_no_esft<unsigned long&, unsigned long&, unsigned long&, unsigned int&, unsigned int&, seastar::io_intent*&>(unsigned long&, unsigned long&, unsigned long&, unsigned int&, unsigned int&, seastar::io_intent*&) at /v/build/v_deps_build/seastar-prefix/src/seastar/include/seastar/core/shared_ptr.hh:165
 (inlined by) seastar::lw_shared_ptr<seastar::internal::file_read_state<unsigned char> > seastar::lw_shared_ptr<seastar::internal::file_read_state<unsigned char> >::make<unsigned long&, unsigned long&, unsigned long&, unsigned int&, unsigned int&, seastar::io_intent*&>(unsigned long&, unsigned long&, unsigned long&, unsigned int&, unsigned int&, seastar::io_intent*&) at /v/build/v_deps_build/seastar-prefix/src/seastar/include/seastar/core/shared_ptr.hh:268
 (inlined by) seastar::lw_shared_ptr<seastar::internal::file_read_state<unsigned char> > seastar::make_lw_shared<seastar::internal::file_read_state<unsigned char>, unsigned long&, unsigned long&, unsigned long&, unsigned int&, unsigned int&, seastar::io_intent*&>(unsigned long&, unsigned long&, unsigned long&, unsigned int&, unsigned int&, seastar::io_intent*&) at /v/build/v_deps_build/seastar-prefix/src/seastar/include/seastar/core/shared_ptr.hh:418
 (inlined by) seastar::posix_file_impl::do_dma_read_bulk(unsigned long, unsigned long, seastar::io_priority_class const&, seastar::io_intent*) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/file.cc:476
{23.1.1-arm64/libexec/redpanda} 0x4fa635f: seastar::file::dma_read_bulk_impl(unsigned long, unsigned long, seastar::io_priority_class const&, seastar::io_intent*) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/file.cc:1170
{23.1.1-arm64/libexec/redpanda} 0x4fcb1e7: seastar::future<seastar::temporary_buffer<char> > seastar::file::dma_read_bulk<char>(unsigned long, unsigned long, seastar::io_priority_class const&, seastar::io_intent*) at /v/build/v_deps_build/seastar-prefix/src/seastar/include/seastar/core/file.hh:545
 (inlined by) seastar::file_data_source_impl::issue_read_aheads(unsigned int)::'lambda'()::operator()() const at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/fstream.cc:295
 (inlined by) seastar::future<seastar::temporary_buffer<char> > seastar::futurize<seastar::future<seastar::temporary_buffer<char> > >::invoke<seastar::file_data_source_impl::issue_read_aheads(unsigned int)::'lambda'()>(seastar::file_data_source_impl::issue_read_aheads(unsigned int)::'lambda'()&&) at /v/build/v_deps_build/seastar-prefix/src/seastar/include/seastar/core/future.hh:2149
 (inlined by) auto seastar::futurize_invoke<seastar::file_data_source_impl::issue_read_aheads(unsigned int)::'lambda'()>(seastar::file_data_source_impl::issue_read_aheads(unsigned int)::'lambda'()&&) at /v/build/v_deps_build/seastar-prefix/src/seastar/include/seastar/core/future.hh:2180
 (inlined by) seastar::file_data_source_impl::issue_read_aheads(unsigned int) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/fstream.cc:294
{23.1.1-arm64/libexec/redpanda} 0x4fc9f43: seastar::file_data_source_impl::get() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/fstream.cc:211
{23.1.1-arm64/libexec/redpanda} 0x4f1f5bf: seastar::data_source::get() at /vectorized/include/seastar/core/iostream.hh:72
 (inlined by) seastar::input_stream<char>::read_up_to(unsigned long) at /vectorized/include/seastar/core/iostream-impl.hh:261
 (inlined by) read_iobuf_exactly(seastar::input_stream<char>&, unsigned long)::$_3::operator()(iobuf&, unsigned long&) const::'lambda0'()::operator()() const at /var/lib/buildkite-agent/builds/buildkite-arm64-builders-i-0ce0a1f3f61d30bef-1/redpanda/redpanda/src/v/bytes/iobuf.cc:64
{23.1.1-arm64/libexec/redpanda} 0x4f1b35f: seastar::future<void> seastar::futurize<seastar::future<void> >::invoke<read_iobuf_exactly(seastar::input_stream<char>&, unsigned long)::$_3::operator()(iobuf&, unsigned long&) const::'lambda0'()&>(read_iobuf_exactly(seastar::input_stream<char>&, unsigned long)::$_3::operator()(iobuf&, unsigned long&) const::'lambda0'()&) at /vectorized/include/seastar/core/future.hh:2149
 (inlined by) auto seastar::futurize_invoke<read_iobuf_exactly(seastar::input_stream<char>&, unsigned long)::$_3::operator()(iobuf&, unsigned long&) const::'lambda0'()&>(read_iobuf_exactly(seastar::input_stream<char>&, unsigned long)::$_3::operator()(iobuf&, unsigned long&) const::'lambda0'()&) at /vectorized/include/seastar/core/future.hh:2180
 (inlined by) seastar::future<void> seastar::do_until<read_iobuf_exactly(seastar::input_stream<char>&, unsigned long)::$_3::operator()(iobuf&, unsigned long&) const::'lambda0'(), read_iobuf_exactly(seastar::input_stream<char>&, unsigned long)::$_3::operator()(iobuf&, unsigned long&) const::'lambda'()>(read_iobuf_exactly(seastar::input_stream<char>&, unsigned long)::$_3::operator()(iobuf&, unsigned long&) const::'lambda'(), read_iobuf_exactly(seastar::input_stream<char>&, unsigned long)::$_3::operator()(iobuf&, unsigned long&) const::'lambda0'()) at /vectorized/include/seastar/core/loop.hh:343
 (inlined by) read_iobuf_exactly(seastar::input_stream<char>&, unsigned long)::$_3::operator()(iobuf&, unsigned long&) const at /var/lib/buildkite-agent/builds/buildkite-arm64-builders-i-0ce0a1f3f61d30bef-1/redpanda/redpanda/src/v/bytes/iobuf.cc:61
 (inlined by) decltype(static_cast<read_iobuf_exactly(seastar::input_stream<char>&, unsigned long)::$_3&>(fp)(static_cast<iobuf&>(fp0), static_cast<unsigned long&>(fp0))) std::__1::__invoke_constexpr<read_iobuf_exactly(seastar::input_stream<char>&, unsigned long)::$_3&, iobuf&, unsigned long&>(read_iobuf_exactly(seastar::input_stream<char>&, unsigned long)::$_3&, iobuf&, unsigned long&) at /vectorized/llvm/bin/../include/c++/v1/type_traits:3648
 (inlined by) decltype(auto) std::__1::__apply_tuple_impl<read_iobuf_exactly(seastar::input_stream<char>&, unsigned long)::$_3&, std::__1::tuple<iobuf, unsigned long>&, 0ul, 1ul>(read_iobuf_exactly(seastar::input_stream<char>&, unsigned long)::$_3&, std::__1::tuple<iobuf, unsigned long>&, std::__1::__tuple_indices<0ul, 1ul>) at /vectorized/llvm/bin/../include/c++/v1/tuple:1595
 (inlined by) decltype(auto) std::__1::apply<read_iobuf_exactly(seastar::input_stream<char>&, unsigned long)::$_3&, std::__1::tuple<iobuf, unsigned long>&>(read_iobuf_exactly(seastar::input_stream<char>&, unsigned long)::$_3&, std::__1::tuple<iobuf, unsigned long>&) at /vectorized/llvm/bin/../include/c++/v1/tuple:1604
 (inlined by) auto seastar::internal::do_with_impl<iobuf, unsigned long&, read_iobuf_exactly(seastar::input_stream<char>&, unsigned long)::$_3>(iobuf&&, unsigned long&, read_iobuf_exactly(seastar::input_stream<char>&, unsigned long)::$_3&&) at /vectorized/include/seastar/core/do_with.hh:96
 (inlined by) seastar::future<iobuf> seastar::futurize<seastar::future<iobuf> >::invoke<seastar::future<iobuf> (*&)(iobuf&&, unsigned long&, read_iobuf_exactly(seastar::input_stream<char>&, unsigned long)::$_3&&), iobuf, unsigned long&, read_iobuf_exactly(seastar::input_stream<char>&, unsigned long)::$_3>(seastar::future<iobuf> (*&)(iobuf&&, unsigned long&, read_iobuf_exactly(seastar::input_stream<char>&, unsigned long)::$_3&&), iobuf&&, unsigned long&, read_iobuf_exactly(seastar::input_stream<char>&, unsigned long)::$_3&&) at /vectorized/include/seastar/core/future.hh:2149
 (inlined by) auto seastar::futurize_invoke<seastar::future<iobuf> (*&)(iobuf&&, unsigned long&, read_iobuf_exactly(seastar::input_stream<char>&, unsigned long)::$_3&&), iobuf, unsigned long&, read_iobuf_exactly(seastar::input_stream<char>&, unsigned long)::$_3>(seastar::future<iobuf> (*&)(iobuf&&, unsigned long&, read_iobuf_exactly(seastar::input_stream<char>&, unsigned long)::$_3&&), iobuf&&, unsigned long&, read_iobuf_exactly(seastar::input_stream<char>&, unsigned long)::$_3&&) at /vectorized/include/seastar/core/future.hh:2180
 (inlined by) auto seastar::do_with<iobuf, unsigned long&, read_iobuf_exactly(seastar::input_stream<char>&, unsigned long)::$_3>(iobuf&&, unsigned long&, read_iobuf_exactly(seastar::input_stream<char>&, unsigned long)::$_3&&) at /vectorized/include/seastar/core/do_with.hh:131
 (inlined by) read_iobuf_exactly(seastar::input_stream<char>&, unsigned long) at /var/lib/buildkite-agent/builds/buildkite-arm64-builders-i-0ce0a1f3f61d30bef-1/redpanda/redpanda/src/v/bytes/iobuf.cc:60
{23.1.1-arm64/libexec/redpanda} 0x47c5d3f: storage::verify_read_iobuf(seastar::input_stream<char>&, unsigned long, char const*, bool) at /var/lib/buildkite-agent/builds/buildkite-arm64-builders-i-0ce0a1f3f61d30bef-1/redpanda/redpanda/src/v/storage/parser.cc:88
{23.1.1-arm64/libexec/redpanda} 0x47caa73: storage::continuous_batch_parser::consume_records() at /var/lib/buildkite-agent/builds/buildkite-arm64-builders-i-0ce0a1f3f61d30bef-1/redpanda/redpanda/src/v/storage/parser.cc:248
 (inlined by) storage::continuous_batch_parser::consume_one()::$_0::operator()(boost::outcome_v2::basic_result<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, void> >) const at /var/lib/buildkite-agent/builds/buildkite-arm64-builders-i-0ce0a1f3f61d30bef-1/redpanda/redpanda/src/v/storage/parser.cc:231
{23.1.1-arm64/libexec/redpanda} 0x47c777b: seastar::future<boost::outcome_v2::basic_result<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, void> > > seastar::futurize<seastar::future<boost::outcome_v2::basic_result<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, void> > > >::invoke<storage::continuous_batch_parser::consume_one()::$_0, boost::outcome_v2::basic_result<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, void> > >(storage::continuous_batch_parser::consume_one()::$_0&&, boost::outcome_v2::basic_result<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, void> >&&) at /vectorized/include/seastar/core/future.hh:2149
 (inlined by) seastar::future<boost::outcome_v2::basic_result<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, void> > > seastar::future<boost::outcome_v2::basic_result<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, void> > >::then_impl<storage::continuous_batch_parser::consume_one()::$_0, seastar::future<boost::outcome_v2::basic_result<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, void> > > >(storage::continuous_batch_parser::consume_one()::$_0&&) at /vectorized/include/seastar/core/future.hh:1615
 (inlined by) seastar::internal::future_result<storage::continuous_batch_parser::consume_one()::$_0, boost::outcome_v2::basic_result<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, void> > >::future_type seastar::internal::call_then_impl<seastar::future<boost::outcome_v2::basic_result<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, void> > > >::run<storage::continuous_batch_parser::consume_one()::$_0>(seastar::future<boost::outcome_v2::basic_result<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, void> > >&, storage::continuous_batch_parser::consume_one()::$_0&&) at /vectorized/include/seastar/core/future.hh:1248
 (inlined by) seastar::future<boost::outcome_v2::basic_result<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, void> > > seastar::future<boost::outcome_v2::basic_result<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, void> > >::then<storage::continuous_batch_parser::consume_one()::$_0, seastar::future<boost::outcome_v2::basic_result<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, void> > > >(storage::continuous_batch_parser::consume_one()::$_0&&) at /vectorized/include/seastar/core/future.hh:1534
 (inlined by) storage::continuous_batch_parser::consume_one() at /var/lib/buildkite-agent/builds/buildkite-arm64-builders-i-0ce0a1f3f61d30bef-1/redpanda/redpanda/src/v/storage/parser.cc:224
 (inlined by) storage::continuous_batch_parser::consume()::$_2::operator()() const at /var/lib/buildkite-agent/builds/buildkite-arm64-builders-i-0ce0a1f3f61d30bef-1/redpanda/redpanda/src/v/storage/parser.cc:264
 (inlined by) seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > seastar::futurize<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > >::invoke<storage::continuous_batch_parser::consume()::$_2&>(storage::continuous_batch_parser::consume()::$_2&) at /vectorized/include/seastar/core/future.hh:2149
 (inlined by) seastar::future<void> seastar::repeat<storage::continuous_batch_parser::consume()::$_2>(storage::continuous_batch_parser::consume()::$_2&&) at /vectorized/include/seastar/core/loop.hh:120
 (inlined by) storage::continuous_batch_parser::consume() at /var/lib/buildkite-agent/builds/buildkite-arm64-builders-i-0ce0a1f3f61d30bef-1/redpanda/redpanda/src/v/storage/parser.cc:263
{23.1.1-arm64/libexec/redpanda} 0x47b8053: storage::log_segment_batch_reader::read_some(std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > >) at /var/lib/buildkite-agent/builds/buildkite-arm64-builders-i-0ce0a1f3f61d30bef-1/redpanda/redpanda/src/v/storage/log_reader.cc:206
{23.1.1-arm64/libexec/redpanda} 0x47ba52b: storage::log_reader::do_load_slice(std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > >)::$_8::operator()() const at /var/lib/buildkite-agent/builds/buildkite-arm64-builders-i-0ce0a1f3f61d30bef-1/redpanda/redpanda/src/v/storage/log_reader.cc:308
 (inlined by) seastar::future<boost::outcome_v2::basic_result<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch> >, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch> >, std::__1::error_code, void> > > seastar::futurize<seastar::future<boost::outcome_v2::basic_result<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch> >, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch> >, std::__1::error_code, void> > > >::invoke<storage::log_reader::do_load_slice(std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > >)::$_8>(storage::log_reader::do_load_slice(std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > >)::$_8&&) at /vectorized/include/seastar/core/future.hh:2149
 (inlined by) seastar::future<boost::outcome_v2::basic_result<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch> >, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch> >, std::__1::error_code, void> > > seastar::futurize<seastar::future<boost::outcome_v2::basic_result<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch> >, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch> >, std::__1::error_code, void> > > >::invoke<storage::log_reader::do_load_slice(std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > >)::$_8>(storage::log_reader::do_load_slice(std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > >)::$_8&&, seastar::internal::monostate) at /vectorized/include/seastar/core/future.hh:1993
 (inlined by) seastar::future<boost::outcome_v2::basic_result<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch> >, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch> >, std::__1::error_code, void> > > seastar::future<void>::then_impl<storage::log_reader::do_load_slice(std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > >)::$_8, seastar::future<boost::outcome_v2::basic_result<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch> >, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch> >, std::__1::error_code, void> > > >(storage::log_reader::do_load_slice(std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > >)::$_8&&) at /vectorized/include/seastar/core/future.hh:1615
 (inlined by) seastar::internal::future_result<storage::log_reader::do_load_slice(std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > >)::$_8, void>::future_type seastar::internal::call_then_impl<seastar::future<void> >::run<storage::log_reader::do_load_slice(std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > >)::$_8>(seastar::future<void>&, storage::log_reader::do_load_slice(std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > >)::$_8&&) at /vectorized/include/seastar/core/future.hh:1248
 (inlined by) seastar::future<boost::outcome_v2::basic_result<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch> >, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch> >, std::__1::error_code, void> > > seastar::future<void>::then<storage::log_reader::do_load_slice(std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > >)::$_8, seastar::future<boost::outcome_v2::basic_result<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch> >, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch> >, std::__1::error_code, void> > > >(storage::log_reader::do_load_slice(std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > >)::$_8&&) at /vectorized/include/seastar/core/future.hh:1534
 (inlined by) storage::log_reader::do_load_slice(std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > >) at /var/lib/buildkite-agent/builds/buildkite-arm64-builders-i-0ce0a1f3f61d30bef-1/redpanda/redpanda/src/v/storage/log_reader.cc:308
{23.1.1-arm64/libexec/redpanda} 0x23de303: model::record_batch_reader::impl::load_slice(std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > >) at /var/lib/buildkite-agent/builds/buildkite-arm64-builders-i-0ce0a1f3f61d30bef-1/redpanda/redpanda/src/v/model/record_batch_reader.h:116
 (inlined by) auto model::record_batch_reader::impl::do_action<raft::state_machine::batch_applicator, auto model::record_batch_reader::impl::do_consume<raft::state_machine::batch_applicator>(raft::state_machine::batch_applicator&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > >)::'lambda'(raft::state_machine::batch_applicator&)>(raft::state_machine::batch_applicator&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > >, auto model::record_batch_reader::impl::do_consume<raft::state_machine::batch_applicator>(raft::state_machine::batch_applicator&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > >)::'lambda'(raft::state_machine::batch_applicator&)&&)::'lambda'()::operator()() const at /var/lib/buildkite-agent/builds/buildkite-arm64-builders-i-0ce0a1f3f61d30bef-1/redpanda/redpanda/src/v/model/record_batch_reader.h:159
 (inlined by) seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > seastar::futurize<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > >::invoke<auto model::record_batch_reader::impl::do_action<raft::state_machine::batch_applicator, auto model::record_batch_reader::impl::do_consume<raft::state_machine::batch_applicator>(raft::state_machine::batch_applicator&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > >)::'lambda'(raft::state_machine::batch_applicator&)>(raft::state_machine::batch_applicator&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > >, auto model::record_batch_reader::impl::do_consume<raft::state_machine::batch_applicator>(raft::state_machine::batch_applicator&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > >)::'lambda'(raft::state_machine::batch_applicator&)&&)::'lambda'()&>(raft::state_machine::batch_applicator&&) at /vectorized/include/seastar/core/future.hh:2149
 (inlined by) auto seastar::futurize_invoke<auto model::record_batch_reader::impl::do_action<raft::state_machine::batch_applicator, auto model::record_batch_reader::impl::do_consume<raft::state_machine::batch_applicator>(raft::state_machine::batch_applicator&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > >)::'lambda'(raft::state_machine::batch_applicator&)>(raft::state_machine::batch_applicator&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > >, auto model::record_batch_reader::impl::do_consume<raft::state_machine::batch_applicator>(raft::state_machine::batch_applicator&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > >)::'lambda'(raft::state_machine::batch_applicator&)&&)::'lambda'()&>(raft::state_machine::batch_applicator&&) at /vectorized/include/seastar/core/future.hh:2180
{23.1.1-arm64/libexec/redpanda} 0x23ddc47: seastar::internal::repeater<auto model::record_batch_reader::impl::do_action<raft::state_machine::batch_applicator, auto model::record_batch_reader::impl::do_consume<raft::state_machine::batch_applicator>(raft::state_machine::batch_applicator&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > >)::'lambda'(raft::state_machine::batch_applicator&)>(raft::state_machine::batch_applicator&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > >, auto model::record_batch_reader::impl::do_consume<raft::state_machine::batch_applicator>(raft::state_machine::batch_applicator&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > >)::'lambda'(raft::state_machine::batch_applicator&)&&)::'lambda'()>::run_and_dispose() at /vectorized/include/seastar/core/loop.hh:73
{23.1.1-arm64/libexec/redpanda} 0x5081d83: seastar::reactor::run_tasks(seastar::reactor::task_queue&) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:2330
 (inlined by) seastar::reactor::run_some_tasks() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:2737
{23.1.1-arm64/libexec/redpanda} 0x5084beb: seastar::reactor::do_run() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:2906
{23.1.1-arm64/libexec/redpanda} 0x50bd1ef: seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_95::operator()() const at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:4140
 (inlined by) decltype(static_cast<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_95&>(fp)()) std::__1::__invoke<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_95&>(seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_95&) at /vectorized/llvm/bin/../include/c++/v1/type_traits:3640
 (inlined by) void std::__1::__invoke_void_return_wrapper<void, true>::__call<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_95&>(seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_95&) at /vectorized/llvm/bin/../include/c++/v1/__functional/invoke.h:61
 (inlined by) std::__1::__function::__alloc_func<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_95, std::__1::allocator<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_95>, void ()>::operator()() at /vectorized/llvm/bin/../include/c++/v1/__functional/function.h:180
 (inlined by) std::__1::__function::__func<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_95, std::__1::allocator<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_95>, void ()>::operator()() at /vectorized/llvm/bin/../include/c++/v1/__functional/function.h:354
{23.1.1-arm64/libexec/redpanda} 0x502fbdb: std::__1::__function::__value_func<void ()>::operator()() const at /vectorized/llvm/bin/../include/c++/v1/__functional/function.h:507
 (inlined by) std::__1::function<void ()>::operator()() const at /vectorized/llvm/bin/../include/c++/v1/__functional/function.h:1184
 (inlined by) seastar::posix_thread::start_routine(void*) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/posix.cc:73
{/opt/redpanda/lib/libc.so.6} 0x843b7: ?? at ??:0
{/opt/redpanda/lib/libc.so.6} 0xef2db: ?? at ??:0

It looks to be like there may be an infinite loop or other problem which prevents forward progress and eventually enough memory accumulates (e.g., via background jobs that cannot drain fast enough) that the OOM ensues.

What should have happened instead?

Not crash.

How to reproduce the issue?

Not sure. It occurrred under a high load test with both producers and consumers active.

Additional information

Full log attached. 0.log

[travisdowns]

Update: it does seem like something weird is going on with shard 9 for hours prior to the crash. Consider the live objects (heap allocated objects) graph prior to the crash:

The crash occurs at 4:23 near the far right of the chart. Shard 9 (the highest line) is accumulating objects at a rapid clip and there was an earlier similar case earlier: this first drop was when I stopped one test (so the clients disconnect) and started another one.

It seems like we are leaking/accumulating objects at least while clients remain connected.

[travisdowns]

Immediately before the crash the reads from disk skyrockets, consistent with the idea that we are doing a lot of reads from disk. Maybe the parser_errc::end_of_stream lo line is just reflecting that there are a lot of reads and we get to the end of many segments

At the same time the batch cache hit rate plunges to 7% from 100%. So maybe memory pressure from the accumulation of objects has become so acute at this point that we don't hit in the batch cache anymore and that causes the heavy read activity, but the actual problem is the memory leak/accumulation.

[travisdowns]

Metrics covering the range of the crash with the relevant instance selected:

https://dev.bamboo.monitoring.dev.vectorized.cloud/grafana/d/f8SxhywGk/redpanda-clusters-v1?orgId=1&from=1679026684698&to=1679027340391&var-node=192.168.30.50:9644&var-node_shard=All&var-aggr_criteria=shard&var-data_cluster=amp13-3

[travisdowns]

After the node came back up, it crashed again 2.5 hours later, with the symptoms being the same.

Full log for the second crash

[travisdowns]

A different node also crashed, in between the two crashes of node-12, with the same symptoms, though the backtrace was different, as shown below:

2023-03-17T05:51:03.383364961Z stderr F Backtrace:
[Backtrace #0]
{23.1.1-arm64/libexec/redpanda} 0x506864b: void seastar::backtrace<seastar::backtrace_buffer::append_backtrace()::'lambda'(seastar::frame)>(seastar::backtrace_buffer::append_backtrace()::'lambda'(seastar::frame)&&) at /v/build/v_deps_build/seastar-prefix/src/seastar/include/seastar/util/backtrace.hh:59
 (inlined by) seastar::backtrace_buffer::append_backtrace() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:754
 (inlined by) seastar::print_with_backtrace(seastar::backtrace_buffer&, bool) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:784
{23.1.1-arm64/libexec/redpanda} 0x50bbecb: seastar::print_with_backtrace(char const*, bool) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:796
 (inlined by) seastar::sigabrt_action() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:3663
 (inlined by) void seastar::install_oneshot_signal_handler<6, (void (*)())(&(seastar::sigabrt_action()))>()::'lambda'(int, siginfo_t*, void*)::operator()(int, siginfo_t*, void*) const at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:3639
 (inlined by) void seastar::install_oneshot_signal_handler<6, (void (*)())(&(seastar::sigabrt_action()))>()::'lambda'(int, siginfo_t*, void*)::__invoke(int, siginfo_t*, void*) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:3635
LLVMSymbolizer: error reading file: No such file or directory
{linux-vdso.so.1} 0x85f: ?? at ??:0
LLVMSymbolizer: error reading file: No such file or directory
{/opt/redpanda/lib/libc.so.6} 0x86067: ?? at ??:0
{/opt/redpanda/lib/libc.so.6} 0x3e87f: ?? at ??:0
{/opt/redpanda/lib/libc.so.6} 0x2aef7: ?? at ??:0
{23.1.1-arm64/libexec/redpanda} 0x4fe4d27: seastar::memory::on_allocation_failure(unsigned long) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:1821
 (inlined by) seastar::memory::allocate(unsigned long) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:1410
{23.1.1-arm64/libexec/redpanda} 0x4ff169f: __libc_malloc at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:1860
{23.1.1-arm64/libexec/redpanda} 0x22d9573: seastar::temporary_buffer<char>::temporary_buffer(unsigned long) at /vectorized/include/seastar/core/temporary_buffer.hh:73
 (inlined by) iobuf::create_new_fragment(unsigned long) at /var/lib/buildkite-agent/builds/buildkite-arm64-builders-i-0ce0a1f3f61d30bef-1/redpanda/redpanda/src/v/bytes/iobuf.h:235
 (inlined by) iobuf::append(char const*, unsigned long) at /var/lib/buildkite-agent/builds/buildkite-arm64-builders-i-0ce0a1f3f61d30bef-1/redpanda/redpanda/src/v/bytes/iobuf.h:289
 (inlined by) iobuf::append(seastar::temporary_buffer<char>) at /var/lib/buildkite-agent/builds/buildkite-arm64-builders-i-0ce0a1f3f61d30bef-1/redpanda/redpanda/src/v/bytes/iobuf.h:307
 (inlined by) iobuf::append(iobuf)::'lambda'(details::io_fragment*)::operator()(details::io_fragment*) const at /var/lib/buildkite-agent/builds/buildkite-arm64-builders-i-0ce0a1f3f61d30bef-1/redpanda/redpanda/src/v/bytes/iobuf.h:326
 (inlined by) void boost::intrusive::list_impl<boost::intrusive::mhtraits<details::io_fragment, boost::intrusive::list_member_hook<boost::intrusive::link_mode<(boost::intrusive::link_mode_type)1> >, &(details::io_fragment::hook)>, unsigned long, false, void>::pop_front_and_dispose<iobuf::append(iobuf)::'lambda'(details::io_fragment*)>(iobuf::append(iobuf)::'lambda'(details::io_fragment*)) at /vectorized/include/boost/intrusive/list.hpp:357
{23.1.1-arm64/libexec/redpanda} 0x343c0bf: iobuf::append(iobuf) at /var/lib/buildkite-agent/builds/buildkite-arm64-builders-i-0ce0a1f3f61d30bef-1/redpanda/redpanda/src/v/bytes/iobuf.h:325
 (inlined by) kafka::response_writer::write(std::__1::optional<iobuf>&&) at /var/lib/buildkite-agent/builds/buildkite-arm64-builders-i-0ce0a1f3f61d30bef-1/redpanda/redpanda/src/v/kafka/protocol/response_writer.h:184
{23.1.1-arm64/libexec/redpanda} 0x32f0303: kafka::response_writer::write(std::__1::optional<kafka::batch_reader>&) at /var/lib/buildkite-agent/builds/buildkite-arm64-builders-i-0ce0a1f3f61d30bef-1/redpanda/redpanda/src/v/kafka/protocol/response_writer.h:209
 (inlined by) kafka::fetch_response_data::encode(kafka::response_writer&, detail::base_named_type<short, kafka::kafka_api_version, std::__1::integral_constant<bool, true> >)::$_192::operator()(kafka::fetchable_topic_response&, kafka::response_writer&) const::'lambda'(kafka::fetchable_partition_response&, kafka::response_writer&)::operator()(kafka::fetchable_partition_response&, kafka::response_writer&) const at /var/lib/buildkite-agent/builds/buildkite-arm64-builders-i-0ce0a1f3f61d30bef-1/redpanda/redpanda/vbuild/release/clang/src/v/kafka/protocol/schemata/protocol.cc:8033
 (inlined by) unsigned int kafka::response_writer::write_array<std::__1::vector<kafka::fetchable_partition_response, std::__1::allocator<kafka::fetchable_partition_response> >, kafka::fetch_response_data::encode(kafka::response_writer&, detail::base_named_type<short, kafka::kafka_api_version, std::__1::integral_constant<bool, true> >)::$_192::operator()(kafka::fetchable_topic_response&, kafka::response_writer&) const::'lambda'(kafka::fetchable_partition_response&, kafka::response_writer&)>(std::__1::vector<kafka::fetchable_partition_response, std::__1::allocator<kafka::fetchable_partition_response> >&, kafka::fetch_response_data::encode(kafka::response_writer&, detail::base_named_type<short, kafka::kafka_api_version, std::__1::integral_constant<bool, true> >)::$_192::operator()(kafka::fetchable_topic_response&, kafka::response_writer&) const::'lambda'(kafka::fetchable_partition_response&, kafka::response_writer&)&&) at /var/lib/buildkite-agent/builds/buildkite-arm64-builders-i-0ce0a1f3f61d30bef-1/redpanda/redpanda/src/v/kafka/protocol/response_writer.h:272
 (inlined by) kafka::fetch_response_data::encode(kafka::response_writer&, detail::base_named_type<short, kafka::kafka_api_version, std::__1::integral_constant<bool, true> >)::$_192::operator()(kafka::fetchable_topic_response&, kafka::response_writer&) const at /var/lib/buildkite-agent/builds/buildkite-arm64-builders-i-0ce0a1f3f61d30bef-1/redpanda/redpanda/vbuild/release/clang/src/v/kafka/protocol/schemata/protocol.cc:8008
 (inlined by) unsigned int kafka::response_writer::write_array<std::__1::vector<kafka::fetchable_topic_response, std::__1::allocator<kafka::fetchable_topic_response> >, kafka::fetch_response_data::encode(kafka::response_writer&, detail::base_named_type<short, kafka::kafka_api_version, std::__1::integral_constant<bool, true> >)::$_192>(std::__1::vector<kafka::fetchable_topic_response, std::__1::allocator<kafka::fetchable_topic_response> >&, kafka::fetch_response_data::encode(kafka::response_writer&, detail::base_named_type<short, kafka::kafka_api_version, std::__1::integral_constant<bool, true> >)::$_192&&) at /var/lib/buildkite-agent/builds/buildkite-arm64-builders-i-0ce0a1f3f61d30bef-1/redpanda/redpanda/src/v/kafka/protocol/response_writer.h:272
 (inlined by) kafka::fetch_response_data::encode(kafka::response_writer&, detail::base_named_type<short, kafka::kafka_api_version, std::__1::integral_constant<bool, true> >) at /var/lib/buildkite-agent/builds/buildkite-arm64-builders-i-0ce0a1f3f61d30bef-1/redpanda/redpanda/vbuild/release/clang/src/v/kafka/protocol/schemata/protocol.cc:8005
{23.1.1-arm64/libexec/redpanda} 0x2a5522b: kafka::fetch_response::encode(kafka::response_writer&, detail::base_named_type<short, kafka::kafka_api_version, std::__1::integral_constant<bool, true> >) at /var/lib/buildkite-agent/builds/buildkite-arm64-builders-i-0ce0a1f3f61d30bef-1/redpanda/redpanda/src/v/kafka/protocol/fetch.h:185
 (inlined by) seastar::future<seastar::foreign_ptr<std::__1::unique_ptr<kafka::response, std::__1::default_delete<kafka::response> > > > kafka::request_context::respond<kafka::fetch_response>(kafka::fetch_response) at /var/lib/buildkite-agent/builds/buildkite-arm64-builders-i-0ce0a1f3f61d30bef-1/redpanda/redpanda/src/v/kafka/server/request_context.h:198
 (inlined by) kafka::op_context::send_response() && at /var/lib/buildkite-agent/builds/buildkite-arm64-builders-i-0ce0a1f3f61d30bef-1/redpanda/redpanda/src/v/kafka/server/handlers/fetch.cc:730
{23.1.1-arm64/libexec/redpanda} 0x2a5f833: kafka::handler_template<kafka::fetch_api, (short)4, (short)11, seastar::future<seastar::foreign_ptr<std::__1::unique_ptr<kafka::response, std::__1::default_delete<kafka::response> > > >, &(kafka::default_estimate_adaptor(unsigned long, kafka::connection_context&))>::handle(kafka::request_context, seastar::smp_service_group)::$_4::operator()(std::__1::unique_ptr<kafka::op_context, std::__1::default_delete<kafka::op_context> >&) const::'lambda0'()::operator()() const at /var/lib/buildkite-agent/builds/buildkite-arm64-builders-i-0ce0a1f3f61d30bef-1/redpanda/redpanda/src/v/kafka/server/handlers/fetch.cc:573
 (inlined by) decltype(static_cast<kafka::handler_template<kafka::fetch_api, (short)4, (short)11, seastar::future<seastar::foreign_ptr<std::__1::unique_ptr<kafka::response, std::__1::default_delete<kafka::response> > > >, &(kafka::default_estimate_adaptor(unsigned long, kafka::connection_context&))>::handle(kafka::request_context, seastar::smp_service_group)::$_4::operator()(std::__1::unique_ptr<kafka::op_context, std::__1::default_delete<kafka::op_context> >&) const::'lambda0'()&>(fp)()) std::__1::__invoke<kafka::handler_template<kafka::fetch_api, (short)4, (short)11, seastar::future<seastar::foreign_ptr<std::__1::unique_ptr<kafka::response, std::__1::default_delete<kafka::response> > > >, &(kafka::default_estimate_adaptor(unsigned long, kafka::connection_context&))>::handle(kafka::request_context, seastar::smp_service_group)::$_4::operator()(std::__1::unique_ptr<kafka::op_context, std::__1::default_delete<kafka::op_context> >&) const::'lambda0'()&>(kafka::handler_template<kafka::fetch_api, (short)4, (short)11, seastar::future<seastar::foreign_ptr<std::__1::unique_ptr<kafka::response, std::__1::default_delete<kafka::response> > > >, &(kafka::default_estimate_adaptor(unsigned long, kafka::connection_context&))>::handle(kafka::request_context, seastar::smp_service_group)::$_4::operator()(std::__1::unique_ptr<kafka::op_context, std::__1::default_delete<kafka::op_context> >&) const::'lambda0'()&) at /vectorized/llvm/bin/../include/c++/v1/type_traits:3640
 (inlined by) std::__1::invoke_result<kafka::handler_template<kafka::fetch_api, (short)4, (short)11, seastar::future<seastar::foreign_ptr<std::__1::unique_ptr<kafka::response, std::__1::default_delete<kafka::response> > > >, &(kafka::default_estimate_adaptor(unsigned long, kafka::connection_context&))>::handle(kafka::request_context, seastar::smp_service_group)::$_4::operator()(std::__1::unique_ptr<kafka::op_context, std::__1::default_delete<kafka::op_context> >&) const::'lambda0'()&>::type std::__1::invoke<kafka::handler_template<kafka::fetch_api, (short)4, (short)11, seastar::future<seastar::foreign_ptr<std::__1::unique_ptr<kafka::response, std::__1::default_delete<kafka::response> > > >, &(kafka::default_estimate_adaptor(unsigned long, kafka::connection_context&))>::handle(kafka::request_context, seastar::smp_service_group)::$_4::operator()(std::__1::unique_ptr<kafka::op_context, std::__1::default_delete<kafka::op_context> >&) const::'lambda0'()&>(kafka::handler_template<kafka::fetch_api, (short)4, (short)11, seastar::future<seastar::foreign_ptr<std::__1::unique_ptr<kafka::response, std::__1::default_delete<kafka::response> > > >, &(kafka::default_estimate_adaptor(unsigned long, kafka::connection_context&))>::handle(kafka::request_context, seastar::smp_service_group)::$_4::operator()(std::__1::unique_ptr<kafka::op_context, std::__1::default_delete<kafka::op_context> >&) const::'lambda0'()&) at /vectorized/llvm/bin/../include/c++/v1/__functional/invoke.h:93
 (inlined by) auto seastar::internal::future_invoke<kafka::handler_template<kafka::fetch_api, (short)4, (short)11, seastar::future<seastar::foreign_ptr<std::__1::unique_ptr<kafka::response, std::__1::default_delete<kafka::response> > > >, &(kafka::default_estimate_adaptor(unsigned long, kafka::connection_context&))>::handle(kafka::request_context, seastar::smp_service_group)::$_4::operator()(std::__1::unique_ptr<kafka::op_context, std::__1::default_delete<kafka::op_context> >&) const::'lambda0'()&, seastar::internal::monostate>(kafka::handler_template<kafka::fetch_api, (short)4, (short)11, seastar::future<seastar::foreign_ptr<std::__1::unique_ptr<kafka::response, std::__1::default_delete<kafka::response> > > >, &(kafka::default_estimate_adaptor(unsigned long, kafka::connection_context&))>::handle(kafka::request_context, seastar::smp_service_group)::$_4::operator()(std::__1::unique_ptr<kafka::op_context, std::__1::default_delete<kafka::op_context> >&) const::'lambda0'()&, seastar::internal::monostate&&) at /vectorized/include/seastar/core/future.hh:1223
 (inlined by) seastar::future<seastar::foreign_ptr<std::__1::unique_ptr<kafka::response, std::__1::default_delete<kafka::response> > > > seastar::future<void>::then_impl_nrvo<kafka::handler_template<kafka::fetch_api, (short)4, (short)11, seastar::future<seastar::foreign_ptr<std::__1::unique_ptr<kafka::response, std::__1::default_delete<kafka::response> > > >, &(kafka::default_estimate_adaptor(unsigned long, kafka::connection_context&))>::handle(kafka::request_context, seastar::smp_service_group)::$_4::operator()(std::__1::unique_ptr<kafka::op_context, std::__1::default_delete<kafka::op_context> >&) const::'lambda0'(), seastar::future<seastar::foreign_ptr<std::__1::unique_ptr<kafka::response, std::__1::default_delete<kafka::response> > > > >(kafka::handler_template<kafka::fetch_api, (short)4, (short)11, seastar::future<seastar::foreign_ptr<std::__1::unique_ptr<kafka::response, std::__1::default_delete<kafka::response> > > >, &(kafka::default_estimate_adaptor(unsigned long, kafka::connection_context&))>::handle(kafka::request_context, seastar::smp_service_group)::$_4::operator()(std::__1::unique_ptr<kafka::op_context, std::__1::default_delete<kafka::op_context> >&) const::'lambda0'()&&)::'lambda'(seastar::internal::promise_base_with_type<seastar::foreign_ptr<std::__1::unique_ptr<kafka::response, std::__1::default_delete<kafka::response> > > >&&, kafka::handler_template<kafka::fetch_api, (short)4, (short)11, seastar::future<seastar::foreign_ptr<std::__1::unique_ptr<kafka::response, std::__1::default_delete<kafka::response> > > >, &(kafka::default_estimate_adaptor(unsigned long, kafka::connection_context&))>::handle(kafka::request_context, seastar::smp_service_group)::$_4::operator()(std::__1::unique_ptr<kafka::op_context, std::__1::default_delete<kafka::op_context> >&) const::'lambda0'()&, seastar::future_state<seastar::internal::monostate>&&)::operator()(seastar::internal::promise_base_with_type<seastar::foreign_ptr<std::__1::unique_ptr<kafka::response, std::__1::default_delete<kafka::response> > > >&&, kafka::handler_template<kafka::fetch_api, (short)4, (short)11, seastar::future<seastar::foreign_ptr<std::__1::unique_ptr<kafka::response, std::__1::default_delete<kafka::response> > > >, &(kafka::default_estimate_adaptor(unsigned long, kafka::connection_context&))>::handle(kafka::request_context, seastar::smp_service_group)::$_4::operator()(std::__1::unique_ptr<kafka::op_context, std::__1::default_delete<kafka::op_context> >&) const::'lambda0'()&, seastar::future_state<seastar::internal::monostate>&&) const::'lambda'()::operator()() const at /vectorized/include/seastar/core/future.hh:1596
 (inlined by) void seastar::futurize<seastar::future<seastar::foreign_ptr<std::__1::unique_ptr<kafka::response, std::__1::default_delete<kafka::response> > > > >::satisfy_with_result_of<seastar::future<seastar::foreign_ptr<std::__1::unique_ptr<kafka::response, std::__1::default_delete<kafka::response> > > > seastar::future<void>::then_impl_nrvo<kafka::handler_template<kafka::fetch_api, (short)4, (short)11, seastar::future<seastar::foreign_ptr<std::__1::unique_ptr<kafka::response, std::__1::default_delete<kafka::response> > > >, &(kafka::default_estimate_adaptor(unsigned long, kafka::connection_context&))>::handle(kafka::request_context, seastar::smp_service_group)::$_4::operator()(std::__1::unique_ptr<kafka::op_context, std::__1::default_delete<kafka::op_context> >&) const::'lambda0'(), seastar::future<seastar::foreign_ptr<std::__1::unique_ptr<kafka::response, std::__1::default_delete<kafka::response> > > > >(kafka::handler_template<kafka::fetch_api, (short)4, (short)11, seastar::future<seastar::foreign_ptr<std::__1::unique_ptr<kafka::response, std::__1::default_delete<kafka::response> > > >, &(kafka::default_estimate_adaptor(unsigned long, kafka::connection_context&))>::handle(kafka::request_context, seastar::smp_service_group)::$_4::operator()(std::__1::unique_ptr<kafka::op_context, std::__1::default_delete<kafka::op_context> >&) const::'lambda0'()&&)::'lambda'(seastar::internal::promise_base_with_type<seastar::foreign_ptr<std::__1::unique_ptr<kafka::response, std::__1::default_delete<kafka::response> > > >&&, kafka::handler_template<kafka::fetch_api, (short)4, (short)11, seastar::future<seastar::foreign_ptr<std::__1::unique_ptr<kafka::response, std::__1::default_delete<kafka::response> > > >, &(kafka::default_estimate_adaptor(unsigned long, kafka::connection_context&))>::handle(kafka::request_context, seastar::smp_service_group)::$_4::operator()(std::__1::unique_ptr<kafka::op_context, std::__1::default_delete<kafka::op_context> >&) const::'lambda0'()&, seastar::future_state<seastar::internal::monostate>&&)::operator()(seastar::internal::promise_base_with_type<seastar::foreign_ptr<std::__1::unique_ptr<kafka::response, std::__1::default_delete<kafka::response> > > >&&, kafka::handler_template<kafka::fetch_api, (short)4, (short)11, seastar::future<seastar::foreign_ptr<std::__1::unique_ptr<kafka::response, std::__1::default_delete<kafka::response> > > >, &(kafka::default_estimate_adaptor(unsigned long, kafka::connection_context&))>::handle(kafka::request_context, seastar::smp_service_group)::$_4::operator()(std::__1::unique_ptr<kafka::op_context, std::__1::default_delete<kafka::op_context> >&) const::'lambda0'()&, seastar::future_state<seastar::internal::monostate>&&) const::'lambda'()>(seastar::internal::promise_base_with_type<seastar::foreign_ptr<std::__1::unique_ptr<kafka::response, std::__1::default_delete<kafka::response> > > >&&, kafka::handler_template<kafka::fetch_api, (short)4, (short)11, seastar::future<seastar::foreign_ptr<std::__1::unique_ptr<kafka::response, std::__1::default_delete<kafka::response> > > >, &(kafka::default_estimate_adaptor(unsigned long, kafka::connection_context&))>::handle(kafka::request_context, seastar::smp_service_group)::$_4::operator()(std::__1::unique_ptr<kafka::op_context, std::__1::default_delete<kafka::op_context> >&) const::'lambda0'()&&) at /vectorized/include/seastar/core/future.hh:2134
 (inlined by) seastar::future<seastar::foreign_ptr<std::__1::unique_ptr<kafka::response, std::__1::default_delete<kafka::response> > > > seastar::future<void>::then_impl_nrvo<kafka::handler_template<kafka::fetch_api, (short)4, (short)11, seastar::future<seastar::foreign_ptr<std::__1::unique_ptr<kafka::response, std::__1::default_delete<kafka::response> > > >, &(kafka::default_estimate_adaptor(unsigned long, kafka::connection_context&))>::handle(kafka::request_context, seastar::smp_service_group)::$_4::operator()(std::__1::unique_ptr<kafka::op_context, std::__1::default_delete<kafka::op_context> >&) const::'lambda0'(), seastar::future<seastar::foreign_ptr<std::__1::unique_ptr<kafka::response, std::__1::default_delete<kafka::response> > > > >(kafka::handler_template<kafka::fetch_api, (short)4, (short)11, seastar::future<seastar::foreign_ptr<std::__1::unique_ptr<kafka::response, std::__1::default_delete<kafka::response> > > >, &(kafka::default_estimate_adaptor(unsigned long, kafka::connection_context&))>::handle(kafka::request_context, seastar::smp_service_group)::$_4::operator()(std::__1::unique_ptr<kafka::op_context, std::__1::default_delete<kafka::op_context> >&) const::'lambda0'()&&)::'lambda'(seastar::internal::promise_base_with_type<seastar::foreign_ptr<std::__1::unique_ptr<kafka::response, std::__1::default_delete<kafka::response> > > >&&, kafka::handler_template<kafka::fetch_api, (short)4, (short)11, seastar::future<seastar::foreign_ptr<std::__1::unique_ptr<kafka::response, std::__1::default_delete<kafka::response> > > >, &(kafka::default_estimate_adaptor(unsigned long, kafka::connection_context&))>::handle(kafka::request_context, seastar::smp_service_group)::$_4::operator()(std::__1::unique_ptr<kafka::op_context, std::__1::default_delete<kafka::op_context> >&) const::'lambda0'()&, seastar::future_state<seastar::internal::monostate>&&)::operator()(seastar::internal::promise_base_with_type<seastar::foreign_ptr<std::__1::unique_ptr<kafka::response, std::__1::default_delete<kafka::response> > > >&&, kafka::handler_template<kafka::fetch_api, (short)4, (short)11, seastar::future<seastar::foreign_ptr<std::__1::unique_ptr<kafka::response, std::__1::default_delete<kafka::response> > > >, &(kafka::default_estimate_adaptor(unsigned long, kafka::connection_context&))>::handle(kafka::request_context, seastar::smp_service_group)::$_4::operator()(std::__1::unique_ptr<kafka::op_context, std::__1::default_delete<kafka::op_context> >&) const::'lambda0'()&, seastar::future_state<seastar::internal::monostate>&&) const at /vectorized/include/seastar/core/future.hh:1589
 (inlined by) seastar::continuation<seastar::internal::promise_base_with_type<seastar::foreign_ptr<std::__1::unique_ptr<kafka::response, std::__1::default_delete<kafka::response> > > >, kafka::handler_template<kafka::fetch_api, (short)4, (short)11, seastar::future<seastar::foreign_ptr<std::__1::unique_ptr<kafka::response, std::__1::default_delete<kafka::response> > > >, &(kafka::default_estimate_adaptor(unsigned long, kafka::connection_context&))>::handle(kafka::request_context, seastar::smp_service_group)::$_4::operator()(std::__1::unique_ptr<kafka::op_context, std::__1::default_delete<kafka::op_context> >&) const::'lambda0'(), seastar::future<seastar::foreign_ptr<std::__1::unique_ptr<kafka::response, std::__1::default_delete<kafka::response> > > > seastar::future<void>::then_impl_nrvo<kafka::handler_template<kafka::fetch_api, (short)4, (short)11, seastar::future<seastar::foreign_ptr<std::__1::unique_ptr<kafka::response, std::__1::default_delete<kafka::response> > > >, &(kafka::default_estimate_adaptor(unsigned long, kafka::connection_context&))>::handle(kafka::request_context, seastar::smp_service_group)::$_4::operator()(std::__1::unique_ptr<kafka::op_context, std::__1::default_delete<kafka::op_context> >&) const::'lambda0'(), seastar::future<seastar::foreign_ptr<std::__1::unique_ptr<kafka::response, std::__1::default_delete<kafka::response> > > > >(kafka::handler_template<kafka::fetch_api, (short)4, (short)11, seastar::future<seastar::foreign_ptr<std::__1::unique_ptr<kafka::response, std::__1::default_delete<kafka::response> > > >, &(kafka::default_estimate_adaptor(unsigned long, kafka::connection_context&))>::handle(kafka::request_context, seastar::smp_service_group)::$_4::operator()(std::__1::unique_ptr<kafka::op_context, std::__1::default_delete<kafka::op_context> >&) const::'lambda0'()&&)::'lambda'(seastar::internal::promise_base_with_type<seastar::foreign_ptr<std::__1::unique_ptr<kafka::response, std::__1::default_delete<kafka::response> > > >&&, kafka::handler_template<kafka::fetch_api, (short)4, (short)11, seastar::future<seastar::foreign_ptr<std::__1::unique_ptr<kafka::response, std::__1::default_delete<kafka::response> > > >, &(kafka::default_estimate_adaptor(unsigned long, kafka::connection_context&))>::handle(kafka::request_context, seastar::smp_service_group)::$_4::operator()(std::__1::unique_ptr<kafka::op_context, std::__1::default_delete<kafka::op_context> >&) const::'lambda0'()&, seastar::future_state<seastar::internal::monostate>&&), void>::run_and_dispose() at /vectorized/include/seastar/core/future.hh:781
{23.1.1-arm64/libexec/redpanda} 0x5081d83: seastar::reactor::run_tasks(seastar::reactor::task_queue&) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:2330
 (inlined by) seastar::reactor::run_some_tasks() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:2737
{23.1.1-arm64/libexec/redpanda} 0x5084beb: seastar::reactor::do_run() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:2906
{23.1.1-arm64/libexec/redpanda} 0x50bd1ef: seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_95::operator()() const at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:4140
 (inlined by) decltype(static_cast<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_95&>(fp)()) std::__1::__invoke<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_95&>(seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_95&) at /vectorized/llvm/bin/../include/c++/v1/type_traits:3640
 (inlined by) void std::__1::__invoke_void_return_wrapper<void, true>::__call<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_95&>(seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_95&) at /vectorized/llvm/bin/../include/c++/v1/__functional/invoke.h:61
 (inlined by) std::__1::__function::__alloc_func<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_95, std::__1::allocator<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_95>, void ()>::operator()() at /vectorized/llvm/bin/../include/c++/v1/__functional/function.h:180
 (inlined by) std::__1::__function::__func<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_95, std::__1::allocator<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_95>, void ()>::operator()() at /vectorized/llvm/bin/../include/c++/v1/__functional/function.h:354
{23.1.1-arm64/libexec/redpanda} 0x502fbdb: std::__1::__function::__value_func<void ()>::operator()() const at /vectorized/llvm/bin/../include/c++/v1/__functional/function.h:507
 (inlined by) std::__1::function<void ()>::operator()() const at /vectorized/llvm/bin/../include/c++/v1/__functional/function.h:1184
 (inlined by) seastar::posix_thread::start_routine(void*) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/posix.cc:73

I don't think that means anything: it's just going to be "some" 128K allocation that fails and in this case it's a buffer we are preparing to send back to the client.

[surikbs]

Hi All, i have reached to the similar error on my redpanda clsuter.

i have attached the full log file. osp-redpanda-0_osp-redpanda.log

[travisdowns]

[travisdowns]

The problem is that the replicate batch flusher call fall behind on background flushes if produce load is high enough and the shard is saturated CPU-wise.

The flushes of the batches are actually happening (since every flush flushes the entire replicate batcher) but the background tasks (most of which will flush nothing) enqueued in replicate_batcher::cache_and_wait_for_result accumulate without bound, eventually causing the shard to run out of memory. Above, the orange shard is consistently at 100% CPU and has a linear increase of waiting flush tasks. Some other shards also seem to be increasing but at a much slower rate and/or occasionally dropping in allocated objects, indicating a period where flushing task completion was outpacing creation.

[travisdowns]

Here's the memory sampling dump:

INFO  2023-04-10 20:07:16,939 [shard 11] main - application.cc:398 - Memory dump 124:
Heap dump 33257/33848613 sampled allocs (1 in 1017.7891271010615), 4478528/4799160320 sampled bytes (1 in 1071.593237778127)
Sites non-zero/total: 158/158
    bytes: 777984, count: 3039, hash: 10246010162384358531, Backtrace: 0x52f766b 0x50690b7 0x236c133 0x236b3bf 0x2368ae3 0x23686e7 0x21e5d33 0x41ba0fb 0x41ffb8f 0x41ac8ff 0x41ac103 0x40d21b7 0x2b39023 0x2accdf7 0x2ad1b37 0x50ea3bb 0x50ed223 0x512582b 0x5098213 /opt/redpanda/lib/libc.so.6+0x7d5c7 /opt/redpanda/lib/libc.so.6+0xe5d1b
    bytes: 622592, count: 19, hash: 654376531747597443, Backtrace: 0x52f766b 0x504d6e7 0x5069d8f 0x2281ff3 0x2368d03 0x23686e7 0x21e5d33 0x41ba0fb 0x41ffb8f 0x41ac8ff 0x41ac103 0x40d21b7 0x2b39023 0x2accdf7 0x2ad1b37 0x50ea3bb 0x50ed223 0x512582b 0x5098213 /opt/redpanda/lib/libc.so.6+0x7d5c7 /opt/redpanda/lib/libc.so.6+0xe5d1b
    bytes: 491520, count: 30, hash: 15973778145999529982, Backtrace: 0x52f766b 0x505bbc3 0x504aa63 0x1dfce9f 0x50ea3bb 0x50ed223 0x512582b 0x5098213 /opt/redpanda/lib/libc.so.6+0x7d5c7 /opt/redpanda/lib/libc.so.6+0xe5d1b
    bytes: 240960, count: 3012, hash: 12810156384384336621, Backtrace: 0x52f766b 0x50690b7 0x2368ddf 0x23686e7 0x21e5d33 0x41ba0fb 0x41ffb8f 0x41ac8ff 0x41ac103 0x40d21b7 0x2b39023 0x2accdf7 0x2ad1b37 0x50ea3bb 0x50ed223 0x512582b 0x5098213 /opt/redpanda/lib/libc.so.6+0x7d5c7 /opt/redpanda/lib/libc.so.6+0xe5d1b
    bytes: 230960, count: 2887, hash: 12263796589439293404, Backtrace: 0x52f766b 0x50690b7 0x22822cb 0x2368d03 0x23686e7 0x21e5d33 0x41ba0fb 0x41ffb8f 0x41ac8ff 0x41ac103 0x40d21b7 0x2b39023 0x2accdf7 0x2ad1b37 0x50ea3bb 0x50ed223 0x512582b 0x5098213 /opt/redpanda/lib/libc.so.6+0x7d5c7 /opt/redpanda/lib/libc.so.6+0xe5d1b
    bytes: 200512, count: 3133, hash: 5311949658652777565, Backtrace: 0x52f766b 0x50690b7 0x236984f 0x23686e7 0x21e5d33 0x41ba0fb 0x41ffb8f 0x41ac8ff 0x41ac103 0x40d21b7 0x2b39023 0x2accdf7 0x2ad1b37 0x50ea3bb 0x50ed223 0x512582b 0x5098213 /opt/redpanda/lib/libc.so.6+0x7d5c7 /opt/redpanda/lib/libc.so.6+0xe5d1b
    bytes: 196992, count: 3078, hash: 6416106912508247201, Backtrace: 0x52f766b 0x50690b7 0x2369623 0x23686e7 0x21e5d33 0x41ba0fb 0x41ffb8f 0x41ac8ff 0x41ac103 0x40d21b7 0x2b39023 0x2accdf7 0x2ad1b37 0x50ea3bb 0x50ed223 0x512582b 0x5098213 /opt/redpanda/lib/libc.so.6+0x7d5c7 /opt/redpanda/lib/libc.so.6+0xe5d1b
    bytes: 195200, count: 3050, hash: 16430223063636387105, Backtrace: 0x52f766b 0x50690b7 0x23693a3 0x23686e7 0x21e5d33 0x41ba0fb 0x41ffb8f 0x41ac8ff 0x41ac103 0x40d21b7 0x2b39023 0x2accdf7 0x2ad1b37 0x50ea3bb 0x50ed223 0x512582b 0x5098213 /opt/redpanda/lib/libc.so.6+0x7d5c7 /opt/redpanda/lib/libc.so.6+0xe5d1b
    bytes: 193088, count: 3017, hash: 13770575252087356513, Backtrace: 0x52f766b 0x50690b7 0x23694e3 0x23686e7 0x21e5d33 0x41ba0fb 0x41ffb8f 0x41ac8ff 0x41ac103 0x40d21b7 0x2b39023 0x2accdf7 0x2ad1b37 0x50ea3bb 0x50ed223 0x512582b 0x5098213 /opt/redpanda/lib/libc.so.6+0x7d5c7 /opt/redpanda/lib/libc.so.6+0xe5d1b
    bytes: 192896, count: 3014, hash: 4076512258210121633, Backtrace: 0x52f766b 0x50690b7 0x2369123 0x23686e7 0x21e5d33 0x41ba0fb 0x41ffb8f 0x41ac8ff 0x41ac103 0x40d21b7 0x2b39023 0x2accdf7 0x2ad1b37 0x50ea3bb 0x50ed223 0x512582b 0x5098213 /opt/redpanda/lib/libc.so.6+0x7d5c7 /opt/redpanda/lib/libc.so.6+0xe5d1b
    bytes: 188544, count: 2946, hash: 9289804037616933873, Backtrace: 0x52f766b 0x50690b7 0x2368f73 0x23686e7 0x21e5d33 0x41ba0fb 0x41ffb8f 0x41ac8ff 0x41ac103 0x40d21b7 0x2b39023 0x2accdf7 0x2ad1b37 0x50ea3bb 0x50ed223 0x512582b 0x5098213 /opt/redpanda/lib/libc.so.6+0x7d5c7 /opt/redpanda/lib/libc.so.6+0xe5d1b
    bytes: 183424, count: 2866, hash: 16356732700079447777, Backtrace: 0x52f766b 0x50690b7 0x2369263 0x23686e7 0x21e5d33 0x41ba0fb 0x41ffb8f 0x41ac8ff 0x41ac103 0x40d21b7 0x2b39023 0x2accdf7 0x2ad1b37 0x50ea3bb 0x50ed223 0x512582b 0x5098213 /opt/redpanda/lib/libc.so.6+0x7d5c7 /opt/redpanda/lib/libc.so.6+0xe5d1b
    bytes: 98304, count: 3, hash: 18323502822051799186, Backtrace: 0x52f766b 0x504d6e7 0x5069d8f 0x2281ff3 0x2368d03 0x23686e7 0x21e5d33 0x41ba0fb 0x41ffb8f 0x41ac8ff 0x41ac103 0x40d21b7 0x2b39023 0x2accdf7 0x2ac8377 0x2ac24f3 0x2abf75f 0x283d167 0x282b97f 0x280f853 0x280f08b 0x280963f 0x282064b 0x1dbcf1f 0x50ea3bb 0x50ed223 0x512582b 0x5098213 /opt/redpanda/lib/libc.so.6+0x7d5c7 /opt/redpanda/lib/libc.so.6+0xe5d1b
    bytes: 90112, count: 11, hash: 15996598970894179544, Backtrace: 0x52f766b 0x50690b7 0x49095cf 0x49024d7 0x48df6e3 0x2d20ccf 0x50ea3bb 0x50ed223 0x512582b 0x5098213 /opt/redpanda/lib/libc.so.6+0x7d5c7 /opt/redpanda/lib/libc.so.6+0xe5d1b
    bytes: 57344, count: 7, hash: 4365568195673421876, Backtrace: 0x52f766b 0x50690b7 0x4908a43 0x49024c7 0x48df6e3 0x2d20ccf 0x50ea3bb 0x50ed223 0x512582b 0x5098213 /opt/redpanda/lib/libc.so.6+0x7d5c7 /opt/redpanda/lib/libc.so.6+0xe5d1b
    bytes: 49152, count: 6, hash: 5128355952192277476, Backtrace: 0x52f766b 0x50690b7 0x4904433 0x48fd1bf 0x48d38b7 0x48a45bf 0x48a582f 0x4812f37 0x222ef33 0x222f65b 0x50ea3bb 0x50ed223 0x512582b 0x5098213 /opt/redpanda/lib/libc.so.6+0x7d5c7 /opt/redpanda/lib/libc.so.6+0xe5d1b
    bytes: 32768, count: 1, hash: 4717383159171747977, Backtrace: 0x52f766b 0x504d6e7 0x5069d8f 0x490cd53 0x490ef63 0x47d893f 0x47d9427 0x21e668f 0x24106f7 0x50ea3bb 0x50ed223 0x512582b 0x5098213 /opt/redpanda/lib/libc.so.6+0x7d5c7 /opt/redpanda/lib/libc.so.6+0xe5d1b
    bytes: 32768, count: 1, hash: 1166850261794791049, Backtrace: 0x52f766b 0x504d6e7 0x5069d8f 0x2281ff3 0x2368d03 0x23686e7 0x21e5d33 0x41ba0fb 0x41ffb8f 0x41ac8ff 0x41ac103 0x40d21b7 0x2b39023 0x2accdf7 0x2ac8377 0x2ac24f3 0x2abf75f 0x283d167 0x282b97f 0x280f853 0x280f08b 0x280963f 0x2805857 0x2821b63 0x1dbcf1f 0x50ea3bb 0x50ed223 0x512582b 0x5098213 /opt/redpanda/lib/libc.so.6+0x7d5c7 /opt/redpanda/lib/libc.so.6+0xe5d1b

All of the objects around the ~3000 count mark are associated with item entries in replicate_batcher::_item_cache. The sampling rate is 0.1% so that means around 3 million objects of each type. Several of the other entries are also related, e.g., some of the big but low-count entries are associated with e.g., the backing array (circular_fifo actually) for the lock object that the tasks queue on.

[travisdowns]

When the load stops, the tasks are able to drain (load comes off at 20:07:30):

[travisdowns]

@surikbs - I don't think it's the same issue based on the size of the objects in your memory diagnostics.

Can you open a new issue and post the full disagnostic output that occurs when the OOM occurs as text, not as a screenshot?

[surikbs]

Hi @travisdowns i have created a new issue and attached the logs. https://github.com/redpanda-data/redpanda/issues/9995