Memory disagnostic report can itself run out of memory when debug logging is on

[travisdowns]

Version & Environment

Redpanda version: 23.1.1

What went wrong?

When an out-of-memory situation occurs, we try to log memory diagnostics (see for example https://github.com/redpanda-data/core-internal/issues/99), but this may itself fail if the logging allocates memory and this allocation fails.

The result is that the diagnostics are not output, and also the backtrace is more confusing because it has many extra frames representing the effort to print the diagnostics.

What should have happened instead?

The diagnostics should be printed if at all possible.

See this series which has a fix and test for a similar issue:

https://github.com/scylladb/seastar/commit/0e4139ed6e905c1f347e3f4caf90e227d2795b06

However, the test misses this case because it does not test the outer maybe_dump_memory_diagnostics function but the lower level do_dump_memory_diagnostics, which does not include the tracing of the backtrace which is doing the allocation here. Furthermore, the issue only occurs if debug logging is on, since at less verbose logging levels the concerned message is not printed.

Additional information

Please attach any relevant logs, backtraces, or metric charts.

Example of a backtrace showing recursive OOM:

Backtrace:
  0x5bb8ca6
  0x5c1c6a2
  /opt/redpanda/lib/libc.so.6+0x42abf
  /opt/redpanda/lib/libc.so.6+0x92e3b
  /opt/redpanda/lib/libc.so.6+0x42a15
  /opt/redpanda/lib/libc.so.6+0x2c82e
  0x5b2bbb6
  0x5b3a2b1
  0x1e5c37a
  0x1e6ea12
  0x5e39af1
  0x1e6d0d0
  0x1e6c73d
  0x1e61457
  0x5b545b5
  0x5e4e23c
  0x5e4d4e9
  0x5b2bc21
  0x5b38755
  0x5231a8d
  0x52344e0
  0x513d4bb
  0x5130484
  0x5148e3e
  0x5149cf4
  0x5bd6ddf
  0x5bdaab7
  0x5bd7e89
  0x5afc131
  0x5afa24f
  0x1dbb28e
  0x5eec899
  /opt/redpanda/lib/libc.so.6+0x2d58f
  /opt/redpanda/lib/libc.so.6+0x2d648
  0x1db54e4
=== DECODED ===
Backtrace:
[Backtrace #0]
void seastar::backtrace<seastar::backtrace_buffer::append_backtrace()::'lambda'(seastar::frame)>(seastar::backtrace_buffer::append_backtrace()::'lambda'(seastar::frame)&&) at /v/build/v_deps_build/seastar-prefix/src/seastar/include/seastar/util/backtrace.hh:59
 (inlined by) seastar::backtrace_buffer::append_backtrace() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:754
 (inlined by) seastar::print_with_backtrace(seastar::backtrace_buffer&, bool) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:784
seastar::print_with_backtrace(char const*, bool) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:796
 (inlined by) seastar::sigabrt_action() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:3663
 (inlined by) void seastar::install_oneshot_signal_handler<6, (void (*)())(&seastar::sigabrt_action())>()::'lambda'(int, siginfo_t*, void*)::operator()(int, siginfo_t*, void*) const at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:3639
 (inlined by) void seastar::install_oneshot_signal_handler<6, (void (*)())(&seastar::sigabrt_action())>()::'lambda'(int, siginfo_t*, void*)::__invoke(int, siginfo_t*, void*) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:3635
LLVMSymbolizer: error reading file: No such file or directory
?? at ??:0
?? at ??:0
?? at ??:0
?? at ??:0
seastar::memory::on_allocation_failure(unsigned long) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:1821
 (inlined by) seastar::memory::allocate(unsigned long) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:1410
operator new(unsigned long) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:2079
void* std::__1::__libcpp_operator_new<unsigned long>(unsigned long) at /vectorized/llvm/bin/../include/c++/v1/new:245
 (inlined by) std::__1::__libcpp_allocate(unsigned long, unsigned long) at /vectorized/llvm/bin/../include/c++/v1/new:271
 (inlined by) std::__1::allocator<char>::allocate(unsigned long) at /vectorized/llvm/bin/../include/c++/v1/__memory/allocator.h:105
 (inlined by) std::__1::allocator_traits<std::__1::allocator<char>>::allocate(std::__1::allocator<char>&, unsigned long) at /vectorized/llvm/bin/../include/c++/v1/__memory/allocator_traits.h:262
 (inlined by) fmt::v8::basic_memory_buffer<char, 500ul, std::__1::allocator<char>>::grow(unsigned long) at /vectorized/include/fmt/format.h:791
fmt::v8::detail::buffer<char>::try_reserve(unsigned long) at /vectorized/include/fmt/core.h:846
 (inlined by) void fmt::v8::detail::buffer<char>::append<char>(char const*, char const*) at /vectorized/include/fmt/format.h:633
 (inlined by) fmt::v8::detail::formatbuf<std::__1::basic_streambuf<char, std::__1::char_traits<char>>>::xsputn(char const*, long) at /vectorized/include/fmt/format.h:281
std::__1::basic_streambuf<char, std::__1::char_traits<char>>::sputn(char const*, long) at /vectorized/llvm/bin/../include/c++/v1/streambuf:231
 (inlined by) std::__1::ostreambuf_iterator<char, std::__1::char_traits<char>> std::__1::__pad_and_output<char, std::__1::char_traits<char>>(std::__1::ostreambuf_iterator<char, std::__1::char_traits<char>>, char const*, char const*, char const*, std::__1::ios_base&, char) at /vectorized/llvm/bin/../include/c++/v1/locale:1440
 (inlined by) std::__1::basic_ostream<char, std::__1::char_traits<char>>& std::__1::__put_character_sequence<char, std::__1::char_traits<char>>(std::__1::basic_ostream<char, std::__1::char_traits<char>>&, char const*, unsigned long) at /vectorized/llvm/bin/../include/c++/v1/ostream:726
 (inlined by) std::__1::basic_ostream<char, std::__1::char_traits<char>>& std::__1::operator<<<std::__1::char_traits<char>>(std::__1::basic_ostream<char, std::__1::char_traits<char>>&, char const*) at /vectorized/llvm/bin/../include/c++/v1/ostream:865
 (inlined by) seastar::operator<<(std::__1::basic_ostream<char, std::__1::char_traits<char>>&, seastar::tasktrace const&) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/util/backtrace.cc:122
void fmt::v8::detail::format_value<char, seastar::tasktrace>(fmt::v8::detail::buffer<char>&, seastar::tasktrace const&, fmt::v8::detail::locale_ref) at /vectorized/include/fmt/ostream.h:77
 (inlined by) fmt::v8::appender fmt::v8::detail::fallback_formatter<seastar::tasktrace, char, void>::format<fmt::v8::appender>(seastar::tasktrace const&, fmt::v8::basic_format_context<fmt::v8::appender, char>&) at /vectorized/include/fmt/ostream.h:92
void fmt::v8::detail::value<fmt::v8::basic_format_context<fmt::v8::appender, char>>::format_custom_arg<seastar::tasktrace, fmt::v8::detail::fallback_formatter<seastar::tasktrace, char, void>>(void*, fmt::v8::basic_format_parse_context<char, fmt::v8::detail::error_handler>&, fmt::v8::basic_format_context<fmt::v8::appender, char>&) at /vectorized/include/fmt/core.h:1282
fmt::v8::basic_format_arg<fmt::v8::basic_format_context<fmt::v8::appender, char>>::handle::format(fmt::v8::basic_format_parse_context<char, fmt::v8::detail::error_handler>&, fmt::v8::basic_format_context<fmt::v8::appender, char>&) const at /vectorized/include/fmt/core.h:1558
 (inlined by) fmt::v8::detail::default_arg_formatter<char>::operator()(fmt::v8::basic_format_arg<fmt::v8::basic_format_context<fmt::v8::appender, char>>::handle) at /vectorized/include/fmt/format.h:2202
 (inlined by) decltype(fp(0)) fmt::v8::visit_format_arg<fmt::v8::detail::default_arg_formatter<char>, fmt::v8::basic_format_context<fmt::v8::appender, char>>(fmt::v8::detail::default_arg_formatter<char>&&, fmt::v8::basic_format_arg<fmt::v8::basic_format_context<fmt::v8::appender, char>> const&) at /vectorized/include/fmt/core.h:1622
 (inlined by) void fmt::v8::detail::vformat_to<char>(fmt::v8::detail::buffer<char>&, fmt::v8::basic_string_view<char>, fmt::v8::basic_format_args<fmt::v8::basic_format_context<std::__1::conditional<std::is_same<fmt::v8::type_identity<char>::type, char>::value, fmt::v8::appender, std::__1::back_insert_iterator<fmt::v8::detail::buffer<fmt::v8::type_identity<char>::type>>>::type, fmt::v8::type_identity<char>::type>>, fmt::v8::detail::locale_ref)::format_handler::on_replacement_field(int, char const*) at /vectorized/include/fmt/format.h:2962
 (inlined by) char const* fmt::v8::detail::parse_replacement_field<char, void fmt::v8::detail::vformat_to<char>(fmt::v8::detail::buffer<char>&, fmt::v8::basic_string_view<char>, fmt::v8::basic_format_args<fmt::v8::basic_format_context<std::__1::conditional<std::is_same<fmt::v8::type_identity<char>::type, char>::value, fmt::v8::appender, std::__1::back_insert_iterator<fmt::v8::detail::buffer<fmt::v8::type_identity<char>::type>>>::type, fmt::v8::type_identity<char>::type>>, fmt::v8::detail::locale_ref)::format_handler&>(char const*, char const*, void fmt::v8::detail::vformat_to<char>(fmt::v8::detail::buffer<char>&, fmt::v8::basic_string_view<char>, fmt::v8::basic_format_args<fmt::v8::basic_format_context<std::__1::conditional<std::is_same<fmt::v8::type_identity<char>::type, char>::value, fmt::v8::appender, std::__1::back_insert_iterator<fmt::v8::detail::buffer<fmt::v8::type_identity<char>::type>>>::type, fmt::v8::type_identity<char>::type>>, fmt::v8::detail::locale_ref)::format_handler&) at /vectorized/include/fmt/core.h:2591
void fmt::v8::detail::parse_format_string<false, char, void fmt::v8::detail::vformat_to<char>(fmt::v8::detail::buffer<char>&, fmt::v8::basic_string_view<char>, fmt::v8::basic_format_args<fmt::v8::basic_format_context<std::__1::conditional<std::is_same<fmt::v8::type_identity<char>::type, char>::value, fmt::v8::appender, std::__1::back_insert_iterator<fmt::v8::detail::buffer<fmt::v8::type_identity<char>::type>>>::type, fmt::v8::type_identity<char>::type>>, fmt::v8::detail::locale_ref)::format_handler>(fmt::v8::basic_string_view<char>, void fmt::v8::detail::vformat_to<char>(fmt::v8::detail::buffer<char>&, fmt::v8::basic_string_view<char>, fmt::v8::basic_format_args<fmt::v8::basic_format_context<std::__1::conditional<std::is_same<fmt::v8::type_identity<char>::type, char>::value, fmt::v8::appender, std::__1::back_insert_iterator<fmt::v8::detail::buffer<fmt::v8::type_identity<char>::type>>>::type, fmt::v8::type_identity<char>::type>>, fmt::v8::detail::locale_ref)::format_handler&&) at /vectorized/include/fmt/core.h:2660
 (inlined by) void fmt::v8::detail::vformat_to<char>(fmt::v8::detail::buffer<char>&, fmt::v8::basic_string_view<char>, fmt::v8::basic_format_args<fmt::v8::basic_format_context<std::__1::conditional<std::is_same<fmt::v8::type_identity<char>::type, char>::value, fmt::v8::appender, std::__1::back_insert_iterator<fmt::v8::detail::buffer<fmt::v8::type_identity<char>::type>>>::type, fmt::v8::type_identity<char>::type>>, fmt::v8::detail::locale_ref) at /vectorized/include/fmt/format.h:2988
 (inlined by) seastar::internal::log_buf::inserter_iterator fmt::v8::vformat_to<seastar::internal::log_buf::inserter_iterator, 0>(seastar::internal::log_buf::inserter_iterator, fmt::v8::basic_string_view<char>, fmt::v8::basic_format_args<fmt::v8::basic_format_context<fmt::v8::appender, char>>) at /vectorized/include/fmt/core.h:3128
 (inlined by) seastar::internal::log_buf::inserter_iterator fmt::v8::format_to<seastar::internal::log_buf::inserter_iterator, unsigned long&, seastar::tasktrace, 0>(seastar::internal::log_buf::inserter_iterator, fmt::v8::basic_format_string<char, fmt::v8::type_identity<unsigned long&>::type, fmt::v8::type_identity<seastar::tasktrace>::type>, unsigned long&, seastar::tasktrace&&) at /vectorized/include/fmt/core.h:3148
 (inlined by) void seastar::logger::log<unsigned long&, seastar::tasktrace>(seastar::log_level, seastar::logger::format_info, unsigned long&, seastar::tasktrace&&)::'lambda'(seastar::internal::log_buf::inserter_iterator)::operator()(seastar::internal::log_buf::inserter_iterator) const at /v/build/v_deps_build/seastar-prefix/src/seastar/include/seastar/util/log.hh:222
 (inlined by) seastar::logger::lambda_log_writer<void seastar::logger::log<unsigned long&, seastar::tasktrace>(seastar::log_level, seastar::logger::format_info, unsigned long&, seastar::tasktrace&&)::'lambda'(seastar::internal::log_buf::inserter_iterator)>::operator()(seastar::internal::log_buf::inserter_iterator) at /v/build/v_deps_build/seastar-prefix/src/seastar/include/seastar/util/log.hh:108
seastar::logger::do_log(seastar::log_level, seastar::logger::log_writer&)::$_0::operator()(seastar::internal::log_buf::inserter_iterator) const at /v/build/v_deps_build/seastar-prefix/src/seastar/src/util/log.cc:323
seastar::logger::do_log(seastar::log_level, seastar::logger::log_writer&) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/util/log.cc:335
void seastar::logger::log<unsigned long&, seastar::tasktrace>(seastar::log_level, seastar::logger::format_info, unsigned long&, seastar::tasktrace&&) at /v/build/v_deps_build/seastar-prefix/src/seastar/include/seastar/util/log.hh:227
 (inlined by) void seastar::logger::debug<unsigned long&, seastar::tasktrace>(seastar::logger::format_info, unsigned long&, seastar::tasktrace&&) at /v/build/v_deps_build/seastar-prefix/src/seastar/include/seastar/util/log.hh:370
 (inlined by) seastar::memory::maybe_dump_memory_diagnostics(unsigned long, bool) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:1784
 (inlined by) seastar::memory::on_allocation_failure(unsigned long) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:1817
 (inlined by) seastar::memory::allocate(unsigned long) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:1410
__libc_malloc at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:1860
seastar::temporary_buffer<char>::temporary_buffer(unsigned long) at /vectorized/include/seastar/core/temporary_buffer.hh:73
 (inlined by) storage::batch_cache::range::range(storage::batch_cache_index&) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-086279b42528bcbef-1/redpanda/redpanda/src/v/storage/batch_cache.cc:32
storage::batch_cache::put(storage::batch_cache_index&, model::record_batch const&) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-086279b42528bcbef-1/redpanda/redpanda/src/v/storage/batch_cache.cc:169
storage::batch_cache_index::put(model::record_batch const&) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-086279b42528bcbef-1/redpanda/redpanda/src/v/storage/batch_cache.h:431
storage::segment::cache_put(model::record_batch const&) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-086279b42528bcbef-1/redpanda/redpanda/src/v/storage/segment.h:389
 (inlined by) storage::log_segment_batch_reader::add_one(model::record_batch&&) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-086279b42528bcbef-1/redpanda/redpanda/src/v/storage/log_reader.cc:161
 (inlined by) storage::skipping_consumer::consume_batch_end() at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-086279b42528bcbef-1/redpanda/redpanda/src/v/storage/log_reader.cc:99
storage::continuous_batch_parser::consume_records()::$_1::operator()(boost::outcome_v2::basic_result<iobuf, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<iobuf, std::__1::error_code, void>>) const at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-086279b42528bcbef-1/redpanda/redpanda/src/v/storage/parser.cc:255
 (inlined by) seastar::future<boost::outcome_v2::basic_result<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, void>>> seastar::futurize<boost::outcome_v2::basic_result<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, void>>>::invoke<storage::continuous_batch_parser::consume_records()::$_1, boost::outcome_v2::basic_result<iobuf, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<iobuf, std::__1::error_code, void>>>(storage::continuous_batch_parser::consume_records()::$_1&&, boost::outcome_v2::basic_result<iobuf, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<iobuf, std::__1::error_code, void>>&&) at /vectorized/include/seastar/core/future.hh:2151
 (inlined by) seastar::future<boost::outcome_v2::basic_result<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, void>>> seastar::future<boost::outcome_v2::basic_result<iobuf, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<iobuf, std::__1::error_code, void>>>::then_impl<storage::continuous_batch_parser::consume_records()::$_1, seastar::future<boost::outcome_v2::basic_result<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, void>>>>(storage::continuous_batch_parser::consume_records()::$_1&&) at /vectorized/include/seastar/core/future.hh:1615
 (inlined by) seastar::internal::future_result<storage::continuous_batch_parser::consume_records()::$_1, boost::outcome_v2::basic_result<iobuf, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<iobuf, std::__1::error_code, void>>>::future_type seastar::internal::call_then_impl<seastar::future<boost::outcome_v2::basic_result<iobuf, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<iobuf, std::__1::error_code, void>>>>::run<storage::continuous_batch_parser::consume_records()::$_1>(seastar::future<boost::outcome_v2::basic_result<iobuf, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<iobuf, std::__1::error_code, void>>>&, storage::continuous_batch_parser::consume_records()::$_1&&) at /vectorized/include/seastar/core/future.hh:1248
 (inlined by) seastar::future<boost::outcome_v2::basic_result<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, void>>> seastar::future<boost::outcome_v2::basic_result<iobuf, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<iobuf, std::__1::error_code, void>>>::then<storage::continuous_batch_parser::consume_records()::$_1, seastar::future<boost::outcome_v2::basic_result<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, void>>>>(storage::continuous_batch_parser::consume_records()::$_1&&) at /vectorized/include/seastar/core/future.hh:1534
 (inlined by) storage::continuous_batch_parser::consume_records() at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-086279b42528bcbef-1/redpanda/redpanda/src/v/storage/parser.cc:250
 (inlined by) storage::continuous_batch_parser::consume_one()::$_0::operator()(boost::outcome_v2::basic_result<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, void>>) const at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-086279b42528bcbef-1/redpanda/redpanda/src/v/storage/parser.cc:231
seastar::future<boost::outcome_v2::basic_result<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, void>>> seastar::futurize<seastar::future<boost::outcome_v2::basic_result<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, void>>>>::invoke<storage::continuous_batch_parser::consume_one()::$_0, boost::outcome_v2::basic_result<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, void>>>(storage::continuous_batch_parser::consume_one()::$_0&&, boost::outcome_v2::basic_result<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, void>>&&) at /vectorized/include/seastar/core/future.hh:2149
 (inlined by) seastar::future<boost::outcome_v2::basic_result<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, void>>> seastar::future<boost::outcome_v2::basic_result<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, void>>>::then_impl<storage::continuous_batch_parser::consume_one()::$_0, seastar::future<boost::outcome_v2::basic_result<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, void>>>>(storage::continuous_batch_parser::consume_one()::$_0&&) at /vectorized/include/seastar/core/future.hh:1615
 (inlined by) seastar::internal::future_result<storage::continuous_batch_parser::consume_one()::$_0, boost::outcome_v2::basic_result<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, void>>>::future_type seastar::internal::call_then_impl<seastar::future<boost::outcome_v2::basic_result<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, void>>>>::run<storage::continuous_batch_parser::consume_one()::$_0>(seastar::future<boost::outcome_v2::basic_result<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, void>>>&, storage::continuous_batch_parser::consume_one()::$_0&&) at /vectorized/include/seastar/core/future.hh:1248
 (inlined by) seastar::future<boost::outcome_v2::basic_result<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, void>>> seastar::future<boost::outcome_v2::basic_result<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, void>>>::then<storage::continuous_batch_parser::consume_one()::$_0, seastar::future<boost::outcome_v2::basic_result<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, void>>>>(storage::continuous_batch_parser::consume_one()::$_0&&) at /vectorized/include/seastar/core/future.hh:1534
 (inlined by) storage::continuous_batch_parser::consume_one() at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-086279b42528bcbef-1/redpanda/redpanda/src/v/storage/parser.cc:224
 (inlined by) storage::continuous_batch_parser::consume()::$_2::operator()() const at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-086279b42528bcbef-1/redpanda/redpanda/src/v/storage/parser.cc:264
 (inlined by) seastar::future<seastar::bool_class<seastar::stop_iteration_tag>> seastar::futurize<seastar::future<seastar::bool_class<seastar::stop_iteration_tag>>>::invoke<storage::continuous_batch_parser::consume()::$_2&>(storage::continuous_batch_parser::consume()::$_2&) at /vectorized/include/seastar/core/future.hh:2149
 (inlined by) auto seastar::futurize_invoke<storage::continuous_batch_parser::consume()::$_2&>(storage::continuous_batch_parser::consume()::$_2&) at /vectorized/include/seastar/core/future.hh:2180
 (inlined by) seastar::internal::repeater<storage::continuous_batch_parser::consume()::$_2>::run_and_dispose() at /vectorized/include/seastar/core/loop.hh:73
seastar::reactor::run_tasks(seastar::reactor::task_queue&) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:2330
 (inlined by) seastar::reactor::run_some_tasks() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:2737
seastar::reactor::do_run() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:2906
seastar::reactor::run() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:2789
seastar::app_template::run_deprecated(int, char**, std::__1::function<void ()>&&) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/app-template.cc:265
seastar::app_template::run(int, char**, std::__1::function<seastar::future<int> ()>&&) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/app-template.cc:156
application::run(int, char**) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-086279b42528bcbef-1/redpanda/redpanda/src/v/redpanda/application.cc:326
main at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-086279b42528bcbef-1/redpanda/redpanda/src/v/redpanda/main.cc:22
?? at ??:0
?? at ??:0
_start at ??:0

JIRA Link: CORE-1212

[travisdowns]

Changed to sev/low since it only happens when debug logging is on.

[github-actions[bot]]

This issue hasn't seen activity in 3 months. If you want to keep it open, post a comment or remove the stale label – otherwise this will be closed in two weeks.