Out of memory (OOM) in continuous batch parser

[travisdowns]

Version & Environment

Redpanda version: 23.1.3

What went wrong?

OOM under load.

What should have happened instead?

No OOM.

Additional information

Decoded backtrace:

{23.1.3-1/opt/redpanda/libexec/redpanda} 0x5ba8296: seastar::memory::on_allocation_failure(unsigned long) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:1821
 (inlined by) seastar::memory::allocate(unsigned long) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:1410
{23.1.3-1/opt/redpanda/libexec/redpanda} 0x5bb4e35: __libc_malloc at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:1860
{23.1.3-1/opt/redpanda/libexec/redpanda} 0x5abcb6e: seastar::temporary_buffer<char>::temporary_buffer(unsigned long) at /vectorized/include/seastar/core/temporary_buffer.hh:73
 (inlined by) iobuf_copy(details::io_iterator_consumer&, unsigned long) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-0bac7bc80c88d96d6-1/redpanda/redpanda/src/v/bytes/iobuf.cc:141
 (inlined by) iobuf::copy() const at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-0bac7bc80c88d96d6-1/redpanda/redpanda/src/v/bytes/iobuf.cc:133
{23.1.3-1/opt/redpanda/libexec/redpanda} 0x52a46e5: storage::batch_cache::range::add(model::record_batch const&) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-0bac7bc80c88d96d6-1/redpanda/redpanda/src/v/storage/batch_cache.cc:107
{23.1.3-1/opt/redpanda/libexec/redpanda} 0x52a454c: storage::batch_cache::range::range(storage::batch_cache_index&, model::record_batch const&) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-0bac7bc80c88d96d6-1/redpanda/redpanda/src/v/storage/batch_cache.cc:39
{23.1.3-1/opt/redpanda/libexec/redpanda} 0x52a6cda: storage::batch_cache::put(storage::batch_cache_index&, model::record_batch const&) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-0bac7bc80c88d96d6-1/redpanda/redpanda/src/v/storage/batch_cache.cc:160
{23.1.3-1/opt/redpanda/libexec/redpanda} 0x51b039b: storage::batch_cache_index::put(model::record_batch const&) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-0bac7bc80c88d96d6-1/redpanda/redpanda/src/v/storage/batch_cache.h:431
{23.1.3-1/opt/redpanda/libexec/redpanda} 0x51a3364: storage::segment::cache_put(model::record_batch const&) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-0bac7bc80c88d96d6-1/redpanda/redpanda/src/v/storage/segment.h:389
 (inlined by) storage::log_segment_batch_reader::add_one(model::record_batch&&) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-0bac7bc80c88d96d6-1/redpanda/redpanda/src/v/storage/log_reader.cc:161
 (inlined by) storage::skipping_consumer::consume_batch_end() at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-0bac7bc80c88d96d6-1/redpanda/redpanda/src/v/storage/log_reader.cc:99
{23.1.3-1/opt/redpanda/libexec/redpanda} 0x51bc61d: storage::continuous_batch_parser::consume_records()::$_1::operator()(boost::outcome_v2::basic_result<iobuf, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<iobuf, std::__1::error_code, void> >) const at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-0bac7bc80c88d96d6-1/redpanda/redpanda/src/v/storage/parser.cc:255
 (inlined by) decltype(static_cast<storage::continuous_batch_parser::consume_records()::$_1&>(fp)(static_cast<boost::outcome_v2::basic_result<iobuf, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<iobuf, std::__1::error_code, void> >>(fp0))) std::__1::__invoke<storage::continuous_batch_parser::consume_records()::$_1&, boost::outcome_v2::basic_result<iobuf, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<iobuf, std::__1::error_code, void> > >(storage::continuous_batch_parser::consume_records()::$_1&, boost::outcome_v2::basic_result<iobuf, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<iobuf, std::__1::error_code, void> >&&) at /vectorized/llvm/bin/../include/c++/v1/type_traits:3640
 (inlined by) std::__1::invoke_result<storage::continuous_batch_parser::consume_records()::$_1&, boost::outcome_v2::basic_result<iobuf, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<iobuf, std::__1::error_code, void> > >::type std::__1::invoke<storage::continuous_batch_parser::consume_records()::$_1&, boost::outcome_v2::basic_result<iobuf, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<iobuf, std::__1::error_code, void> > >(storage::continuous_batch_parser::consume_records()::$_1&, boost::outcome_v2::basic_result<iobuf, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<iobuf, std::__1::error_code, void> >&&) at /vectorized/llvm/bin/../include/c++/v1/__functional/invoke.h:93
 (inlined by) auto seastar::internal::future_invoke<storage::continuous_batch_parser::consume_records()::$_1&, boost::outcome_v2::basic_result<iobuf, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<iobuf, std::__1::error_code, void> > >(storage::continuous_batch_parser::consume_records()::$_1&, boost::outcome_v2::basic_result<iobuf, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<iobuf, std::__1::error_code, void> >&&) at /vectorized/include/seastar/core/future.hh:1225
 (inlined by) seastar::future<boost::outcome_v2::basic_result<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, void> > > seastar::future<boost::outcome_v2::basic_result<iobuf, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<iobuf, std::__1::error_code, void> > >::then_impl_nrvo<storage::continuous_batch_parser::consume_records()::$_1, seastar::future<boost::outcome_v2::basic_result<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, void> > > >(storage::continuous_batch_parser::consume_records()::$_1&&)::'lambda'(seastar::internal::promise_base_with_type<boost::outcome_v2::basic_result<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, void> > >&&, storage::continuous_batch_parser::consume_records()::$_1&, seastar::future_state<boost::outcome_v2::basic_result<iobuf, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<iobuf, std::__1::error_code, void> > >&&)::operator()(seastar::internal::promise_base_with_type<boost::outcome_v2::basic_result<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, void> > >&&, storage::continuous_batch_parser::consume_records()::$_1&, seastar::future_state<boost::outcome_v2::basic_result<iobuf, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<iobuf, std::__1::error_code, void> > >&&) const::'lambda'()::operator()() const at /vectorized/include/seastar/core/future.hh:1596
 (inlined by) void seastar::futurize<boost::outcome_v2::basic_result<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, void> > >::satisfy_with_result_of<seastar::future<boost::outcome_v2::basic_result<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, void> > > seastar::future<boost::outcome_v2::basic_result<iobuf, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<iobuf, std::__1::error_code, void> > >::then_impl_nrvo<storage::continuous_batch_parser::consume_records()::$_1, seastar::future<boost::outcome_v2::basic_result<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, void> > > >(storage::continuous_batch_parser::consume_records()::$_1&&)::'lambda'(seastar::internal::promise_base_with_type<boost::outcome_v2::basic_result<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, void> > >&&, storage::continuous_batch_parser::consume_records()::$_1&, seastar::future_state<boost::outcome_v2::basic_result<iobuf, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<iobuf, std::__1::error_code, void> > >&&)::operator()(seastar::internal::promise_base_with_type<boost::outcome_v2::basic_result<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, void> > >&&, storage::continuous_batch_parser::consume_records()::$_1&, seastar::future_state<boost::outcome_v2::basic_result<iobuf, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<iobuf, std::__1::error_code, void> > >&&) const::'lambda'()>(seastar::internal::promise_base_with_type<boost::outcome_v2::basic_result<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, void> > >&&, storage::continuous_batch_parser::consume_records()::$_1&&) at /vectorized/include/seastar/core/future.hh:2136
 (inlined by) seastar::future<boost::outcome_v2::basic_result<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, void> > > seastar::future<boost::outcome_v2::basic_result<iobuf, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<iobuf, std::__1::error_code, void> > >::then_impl_nrvo<storage::continuous_batch_parser::consume_records()::$_1, seastar::future<boost::outcome_v2::basic_result<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, void> > > >(storage::continuous_batch_parser::consume_records()::$_1&&)::'lambda'(seastar::internal::promise_base_with_type<boost::outcome_v2::basic_result<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, void> > >&&, storage::continuous_batch_parser::consume_records()::$_1&, seastar::future_state<boost::outcome_v2::basic_result<iobuf, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<iobuf, std::__1::error_code, void> > >&&)::operator()(seastar::internal::promise_base_with_type<boost::outcome_v2::basic_result<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, void> > >&&, storage::continuous_batch_parser::consume_records()::$_1&, seastar::future_state<boost::outcome_v2::basic_result<iobuf, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<iobuf, std::__1::error_code, void> > >&&) const at /vectorized/include/seastar/core/future.hh:1589
 (inlined by) seastar::continuation<seastar::internal::promise_base_with_type<boost::outcome_v2::basic_result<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, void> > >, storage::continuous_batch_parser::consume_records()::$_1, seastar::future<boost::outcome_v2::basic_result<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, void> > > seastar::future<boost::outcome_v2::basic_result<iobuf, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<iobuf, std::__1::error_code, void> > >::then_impl_nrvo<storage::continuous_batch_parser::consume_records()::$_1, seastar::future<boost::outcome_v2::basic_result<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, void> > > >(storage::continuous_batch_parser::consume_records()::$_1&&)::'lambda'(seastar::internal::promise_base_with_type<boost::outcome_v2::basic_result<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, void> > >&&, storage::continuous_batch_parser::consume_records()::$_1&, seastar::future_state<boost::outcome_v2::basic_result<iobuf, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<iobuf, std::__1::error_code, void> > >&&), boost::outcome_v2::basic_result<iobuf, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<iobuf, std::__1::error_code, void> > >::run_and_dispose() at /vectorized/include/seastar/core/future.hh:781
{23.1.3-1/opt/redpanda/libexec/redpanda} 0x5c534bf: seastar::reactor::run_tasks(seastar::reactor::task_queue&) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:2330
 (inlined by) seastar::reactor::run_some_tasks() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:2737
{23.1.3-1/opt/redpanda/libexec/redpanda} 0x5c57197: seastar::reactor::do_run() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:2906
{23.1.3-1/opt/redpanda/libexec/redpanda} 0x5c9a4e5: seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_95::operator()() const at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:4140
 (inlined by) decltype(static_cast<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_95&>(fp)()) std::__1::__invoke<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_95&>(seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_95&) at /vectorized/llvm/bin/../include/c++/v1/type_traits:3640
 (inlined by) void std::__1::__invoke_void_return_wrapper<void, true>::__call<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_95&>(seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_95&) at /vectorized/llvm/bin/../include/c++/v1/__functional/invoke.h:61
 (inlined by) std::__1::__function::__alloc_func<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_95, std::__1::allocator<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_95>, void ()>::operator()() at /vectorized/llvm/bin/../include/c++/v1/__functional/function.h:180
 (inlined by) std::__1::__function::__func<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_95, std::__1::allocator<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_95>, void ()>::operator()() at /vectorized/llvm/bin/../include/c++/v1/__functional/function.h:354
{23.1.3-1/opt/redpanda/libexec/redpanda} 0x5bf528f: std::__1::__function::__value_func<void ()>::operator()() const at /vectorized/llvm/bin/../include/c++/v1/__functional/function.h:507
 (inlined by) std::__1::function<void ()>::operator()() const at /vectorized/llvm/bin/../include/c++/v1/__functional/function.h:1184
 (inlined by) seastar::posix_thread::start_routine(void*) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/posix.cc:73
{/opt/redpanda/lib/libc.so.6} 0x91016: ?? at ??:0
{/opt/redpanda/lib/libc.so.6} 0x1166cf: ?? at ??:0

Raw backtrace:

ERROR 2023-04-04 21:40:44,959 [shard 17] seastar - Failed to allocate 131072 bytes
Aborting on shard 17
Backtrace:
0x5c35386
0x5c98d82
/opt/redpanda/lib/libc.so.6+0x42abf
/opt/redpanda/lib/libc.so.6+0x92e3b
/opt/redpanda/lib/libc.so.6+0x42a15
/opt/redpanda/lib/libc.so.6+0x2c82e
0x5ba8296
0x5bb4e35
0x5abcb6e
0x52a46e5
0x52a454c
0x52a6cda
0x51b039b
0x51a3364
0x51bc61d
0x5c534bf
0x5c57197
0x5c9a4e5
0x5bf528f
/opt/redpanda/lib/libc.so.6+0x91016
/opt/redpanda/lib/libc.so.6+0x1166cf

JIRA Link: CORE-1241

[travisdowns]

The trace above accounted for 46 out of 58 OOMs.

Here's a different backtrace which accounted for (TBD) of the remaining OOMS:

{23.1.3-1/opt/redpanda/libexec/redpanda} 0x5ba8296: seastar::memory::on_allocation_failure(unsigned long) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:1821
 (inlined by) seastar::memory::allocate(unsigned long) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:1410
{23.1.3-1/opt/redpanda/libexec/redpanda} 0x5bb4e35: __libc_malloc at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:1860
{23.1.3-1/opt/redpanda/libexec/redpanda} 0x5ac10a9: seastar::temporary_buffer<char>::temporary_buffer(unsigned long) at /vectorized/include/seastar/core/temporary_buffer.hh:73
 (inlined by) iobuf::create_new_fragment(unsigned long) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-0bac7bc80c88d96d6-1/redpanda/redpanda/src/v/bytes/iobuf.h:235
 (inlined by) iobuf::append(char const*, unsigned long) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-0bac7bc80c88d96d6-1/redpanda/redpanda/src/v/bytes/iobuf.h:289
 (inlined by) iobuf::append(seastar::temporary_buffer<char>) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-0bac7bc80c88d96d6-1/redpanda/redpanda/src/v/bytes/iobuf.h:307
 (inlined by) read_iobuf_exactly(seastar::input_stream<char>&, unsigned long)::$_3::operator()(iobuf&, unsigned long&) const::'lambda0'()::operator()() const::'lambda'(seastar::temporary_buffer<char>)::operator()(seastar::temporary_buffer<char>) const at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-0bac7bc80c88d96d6-1/redpanda/redpanda/src/v/bytes/iobuf.cc:71
 (inlined by) seastar::future<void> seastar::futurize<void>::invoke<read_iobuf_exactly(seastar::input_stream<char>&, unsigned long)::$_3::operator()(iobuf&, unsigned long&) const::'lambda0'()::operator()() const::'lambda'(seastar::temporary_buffer<char>), seastar::temporary_buffer<char> >(read_iobuf_exactly(seastar::input_stream<char>&, unsigned long)::$_3::operator()(iobuf&, unsigned long&) const::'lambda0'()::operator()() const::'lambda'(seastar::temporary_buffer<char>)&&, seastar::temporary_buffer<char>&&) at /vectorized/include/seastar/core/future.hh:2146
 (inlined by) seastar::future<void> seastar::future<seastar::temporary_buffer<char> >::then_impl<read_iobuf_exactly(seastar::input_stream<char>&, unsigned long)::$_3::operator()(iobuf&, unsigned long&) const::'lambda0'()::operator()() const::'lambda'(seastar::temporary_buffer<char>), seastar::future<void> >(read_iobuf_exactly(seastar::input_stream<char>&, unsigned long)::$_3::operator()(iobuf&, unsigned long&) const::'lambda0'()::operator()() const::'lambda'(seastar::temporary_buffer<char>)&&) at /vectorized/include/seastar/core/future.hh:1615
 (inlined by) seastar::internal::future_result<read_iobuf_exactly(seastar::input_stream<char>&, unsigned long)::$_3::operator()(iobuf&, unsigned long&) const::'lambda0'()::operator()() const::'lambda'(seastar::temporary_buffer<char>), seastar::temporary_buffer<char> >::future_type seastar::internal::call_then_impl<seastar::future<seastar::temporary_buffer<char> > >::run<read_iobuf_exactly(seastar::input_stream<char>&, unsigned long)::$_3::operator()(iobuf&, unsigned long&) const::'lambda0'()::operator()() const::'lambda'(seastar::temporary_buffer<char>)>(seastar::future<seastar::temporary_buffer<char> >&, read_iobuf_exactly(seastar::input_stream<char>&, unsigned long)::$_3::operator()(iobuf&, unsigned long&) const::'lambda0'()::operator()() const::'lambda'(seastar::temporary_buffer<char>)&&) at /vectorized/include/seastar/core/future.hh:1248
 (inlined by) seastar::future<void> seastar::future<seastar::temporary_buffer<char> >::then<read_iobuf_exactly(seastar::input_stream<char>&, unsigned long)::$_3::operator()(iobuf&, unsigned long&) const::'lambda0'()::operator()() const::'lambda'(seastar::temporary_buffer<char>), seastar::future<void> >(read_iobuf_exactly(seastar::input_stream<char>&, unsigned long)::$_3::operator()(iobuf&, unsigned long&) const::'lambda0'()::operator()() const::'lambda'(seastar::temporary_buffer<char>)&&) at /vectorized/include/seastar/core/future.hh:1534
 (inlined by) read_iobuf_exactly(seastar::input_stream<char>&, unsigned long)::$_3::operator()(iobuf&, unsigned long&) const::'lambda0'()::operator()() const at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-0bac7bc80c88d96d6-1/redpanda/redpanda/src/v/bytes/iobuf.cc:64
{23.1.3-1/opt/redpanda/libexec/redpanda} 0x5ac1d28: seastar::internal::do_until_state<read_iobuf_exactly(seastar::input_stream<char>&, unsigned long)::$_3::operator()(iobuf&, unsigned long&) const::'lambda'(), read_iobuf_exactly(seastar::input_stream<char>&, unsigned long)::$_3::operator()(iobuf&, unsigned long&) const::'lambda0'()>::run_and_dispose() at /vectorized/include/seastar/core/loop.hh:297
{23.1.3-1/opt/redpanda/libexec/redpanda} 0x5c534bf: seastar::reactor::run_tasks(seastar::reactor::task_queue&) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:2330
 (inlined by) seastar::reactor::run_some_tasks() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:2737
{23.1.3-1/opt/redpanda/libexec/redpanda} 0x5c57197: seastar::reactor::do_run() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:2906

It's in the iobuf_read_exactly code, but the loop has suspended lacks any context about the caller. This is called in many places in including the continuous_batch_parser implicated in most of the backtraces.