redpanda-data / redpanda

Redpanda is a streaming data platform for developers. Kafka API compatible. 10x faster. No ZooKeeper. No JVM!
https://redpanda.com
9.73k stars 591 forks source link

Segmentation fault in append_challenged_posix_file_impl::op allocation, during dma_write #3518

Closed NyaliaLui closed 1 year ago

NyaliaLui commented 2 years ago

Version & Environment

Redpanda version: v21.11.3-si-beta8

The following backtrace was seen on BYOC during long running tests for shadow indexing.

[Backtrace #0]
void seastar::backtrace<seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}>(seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}&&) at /v/build/v_deps_build/seastar-prefix/src/seastar/include/seastar/util/backtrace.hh:59
 (inlined by) seastar::backtrace_buffer::append_backtrace() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:754
 (inlined by) seastar::print_with_backtrace(seastar::backtrace_buffer&, bool) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:784
 (inlined by) seastar::print_with_backtrace(char const*, bool) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:796
 (inlined by) seastar::sigsegv_action() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:3568
 (inlined by) operator() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:3554
 (inlined by) __invoke at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:3550
?? ??:0
op at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/file-impl.hh:212
 (inlined by) seastar::append_challenged_posix_file_impl::op* std::__1::construct_at<seastar::append_challenged_posix_file_impl::op, seastar::append_challenged_posix_file_impl::op, seastar::append_challenged_posix_file_impl::op*>(seastar::append_challenged_posix_file_impl::op*, seastar::append_challenged_posix_file_impl::op&&) at /vectorized/llvm/bin/../include/c++/v1/__memory/base.h:105
 (inlined by) void std::__1::allocator_traits<std::__1::allocator<seastar::append_challenged_posix_file_impl::op> >::construct<seastar::append_challenged_posix_file_impl::op, seastar::append_challenged_posix_file_impl::op, void, void>(std::__1::allocator<seastar::append_challenged_posix_file_impl::op>&, seastar::append_challenged_posix_file_impl::op*, seastar::append_challenged_posix_file_impl::op&&) at /vectorized/llvm/bin/../include/c++/v1/__memory/allocator_traits.h:296
 (inlined by) std::__1::deque<seastar::append_challenged_posix_file_impl::op, std::__1::allocator<seastar::append_challenged_posix_file_impl::op> >::push_back(seastar::append_challenged_posix_file_impl::op&&) at /vectorized/llvm/bin/../include/c++/v1/deque:1958
 (inlined by) seastar::append_challenged_posix_file_impl::enqueue_op(seastar::append_challenged_posix_file_impl::op&&) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/file.cc:730
 (inlined by) seastar::future<unsigned long> seastar::append_challenged_posix_file_impl::enqueue<unsigned long, seastar::append_challenged_posix_file_impl::write_dma(unsigned long, void const*, unsigned long, seastar::io_priority_class const&, seastar::io_intent*)::$_30>(seastar::append_challenged_posix_file_impl::opcode, unsigned long, unsigned long, seastar::append_challenged_posix_file_impl::write_dma(unsigned long, void const*, unsigned long, seastar::io_priority_class const&, seastar::io_intent*)::$_30&&) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/file-impl.hh:256
 (inlined by) seastar::append_challenged_posix_file_impl::write_dma(unsigned long, void const*, unsigned long, seastar::io_priority_class const&, seastar::io_intent*) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/file.cc:801
seastar::file::dma_write_impl(unsigned long, unsigned char const*, unsigned long, seastar::io_priority_class const&, seastar::io_intent*) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/file.cc:1182
seastar::future<unsigned long> seastar::file::dma_write<char>(unsigned long, char const*, unsigned long, seastar::io_priority_class const&, seastar::io_intent*) at /vectorized/include/seastar/core/file.hh:351
 (inlined by) operator() at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-09f40697b81d3241f-1/vectorized/redpanda/vbuild/release/clang/../../../src/v/storage/segment_appender.cc:473
seastar::future<void> seastar::futurize<seastar::future<void> >::invoke<storage::segment_appender::dispatch_background_head_write()::$_20::operator()()::{lambda(seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>)#1}, seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock> >(storage::segment_appender::dispatch_background_head_write()::$_20::operator()()::{lambda(seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>)#1}&&, seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>&&) at /vectorized/include/seastar/core/future.hh:2135
 (inlined by) seastar::future<void> seastar::future<seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock> >::then_impl<storage::segment_appender::dispatch_background_head_write()::$_20::operator()()::{lambda(seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>)#1}, seastar::future<void> >(storage::segment_appender::dispatch_background_head_write()::$_20::operator()()::{lambda(seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>)#1}&&) at /vectorized/include/seastar/core/future.hh:1601
 (inlined by) seastar::internal::future_result<storage::segment_appender::dispatch_background_head_write()::$_20::operator()()::{lambda(seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>)#1}, seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock> >::future_type seastar::internal::call_then_impl<seastar::future<seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock> > >::run<storage::segment_appender::dispatch_background_head_write()::$_20::operator()()::{lambda(seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>)#1}>(seastar::future<seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock> >&, storage::segment_appender::dispatch_background_head_write()::$_20::operator()()::{lambda(seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>)#1}&&) at /vectorized/include/seastar/core/future.hh:1234
 (inlined by) seastar::future<void> seastar::future<seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock> >::then<storage::segment_appender::dispatch_background_head_write()::$_20::operator()()::{lambda(seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>)#1}, seastar::future<void> >(storage::segment_appender::dispatch_background_head_write()::$_20::operator()()::{lambda(seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>)#1}&&) at /vectorized/include/seastar/core/future.hh:1520
 (inlined by) operator() at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-09f40697b81d3241f-1/vectorized/redpanda/vbuild/release/clang/../../../src/v/storage/segment_appender.cc:470
 (inlined by) seastar::future<void> seastar::futurize<seastar::future<void> >::invoke<storage::segment_appender::dispatch_background_head_write()::$_20>(storage::segment_appender::dispatch_background_head_write()::$_20&&) at /vectorized/include/seastar/core/future.hh:2135
 (inlined by) auto seastar::futurize_invoke<storage::segment_appender::dispatch_background_head_write()::$_20>(storage::segment_appender::dispatch_background_head_write()::$_20&&) at /vectorized/include/seastar/core/future.hh:2166
 (inlined by) operator()<seastar::semaphore_units<> > at /vectorized/include/seastar/core/semaphore.hh:528
seastar::future<void> seastar::futurize<seastar::future<void> >::invoke<seastar::with_semaphore<seastar::semaphore_default_exception_factory, storage::segment_appender::dispatch_background_head_write()::$_20, std::__1::chrono::steady_clock>(seastar::basic_semaphore<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>&, unsigned long, storage::segment_appender::dispatch_background_head_write()::$_20&&)::{lambda(auto:1)#1}, seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock> >(seastar::with_semaphore<seastar::semaphore_default_exception_factory, storage::segment_appender::dispatch_background_head_write()::$_20, std::__1::chrono::steady_clock>(seastar::basic_semaphore<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>&, unsigned long, storage::segment_appender::dispatch_background_head_write()::$_20&&)::{lambda(auto:1)#1}&&, seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>&&) at /vectorized/include/seastar/core/future.hh:2135
 (inlined by) seastar::future<void> seastar::future<seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock> >::then_impl<seastar::with_semaphore<seastar::semaphore_default_exception_factory, storage::segment_appender::dispatch_background_head_write()::$_20, std::__1::chrono::steady_clock>(seastar::basic_semaphore<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>&, unsigned long, storage::segment_appender::dispatch_background_head_write()::$_20&&)::{lambda(auto:1)#1}, seastar::future<void> >(seastar::with_semaphore<seastar::semaphore_default_exception_factory, storage::segment_appender::dispatch_background_head_write()::$_20, std::__1::chrono::steady_clock>(seastar::basic_semaphore<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>&, unsigned long, storage::segment_appender::dispatch_background_head_write()::$_20&&)::{lambda(auto:1)#1}&&) at /vectorized/include/seastar/core/future.hh:1601
 (inlined by) seastar::internal::future_result<seastar::with_semaphore<seastar::semaphore_default_exception_factory, storage::segment_appender::dispatch_background_head_write()::$_20, std::__1::chrono::steady_clock>(seastar::basic_semaphore<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>&, unsigned long, storage::segment_appender::dispatch_background_head_write()::$_20&&)::{lambda(auto:1)#1}, seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock> >::future_type seastar::internal::call_then_impl<seastar::future<seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock> > >::run<seastar::with_semaphore<seastar::semaphore_default_exception_factory, storage::segment_appender::dispatch_background_head_write()::$_20, std::__1::chrono::steady_clock>(seastar::basic_semaphore<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>&, unsigned long, storage::segment_appender::dispatch_background_head_write()::$_20&&)::{lambda(auto:1)#1}>(seastar::future<seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock> >&, seastar::with_semaphore<seastar::semaphore_default_exception_factory, storage::segment_appender::dispatch_background_head_write()::$_20, std::__1::chrono::steady_clock>(seastar::basic_semaphore<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>&, unsigned long, storage::segment_appender::dispatch_background_head_write()::$_20&&)::{lambda(auto:1)#1}&&) at /vectorized/include/seastar/core/future.hh:1234
 (inlined by) seastar::future<void> seastar::future<seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock> >::then<seastar::with_semaphore<seastar::semaphore_default_exception_factory, storage::segment_appender::dispatch_background_head_write()::$_20, std::__1::chrono::steady_clock>(seastar::basic_semaphore<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>&, unsigned long, storage::segment_appender::dispatch_background_head_write()::$_20&&)::{lambda(auto:1)#1}, seastar::future<void> >(seastar::with_semaphore<seastar::semaphore_default_exception_factory, storage::segment_appender::dispatch_background_head_write()::$_20, std::__1::chrono::steady_clock>(seastar::basic_semaphore<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>&, unsigned long, storage::segment_appender::dispatch_background_head_write()::$_20&&)::{lambda(auto:1)#1}&&) at /vectorized/include/seastar/core/future.hh:1520
 (inlined by) seastar::futurize<std::__1::result_of<storage::segment_appender::dispatch_background_head_write()::$_20 ()>::type>::type seastar::with_semaphore<seastar::semaphore_default_exception_factory, storage::segment_appender::dispatch_background_head_write()::$_20, std::__1::chrono::steady_clock>(seastar::basic_semaphore<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>&, unsigned long, storage::segment_appender::dispatch_background_head_write()::$_20&&) at /vectorized/include/seastar/core/semaphore.hh:527
 (inlined by) storage::segment_appender::dispatch_background_head_write() at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-09f40697b81d3241f-1/vectorized/redpanda/vbuild/release/clang/../../../src/v/storage/segment_appender.cc:457
storage::segment_appender::do_append(char const*, unsigned long) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-09f40697b81d3241f-1/vectorized/redpanda/vbuild/release/clang/../../../src/v/storage/segment_appender.cc:163
storage::segment_appender::append(char const*, unsigned long) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-09f40697b81d3241f-1/vectorized/redpanda/vbuild/release/clang/../../../src/v/storage/segment_appender.cc:124
 (inlined by) operator() at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-09f40697b81d3241f-1/vectorized/redpanda/vbuild/release/clang/../../../src/v/storage/segment_appender.cc:115
 (inlined by) seastar::future<void> seastar::futurize<seastar::future<void> >::invoke<storage::segment_appender::append(iobuf const&)::$_3&, details::io_fragment const&>(storage::segment_appender::append(iobuf const&)::$_3&, details::io_fragment const&) at /vectorized/include/seastar/core/future.hh:2135
 (inlined by) auto seastar::futurize_invoke<storage::segment_appender::append(iobuf const&)::$_3&, details::io_fragment const&>(storage::segment_appender::append(iobuf const&)::$_3&, details::io_fragment const&) at /vectorized/include/seastar/core/future.hh:2166
 (inlined by) seastar::future<void> seastar::internal::do_for_each_impl<boost::intrusive::list_iterator<boost::intrusive::mhtraits<details::io_fragment, boost::intrusive::list_member_hook<boost::intrusive::link_mode<(boost::intrusive::link_mode_type)1> >, &details::io_fragment::hook>, true>, storage::segment_appender::append(iobuf const&)::$_3>(boost::intrusive::list_iterator<boost::intrusive::mhtraits<details::io_fragment, boost::intrusive::list_member_hook<boost::intrusive::link_mode<(boost::intrusive::link_mode_type)1> >, &details::io_fragment::hook>, true>, boost::intrusive::list_iterator<boost::intrusive::mhtraits<details::io_fragment, boost::intrusive::list_member_hook<boost::intrusive::link_mode<(boost::intrusive::link_mode_type)1> >, &details::io_fragment::hook>, true>, storage::segment_appender::append(iobuf const&)::$_3) at /vectorized/include/seastar/core/loop.hh:421
 (inlined by) seastar::future<void> seastar::do_for_each<boost::intrusive::list_iterator<boost::intrusive::mhtraits<details::io_fragment, boost::intrusive::list_member_hook<boost::intrusive::link_mode<(boost::intrusive::link_mode_type)1> >, &details::io_fragment::hook>, true>, storage::segment_appender::append(iobuf const&)::$_3>(boost::intrusive::list_iterator<boost::intrusive::mhtraits<details::io_fragment, boost::intrusive::list_member_hook<boost::intrusive::link_mode<(boost::intrusive::link_mode_type)1> >, &details::io_fragment::hook>, true>, boost::intrusive::list_iterator<boost::intrusive::mhtraits<details::io_fragment, boost::intrusive::list_member_hook<boost::intrusive::link_mode<(boost::intrusive::link_mode_type)1> >, &details::io_fragment::hook>, true>, storage::segment_appender::append(iobuf const&)::$_3) at /vectorized/include/seastar/core/loop.hh:456
 (inlined by) storage::segment_appender::append(iobuf const&) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-09f40697b81d3241f-1/vectorized/redpanda/vbuild/release/clang/../../../src/v/storage/segment_appender.cc:113
operator() at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-09f40697b81d3241f-1/vectorized/redpanda/vbuild/release/clang/../../../src/v/storage/segment_appender_utils.cc:62
 (inlined by) seastar::future<void> seastar::futurize<seastar::future<void> >::invoke<storage::write(storage::segment_appender&, model::record_batch const&)::$_0>(storage::write(storage::segment_appender&, model::record_batch const&)::$_0&&) at /vectorized/include/seastar/core/future.hh:2135
 (inlined by) seastar::future<void> seastar::futurize<seastar::future<void> >::invoke<storage::write(storage::segment_appender&, model::record_batch const&)::$_0>(storage::write(storage::segment_appender&, model::record_batch const&)::$_0&&, seastar::internal::monostate) at /vectorized/include/seastar/core/future.hh:1979
 (inlined by) seastar::future<void> seastar::future<void>::then_impl<storage::write(storage::segment_appender&, model::record_batch const&)::$_0, seastar::future<void> >(storage::write(storage::segment_appender&, model::record_batch const&)::$_0&&) at /vectorized/include/seastar/core/future.hh:1601
 (inlined by) seastar::internal::future_result<storage::write(storage::segment_appender&, model::record_batch const&)::$_0, void>::future_type seastar::internal::call_then_impl<seastar::future<void> >::run<storage::write(storage::segment_appender&, model::record_batch const&)::$_0>(seastar::future<void>&, storage::write(storage::segment_appender&, model::record_batch const&)::$_0&&) at /vectorized/include/seastar/core/future.hh:1234
 (inlined by) seastar::future<void> seastar::future<void>::then<storage::write(storage::segment_appender&, model::record_batch const&)::$_0, seastar::future<void> >(storage::write(storage::segment_appender&, model::record_batch const&)::$_0&&) at /vectorized/include/seastar/core/future.hh:1520
 (inlined by) storage::write(storage::segment_appender&, model::record_batch const&) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-09f40697b81d3241f-1/vectorized/redpanda/vbuild/release/clang/../../../src/v/storage/segment_appender_utils.cc:60
storage::segment::append(model::record_batch const&) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-09f40697b81d3241f-1/vectorized/redpanda/vbuild/release/clang/../../../src/v/storage/segment.cc:391
storage::disk_log_appender::append_batch_to_segment(model::record_batch const&) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-09f40697b81d3241f-1/vectorized/redpanda/vbuild/release/clang/../../../src/v/storage/disk_log_appender.cc:118
 (inlined by) operator() at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-09f40697b81d3241f-1/vectorized/redpanda/vbuild/release/clang/../../../src/v/storage/disk_log_appender.cc:99
 (inlined by) seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > seastar::futurize<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > >::invoke<storage::disk_log_appender::operator()(model::record_batch&)::$_3>(storage::disk_log_appender::operator()(model::record_batch&)::$_3&&) at /vectorized/include/seastar/core/future.hh:2135
 (inlined by) seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > seastar::futurize<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > >::invoke<storage::disk_log_appender::operator()(model::record_batch&)::$_3>(storage::disk_log_appender::operator()(model::record_batch&)::$_3&&, seastar::internal::monostate) at /vectorized/include/seastar/core/future.hh:1979
 (inlined by) seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > seastar::future<void>::then_impl<storage::disk_log_appender::operator()(model::record_batch&)::$_3, seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > >(storage::disk_log_appender::operator()(model::record_batch&)::$_3&&) at /vectorized/include/seastar/core/future.hh:1601
 (inlined by) seastar::internal::future_result<storage::disk_log_appender::operator()(model::record_batch&)::$_3, void>::future_type seastar::internal::call_then_impl<seastar::future<void> >::run<storage::disk_log_appender::operator()(model::record_batch&)::$_3>(seastar::future<void>&, storage::disk_log_appender::operator()(model::record_batch&)::$_3&&) at /vectorized/include/seastar/core/future.hh:1234
 (inlined by) seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > seastar::future<void>::then<storage::disk_log_appender::operator()(model::record_batch&)::$_3, seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > >(storage::disk_log_appender::operator()(model::record_batch&)::$_3&&) at /vectorized/include/seastar/core/future.hh:1520
 (inlined by) storage::disk_log_appender::operator()(model::record_batch&) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-09f40697b81d3241f-1/vectorized/redpanda/vbuild/release/clang/../../../src/v/storage/disk_log_appender.cc:99
storage::log_appender::operator()(model::record_batch&) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-09f40697b81d3241f-1/vectorized/redpanda/vbuild/release/clang/../../../src/v/storage/log_appender.h:62
 (inlined by) operator() at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-09f40697b81d3241f-1/vectorized/redpanda/vbuild/release/clang/../../../src/v/raft/consensus.cc:2123
 (inlined by) operator() at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-09f40697b81d3241f-1/vectorized/redpanda/vbuild/release/clang/../../../src/v/model/record_batch_reader.h:136
 (inlined by) decltype ((std::__1::forward<seastar::internal::variant_visitor<model::record_batch_reader::impl::do_for_each_ref<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::{lambda(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&)#1}::operator()(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&) const::{lambda(seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch> >&)#1}, model::record_batch_reader::impl::do_for_each_ref<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::{lambda(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&)#1}::operator()(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&) const::{lambda(model::record_batch_reader::foreign_data_t&)#1}> >({parm#1}))(std::__1::forward<model::record_batch_reader::foreign_data_t&>({parm#2}))) std::__1::__invoke_constexpr<seastar::internal::variant_visitor<model::record_batch_reader::impl::do_for_each_ref<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::{lambda(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&)#1}::operator()(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&) const::{lambda(seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch> >&)#1}, model::record_batch_reader::impl::do_for_each_ref<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::{lambda(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&)#1}::operator()(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&) const::{lambda(model::record_batch_reader::foreign_data_t&)#1}>, model::record_batch_reader::foreign_data_t&>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&&, model::record_batch_reader::foreign_data_t&) at /vectorized/llvm/bin/../include/c++/v1/type_traits:3700
 (inlined by) decltype(auto) std::__1::__variant_detail::__visitation::__variant::__value_visitor<seastar::internal::variant_visitor<model::record_batch_reader::impl::do_for_each_ref<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::{lambda(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&)#1}::operator()(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&) const::{lambda(seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch> >&)#1}, model::record_batch_reader::impl::do_for_each_ref<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::{lambda(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&)#1}::operator()(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&) const::{lambda(model::record_batch_reader::foreign_data_t&)#1}> >::operator()<std::__1::__variant_detail::__alt<1ul, model::record_batch_reader::foreign_data_t>&>(std::__1::__variant_detail::__alt<1ul, model::record_batch_reader::foreign_data_t>&) const at /vectorized/llvm/bin/../include/c++/v1/variant:615
 (inlined by) decltype ((std::__1::forward<std::__1::__variant_detail::__visitation::__variant::__value_visitor<seastar::internal::variant_visitor<model::record_batch_reader::impl::do_for_each_ref<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::{lambda(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&)#1}::operator()(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&) const::{lambda(seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch> >&)#1}, model::record_batch_reader::impl::do_for_each_ref<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::{lambda(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&)#1}::operator()(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&) const::{lambda(model::record_batch_reader::foreign_data_t&)#1}> > >({parm#1}))(std::__1::forward<std::__1::__variant_detail::__alt<1ul, model::record_batch_reader::foreign_data_t>&>({parm#2}))) std::__1::__invoke_constexpr<std::__1::__variant_detail::__visitation::__variant::__value_visitor<seastar::internal::variant_visitor<model::record_batch_reader::impl::do_for_each_ref<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::{lambda(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&)#1}::operator()(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&) const::{lambda(seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch> >&)#1}, model::record_batch_reader::impl::do_for_each_ref<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::{lambda(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&)#1}::operator()(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&) const::{lambda(model::record_batch_reader::foreign_data_t&)#1}> >, std::__1::__variant_detail::__alt<1ul, model::record_batch_reader::foreign_data_t>&>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&&, std::__1::__variant_detail::__alt<1ul, model::record_batch_reader::foreign_data_t>&) at /vectorized/llvm/bin/../include/c++/v1/type_traits:3700
 (inlined by) decltype(auto) std::__1::__variant_detail::__visitation::__base::__dispatcher<1ul>::__dispatch<std::__1::__variant_detail::__visitation::__variant::__value_visitor<seastar::internal::variant_visitor<model::record_batch_reader::impl::do_for_each_ref<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::{lambda(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&)#1}::operator()(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&) const::{lambda(seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch> >&)#1}, model::record_batch_reader::impl::do_for_each_ref<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::{lambda(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&)#1}::operator()(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&) const::{lambda(model::record_batch_reader::foreign_data_t&)#1}> >&&, std::__1::__variant_detail::__base<(std::__1::__variant_detail::_Trait)1, seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch> >, model::record_batch_reader::foreign_data_t>&>(std::__1::__variant_detail::__visitation::__variant::__value_visitor<seastar::internal::variant_visitor<model::record_batch_reader::impl::do_for_each_ref<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::{lambda(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&)#1}::operator()(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&) const::{lambda(seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch> >&)#1}, model::record_batch_reader::impl::do_for_each_ref<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::{lambda(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&)#1}::operator()(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&) const::{lambda(model::record_batch_reader::foreign_data_t&)#1}> >&&, std::__1::__variant_detail::__base<(std::__1::__variant_detail::_Trait)1, seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch> >, model::record_batch_reader::foreign_data_t>&) at /vectorized/llvm/bin/../include/c++/v1/variant:497
decltype(auto) std::__1::__variant_detail::__visitation::__base::__visit_alt<std::__1::__variant_detail::__visitation::__variant::__value_visitor<seastar::internal::variant_visitor<model::record_batch_reader::impl::do_for_each_ref<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::{lambda(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&)#1}::operator()(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&) const::{lambda(seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch> >&)#1}, model::record_batch_reader::impl::do_for_each_ref<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::{lambda(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&)#1}::operator()(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&) const::{lambda(model::record_batch_reader::foreign_data_t&)#1}> >, std::__1::__variant_detail::__impl<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch> >, model::record_batch_reader::foreign_data_t>&>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&&, std::__1::__variant_detail::__impl<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch> >, model::record_batch_reader::foreign_data_t>&) at /vectorized/llvm/bin/../include/c++/v1/variant:460
 (inlined by) decltype(auto) std::__1::__variant_detail::__visitation::__variant::__visit_alt<std::__1::__variant_detail::__visitation::__variant::__value_visitor<seastar::internal::variant_visitor<model::record_batch_reader::impl::do_for_each_ref<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::{lambda(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&)#1}::operator()(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&) const::{lambda(seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch> >&)#1}, model::record_batch_reader::impl::do_for_each_ref<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::{lambda(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&)#1}::operator()(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&) const::{lambda(model::record_batch_reader::foreign_data_t&)#1}> >, std::__1::variant<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch> >, model::record_batch_reader::foreign_data_t>&>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&&, std::__1::variant<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch> >, model::record_batch_reader::foreign_data_t>&) at /vectorized/llvm/bin/../include/c++/v1/variant:567
 (inlined by) decltype(auto) std::__1::__variant_detail::__visitation::__variant::__visit_value<seastar::internal::variant_visitor<model::record_batch_reader::impl::do_for_each_ref<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::{lambda(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&)#1}::operator()(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&) const::{lambda(seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch> >&)#1}, model::record_batch_reader::impl::do_for_each_ref<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::{lambda(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&)#1}::operator()(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&) const::{lambda(model::record_batch_reader::foreign_data_t&)#1}>, std::__1::variant<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch> >, model::record_batch_reader::foreign_data_t>&>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&&, std::__1::variant<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch> >, model::record_batch_reader::foreign_data_t>&) at /vectorized/llvm/bin/../include/c++/v1/variant:585
 (inlined by) decltype(auto) std::__1::visit<seastar::internal::variant_visitor<model::record_batch_reader::impl::do_for_each_ref<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::{lambda(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&)#1}::operator()(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&) const::{lambda(seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch> >&)#1}, model::record_batch_reader::impl::do_for_each_ref<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::{lambda(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&)#1}::operator()(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&) const::{lambda(model::record_batch_reader::foreign_data_t&)#1}>, std::__1::variant<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch> >, model::record_batch_reader::foreign_data_t>&>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&&, std::__1::variant<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch> >, model::record_batch_reader::foreign_data_t>&) at /vectorized/llvm/bin/../include/c++/v1/variant:1654
 (inlined by) auto seastar::visit<std::__1::variant<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch> >, model::record_batch_reader::foreign_data_t>&, model::record_batch_reader::impl::do_for_each_ref<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::{lambda(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&)#1}::operator()(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&) const::{lambda(seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch> >&)#1}, model::record_batch_reader::impl::do_for_each_ref<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::{lambda(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&)#1}::operator()(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&) const::{lambda(model::record_batch_reader::foreign_data_t&)#1}>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&&, model::record_batch_reader::impl::do_for_each_ref<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::{lambda(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&)#1}::operator()(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&) const::{lambda(seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch> >&)#1}&&, model::record_batch_reader::impl::do_for_each_ref<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::{lambda(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&)#1}::operator()(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&) const::{lambda(model::record_batch_reader::foreign_data_t&)#1}&&) at /vectorized/include/seastar/util/variant_utils.hh:71
 (inlined by) operator() at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-09f40697b81d3241f-1/vectorized/redpanda/vbuild/release/clang/../../../src/v/model/record_batch_reader.h:128
 (inlined by) operator() at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-09f40697b81d3241f-1/vectorized/redpanda/vbuild/release/clang/../../../src/v/model/record_batch_reader.h:156
 (inlined by) seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > seastar::futurize<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > >::invoke<model::record_batch_reader::impl::do_action<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer, model::record_batch_reader::impl::do_for_each_ref<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::{lambda(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&)#1}>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >, model::record_batch_reader::impl::do_for_each_ref<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::{lambda(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&)#1}&&)::{lambda()#1}&>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&&) at /vectorized/include/seastar/core/future.hh:2135
 (inlined by) seastar::future<void> seastar::repeat<model::record_batch_reader::impl::do_action<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer, model::record_batch_reader::impl::do_for_each_ref<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::{lambda(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&)#1}>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >, model::record_batch_reader::impl::do_for_each_ref<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::{lambda(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&)#1}&&)::{lambda()#1}>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&&) at /vectorized/include/seastar/core/loop.hh:118
 (inlined by) auto model::record_batch_reader::impl::do_action<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer, model::record_batch_reader::impl::do_for_each_ref<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::{lambda(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&)#1}>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >, model::record_batch_reader::impl::do_for_each_ref<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::{lambda(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&)#1}&&) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-09f40697b81d3241f-1/vectorized/redpanda/vbuild/release/clang/../../../src/v/model/record_batch_reader.h:151
 (inlined by) auto model::record_batch_reader::impl::do_for_each_ref<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-09f40697b81d3241f-1/vectorized/redpanda/vbuild/release/clang/../../../src/v/model/record_batch_reader.h:127
 (inlined by) operator() at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-09f40697b81d3241f-1/vectorized/redpanda/vbuild/release/clang/../../../src/v/model/record_batch_reader.h:92
 (inlined by) decltype ((std::__1::forward<model::record_batch_reader::impl::for_each_ref<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::{lambda(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&)#1}&>({parm#1}))(std::__1::forward<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&>({parm#2}))) std::__1::__invoke_constexpr<model::record_batch_reader::impl::for_each_ref<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::{lambda(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&)#1}&, raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&>(model::record_batch_reader::impl::for_each_ref<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::{lambda(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&)#1}&, raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&) at /vectorized/llvm/bin/../include/c++/v1/type_traits:3700
 (inlined by) decltype(auto) std::__1::__apply_tuple_impl<model::record_batch_reader::impl::for_each_ref<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::{lambda(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&)#1}&, std::__1::tuple<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer>&, 0ul>(model::record_batch_reader::impl::for_each_ref<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::{lambda(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&)#1}&, std::__1::tuple<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer>&, std::__1::__tuple_indices<0ul>) at /vectorized/llvm/bin/../include/c++/v1/tuple:1415
 (inlined by) decltype(auto) std::__1::apply<model::record_batch_reader::impl::for_each_ref<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::{lambda(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&)#1}&, std::__1::tuple<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer>&>(model::record_batch_reader::impl::for_each_ref<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::{lambda(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&)#1}&, std::__1::tuple<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer>&) at /vectorized/llvm/bin/../include/c++/v1/tuple:1424
 (inlined by) auto seastar::internal::do_with_impl<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer, model::record_batch_reader::impl::for_each_ref<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::{lambda(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&)#1}>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&&, model::record_batch_reader::impl::for_each_ref<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::{lambda(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&)#1}&&) at /vectorized/include/seastar/core/do_with.hh:96
 (inlined by) seastar::future<storage::append_result> seastar::futurize<seastar::future<storage::append_result> >::invoke<seastar::future<storage::append_result> (*&)(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&&, model::record_batch_reader::impl::for_each_ref<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::{lambda(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&)#1}&&), raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer, model::record_batch_reader::impl::for_each_ref<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::{lambda(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&)#1}>(seastar::future<storage::append_result> (*&)(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&&, model::record_batch_reader::impl::for_each_ref<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::{lambda(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&)#1}&&), raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&&, model::record_batch_reader::impl::for_each_ref<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::{lambda(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&)#1}&&) at /vectorized/include/seastar/core/future.hh:2135
 (inlined by) auto seastar::futurize_invoke<seastar::future<storage::append_result> (*&)(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&&, model::record_batch_reader::impl::for_each_ref<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::{lambda(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&)#1}&&), raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer, model::record_batch_reader::impl::for_each_ref<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::{lambda(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&)#1}>(seastar::future<storage::append_result> (*&)(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&&, model::record_batch_reader::impl::for_each_ref<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::{lambda(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&)#1}&&), raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&&, model::record_batch_reader::impl::for_each_ref<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::{lambda(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&)#1}&&) at /vectorized/include/seastar/core/future.hh:2166
 (inlined by) auto seastar::do_with<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer, model::record_batch_reader::impl::for_each_ref<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::{lambda(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&)#1}>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&&, model::record_batch_reader::impl::for_each_ref<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::{lambda(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer&)#1}&&) at /vectorized/include/seastar/core/do_with.hh:131
 (inlined by) auto model::record_batch_reader::impl::for_each_ref<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-09f40697b81d3241f-1/vectorized/redpanda/vbuild/release/clang/../../../src/v/model/record_batch_reader.h:91
 (inlined by) auto model::record_batch_reader::for_each_ref<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer>(raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >) && at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-09f40697b81d3241f-1/vectorized/redpanda/vbuild/release/clang/../../../src/v/model/record_batch_reader.h:203
 (inlined by) operator() at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-09f40697b81d3241f-1/vectorized/redpanda/vbuild/release/clang/../../../src/v/raft/consensus_utils.h:173
 (inlined by) decltype ((std::__1::forward<raft::details::for_each_ref_extract_configuration<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer>(detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true> >, model::record_batch_reader&&, raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::{lambda(std::__1::vector<raft::offset_configuration, std::__1::allocator<raft::offset_configuration> >&)#1}&>({parm#1}))(std::__1::forward<std::__1::vector<raft::offset_configuration, std::__1::allocator<raft::offset_configuration> >&>({parm#2}))) std::__1::__invoke_constexpr<raft::details::for_each_ref_extract_configuration<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer>(detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true> >, model::record_batch_reader&&, raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::{lambda(std::__1::vector<raft::offset_configuration, std::__1::allocator<raft::offset_configuration> >&)#1}&, std::__1::vector<raft::offset_configuration, std::__1::allocator<raft::offset_configuration> >&>(raft::details::for_each_ref_extract_configuration<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer>(detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true> >, model::record_batch_reader&&, raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::{lambda(std::__1::vector<raft::offset_configuration, std::__1::allocator<raft::offset_configuration> >&)#1}&, std::__1::vector<raft::offset_configuration, std::__1::allocator<raft::offset_configuration> >&) at /vectorized/llvm/bin/../include/c++/v1/type_traits:3700
 (inlined by) decltype(auto) std::__1::__apply_tuple_impl<raft::details::for_each_ref_extract_configuration<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer>(detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true> >, model::record_batch_reader&&, raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::{lambda(std::__1::vector<raft::offset_configuration, std::__1::allocator<raft::offset_configuration> >&)#1}&, std::__1::tuple<std::__1::vector<raft::offset_configuration, std::__1::allocator<raft::offset_configuration> > >&, 0ul>(raft::details::for_each_ref_extract_configuration<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer>(detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true> >, model::record_batch_reader&&, raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::{lambda(std::__1::vector<raft::offset_configuration, std::__1::allocator<raft::offset_configuration> >&)#1}&, std::__1::tuple<std::__1::vector<raft::offset_configuration, std::__1::allocator<raft::offset_configuration> > >&, std::__1::__tuple_indices<0ul>) at /vectorized/llvm/bin/../include/c++/v1/tuple:1415
 (inlined by) decltype(auto) std::__1::apply<raft::details::for_each_ref_extract_configuration<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer>(detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true> >, model::record_batch_reader&&, raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::{lambda(std::__1::vector<raft::offset_configuration, std::__1::allocator<raft::offset_configuration> >&)#1}&, std::__1::tuple<std::__1::vector<raft::offset_configuration, std::__1::allocator<raft::offset_configuration> > >&>(raft::details::for_each_ref_extract_configuration<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer>(detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true> >, model::record_batch_reader&&, raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::{lambda(std::__1::vector<raft::offset_configuration, std::__1::allocator<raft::offset_configuration> >&)#1}&, std::__1::tuple<std::__1::vector<raft::offset_configuration, std::__1::allocator<raft::offset_configuration> > >&) at /vectorized/llvm/bin/../include/c++/v1/tuple:1424
 (inlined by) auto seastar::internal::do_with_impl<std::__1::vector<raft::offset_configuration, std::__1::allocator<raft::offset_configuration> >, raft::details::for_each_ref_extract_configuration<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer>(detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true> >, model::record_batch_reader&&, raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::{lambda(std::__1::vector<raft::offset_configuration, std::__1::allocator<raft::offset_configuration> >&)#1}>(std::__1::vector<raft::offset_configuration, std::__1::allocator<raft::offset_configuration> >&&, raft::details::for_each_ref_extract_configuration<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer>(detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true> >, model::record_batch_reader&&, raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::{lambda(std::__1::vector<raft::offset_configuration, std::__1::allocator<raft::offset_configuration> >&)#1}&&) at /vectorized/include/seastar/core/do_with.hh:96
 (inlined by) seastar::future<std::__1::tuple<storage::append_result, std::__1::vector<raft::offset_configuration, std::__1::allocator<raft::offset_configuration> > > > seastar::futurize<seastar::future<std::__1::tuple<storage::append_result, std::__1::vector<raft::offset_configuration, std::__1::allocator<raft::offset_configuration> > > > >::invoke<seastar::future<std::__1::tuple<storage::append_result, std::__1::vector<raft::offset_configuration, std::__1::allocator<raft::offset_configuration> > > > (*&)(std::__1::vector<raft::offset_configuration, std::__1::allocator<raft::offset_configuration> >&&, raft::details::for_each_ref_extract_configuration<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer>(detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true> >, model::record_batch_reader&&, raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::{lambda(std::__1::vector<raft::offset_configuration, std::__1::allocator<raft::offset_configuration> >&)#1}&&), std::__1::vector<raft::offset_configuration, std::__1::allocator<raft::offset_configuration> >, raft::details::for_each_ref_extract_configuration<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer>(detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true> >, model::record_batch_reader&&, raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::{lambda(std::__1::vector<raft::offset_configuration, std::__1::allocator<raft::offset_configuration> >&)#1}>(seastar::future<std::__1::tuple<storage::append_result, std::__1::vector<raft::offset_configuration, std::__1::allocator<raft::offset_configuration> > > > (*&)(std::__1::vector<raft::offset_configuration, std::__1::allocator<raft::offset_configuration> >&&, raft::details::for_each_ref_extract_configuration<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer>(detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true> >, model::record_batch_reader&&, raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::{lambda(std::__1::vector<raft::offset_configuration, std::__1::allocator<raft::offset_configuration> >&)#1}&&), std::__1::vector<raft::offset_configuration, std::__1::allocator<raft::offset_configuration> >&&, raft::details::for_each_ref_extract_configuration<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer>(detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true> >, model::record_batch_reader&&, raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::{lambda(std::__1::vector<raft::offset_configuration, std::__1::allocator<raft::offset_configuration> >&)#1}&&) at /vectorized/include/seastar/core/future.hh:2135
 (inlined by) auto seastar::futurize_invoke<seastar::future<std::__1::tuple<storage::append_result, std::__1::vector<raft::offset_configuration, std::__1::allocator<raft::offset_configuration> > > > (*&)(std::__1::vector<raft::offset_configuration, std::__1::allocator<raft::offset_configuration> >&&, raft::details::for_each_ref_extract_configuration<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer>(detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true> >, model::record_batch_reader&&, raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::{lambda(std::__1::vector<raft::offset_configuration, std::__1::allocator<raft::offset_configuration> >&)#1}&&), std::__1::vector<raft::offset_configuration, std::__1::allocator<raft::offset_configuration> >, raft::details::for_each_ref_extract_configuration<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer>(detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true> >, model::record_batch_reader&&, raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::{lambda(std::__1::vector<raft::offset_configuration, std::__1::allocator<raft::offset_configuration> >&)#1}>(seastar::future<std::__1::tuple<storage::append_result, std::__1::vector<raft::offset_configuration, std::__1::allocator<raft::offset_configuration> > > > (*&)(std::__1::vector<raft::offset_configuration, std::__1::allocator<raft::offset_configuration> >&&, raft::details::for_each_ref_extract_configuration<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer>(detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true> >, model::record_batch_reader&&, raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::{lambda(std::__1::vector<raft::offset_configuration, std::__1::allocator<raft::offset_configuration> >&)#1}&&), std::__1::vector<raft::offset_configuration, std::__1::allocator<raft::offset_configuration> >&&, raft::details::for_each_ref_extract_configuration<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer>(detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true> >, model::record_batch_reader&&, raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::{lambda(std::__1::vector<raft::offset_configuration, std::__1::allocator<raft::offset_configuration> >&)#1}&&) at /vectorized/include/seastar/core/future.hh:2166
 (inlined by) auto seastar::do_with<std::__1::vector<raft::offset_configuration, std::__1::allocator<raft::offset_configuration> >, raft::details::for_each_ref_extract_configuration<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer>(detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true> >, model::record_batch_reader&&, raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::{lambda(std::__1::vector<raft::offset_configuration, std::__1::allocator<raft::offset_configuration> >&)#1}>(std::__1::vector<raft::offset_configuration, std::__1::allocator<raft::offset_configuration> >&&, raft::details::for_each_ref_extract_configuration<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer>(detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true> >, model::record_batch_reader&&, raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::{lambda(std::__1::vector<raft::offset_configuration, std::__1::allocator<raft::offset_configuration> >&)#1}&&) at /vectorized/include/seastar/core/do_with.hh:131
auto raft::details::for_each_ref_extract_configuration<raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer>(detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true> >, model::record_batch_reader&&, raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>)::consumer, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-09f40697b81d3241f-1/vectorized/redpanda/vbuild/release/clang/../../../src/v/raft/consensus_utils.h:167
 (inlined by) raft::consensus::disk_append(model::record_batch_reader&&, seastar::bool_class<raft::update_last_quorum_index>) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-09f40697b81d3241f-1/vectorized/redpanda/vbuild/release/clang/../../../src/v/raft/consensus.cc:2137
operator() at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-09f40697b81d3241f-1/vectorized/redpanda/vbuild/release/clang/../../../src/v/raft/replicate_entries_stm.cc:180
 (inlined by) seastar::future<storage::append_result> seastar::futurize<seastar::future<storage::append_result> >::invoke<raft::replicate_entries_stm::append_to_self()::$_9, raft::append_entries_request>(raft::replicate_entries_stm::append_to_self()::$_9&&, raft::append_entries_request&&) at /vectorized/include/seastar/core/future.hh:2135
 (inlined by) seastar::future<storage::append_result> seastar::future<raft::append_entries_request>::then_impl<raft::replicate_entries_stm::append_to_self()::$_9, seastar::future<storage::append_result> >(raft::replicate_entries_stm::append_to_self()::$_9&&) at /vectorized/include/seastar/core/future.hh:1601
 (inlined by) seastar::internal::future_result<raft::replicate_entries_stm::append_to_self()::$_9, raft::append_entries_request>::future_type seastar::internal::call_then_impl<seastar::future<raft::append_entries_request> >::run<raft::replicate_entries_stm::append_to_self()::$_9>(seastar::future<raft::append_entries_request>&, raft::replicate_entries_stm::append_to_self()::$_9&&) at /vectorized/include/seastar/core/future.hh:1234
 (inlined by) seastar::future<storage::append_result> seastar::future<raft::append_entries_request>::then<raft::replicate_entries_stm::append_to_self()::$_9, seastar::future<storage::append_result> >(raft::replicate_entries_stm::append_to_self()::$_9&&) at /vectorized/include/seastar/core/future.hh:1520
 (inlined by) raft::replicate_entries_stm::append_to_self() at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-09f40697b81d3241f-1/vectorized/redpanda/vbuild/release/clang/../../../src/v/raft/replicate_entries_stm.cc:177
raft::replicate_entries_stm::apply(std::__1::vector<seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>, std::__1::allocator<seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock> > >) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-09f40697b81d3241f-1/vectorized/redpanda/vbuild/release/clang/../../../src/v/raft/replicate_entries_stm.cc:231
raft::consensus::dispatch_replicate(raft::append_entries_request, std::__1::vector<seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>, std::__1::allocator<seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock> > >, absl::lts_20210324::flat_hash_map<raft::vnode, detail::base_named_type<unsigned long, raft::follower_req_seq_tag, std::__1::integral_constant<bool, true> >, absl::lts_20210324::hash_internal::Hash<raft::vnode>, std::__1::equal_to<raft::vnode>, std::__1::allocator<std::__1::pair<raft::vnode const, detail::base_named_type<unsigned long, raft::follower_req_seq_tag, std::__1::integral_constant<bool, true> > > > >) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-09f40697b81d3241f-1/vectorized/redpanda/vbuild/release/clang/../../../src/v/raft/consensus.cc:2044
raft::replicate_batcher::do_flush(std::__1::vector<seastar::lw_shared_ptr<raft::replicate_batcher::item>, std::__1::allocator<seastar::lw_shared_ptr<raft::replicate_batcher::item> > >, raft::append_entries_request, std::__1::vector<seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>, std::__1::allocator<seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock> > >, absl::lts_20210324::flat_hash_map<raft::vnode, detail::base_named_type<unsigned long, raft::follower_req_seq_tag, std::__1::integral_constant<bool, true> >, absl::lts_20210324::hash_internal::Hash<raft::vnode>, std::__1::equal_to<raft::vnode>, std::__1::allocator<std::__1::pair<raft::vnode const, detail::base_named_type<unsigned long, raft::follower_req_seq_tag, std::__1::integral_constant<bool, true> > > > >) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-09f40697b81d3241f-1/vectorized/redpanda/vbuild/release/clang/../../../src/v/raft/replicate_batcher.cc:286
operator() at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-09f40697b81d3241f-1/vectorized/redpanda/vbuild/release/clang/../../../src/v/raft/replicate_batcher.cc:242
seastar::future<void> seastar::futurize<seastar::future<void> >::invoke<raft::replicate_batcher::flush(seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>, bool)::$_2::operator()()::{lambda(seastar::future<seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock> >)#1}, seastar::future<seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock> > >(raft::replicate_batcher::flush(seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>, bool)::$_2::operator()()::{lambda(seastar::future<seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock> >)#1}&&, seastar::future<seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock> >&&) at /vectorized/include/seastar/core/future.hh:2135
 (inlined by) seastar::futurize<seastar::future<void> >::type seastar::future<seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock> >::then_wrapped_common<true, seastar::future<void>, raft::replicate_batcher::flush(seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>, bool)::$_2::operator()()::{lambda(seastar::future<seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock> >)#1}>(raft::replicate_batcher::flush(seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>, bool)::$_2::operator()()::{lambda(seastar::future<seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock> >)#1}&&) at /vectorized/include/seastar/core/future.hh:1685
 (inlined by) seastar::futurize<seastar::future<void> >::type seastar::future<seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock> >::then_wrapped_maybe_erase<true, seastar::future<void>, raft::replicate_batcher::flush(seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>, bool)::$_2::operator()()::{lambda(seastar::future<seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock> >)#1}>(raft::replicate_batcher::flush(seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>, bool)::$_2::operator()()::{lambda(seastar::future<seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock> >)#1}&&) at /vectorized/include/seastar/core/future.hh:1644
 (inlined by) seastar::futurize<seastar::future<void> >::type seastar::future<seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock> >::then_wrapped<raft::replicate_batcher::flush(seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>, bool)::$_2::operator()()::{lambda(seastar::future<seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock> >)#1}, seastar::future<void> >(raft::replicate_batcher::flush(seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>, bool)::$_2::operator()()::{lambda(seastar::future<seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock> >)#1}&&) && at /vectorized/include/seastar/core/future.hh:1635
 (inlined by) operator() at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-09f40697b81d3241f-1/vectorized/redpanda/vbuild/release/clang/../../../src/v/raft/replicate_batcher.cc:167
 (inlined by) seastar::future<void> seastar::futurize<seastar::future<void> >::invoke<raft::replicate_batcher::flush(seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>, bool)::$_2>(raft::replicate_batcher::flush(seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>, bool)::$_2&&) at /vectorized/include/seastar/core/future.hh:2135
 (inlined by) auto seastar::futurize_invoke<raft::replicate_batcher::flush(seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>, bool)::$_2>(raft::replicate_batcher::flush(seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>, bool)::$_2&&) at /vectorized/include/seastar/core/future.hh:2166
 (inlined by) auto seastar::internal::invoke_func_with_gate<raft::replicate_batcher::flush(seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>, bool)::$_2>(seastar::gate&, raft::replicate_batcher::flush(seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>, bool)::$_2&&) at /vectorized/include/seastar/core/gate.hh:221
 (inlined by) auto seastar::try_with_gate<raft::replicate_batcher::flush(seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>, bool)::$_2>(seastar::gate&, raft::replicate_batcher::flush(seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>, bool)::$_2&&) at /vectorized/include/seastar/core/gate.hh:261
 (inlined by) raft::replicate_batcher::flush(seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>, bool) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-09f40697b81d3241f-1/vectorized/redpanda/vbuild/release/clang/../../../src/v/raft/replicate_batcher.cc:160
operator() at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-09f40697b81d3241f-1/vectorized/redpanda/vbuild/release/clang/../../../src/v/raft/replicate_batcher.cc:62
 (inlined by) seastar::future<void> seastar::futurize<seastar::future<void> >::invoke<raft::replicate_batcher::replicate(std::__1::optional<detail::base_named_type<long, model::model_raft_term_id_type, std::__1::integral_constant<bool, true> > >, model::record_batch_reader&&)::$_3::operator()(seastar::future<seastar::lw_shared_ptr<raft::replicate_batcher::item> >)::{lambda(seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>)#1}, seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock> >(raft::replicate_batcher::replicate(std::__1::optional<detail::base_named_type<long, model::model_raft_term_id_type, std::__1::integral_constant<bool, true> > >, model::record_batch_reader&&)::$_3::operator()(seastar::future<seastar::lw_shared_ptr<raft::replicate_batcher::item> >)::{lambda(seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>)#1}&&, seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>&&) at /vectorized/include/seastar/core/future.hh:2135
 (inlined by) seastar::future<void> seastar::future<seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock> >::then_impl<raft::replicate_batcher::replicate(std::__1::optional<detail::base_named_type<long, model::model_raft_term_id_type, std::__1::integral_constant<bool, true> > >, model::record_batch_reader&&)::$_3::operator()(seastar::future<seastar::lw_shared_ptr<raft::replicate_batcher::item> >)::{lambda(seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>)#1}, seastar::future<void> >(raft::replicate_batcher::replicate(std::__1::optional<detail::base_named_type<long, model::model_raft_term_id_type, std::__1::integral_constant<bool, true> > >, model::record_batch_reader&&)::$_3::operator()(seastar::future<seastar::lw_shared_ptr<raft::replicate_batcher::item> >)::{lambda(seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>)#1}&&) at /vectorized/include/seastar/core/future.hh:1601
 (inlined by) seastar::internal::future_result<raft::replicate_batcher::replicate(std::__1::optional<detail::base_named_type<long, model::model_raft_term_id_type, std::__1::integral_constant<bool, true> > >, model::record_batch_reader&&)::$_3::operator()(seastar::future<seastar::lw_shared_ptr<raft::replicate_batcher::item> >)::{lambda(seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>)#1}, seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock> >::future_type seastar::internal::call_then_impl<seastar::future<seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock> > >::run<raft::replicate_batcher::replicate(std::__1::optional<detail::base_named_type<long, model::model_raft_term_id_type, std::__1::integral_constant<bool, true> > >, model::record_batch_reader&&)::$_3::operator()(seastar::future<seastar::lw_shared_ptr<raft::replicate_batcher::item> >)::{lambda(seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>)#1}>(seastar::future<seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock> >&, raft::replicate_batcher::replicate(std::__1::optional<detail::base_named_type<long, model::model_raft_term_id_type, std::__1::integral_constant<bool, true> > >, model::record_batch_reader&&)::$_3::operator()(seastar::future<seastar::lw_shared_ptr<raft::replicate_batcher::item> >)::{lambda(seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>)#1}&&) at /vectorized/include/seastar/core/future.hh:1234
 (inlined by) seastar::future<void> seastar::future<seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock> >::then<raft::replicate_batcher::replicate(std::__1::optional<detail::base_named_type<long, model::model_raft_term_id_type, std::__1::integral_constant<bool, true> > >, model::record_batch_reader&&)::$_3::operator()(seastar::future<seastar::lw_shared_ptr<raft::replicate_batcher::item> >)::{lambda(seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>)#1}, seastar::future<void> >(raft::replicate_batcher::replicate(std::__1::optional<detail::base_named_type<long, model::model_raft_term_id_type, std::__1::integral_constant<bool, true> > >, model::record_batch_reader&&)::$_3::operator()(seastar::future<seastar::lw_shared_ptr<raft::replicate_batcher::item> >)::{lambda(seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>)#1}&&) at /vectorized/include/seastar/core/future.hh:1520
 (inlined by) operator() at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-09f40697b81d3241f-1/vectorized/redpanda/vbuild/release/clang/../../../src/v/raft/replicate_batcher.cc:61
seastar::future<boost::outcome_v2::basic_result<raft::replicate_result, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<raft::replicate_result, std::__1::error_code, void> > > seastar::futurize<seastar::future<boost::outcome_v2::basic_result<raft::replicate_result, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<raft::replicate_result, std::__1::error_code, void> > > >::invoke<raft::replicate_batcher::replicate(std::__1::optional<detail::base_named_type<long, model::model_raft_term_id_type, std::__1::integral_constant<bool, true> > >, model::record_batch_reader&&)::$_3, seastar::future<seastar::lw_shared_ptr<raft::replicate_batcher::item> > >(raft::replicate_batcher::replicate(std::__1::optional<detail::base_named_type<long, model::model_raft_term_id_type, std::__1::integral_constant<bool, true> > >, model::record_batch_reader&&)::$_3&&, seastar::future<seastar::lw_shared_ptr<raft::replicate_batcher::item> >&&) at /vectorized/include/seastar/core/future.hh:2135
 (inlined by) seastar::futurize<seastar::future<boost::outcome_v2::basic_result<raft::replicate_result, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<raft::replicate_result, std::__1::error_code, void> > > >::type seastar::future<seastar::lw_shared_ptr<raft::replicate_batcher::item> >::then_wrapped_common<true, seastar::future<boost::outcome_v2::basic_result<raft::replicate_result, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<raft::replicate_result, std::__1::error_code, void> > >, raft::replicate_batcher::replicate(std::__1::optional<detail::base_named_type<long, model::model_raft_term_id_type, std::__1::integral_constant<bool, true> > >, model::record_batch_reader&&)::$_3>(raft::replicate_batcher::replicate(std::__1::optional<detail::base_named_type<long, model::model_raft_term_id_type, std::__1::integral_constant<bool, true> > >, model::record_batch_reader&&)::$_3&&) at /vectorized/include/seastar/core/future.hh:1685
 (inlined by) seastar::futurize<seastar::future<boost::outcome_v2::basic_result<raft::replicate_result, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<raft::replicate_result, std::__1::error_code, void> > > >::type seastar::future<seastar::lw_shared_ptr<raft::replicate_batcher::item> >::then_wrapped_maybe_erase<true, seastar::future<boost::outcome_v2::basic_result<raft::replicate_result, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<raft::replicate_result, std::__1::error_code, void> > >, raft::replicate_batcher::replicate(std::__1::optional<detail::base_named_type<long, model::model_raft_term_id_type, std::__1::integral_constant<bool, true> > >, model::record_batch_reader&&)::$_3>(raft::replicate_batcher::replicate(std::__1::optional<detail::base_named_type<long, model::model_raft_term_id_type, std::__1::integral_constant<bool, true> > >, model::record_batch_reader&&)::$_3&&) at /vectorized/include/seastar/core/future.hh:1644
 (inlined by) seastar::futurize<seastar::future<boost::outcome_v2::basic_result<raft::replicate_result, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<raft::replicate_result, std::__1::error_code, void> > > >::type seastar::future<seastar::lw_shared_ptr<raft::replicate_batcher::item> >::then_wrapped<raft::replicate_batcher::replicate(std::__1::optional<detail::base_named_type<long, model::model_raft_term_id_type, std::__1::integral_constant<bool, true> > >, model::record_batch_reader&&)::$_3, seastar::future<boost::outcome_v2::basic_result<raft::replicate_result, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<raft::replicate_result, std::__1::error_code, void> > > >(raft::replicate_batcher::replicate(std::__1::optional<detail::base_named_type<long, model::model_raft_term_id_type, std::__1::integral_constant<bool, true> > >, model::record_batch_reader&&)::$_3&&) && at /vectorized/include/seastar/core/future.hh:1635
 (inlined by) raft::replicate_batcher::replicate(std::__1::optional<detail::base_named_type<long, model::model_raft_term_id_type, std::__1::integral_constant<bool, true> > >, model::record_batch_reader&&) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-09f40697b81d3241f-1/vectorized/redpanda/vbuild/release/clang/../../../src/v/raft/replicate_batcher.cc:48
raft::consensus::do_replicate(std::__1::optional<detail::base_named_type<long, model::model_raft_term_id_type, std::__1::integral_constant<bool, true> > >, model::record_batch_reader&&, raft::replicate_options) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-09f40697b81d3241f-1/vectorized/redpanda/vbuild/release/clang/../../../src/v/raft/consensus.cc:659
raft::consensus::replicate(detail::base_named_type<long, model::model_raft_term_id_type, std::__1::integral_constant<bool, true> >, model::record_batch_reader&&, raft::replicate_options) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-09f40697b81d3241f-1/vectorized/redpanda/vbuild/release/clang/../../../src/v/raft/consensus.cc:614
cluster::rm_stm::replicate_seq(model::batch_identity, model::record_batch_reader, raft::replicate_options) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-09f40697b81d3241f-1/vectorized/redpanda/vbuild/release/clang/../../../src/v/cluster/rm_stm.cc:901
operator() at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-09f40697b81d3241f-1/vectorized/redpanda/vbuild/release/clang/../../../src/v/cluster/rm_stm.cc:703
seastar::future<boost::outcome_v2::basic_result<raft::replicate_result, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<raft::replicate_result, std::__1::error_code, void> > > seastar::futurize<seastar::future<boost::outcome_v2::basic_result<raft::replicate_result, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<raft::replicate_result, std::__1::error_code, void> > > >::invoke<cluster::rm_stm::replicate(model::batch_identity, model::record_batch_reader, raft::replicate_options)::$_4, seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock> >(cluster::rm_stm::replicate(model::batch_identity, model::record_batch_reader, raft::replicate_options)::$_4&&, seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>&&) at /vectorized/include/seastar/core/future.hh:2135
 (inlined by) seastar::future<boost::outcome_v2::basic_result<raft::replicate_result, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<raft::replicate_result, std::__1::error_code, void> > > seastar::future<seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock> >::then_impl<cluster::rm_stm::replicate(model::batch_identity, model::record_batch_reader, raft::replicate_options)::$_4, seastar::future<boost::outcome_v2::basic_result<raft::replicate_result, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<raft::replicate_result, std::__1::error_code, void> > > >(cluster::rm_stm::replicate(model::batch_identity, model::record_batch_reader, raft::replicate_options)::$_4&&) at /vectorized/include/seastar/core/future.hh:1601
 (inlined by) seastar::internal::future_result<cluster::rm_stm::replicate(model::batch_identity, model::record_batch_reader, raft::replicate_options)::$_4, seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock> >::future_type seastar::internal::call_then_impl<seastar::future<seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock> > >::run<cluster::rm_stm::replicate(model::batch_identity, model::record_batch_reader, raft::replicate_options)::$_4>(seastar::future<seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock> >&, cluster::rm_stm::replicate(model::batch_identity, model::record_batch_reader, raft::replicate_options)::$_4&&) at /vectorized/include/seastar/core/future.hh:1234
 (inlined by) seastar::future<boost::outcome_v2::basic_result<raft::replicate_result, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<raft::replicate_result, std::__1::error_code, void> > > seastar::future<seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock> >::then<cluster::rm_stm::replicate(model::batch_identity, model::record_batch_reader, raft::replicate_options)::$_4, seastar::future<boost::outcome_v2::basic_result<raft::replicate_result, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<raft::replicate_result, std::__1::error_code, void> > > >(cluster::rm_stm::replicate(model::batch_identity, model::record_batch_reader, raft::replicate_options)::$_4&&) at /vectorized/include/seastar/core/future.hh:1520
 (inlined by) cluster::rm_stm::replicate(model::batch_identity, model::record_batch_reader, raft::replicate_options) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-09f40697b81d3241f-1/vectorized/redpanda/vbuild/release/clang/../../../src/v/cluster/rm_stm.cc:692
cluster::partition::replicate_in_stages(model::batch_identity, model::record_batch_reader&&, raft::replicate_options) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-09f40697b81d3241f-1/vectorized/redpanda/vbuild/release/clang/../../../src/v/cluster/partition.cc:211
kafka::replicated_partition::replicate(model::batch_identity, model::record_batch_reader&&, raft::replicate_options) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-09f40697b81d3241f-1/vectorized/redpanda/vbuild/release/clang/../../../src/v/kafka/server/replicated_partition.cc:177
kafka::partition_append(detail::base_named_type<int, model::model_partition_id_type, std::__1::integral_constant<bool, true> >, seastar::lw_shared_ptr<kafka::replicated_partition>, model::batch_identity, model::record_batch_reader, short, int) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-09f40697b81d3241f-1/vectorized/redpanda/vbuild/release/clang/../../../src/v/kafka/server/handlers/produce.cc:170
 (inlined by) operator() at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-09f40697b81d3241f-1/vectorized/redpanda/vbuild/release/clang/../../../src/v/kafka/server/handlers/produce.cc:287
 (inlined by) decltype ((std::__1::forward<kafka::produce_topic_partition(kafka::produce_ctx&, kafka::topic_produce_data&, kafka::partition_produce_data&)::$_2>({parm#1}))(std::__1::forward<cluster::partition_manager&>({parm#2}))) std::__1::__invoke_constexpr<kafka::produce_topic_partition(kafka::produce_ctx&, kafka::topic_produce_data&, kafka::partition_produce_data&)::$_2, cluster::partition_manager&>(kafka::produce_topic_partition(kafka::produce_ctx&, kafka::topic_produce_data&, kafka::partition_produce_data&)::$_2&&, cluster::partition_manager&) at /vectorized/llvm/bin/../include/c++/v1/type_traits:3700
 (inlined by) decltype(auto) std::__1::__apply_tuple_impl<kafka::produce_topic_partition(kafka::produce_ctx&, kafka::topic_produce_data&, kafka::partition_produce_data&)::$_2, std::__1::tuple<cluster::partition_manager&>, 0ul>(kafka::produce_topic_partition(kafka::produce_ctx&, kafka::topic_produce_data&, kafka::partition_produce_data&)::$_2&&, std::__1::tuple<cluster::partition_manager&>&&, std::__1::__tuple_indices<0ul>) at /vectorized/llvm/bin/../include/c++/v1/tuple:1415
 (inlined by) decltype(auto) std::__1::apply<kafka::produce_topic_partition(kafka::produce_ctx&, kafka::topic_produce_data&, kafka::partition_produce_data&)::$_2, std::__1::tuple<cluster::partition_manager&> >(kafka::produce_topic_partition(kafka::produce_ctx&, kafka::topic_produce_data&, kafka::partition_produce_data&)::$_2&&, std::__1::tuple<cluster::partition_manager&>&&) at /vectorized/llvm/bin/../include/c++/v1/tuple:1424
 (inlined by) operator() at /vectorized/include/seastar/core/sharded.hh:411
 (inlined by) seastar::future<kafka::partition_produce_response> seastar::futurize<seastar::future<kafka::partition_produce_response> >::invoke<seastar::sharded<cluster::partition_manager>::invoke_on<kafka::produce_topic_partition(kafka::produce_ctx&, kafka::topic_produce_data&, kafka::partition_produce_data&)::$_2, , seastar::future<kafka::partition_produce_response> >(unsigned int, seastar::smp_submit_to_options, kafka::produce_topic_partition(kafka::produce_ctx&, kafka::topic_produce_data&, kafka::partition_produce_data&)::$_2&&)::{lambda()#1}&>(kafka::produce_topic_partition(kafka::produce_ctx&, kafka::topic_produce_data&, kafka::partition_produce_data&)::$_2&&) at /vectorized/include/seastar/core/future.hh:2135
seastar::smp_message_queue::async_work_item<seastar::sharded<cluster::partition_manager>::invoke_on<kafka::produce_topic_partition(kafka::produce_ctx&, kafka::topic_produce_data&, kafka::partition_produce_data&)::$_2, , seastar::future<kafka::partition_produce_response> >(unsigned int, seastar::smp_submit_to_options, kafka::produce_topic_partition(kafka::produce_ctx&, kafka::topic_produce_data&, kafka::partition_produce_data&)::$_2&&)::{lambda()#1}>::run_and_dispose() at /vectorized/include/seastar/core/smp.hh:231
seastar::reactor::run_tasks(seastar::reactor::task_queue&) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:2230
 (inlined by) seastar::reactor::run_some_tasks() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:2639
seastar::reactor::do_run() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:2808
operator() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:3996
 (inlined by) decltype ((std::__1::forward<seastar::smp::configure(boost::program_options::variables_map, seastar::reactor_config)::$_92&>({parm#1}))()) std::__1::__invoke<seastar::smp::configure(boost::program_options::variables_map, seastar::reactor_config)::$_92&>(seastar::smp::configure(boost::program_options::variables_map, seastar::reactor_config)::$_92&) at /vectorized/llvm/bin/../include/c++/v1/type_traits:3694
 (inlined by) void std::__1::__invoke_void_return_wrapper<void, true>::__call<seastar::smp::configure(boost::program_options::variables_map, seastar::reactor_config)::$_92&>(seastar::smp::configure(boost::program_options::variables_map, seastar::reactor_config)::$_92&) at /vectorized/llvm/bin/../include/c++/v1/__functional_base:348
 (inlined by) std::__1::__function::__alloc_func<seastar::smp::configure(boost::program_options::variables_map, seastar::reactor_config)::$_92, std::__1::allocator<seastar::smp::configure(boost::program_options::variables_map, seastar::reactor_config)::$_92>, void ()>::operator()() at /vectorized/llvm/bin/../include/c++/v1/functional:1558
 (inlined by) std::__1::__function::__func<seastar::smp::configure(boost::program_options::variables_map, seastar::reactor_config)::$_92, std::__1::allocator<seastar::smp::configure(boost::program_options::variables_map, seastar::reactor_config)::$_92>, void ()>::operator()() at /vectorized/llvm/bin/../include/c++/v1/functional:1732
std::__1::__function::__value_func<void ()>::operator()() const at /vectorized/llvm/bin/../include/c++/v1/functional:1885
 (inlined by) std::__1::function<void ()>::operator()() const at /vectorized/llvm/bin/../include/c++/v1/functional:2560
 (inlined by) seastar::posix_thread::start_routine(void*) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/posix.cc:60
/opt/redpanda/lib/libpthread.so.0: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=7b0cdaf878ab4f99078439d864af70a5fd7b5a2c, for GNU/Linux 3.2.0, stripped
NyaliaLui commented 2 years ago

the grafana logs before the segfault:

    2022-01-16T15:59:51.280041202Z stderr F INFO  2022-01-16 15:59:51,279 [shard 0] kafka - fetch.cc:346 - Fetch requested very large response (220200960), clamping each partition's max_bytes to 3195660 bytes
    2022-01-16T15:59:51.28004731Z stderr F INFO  2022-01-16 15:59:51,279 [shard 11] kafka - fetch.cc:346 - Fetch requested very large response (230686720), clamping each partition's max_bytes to 3050402 bytes
    2022-01-16T15:59:51.280051187Z stderr F INFO  2022-01-16 15:59:51,279 [shard 10] kafka - fetch.cc:346 - Fetch requested very large response (230686720), clamping each partition's max_bytes to 3050402 bytes
    2022-01-16T15:59:51.280055828Z stderr F INFO  2022-01-16 15:59:51,279 [shard 12] kafka - fetch.cc:346 - Fetch requested very large response (220200960), clamping each partition's max_bytes to 3195660 bytes
    2022-01-16T15:59:51.280059719Z stderr F INFO  2022-01-16 15:59:51,279 [shard 13] kafka - fetch.cc:346 - Fetch requested very large response (220200960), clamping each partition's max_bytes to 3195660 bytes
    2022-01-16T15:59:51.280063513Z stderr F INFO  2022-01-16 15:59:51,279 [shard 15] kafka - fetch.cc:346 - Fetch requested very large response (220200960), clamping each partition's max_bytes to 3195660 bytes
    2022-01-16T15:59:51.280067244Z stderr F INFO  2022-01-16 15:59:51,279 [shard 9] kafka - fetch.cc:346 - Fetch requested very large response (230686720), clamping each partition's max_bytes to 3050402 bytes
    2022-01-16T15:59:51.280078468Z stderr F INFO  2022-01-16 15:59:51,279 [shard 14] kafka - fetch.cc:346 - Fetch requested very large response (220200960), clamping each partition's max_bytes to 3195660 bytes
    2022-01-16T15:59:51.280155065Z stderr F INFO  2022-01-16 15:59:51,280 [shard 3] kafka - fetch.cc:346 - Fetch requested very large response (230686720), clamping each partition's max_bytes to 3050402 bytes
    2022-01-16T15:59:51.280373543Z stderr F INFO  2022-01-16 15:59:51,280 [shard 1] kafka - fetch.cc:346 - Fetch requested very large response (230686720), clamping each partition's max_bytes to 3050402 bytes
NyaliaLui commented 2 years ago

Brokers were configured with the new storage_read_buffer_size and storage_read_readahead_count configs: Do kubectl edit cluster -n <namespace> <cluster name> to get

spec:
   additionalConfiguration:
     redpanda.default_topic_replications: "3"
     redpanda.id_allocator_replication: "3"
     redpanda.storage_read_buffer_size: "32768"
     redpanda.storage_read_readahead_count: "2"

The topic:

SUMMARY
=======
NAME        test-2k-e
PARTITIONS  2048
REPLICAS    3

CONFIGS
=======
KEY                     VALUE                          SOURCE
cleanup.policy          delete                         DYNAMIC_TOPIC_CONFIG
compression.type        producer                       DEFAULT_CONFIG
message.timestamp.type  CreateTime                     DEFAULT_CONFIG
partition_count         2048                           DYNAMIC_TOPIC_CONFIG
redpanda.datapolicy     function_name:  script_name:   DEFAULT_CONFIG
redpanda.remote.read    true                           DYNAMIC_TOPIC_CONFIG
redpanda.remote.write   true                           DYNAMIC_TOPIC_CONFIG
replication_factor      3                              DYNAMIC_TOPIC_CONFIG
retention.bytes         2147483648                     DYNAMIC_TOPIC_CONFIG
retention.ms            604800000                      DEFAULT_CONFIG
segment.bytes           1073741824                     DYNAMIC_TOPIC_CONFIG
jcsp commented 2 years ago

This is the constructor for an append_challenged_posix_file_impl::op failing, presumably on failure to allocate.

jcsp commented 1 year ago

Closing individual bad_alloc tickets: our mid-term position is probably going to be to assert out on bad alloc.