Segmentation fault after uncaught bad_alloc exception

[NyaliaLui]

Version & Environment

Redpanda version: v21.11.3-si-beta7 Shadow Index Cache has 300GB of space On BYOC

The segfault and backtrace

libc++abi: terminating with uncaught exception of type std::bad_alloc: std::bad_alloc
Aborting on shard 0.
Backtrace:
  0x381bfc8
  0x2968795699ff
  /opt/redpanda/lib/libc.so.6+0x3d291
  /opt/redpanda/lib/libc.so.6+0x268a3
  0x2968793d9f85
  /opt/redpanda/lib/libc++abi.so.1+0xf90e
  0x2968793d9152
  0x2968793d90f7
  0x3b1256a
  0x12e7e19
  0x12e5dd9
  0x12e5639
  0x12e9ddb
  0x37cb5d4
  0x37cea97
  0x37cbe75
  0x371dc99
  0x371bd30
  0x116e014
  0x3b1291c
  /opt/redpanda/lib/libc.so.6+0x27b74
  0x116ae6d

Topic:

SUMMARY
=======
NAME        test-2k
PARTITIONS  2048
REPLICAS    3

CONFIGS
=======
KEY                     VALUE                          SOURCE
cleanup.policy          delete                         DYNAMIC_TOPIC_CONFIG
compression.type        producer                       DEFAULT_CONFIG
message.timestamp.type  CreateTime                     DEFAULT_CONFIG
partition_count         2048                           DYNAMIC_TOPIC_CONFIG
redpanda.datapolicy     function_name:  script_name:   DEFAULT_CONFIG
redpanda.remote.read    true                           DYNAMIC_TOPIC_CONFIG
redpanda.remote.write   true                           DYNAMIC_TOPIC_CONFIG
replication_factor      3                              DYNAMIC_TOPIC_CONFIG
retention.bytes         134217728                      DYNAMIC_TOPIC_CONFIG
retention.ms            300000                         DYNAMIC_TOPIC_CONFIG
segment.bytes           67108864                       DYNAMIC_TOPIC_CONFIG

Brokers were configured with the new storage_read_buffer_size and storage_read_readahead_count configs: Do kubectl edit cluster -n <namespace> <cluster name> to get

spec:
   additionalConfiguration:
     redpanda.default_topic_replications: "3"
     redpanda.id_allocator_replication: "3"
     redpanda.storage_read_buffer_size: "4096"
     redpanda.storage_read_readahead_count: "1"

[NyaliaLui]

A piece from the large backtrace:

__clang_call_terminate at main.cc:?
void seastar::future<raft::append_entries_reply>::schedule<seastar::internal::promise_base_with_type<raft::append_entries_reply>, seastar::future<raft::append_entries_reply>::handle_exception_type<raft::service<cluster::partition_manager, cluster::shard_table>::dispatch_hbeats_to_groups(cluster::partition_manager&, seastar::foreign_ptr<std::__1::unique_ptr<std::__1::vector<raft::append_entries_request, std::__1::allocator<raft::append_entries_request> >, std::__1::default_delete<std::__1::vector<raft::append_entries_request, std::__1::allocator<raft::append_entries_request> > > > >)::{lambda(raft::append_entries_request&)#1}::operator()(raft::append_entries_request&)::{lambda(seastar::timed_out_error const&)#1}>(raft::service<cluster::partition_manager, cluster::shard_table>::dispatch_hbeats_to_groups(cluster::partition_manager&, seastar::foreign_ptr<std::__1::unique_ptr<std::__1::vector<raft::append_entries_request, std::__1::allocator<raft::append_entries_request> >, std::__1::default_delete<std::__1::vector<raft::append_entries_request, std::__1::allocator<raft::append_entries_request> > > > >)::{lambda(raft::append_entries_request&)#1}::operator()(raft::append_entries_request&)::{lambda(seastar::timed_out_error const&)#1}&&)::{lambda(auto:1&&)#1}, seastar::future<raft::append_entries_reply>::then_wrapped_nrvo<seastar::future<raft::append_entries_reply>, seastar::future<raft::append_entries_reply>::handle_exception_type<raft::service<cluster::partition_manager, cluster::shard_table>::dispatch_hbeats_to_groups(cluster::partition_manager&, seastar::foreign_ptr<std::__1::unique_ptr<std::__1::vector<raft::append_entries_request, std::__1::allocator<raft::append_entries_request> >, std::__1::default_delete<std::__1::vector<raft::append_entries_request, std::__1::allocator<raft::append_entries_request> > > > >)::{lambda(raft::append_entries_request&)#1}::operator()(raft::append_entries_request&)::{lambda(seastar::timed_out_error const&)#1}>(raft::service<cluster::partition_manager, cluster::shard_table>::dispatch_hbeats_to_groups(cluster::partition_manager&, seastar::foreign_ptr<std::__1::unique_ptr<std::__1::vector<raft::append_entries_request, std::__1::allocator<raft::append_entries_request> >, std::__1::default_delete<std::__1::vector<raft::append_entries_request, std::__1::allocator<raft::append_entries_request> > > > >)::{lambda(raft::append_entries_request&)#1}::operator()(raft::append_entries_request&)::{lambda(seastar::timed_out_error const&)#1}&&)::{lambda(auto:1&&)#1}>(seastar::future<raft::append_entries_reply>::handle_exception_type<raft::service<cluster::partition_manager, cluster::shard_table>::dispatch_hbeats_to_groups(cluster::partition_manager&, seastar::foreign_ptr<std::__1::unique_ptr<std::__1::vector<raft::append_entries_request, std::__1::allocator<raft::append_entries_request> >, std::__1::default_delete<std::__1::vector<raft::append_entries_request, std::__1::allocator<raft::append_entries_request> > > > >)::{lambda(raft::append_entries_request&)#1}::operator()(raft::append_entries_request&)::{lambda(seastar::timed_out_error const&)#1}>(raft::service<cluster::partition_manager, cluster::shard_table>::dispatch_hbeats_to_groups(cluster::partition_manager&, seastar::foreign_ptr<std::__1::unique_ptr<std::__1::vector<raft::append_entries_request, std::__1::allocator<raft::append_entries_request> >, std::__1::default_delete<std::__1::vector<raft::append_entries_request, std::__1::allocator<raft::append_entries_request> > > > >)::{lambda(raft::append_entries_request&)#1}::operator()(raft::append_entries_request&)::{lambda(seastar::timed_out_error const&)#1}&&)::{lambda(auto:1&&)#1}&&)::{lambda(seastar::internal::promise_base_with_type<raft::append_entries_reply>&&, seastar::future<raft::append_entries_reply>::handle_exception_type<raft::service<cluster::partition_manager, cluster::shard_table>::dispatch_hbeats_to_groups(cluster::partition_manager&, seastar::foreign_ptr<std::__1::unique_ptr<std::__1::vector<raft::append_entries_request, std::__1::allocator<raft::append_entries_request> >, std::__1::default_delete<std::__1::vector<raft::append_entries_request, std::__1::allocator<raft::append_entries_request> > > > >)::{lambda(raft::append_entries_request&)#1}::operator()(raft::append_entries_request&)::{lambda(seastar::timed_out_error const&)#1}>(auto:1&&)::{lambda(auto:1&&)#1}&, seastar::future_state<raft::append_entries_reply>&&)#1}>(raft::service<cluster::partition_manager, cluster::shard_table>::dispatch_hbeats_to_groups(cluster::partition_manager&, seastar::foreign_ptr<std::__1::unique_ptr<std::__1::vector<raft::append_entries_request, std::__1::allocator<raft::append_entries_request> >, std::__1::default_delete<std::__1::vector<raft::append_entries_request, std::__1::allocator<raft::append_entries_request> > > > >)::{lambda(raft::append_entries_request&)#1}::operator()(raft::append_entries_request&)::{lambda(seastar::timed_out_error const&)#1}&&, seastar::future<raft::append_entries_reply>::handle_exception_type<raft::service<cluster::partition_manager, cluster::shard_table>::dispatch_hbeats_to_groups(cluster::partition_manager&, seastar::foreign_ptr<std::__1::unique_ptr<std::__1::vector<raft::append_entries_request, std::__1::allocator<raft::append_entries_request> >, std::__1::default_delete<std::__1::vector<raft::append_entries_request, std::__1::allocator<raft::append_entries_request> > > > >)::{lambda(raft::append_entries_request&)#1}::operator()(raft::append_entries_request&)::{lambda(seastar::timed_out_error const&)#1}>(raft::service<cluster::partition_manager, cluster::shard_table>::dispatch_hbeats_to_groups(cluster::partition_manager&, seastar::foreign_ptr<std::__1::unique_ptr<std::__1::vector<raft::append_entries_request, std::__1::allocator<raft::append_entries_request> >, std::__1::default_delete<std::__1::vector<raft::append_entries_request, std::__1::allocator<raft::append_entries_request> > > > >)::{lambda(raft::append_entries_request&)#1}::operator()(raft::append_entries_request&)::{lambda(seastar::timed_out_error const&)#1}&&)::{lambda(auto:1&&)#1}&&, seastar::future<raft::append_entries_reply>::then_wrapped_nrvo<seastar::future<raft::append_entries_reply>, seastar::future<raft::append_entries_reply>::handle_exception_type<raft::service<cluster::partition_manager, cluster::shard_table>::dispatch_hbeats_to_groups(cluster::partition_manager&, seastar::foreign_ptr<std::__1::unique_ptr<std::__1::vector<raft::append_entries_request, std::__1::allocator<raft::append_entries_request> >, std::__1::default_delete<std::__1::vector<raft::append_entries_request, std::__1::allocator<raft::append_entries_request> > > > >)::{lambda(raft::append_entries_request&)#1}::operator()(raft::append_entries_request&)::{lambda(seastar::timed_out_error const&)#1}>(raft::service<cluster::partition_manager, cluster::shard_table>::dispatch_hbeats_to_groups(cluster::partition_manager&, seastar::foreign_ptr<std::__1::unique_ptr<std::__1::vector<raft::append_entries_request, std::__1::allocator<raft::append_entries_request> >, std::__1::default_delete<std::__1::vector<raft::append_entries_request, std::__1::allocator<raft::append_entries_request> > > > >)::{lambda(raft::append_entries_request&)#1}::operator()(raft::append_entries_request&)::{lambda(seastar::timed_out_error const&)#1}&&)::{lambda(auto:1&&)#1}>(seastar::future<raft::append_entries_reply>::handle_exception_type<raft::service<cluster::partition_manager, cluster::shard_table>::dispatch_hbeats_to_groups(cluster::partition_manager&, seastar::foreign_ptr<std::__1::unique_ptr<std::__1::vector<raft::append_entries_request, std::__1::allocator<raft::append_entries_request> >, std::__1::default_delete<std::__1::vector<raft::append_entries_request, std::__1::allocator<raft::append_entries_request> > > > >)::{lambda(raft::append_entries_request&)#1}::operator()(raft::append_entries_request&)::{lambda(seastar::timed_out_error const&)#1}>(raft::service<cluster::partition_manager, cluster::shard_table>::dispatch_hbeats_to_groups(cluster::partition_manager&, seastar::foreign_ptr<std::__1::unique_ptr<std::__1::vector<raft::append_entries_request, std::__1::allocator<raft::append_entries_request> >, std::__1::default_delete<std::__1::vector<raft::append_entries_request, std::__1::allocator<raft::append_entries_request> > > > >)::{lambda(raft::append_entries_request&)#1}::operator()(raft::append_entries_request&)::{lambda(seastar::timed_out_error const&)#1}&&)::{lambda(auto:1&&)#1}&&)::{lambda(seastar::internal::promise_base_with_type<raft::append_entries_reply>&&, seastar::future<raft::append_entries_reply>::handle_exception_type<raft::service<cluster::partition_manager, cluster::shard_table>::dispatch_hbeats_to_groups(cluster::partition_manager&, seastar::foreign_ptr<std::__1::unique_ptr<std::__1::vector<raft::append_entries_request, std::__1::allocator<raft::append_entries_request> >, std::__1::default_delete<std::__1::vector<raft::append_entries_request, std::__1::allocator<raft::append_entries_request> > > > >)::{lambda(raft::append_entries_request&)#1}::operator()(raft::append_entries_request&)::{lambda(seastar::timed_out_error const&)#1}>(auto:1&&)::{lambda(auto:1&&)#1}&, seastar::future_state<raft::append_entries_reply>&&)#1}&&) at /vectorized/include/seastar/core/future.hh:1393
 (inlined by) seastar::futurize<seastar::future<raft::append_entries_reply> >::type seastar::future<raft::append_entries_reply>::then_wrapped_nrvo<seastar::future<raft::append_entries_reply>, seastar::future<raft::append_entries_reply>::handle_exception_type<raft::service<cluster::partition_manager, cluster::shard_table>::dispatch_hbeats_to_groups(cluster::partition_manager&, seastar::foreign_ptr<std::__1::unique_ptr<std::__1::vector<raft::append_entries_request, std::__1::allocator<raft::append_entries_request> >, std::__1::default_delete<std::__1::vector<raft::append_entries_request, std::__1::allocator<raft::append_entries_request> > > > >)::{lambda(raft::append_entries_request&)#1}::operator()(raft::append_entries_request&)::{lambda(seastar::timed_out_error const&)#1}>(raft::service<cluster::partition_manager, cluster::shard_table>::dispatch_hbeats_to_groups(cluster::partition_manager&, seastar::foreign_ptr<std::__1::unique_ptr<std::__1::vector<raft::append_entries_request, std::__1::allocator<raft::append_entries_request> >, std::__1::default_delete<std::__1::vector<raft::append_entries_request, std::__1::allocator<raft::append_entries_request> > > > >)::{lambda(raft::append_entries_request&)#1}::operator()(raft::append_entries_request&)::{lambda(seastar::timed_out_error const&)#1}&&)::{lambda(auto:1&&)#1}>(seastar::future<raft::append_entries_reply>::handle_exception_type<raft::service<cluster::partition_manager, cluster::shard_table>::dispatch_hbeats_to_groups(cluster::partition_manager&, seastar::foreign_ptr<std::__1::unique_ptr<std::__1::vector<raft::append_entries_request, std::__1::allocator<raft::append_entries_request> >, std::__1::default_delete<std::__1::vector<raft::append_entries_request, std::__1::allocator<raft::append_entries_request> > > > >)::{lambda(raft::append_entries_request&)#1}::operator()(raft::append_entries_request&)::{lambda(seastar::timed_out_error const&)#1}>(raft::service<cluster::partition_manager, cluster::shard_table>::dispatch_hbeats_to_groups(cluster::partition_manager&, seastar::foreign_ptr<std::__1::unique_ptr<std::__1::vector<raft::append_entries_request, std::__1::allocator<raft::append_entries_request> >, std::__1::default_delete<std::__1::vector<raft::append_entries_request, std::__1::allocator<raft::append_entries_request> > > > >)::{lambda(raft::append_entries_request&)#1}::operator()(raft::append_entries_request&)::{lambda(seastar::timed_out_error const&)#1}&&)::{lambda(auto:1&&)#1}&&) at /vectorized/include/seastar/core/future.hh:1666
 (inlined by) seastar::futurize<seastar::future<raft::append_entries_reply> >::type seastar::future<raft::append_entries_reply>::then_wrapped_common<false, seastar::future<raft::append_entries_reply>, seastar::future<raft::append_entries_reply>::handle_exception_type<raft::service<cluster::partition_manager, cluster::shard_table>::dispatch_hbeats_to_groups(cluster::partition_manager&, seastar::foreign_ptr<std::__1::unique_ptr<std::__1::vector<raft::append_entries_request, std::__1::allocator<raft::append_entries_request> >, std::__1::default_delete<std::__1::vector<raft::append_entries_request, std::__1::allocator<raft::append_entries_request> > > > >)::{lambda(raft::append_entries_request&)#1}::operator()(raft::append_entries_request&)::{lambda(seastar::timed_out_error const&)#1}>(raft::service<cluster::partition_manager, cluster::shard_table>::dispatch_hbeats_to_groups(cluster::partition_manager&, seastar::foreign_ptr<std::__1::unique_ptr<std::__1::vector<raft::append_entries_request, std::__1::allocator<raft::append_entries_request> >, std::__1::default_delete<std::__1::vector<raft::append_entries_request, std::__1::allocator<raft::append_entries_request> > > > >)::{lambda(raft::append_entries_request&)#1}::operator()(raft::append_entries_request&)::{lambda(seastar::timed_out_error const&)#1}&&)::{lambda(auto:1&&)#1}>(seastar::future<raft::append_entries_reply>::handle_exception_type<raft::service<cluster::partition_manager, cluster::shard_table>::dispatch_hbeats_to_groups(cluster::partition_manager&, seastar::foreign_ptr<std::__1::unique_ptr<std::__1::vector<raft::append_entries_request, std::__1::allocator<raft::append_entries_request> >, std::__1::default_delete<std::__1::vector<raft::append_entries_request, std::__1::allocator<raft::append_entries_request> > > > >)::{lambda(raft::append_entries_request&)#1}::operator()(raft::append_entries_request&)::{lambda(seastar::timed_out_error const&)#1}>(raft::service<cluster::partition_manager, cluster::shard_table>::dispatch_hbeats_to_groups(cluster::partition_manager&, seastar::foreign_ptr<std::__1::unique_ptr<std::__1::vector<raft::append_entries_request, std::__1::allocator<raft::append_entries_request> >, std::__1::default_delete<std::__1::vector<raft::append_entries_request, std::__1::allocator<raft::append_entries_request> > > > >)::{lambda(raft::append_entries_request&)#1}::operator()(raft::append_entries_request&)::{lambda(seastar::timed_out_error const&)#1}&&)::{lambda(auto:1&&)#1}&&) at /vectorized/include/seastar/core/future.hh:1691
 (inlined by) seastar::futurize<seastar::future<raft::append_entries_reply> >::type seastar::future<raft::append_entries_reply>::then_wrapped_maybe_erase<false, seastar::future<raft::append_entries_reply>, seastar::future<raft::append_entries_reply>::handle_exception_type<raft::service<cluster::partition_manager, cluster::shard_table>::dispatch_hbeats_to_groups(cluster::partition_manager&, seastar::foreign_ptr<std::__1::unique_ptr<std::__1::vector<raft::append_entries_request, std::__1::allocator<raft::append_entries_request> >, std::__1::default_delete<std::__1::vector<raft::append_entries_request, std::__1::allocator<raft::append_entries_request> > > > >)::{lambda(raft::append_entries_request&)#1}::operator()(raft::append_entries_request&)::{lambda(seastar::timed_out_error const&)#1}>(raft::service<cluster::partition_manager, cluster::shard_table>::dispatch_hbeats_to_groups(cluster::partition_manager&, seastar::foreign_ptr<std::__1::unique_ptr<std::__1::vector<raft::append_entries_request, std::__1::allocator<raft::append_entries_request> >, std::__1::default_delete<std::__1::vector<raft::append_entries_request, std::__1::allocator<raft::append_entries_request> > > > >)::{lambda(raft::append_entries_request&)#1}::operator()(raft::append_entries_request&)::{lambda(seastar::timed_out_error const&)#1}&&)::{lambda(auto:1&&)#1}>(seastar::future<raft::append_entries_reply>::handle_exception_type<raft::service<cluster::partition_manager, cluster::shard_table>::dispatch_hbeats_to_groups(cluster::partition_manager&, seastar::foreign_ptr<std::__1::unique_ptr<std::__1::vector<raft::append_entries_request, std::__1::allocator<raft::append_entries_request> >, std::__1::default_delete<std::__1::vector<raft::append_entries_request, std::__1::allocator<raft::append_entries_request> > > > >)::{lambda(raft::append_entries_request&)#1}::operator()(raft::append_entries_request&)::{lambda(seastar::timed_out_error const&)#1}>(raft::service<cluster::partition_manager, cluster::shard_table>::dispatch_hbeats_to_groups(cluster::partition_manager&, seastar::foreign_ptr<std::__1::unique_ptr<std::__1::vector<raft::append_entries_request, std::__1::allocator<raft::append_entries_request> >, std::__1::default_delete<std::__1::vector<raft::append_entries_request, std::__1::allocator<raft::append_entries_request> > > > >)::{lambda(raft::append_entries_request&)#1}::operator()(raft::append_entries_request&)::{lambda(seastar::timed_out_error const&)#1}&&)::{lambda(auto:1&&)#1}&&) at /vectorized/include/seastar/core/future.hh:1644
 (inlined by) seastar::futurize<seastar::future<raft::append_entries_reply> >::type seastar::future<raft::append_entries_reply>::then_wrapped<seastar::future<raft::append_entries_reply>::handle_exception_type<raft::service<cluster::partition_manager, cluster::shard_table>::dispatch_hbeats_to_groups(cluster::partition_manager&, seastar::foreign_ptr<std::__1::unique_ptr<std::__1::vector<raft::append_entries_request, std::__1::allocator<raft::append_entries_request> >, std::__1::default_delete<std::__1::vector<raft::append_entries_request, std::__1::allocator<raft::append_entries_request> > > > >)::{lambda(raft::append_entries_request&)#1}::operator()(raft::append_entries_request&)::{lambda(seastar::timed_out_error const&)#1}>(raft::service<cluster::partition_manager, cluster::shard_table>::dispatch_hbeats_to_groups(cluster::partition_manager&, seastar::foreign_ptr<std::__1::unique_ptr<std::__1::vector<raft::append_entries_request, std::__1::allocator<raft::append_entries_request> >, std::__1::default_delete<std::__1::vector<raft::append_entries_request, std::__1::allocator<raft::append_entries_request> > > > >)::{lambda(raft::append_entries_request&)#1}::operator()(raft::append_entries_request&)::{lambda(seastar::timed_out_error const&)#1}&&)::{lambda(auto:1&&)#1}, seastar::future<raft::append_entries_reply> >(raft::service<cluster::partition_manager, cluster::shard_table>::dispatch_hbeats_to_groups(cluster::partition_manager&, seastar::foreign_ptr<std::__1::unique_ptr<std::__1::vector<raft::append_entries_request, std::__1::allocator<raft::append_entries_request> >, std::__1::default_delete<std::__1::vector<raft::append_entries_request, std::__1::allocator<raft::append_entries_request> > > > >)::{lambda(raft::append_entries_request&)#1}::operator()(raft::append_entries_request&)::{lambda(seastar::timed_out_error const&)#1}&&) & at /vectorized/include/seastar/core/future.hh:1628
 (inlined by) seastar::future<raft::append_entries_reply> seastar::future<raft::append_entries_reply>::handle_exception_type<raft::service<cluster::partition_manager, cluster::shard_table>::dispatch_hbeats_to_groups(cluster::partition_manager&, seastar::foreign_ptr<std::__1::unique_ptr<std::__1::vector<raft::append_entries_request, std::__1::allocator<raft::append_entries_request> >, std::__1::default_delete<std::__1::vector<raft::append_entries_request, std::__1::allocator<raft::append_entries_request> > > > >)::{lambda(raft::append_entries_request&)#1}::operator()(raft::append_entries_request&)::{lambda(seastar::timed_out_error const&)#1}>(raft::service<cluster::partition_manager, cluster::shard_table>::dispatch_hbeats_to_groups(cluster::partition_manager&, seastar::foreign_ptr<std::__1::unique_ptr<std::__1::vector<raft::append_entries_request, std::__1::allocator<raft::append_entries_request> >, std::__1::default_delete<std::__1::vector<raft::append_entries_request, std::__1::allocator<raft::append_entries_request> > > > >)::{lambda(raft::append_entries_request&)#1}::operator()(raft::append_entries_request&)::{lambda(seastar::timed_out_error const&)#1}&&) at /vectorized/include/seastar/core/future.hh:1856
 (inlined by) operator() at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-007ec1632d6275385-1/vectorized/redpanda/vbuild/release/clang/../../../src/v/raft/service.h:283
 (inlined by) std::__1::back_insert_iterator<std::__1::vector<seastar::future<raft::append_entries_reply>, std::__1::allocator<seastar::future<raft::append_entries_reply> > > > std::__1::transform<std::__1::__wrap_iter<raft::append_entries_request*>, std::__1::back_insert_iterator<std::__1::vector<seastar::future<raft::append_entries_reply>, std::__1::allocator<seastar::future<raft::append_entries_reply> > > >, raft::service<cluster::partition_manager, cluster::shard_table>::dispatch_hbeats_to_groups(cluster::partition_manager&, seastar::foreign_ptr<std::__1::unique_ptr<std::__1::vector<raft::append_entries_request, std::__1::allocator<raft::append_entries_request> >, std::__1::default_delete<std::__1::vector<raft::append_entries_request, std::__1::allocator<raft::append_entries_request> > > > >)::{lambda(raft::append_entries_request&)#1}>(std::__1::__wrap_iter<raft::append_entries_request*>, std::__1::__wrap_iter<raft::append_entries_request*>, std::__1::back_insert_iterator<std::__1::vector<seastar::future<raft::append_entries_reply>, std::__1::allocator<seastar::future<raft::append_entries_reply> > > >, raft::service<cluster::partition_manager, cluster::shard_table>::dispatch_hbeats_to_groups(cluster::partition_manager&, seastar::foreign_ptr<std::__1::unique_ptr<std::__1::vector<raft::append_entries_request, std::__1::allocator<raft::append_entries_request> >, std::__1::default_delete<std::__1::vector<raft::append_entries_request, std::__1::allocator<raft::append_entries_request> > > > >)::{lambda(raft::append_entries_request&)#1}) at /vectorized/llvm/bin/../include/c++/v1/algorithm:1982
raft::service<cluster::partition_manager, cluster::shard_table>::dispatch_hbeats_to_groups(cluster::partition_manager&, seastar::foreign_ptr<std::__1::unique_ptr<std::__1::vector<raft::append_entries_request, std::__1::allocator<raft::append_entries_request> >, std::__1::default_delete<std::__1::vector<raft::append_entries_request, std::__1::allocator<raft::append_entries_request> > > > >) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-007ec1632d6275385-1/vectorized/redpanda/vbuild/release/clang/../../../src/v/raft/service.h:275
operator() at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-007ec1632d6275385-1/vectorized/redpanda/vbuild/release/clang/../../../src/v/raft/service.h:264
 (inlined by) decltype ((std::__1::forward<raft::service<cluster::partition_manager, cluster::shard_table>::dispatch_hbeats_to_core(unsigned int, seastar::foreign_ptr<std::__1::unique_ptr<std::__1::vector<raft::append_entries_request, std::__1::allocator<raft::append_entries_request> >, std::__1::default_delete<std::__1::vector<raft::append_entries_request, std::__1::allocator<raft::append_entries_request> > > > >)::{lambda()#1}::operator()()::{lambda(cluster::partition_manager&)#1}>({parm#1}))(std::__1::forward<cluster::partition_manager&>({parm#2}))) std::__1::__invoke_constexpr<raft::service<cluster::partition_manager, cluster::shard_table>::dispatch_hbeats_to_core(unsigned int, seastar::foreign_ptr<std::__1::unique_ptr<std::__1::vector<raft::append_entries_request, std::__1::allocator<raft::append_entries_request> >, std::__1::default_delete<std::__1::vector<raft::append_entries_request, std::__1::allocator<raft::append_entries_request> > > > >)::{lambda()#1}::operator()()::{lambda(cluster::partition_manager&)#1}, cluster::partition_manager&>(raft::service<cluster::partition_manager, cluster::shard_table>::dispatch_hbeats_to_core(unsigned int, seastar::foreign_ptr<std::__1::unique_ptr<std::__1::vector<raft::append_entries_request, std::__1::allocator<raft::append_entries_request> >, std::__1::default_delete<std::__1::vector<raft::append_entries_request, std::__1::allocator<raft::append_entries_request> > > > >)::{lambda()#1}::operator()()::{lambda(cluster::partition_manager&)#1}&&, cluster::partition_manager&) at /vectorized/llvm/bin/../include/c++/v1/type_traits:3700
 (inlined by) decltype(auto) std::__1::__apply_tuple_impl<raft::service<cluster::partition_manager, cluster::shard_table>::dispatch_hbeats_to_core(unsigned int, seastar::foreign_ptr<std::__1::unique_ptr<std::__1::vector<raft::append_entries_request, std::__1::allocator<raft::append_entries_request> >, std::__1::default_delete<std::__1::vector<raft::append_entries_request, std::__1::allocator<raft::append_entries_request> > > > >)::{lambda()#1}::operator()()::{lambda(cluster::partition_manager&)#1}, std::__1::tuple<cluster::partition_manager&>, 0ul>(raft::service<cluster::partition_manager, cluster::shard_table>::dispatch_hbeats_to_core(unsigned int, seastar::foreign_ptr<std::__1::unique_ptr<std::__1::vector<raft::append_entries_request, std::__1::allocator<raft::append_entries_request> >, std::__1::default_delete<std::__1::vector<raft::append_entries_request, std::__1::allocator<raft::append_entries_request> > > > >)::{lambda()#1}::operator()()::{lambda(cluster::partition_manager&)#1}&&, std::__1::tuple<cluster::partition_manager&>&&, std::__1::__tuple_indices<0ul>) at /vectorized/llvm/bin/../include/c++/v1/tuple:1415
 (inlined by) decltype(auto) std::__1::apply<raft::service<cluster::partition_manager, cluster::shard_table>::dispatch_hbeats_to_core(unsigned int, seastar::foreign_ptr<std::__1::unique_ptr<std::__1::vector<raft::append_entries_request, std::__1::allocator<raft::append_entries_request> >, std::__1::default_delete<std::__1::vector<raft::append_entries_request, std::__1::allocator<raft::append_entries_request> > > > >)::{lambda()#1}::operator()()::{lambda(cluster::partition_manager&)#1}, std::__1::tuple<cluster::partition_manager&> >(raft::service<cluster::partition_manager, cluster::shard_table>::dispatch_hbeats_to_core(unsigned int, seastar::foreign_ptr<std::__1::unique_ptr<std::__1::vector<raft::append_entries_request, std::__1::allocator<raft::append_entries_request> >, std::__1::default_delete<std::__1::vector<raft::append_entries_request, std::__1::allocator<raft::append_entries_request> > > > >)::{lambda()#1}::operator()()::{lambda(cluster::partition_manager&)#1}&&, std::__1::tuple<cluster::partition_manager&>&&) at /vectorized/llvm/bin/../include/c++/v1/tuple:1424
 (inlined by) operator() at /vectorized/include/seastar/core/sharded.hh:411
 (inlined by) seastar::future<std::__1::vector<raft::append_entries_reply, std::__1::allocator<raft::append_entries_reply> > > seastar::futurize<seastar::future<std::__1::vector<raft::append_entries_reply, std::__1::allocator<raft::append_entries_reply> > > >::invoke<seastar::sharded<cluster::partition_manager>::invoke_on<raft::service<cluster::partition_manager, cluster::shard_table>::dispatch_hbeats_to_core(unsigned int, seastar::foreign_ptr<std::__1::unique_ptr<std::__1::vector<raft::append_entries_request, std::__1::allocator<raft::append_entries_request> >, std::__1::default_delete<std::__1::vector<raft::append_entries_request, std::__1::allocator<raft::append_entries_request> > > > >)::{lambda()#1}::operator()()::{lambda(cluster::partition_manager&)#1}, , seastar::future<std::__1::vector<raft::append_entries_reply, std::__1::allocator<raft::append_entries_reply> > > >(unsigned int, seastar::smp_submit_to_options, raft::service<cluster::partition_manager, cluster::shard_table>::dispatch_hbeats_to_core(unsigned int, seastar::foreign_ptr<std::__1::unique_ptr<std::__1::vector<raft::append_entries_request, std::__1::allocator<raft::append_entries_request> >, std::__1::default_delete<std::__1::vector<raft::append_entries_request, std::__1::allocator<raft::append_entries_request> > > > >)::{lambda()#1}::operator()()::{lambda(cluster::partition_manager&)#1}&&)::{lambda()#1}&>(raft::service<cluster::partition_manager, cluster::shard_table>::dispatch_hbeats_to_core(unsigned int, seastar::foreign_ptr<std::__1::unique_ptr<std::__1::vector<raft::append_entries_request, std::__1::allocator<raft::append_entries_request> >, std::__1::default_delete<std::__1::vector<raft::append_entries_request, std::__1::allocator<raft::append_entries_request> > > > >)::{lambda()#1}::operator()()::{lambda(cluster::partition_manager&)#1}&&) at /vectorized/include/seastar/core/future.hh:2135
seastar::smp_message_queue::async_work_item<seastar::sharded<cluster::partition_manager>::invoke_on<raft::service<cluster::partition_manager, cluster::shard_table>::dispatch_hbeats_to_core(unsigned int, seastar::foreign_ptr<std::__1::unique_ptr<std::__1::vector<raft::append_entries_request, std::__1::allocator<raft::append_entries_request> >, std::__1::default_delete<std::__1::vector<raft::append_entries_request, std::__1::allocator<raft::append_entries_request> > > > >)::{lambda()#1}::operator()()::{lambda(cluster::partition_manager&)#1}, , seastar::future<std::__1::vector<raft::append_entries_reply, std::__1::allocator<raft::append_entries_reply> > > >(unsigned int, seastar::smp_submit_to_options, raft::service<cluster::partition_manager, cluster::shard_table>::dispatch_hbeats_to_core(unsigned int, seastar::foreign_ptr<std::__1::unique_ptr<std::__1::vector<raft::append_entries_request, std::__1::allocator<raft::append_entries_request> >, std::__1::default_delete<std::__1::vector<raft::append_entries_request, std::__1::allocator<raft::append_entries_request> > > > >)::{lambda()#1}::operator()()::{lambda(cluster::partition_manager&)#1}&&)::{lambda()#1}>::run_and_dispose() at /vectorized/include/seastar/core/smp.hh:231
seastar::reactor::run_tasks(seastar::reactor::task_queue&) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:2230
 (inlined by) seastar::reactor::run_some_tasks() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:2639
seastar::reactor::do_run() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:2808
seastar::reactor::run() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:2691
seastar::app_template::run_deprecated(int, char**, std::__1::function<void ()>&&) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/app-template.cc:217
seastar::app_template::run(int, char**, std::__1::function<seastar::future<int> ()>&&) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/app-template.cc:125
application::run(int, char**) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-007ec1632d6275385-1/vectorized/redpanda/vbuild/release/clang/../../../src/v/redpanda/application.cc:175
main at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-007ec1632d6275385-1/vectorized/redpanda/vbuild/release/clang/../../../src/v/redpanda/main.cc:38

[jcsp]

Is this a duplicate of #3498?

[NyaliaLui]

Is this a duplicate of #3498?

Did you mean to link the same PR in the PR?

[jcsp]

Did you mean to link the same PR in the PR?

I did not... meant to link https://github.com/vectorizedio/redpanda/issues/3497

[NyaliaLui]

Oh it is a duplicate. I knew I posted that backtrace somewhere. I'll close this.