search

redpanda-data / redpanda

Redpanda is a streaming data platform for developers. Kafka API compatible. 10x faster. No ZooKeeper. No JVM!
https://redpanda.com
9.39k stars 577 forks source link

2.75GiB allocation in topic_table #9734

Closed ballard26 closed 2 weeks ago

ballard26 commented 1 year ago

Version & Environment

Redpanda version: (use rpk version):

What went wrong?

Out of memory while trying to start Redpanda;

ERROR 2023-03-30 05:22:47,856 [shard 0] seastar - Failed to allocate 2952790016 bytes

The exact backtrace for the allocation can be found in the "Additional information" section below. The gist this that we are trying to allocate 2.75GiB for the _pending_deltas vector in topic_table. This seems to be due to the 429,459 update_topic_properties commands in the controller log;

$ cat controller.json | jq '.[] | select(.type == "topic_management_cmd") | .data.type_string' | sort | uniq -c | sort -n -r
429459 "update_topic_properties"
 125 "create_topic"
   2 "delete_topic"

(from @daisukebe)

While replaying the controller log the topic_table collects a std::vector of pending deltas to the table. And for each update_topic_properties we add a pending delta for each partition in the topic. A bunch of the topics in the cluster have 90 partitions;

[ballard@fedora offline_log_viewer]$ cat controller.json | jq '.[] | select(.type == "topic_management_cmd" and .data.type_string == "create_topic") | .data.cfg.partitions' | sort -nr
90
90
90
90
90
90
30
30
30
30
30

Mitigations

Solution

Snapshotting the controller log

Additional information

[Backtrace #0]                                                                                                                                                                      [473/1617]
void seastar::backtrace<seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}>(seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}&&) at /v/build/v_
deps_build/seastar-prefix/src/seastar/include/seastar/util/backtrace.hh:59
 (inlined by) seastar::backtrace_buffer::append_backtrace() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:754
 (inlined by) seastar::print_with_backtrace(seastar::backtrace_buffer&, bool) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:784
seastar::print_with_backtrace(char const*, bool) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:796
 (inlined by) seastar::sigabrt_action() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:3663
 (inlined by) operator() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:3639
 (inlined by) __invoke at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:3635
/opt/redpanda/lib/libc.so.6: ELF 64-bit LSB shared object, x86-64, version 1 (GNU/Linux), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=6e7b96dfb83f0bdcb6a410469
b82f86415e5ada3, for GNU/Linux 3.2.0, stripped

__sigaction at ??:?
pthread_key_delete at ??:?
raise at ??:?
abort at ??:?
seastar::memory::on_allocation_failure(unsigned long) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:1806
 (inlined by) seastar::memory::allocate(unsigned long) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:1406
operator new(unsigned long) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:2064
void* std::__1::__libcpp_operator_new<unsigned long>(unsigned long) at /vectorized/llvm/bin/../include/c++/v1/new:245
 (inlined by) std::__1::__libcpp_allocate(unsigned long, unsigned long) at /vectorized/llvm/bin/../include/c++/v1/new:271
 (inlined by) std::__1::allocator<cluster::topic_table_delta>::allocate(unsigned long) at /vectorized/llvm/bin/../include/c++/v1/__memory/allocator.h:105
 (inlined by) std::__1::allocator_traits<std::__1::allocator<cluster::topic_table_delta> >::allocate(std::__1::allocator<cluster::topic_table_delta>&, unsigned long) at /vectorized/llvm/bin/
../include/c++/v1/__memory/allocator_traits.h:262
 (inlined by) __split_buffer at /vectorized/llvm/bin/../include/c++/v1/__split_buffer:306
 (inlined by) void std::__1::vector<cluster::topic_table_delta, std::__1::allocator<cluster::topic_table_delta> >::__push_back_slow_path<cluster::topic_table_delta>(cluster::topic_table_delt
a&&) at /vectorized/llvm/bin/../include/c++/v1/vector:1517
 (inlined by) std::__1::vector<cluster::topic_table_delta, std::__1::allocator<cluster::topic_table_delta> >::push_back(cluster::topic_table_delta&&) at /vectorized/llvm/bin/../include/c++/v
1/vector:1549
 (inlined by) std::__1::back_insert_iterator<std::__1::vector<cluster::topic_table_delta, std::__1::allocator<cluster::topic_table_delta> > >::operator=(cluster::topic_table_delta&&) at /vec
torized/llvm/bin/../include/c++/v1/__iterator/back_insert_iterator.h:53
 (inlined by) std::__1::back_insert_iterator<std::__1::vector<cluster::topic_table_delta, std::__1::allocator<cluster::topic_table_delta> > > std::__1::__move_constexpr<cluster::topic_table_
delta*, std::__1::back_insert_iterator<std::__1::vector<cluster::topic_table_delta, std::__1::allocator<cluster::topic_table_delta> > > >(cluster::topic_table_delta*, cluster::topic_table_de
lta*, std::__1::back_insert_iterator<std::__1::vector<cluster::topic_table_delta, std::__1::allocator<cluster::topic_table_delta> > >) at /vectorized/llvm/bin/../include/c++/v1/__algorithm/m
ove.h:33
 (inlined by) std::__1::back_insert_iterator<std::__1::vector<cluster::topic_table_delta, std::__1::allocator<cluster::topic_table_delta> > > std::__1::__move<cluster::topic_table_delta*, st
d::__1::back_insert_iterator<std::__1::vector<cluster::topic_table_delta, std::__1::allocator<cluster::topic_table_delta> > > >(cluster::topic_table_delta*, cluster::topic_table_delta*, std:
:__1::back_insert_iterator<std::__1::vector<cluster::topic_table_delta, std::__1::allocator<cluster::topic_table_delta> > >) at /vectorized/llvm/bin/../include/c++/v1/__algorithm/move.h:42
 (inlined by) std::__1::back_insert_iterator<std::__1::vector<cluster::topic_table_delta, std::__1::allocator<cluster::topic_table_delta> > > std::__1::move<std::__1::__wrap_iter<cluster::to
pic_table_delta*>, std::__1::back_insert_iterator<std::__1::vector<cluster::topic_table_delta, std::__1::allocator<cluster::topic_table_delta> > > >(std::__1::__wrap_iter<cluster::topic_tabl
e_delta*>, std::__1::__wrap_iter<cluster::topic_table_delta*>, std::__1::back_insert_iterator<std::__1::vector<cluster::topic_table_delta, std::__1::allocator<cluster::topic_table_delta> > >
) at /vectorized/llvm/bin/../include/c++/v1/__algorithm/move.h:70
 (inlined by) cluster::topic_table::apply(cluster::controller_command<model::topic_namespace, cluster::incremental_topic_updates, (signed char)4, (model::record_batch_type)6, (cluster::serde
_opts)0>, detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true> >) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-034adfcdac980be70-1
/redpanda/redpanda/src/v/cluster/topic_table.cc:590
operator() at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-034adfcdac980be70-1/redpanda/redpanda/src/v/cluster/topic_updates_dispatcher.cc:348
 (inlined by) decltype ((static_cast<cluster::do_apply<cluster::controller_command<model::topic_namespace, cluster::incremental_topic_updates, (signed char)4, (model::record_batch_type)6, (c
luster::serde_opts)0> >(unsigned int, cluster::controller_command<model::topic_namespace, cluster::incremental_topic_updates, (signed char)4, (model::record_batch_type)6, (cluster::serde_opt
s)0>, seastar::sharded<cluster::topic_table>&, detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true> >)::{lambda(cluster::topic_table&)#1}>({parm#1}
))(static_cast<cluster::topic_table&>({parm#2}))) std::__1::__invoke_constexpr<cluster::do_apply<cluster::controller_command<model::topic_namespace, cluster::incremental_topic_updates, (sign
ed char)4, (model::record_batch_type)6, (cluster::serde_opts)0> >(unsigned int, cluster::controller_command<model::topic_namespace, cluster::incremental_topic_updates, (signed char)4, (model
::record_batch_type)6, (cluster::serde_opts)0>, seastar::sharded<cluster::topic_table>&, detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true> >)::{
lambda(cluster::topic_table&)#1}, cluster::topic_table&>(cluster::do_apply<cluster::controller_command<model::topic_namespace, cluster::incremental_topic_updates, (signed char)4, (model::rec
ord_batch_type)6, (cluster::serde_opts)0> >(unsigned int, cluster::controller_command<model::topic_namespace, cluster::incremental_topic_updates, (signed char)4, (model::record_batch_type)6,
 (cluster::serde_opts)0>, seastar::sharded<cluster::topic_table>&, detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true> >)::{lambda(cluster::topic_
table&)#1}&&, cluster::topic_table&) at /vectorized/llvm/bin/../include/c++/v1/type_traits:3648 
 (inlined by) decltype(auto) std::__1::__apply_tuple_impl<cluster::do_apply<cluster::controller_command<model::topic_namespace, cluster::incremental_topic_updates, (signed char)4, (model::re
cord_batch_type)6, (cluster::serde_opts)0> >(unsigned int, cluster::controller_command<model::topic_namespace, cluster::incremental_topic_updates, (signed char)4, (model::record_batch_type)6
, (cluster::serde_opts)0>, seastar::sharded<cluster::topic_table>&, detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true> >)::{lambda(cluster::topic
_table&)#1}, std::__1::tuple<cluster::topic_table&>, 0ul>(cluster::do_apply<cluster::controller_command<model::topic_namespace, cluster::incremental_topic_updates, (signed char)4, (model::re
cord_batch_type)6, (cluster::serde_opts)0> >(unsigned int, cluster::controller_command<model::topic_namespace, cluster::incremental_topic_updates, (signed char)4, (model::record_batch_type)6
, (cluster::serde_opts)0>, seastar::sharded<cluster::topic_table>&, detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true> >)::{lambda(cluster::topic
_table&)#1}&&, std::__1::tuple<cluster::topic_table&>&&, std::__1::__tuple_indices<0ul>) at /vectorized/llvm/bin/../include/c++/v1/tuple:1595
 (inlined by) decltype(auto) std::__1::apply<cluster::do_apply<cluster::controller_command<model::topic_namespace, cluster::incremental_topic_updates, (signed char)4, (model::record_batch_ty
pe)6, (cluster::serde_opts)0> >(unsigned int, cluster::controller_command<model::topic_namespace, cluster::incremental_topic_updates, (signed char)4, (model::record_batch_type)6, (cluster::s
erde_opts)0>, seastar::sharded<cluster::topic_table>&, detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true> >)::{lambda(cluster::topic_table&)#1}, 
std::__1::tuple<cluster::topic_table&> >(cluster::do_apply<cluster::controller_command<model::topic_namespace, cluster::incremental_topic_updates, (signed char)4, (model::record_batch_type)6
, (cluster::serde_opts)0> >(unsigned int, cluster::controller_command<model::topic_namespace, cluster::incremental_topic_updates, (signed char)4, (model::record_batch_type)6, (cluster::serde
_opts)0>, seastar::sharded<cluster::topic_table>&, detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true> >)::{lambda(cluster::topic_table&)#1}&&, st
d::__1::tuple<cluster::topic_table&>&&) at /vectorized/llvm/bin/../include/c++/v1/tuple:1604
 (inlined by) operator() at /vectorized/include/seastar/core/sharded.hh:457
 (inlined by) seastar::future<std::__1::error_code> seastar::futurize<seastar::future<std::__1::error_code> >::invoke<seastar::sharded<cluster::topic_table>::invoke_on<cluster::do_apply<clus
ter::controller_command<model::topic_namespace, cluster::incremental_topic_updates, (signed char)4, (model::record_batch_type)6, (cluster::serde_opts)0> >(unsigned int, cluster::controller_c
ommand<model::topic_namespace, cluster::incremental_topic_updates, (signed char)4, (model::record_batch_type)6, (cluster::serde_opts)0>, seastar::sharded<cluster::topic_table>&, detail::base
_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true> >)::{lambda(cluster::topic_table&)#1}, , seastar::future<std::__1::error_code> >(unsigned int, seastar::sm
p_submit_to_options, cluster::do_apply<cluster::controller_command<model::topic_namespace, cluster::incremental_topic_updates, (signed char)4, (model::record_batch_type)6, (cluster::serde_op
ts)0> >(unsigned int, cluster::controller_command<model::topic_namespace, cluster::incremental_topic_updates, (signed char)4, (model::record_batch_type)6, (cluster::serde_opts)0>, seastar::s
harded<cluster::topic_table>&, detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true> >)::{lambda(cluster::topic_table&)#1}&&)::{lambda()#1}&>(cluste
r::do_apply<cluster::controller_command<model::topic_namespace, cluster::incremental_topic_updates, (signed char)4, (model::record_batch_type)6, (cluster::serde_opts)0> >(unsigned int, clust
er::controller_command<model::topic_namespace, cluster::incremental_topic_updates, (signed char)4, (model::record_batch_type)6, (cluster::serde_opts)0>, seastar::sharded<cluster::topic_table
>&, detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true> >)::{lambda(cluster::topic_table&)#1}&&) at /vectorized/include/seastar/core/future.hh:214
9
 (inlined by) _ZN7seastar3smp9submit_toIZNS_7shardedIN7cluster11topic_tableEE9invoke_onIZNS3_8do_applyINS3_18controller_commandIN5model15topic_namespaceENS3_25incremental_topic_updatesELa4EL
NS9_17record_batch_typeE6ELNS3_10serde_optsE0EEEEENS_6futureINSt3__110error_codeEEEjT_RS5_N6detail15base_named_typeIlNS9_17model_offset_typeENSG_17integral_constantIbLb1EEEEEEUlRS4_E_JESI_EE
T1_jNS_21smp_submit_to_optionsEOSJ_DpOT0_EUlvE_EENS_8futurizeINSG_13invoke_resultISJ_JEE4typeEE4typeEjSU_SV_ at /vectorized/include/seastar/core/smp.hh:349

JIRA Link: CORE-1232

dotnwat commented 1 year ago

Switch from std::vector to fragment_vector

At 3GB we should probably think twice about this as even a mitigation. That's a lot of data.

ballard26 commented 6 months ago

Likely the same issue as the following;

ballard26 commented 6 months ago

Similar issue from recent testing;

  ./tests/results/latest/ManyTopicsTest/test_many_topics_throughput/1/RedpandaService-0-140400864584720/ip-172-31-2-236/redpanda.log:3647:WARN  2024-03-07 07:02:07,829 [shard  1:main] seasta  r_memory - oversized allocation: 151552 bytes. This is non-fatal, but could lead to latency and/or fragmentation issues. Please report: at                                                  
  [Backtrace #4]                                                                                                                                                                              
  {/opt/redpanda/libexec/redpanda} 0x94ad7a3: void seastar::backtrace<seastar::current_backtrace_tasklocal()::$_0>(seastar::current_backtrace_tasklocal()::$_0&&) at /v/build/v_deps_build/sea  star-prefix/src/seastar/include/seastar/util/backtrace.hh:64                                                                                                                                
   (inlined by) seastar::current_backtrace_tasklocal() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/util/backtrace.cc:98                                                           
   (inlined by) seastar::current_tasktrace() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/util/backtrace.cc:149                                                                    
   (inlined by) seastar::current_backtrace() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/util/backtrace.cc:182                                                                    
  {/opt/redpanda/libexec/redpanda} 0x91152cc: seastar::memory::cpu_pages::warn_large_allocation(unsigned long) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:839     
   (inlined by) seastar::memory::cpu_pages::check_large_allocation(unsigned long) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:904                                  
   (inlined by) seastar::memory::cpu_pages::allocate_large(unsigned int, bool) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:911                                     
   (inlined by) seastar::memory::allocate_large(unsigned long, bool) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:1521                                              
   (inlined by) seastar::memory::allocate_slowpath(unsigned long) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:1645                                                 
  {/opt/redpanda/libexec/redpanda} 0x91225a0: seastar::memory::allocate(unsigned long) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:1658                            
   (inlined by) operator new(unsigned long) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:2355                                                                       
  {/opt/redpanda/libexec/redpanda} 0x3709dc5: void* std::__1::__libcpp_operator_new[abi:v160004]<unsigned long>(unsigned long) at /vectorized/llvm/bin/../include/c++/v1/new:266              
   (inlined by) std::__1::__libcpp_allocate[abi:v160004](unsigned long, unsigned long) at /vectorized/llvm/bin/../include/c++/v1/new:292                                                      
   (inlined by) std::__1::allocator<absl::lts_20230802::container_internal::AlignedType<8ul> >::allocate[abi:v160004](unsigned long) at /vectorized/llvm/bin/../include/c++/v1/__memory/alloca  tor.h:115                                                                                                                                                                                   
   (inlined by) std::__1::allocator_traits<std::__1::allocator<absl::lts_20230802::container_internal::AlignedType<8ul> > >::allocate[abi:v160004](std::__1::allocator<absl::lts_20230802::con  tainer_internal::AlignedType<8ul> >&, unsigned long) at /vectorized/llvm/bin/../include/c++/v1/__memory/allocator_traits.h:268                                                              
   (inlined by) void* absl::lts_20230802::container_internal::Allocate<8ul, std::__1::allocator<char> >(std::__1::allocator<char>*, unsigned long) at /vectorized/include/absl/container/inter  nal/container_memory.h:65                                                                                                                                                                   
   (inlined by) void absl::lts_20230802::container_internal::InitializeSlots<std::__1::allocator<char>, 8ul, 8ul>(absl::lts_20230802::container_internal::CommonFields&, std::__1::allocator<c  har>) at /vectorized/include/absl/container/internal/raw_hash_set.h:1407                                                                                                                    
  {/opt/redpanda/libexec/redpanda} 0x61112f1: absl::lts_20230802::container_internal::raw_hash_set<absl::lts_20230802::container_internal::NodeHashMapPolicy<model::topic_namespace, cluster::  topic_table::topic_metadata_item>, model::topic_namespace_hash, model::topic_namespace_eq, std::__1::allocator<std::__1::pair<model::topic_namespace const, cluster::topic_table::topic_meta  data_item> > >::initialize_slots() at /vectorized/include/absl/container/internal/raw_hash_set.h:2505                                                                                       
   (inlined by) absl::lts_20230802::container_internal::raw_hash_set<absl::lts_20230802::container_internal::NodeHashMapPolicy<model::topic_namespace, cluster::topic_table::topic_metadata_it  em>, model::topic_namespace_hash, model::topic_namespace_eq, std::__1::allocator<std::__1::pair<model::topic_namespace const, cluster::topic_table::topic_metadata_item> > >::resize(unsigne  d long) at /vectorized/include/absl/container/internal/raw_hash_set.h:2515                                                                                                                  
  {/opt/redpanda/libexec/redpanda} 0x61112b0: absl::lts_20230802::container_internal::raw_hash_set<absl::lts_20230802::container_internal::NodeHashMapPolicy<model::topic_namespace, cluster::  topic_table::topic_metadata_item>, model::topic_namespace_hash, model::topic_namespace_eq, std::__1::allocator<std::__1::pair<model::topic_namespace const, cluster::topic_table::topic_meta  data_item> > >::prepare_insert(unsigned long) at /vectorized/include/absl/container/internal/raw_hash_set.h:2678                                                                            
  {/opt/redpanda/libexec/redpanda} 0x60cbf09: std::__1::pair<unsigned long, bool> absl::lts_20230802::container_internal::raw_hash_set<absl::lts_20230802::container_internal::NodeHashMapPoli  cy<model::topic_namespace, cluster::topic_table::topic_metadata_item>, model::topic_namespace_hash, model::topic_namespace_eq, std::__1::allocator<std::__1::pair<model::topic_namespace con  st, cluster::topic_table::topic_metadata_item> > >::find_or_prepare_insert<model::topic_namespace>(model::topic_namespace const&) at /vectorized/include/absl/container/internal/raw_hash_se  t.h:2659                                                                                                                                                                                    
   (inlined by) std::__1::pair<absl::lts_20230802::container_internal::raw_hash_set<absl::lts_20230802::container_internal::NodeHashMapPolicy<model::topic_namespace, cluster::topic_table::to  pic_metadata_item>, model::topic_namespace_hash, model::topic_namespace_eq, std::__1::allocator<std::__1::pair<model::topic_namespace const, cluster::topic_table::topic_metadata_item> > >:  :iterator, bool> absl::lts_20230802::container_internal::raw_hash_set<absl::lts_20230802::container_internal::NodeHashMapPolicy<model::topic_namespace, cluster::topic_table::topic_metadata  _item>, model::topic_namespace_hash, model::topic_namespace_eq, std::__1::allocator<std::__1::pair<model::topic_namespace const, cluster::topic_table::topic_metadata_item> > >::EmplaceDeco  mposable::operator()<model::topic_namespace, std::__1::piecewise_construct_t const&, std::__1::tuple<model::topic_namespace&&>, std::__1::tuple<cluster::topic_table::topic_metadata_item&&>   >(model::topic_namespace const&, std::__1::piecewise_construct_t const&, std::__1::tuple<model::topic_namespace&&>&&, std::__1::tuple<cluster::topic_table::topic_metadata_item&&>&&) const   at /vectorized/include/absl/container/internal/raw_hash_set.h:2459         
   (inlined by) decltype(std::declval<absl::lts_20230802::container_internal::raw_hash_set<absl::lts_20230802::container_internal::NodeHashMapPolicy<model::topic_namespace, cluster::topic_ta  ble::topic_metadata_item>, model::topic_namespace_hash, model::topic_namespace_eq, std::__1::allocator<std::__1::pair<model::topic_namespace const, cluster::topic_table::topic_metadata_ite  m> > >::EmplaceDecomposable>()(std::declval<model::topic_namespace&& const&>(), std::__1::piecewise_construct, std::declval<std::__1::tuple<model::topic_namespace&&> >(), std::declval<std:  :__1::tuple<cluster::topic_table::topic_metadata_item&&> >())) absl::lts_20230802::container_internal::memory_internal::DecomposePairImpl<absl::lts_20230802::container_internal::raw_hash_s  et<absl::lts_20230802::container_internal::NodeHashMapPolicy<model::topic_namespace, cluster::topic_table::topic_metadata_item>, model::topic_namespace_hash, model::topic_namespace_eq, std  ::__1::allocator<std::__1::pair<model::topic_namespace const, cluster::topic_table::topic_metadata_item> > >::EmplaceDecomposable, model::topic_namespace&&, std::__1::tuple<cluster::topic_  table::topic_metadata_item&&> >(absl::lts_20230802::container_internal::raw_hash_set<absl::lts_20230802::container_internal::NodeHashMapPolicy<model::topic_namespace, cluster::topic_table:  :topic_metadata_item>, model::topic_namespace_hash, model::topic_namespace_eq, std::__1::allocator<std::__1::pair<model::topic_namespace const, cluster::topic_table::topic_metadata_item> >   >::EmplaceDecomposable&&, std::__1::pair<std::__1::tuple<model::topic_namespace&&>, std::__1::tuple<cluster::topic_table::topic_metadata_item&&> >) at /vectorized/include/absl/container/i  nternal/container_memory.h:140                                                                                                                                                              
   (inlined by) decltype(memory_internal::DecomposePairImpl(std::forward<absl::lts_20230802::container_internal::raw_hash_set<absl::lts_20230802::container_internal::NodeHashMapPolicy<model:  :topic_namespace, cluster::topic_table::topic_metadata_item>, model::topic_namespace_hash, model::topic_namespace_eq, std::__1::allocator<std::__1::pair<model::topic_namespace const, clust  er::topic_table::topic_metadata_item> > >::EmplaceDecomposable>(fp), PairArgs(std::forward<std::__1::pair<model::topic_namespace, cluster::topic_table::topic_metadata_item> >(fp0)))) absl:  :lts_20230802::container_internal::DecomposePair<absl::lts_20230802::container_internal::raw_hash_set<absl::lts_20230802::container_internal::NodeHashMapPolicy<model::topic_namespace, clus  ter::topic_table::topic_metadata_item>, model::topic_namespace_hash, model::topic_namespace_eq, std::__1::allocator<std::__1::pair<model::topic_namespace const, cluster::topic_table::topic  _metadata_item> > >::EmplaceDecomposable, std::__1::pair<model::topic_namespace, cluster::topic_table::topic_metadata_item> >(absl::lts_20230802::container_internal::raw_hash_set<absl::lts  _20230802::container_internal::NodeHashMapPolicy<model::topic_namespace, cluster::topic_table::topic_metadata_item>, model::topic_namespace_hash, model::topic_namespace_eq, std::__1::alloc  ator<std::__1::pair<model::topic_namespace const, cluster::topic_table::topic_metadata_item> > >::EmplaceDecomposable&&, std::__1::pair<model::topic_namespace, cluster::topic_table::topic_  metadata_item>&&) at /vectorized/include/absl/container/internal/container_memory.h:207                                                                                                     
   (inlined by) decltype(absl::container_internal::DecomposePair(std::declval<absl::lts_20230802::container_internal::raw_hash_set<absl::lts_20230802::container_internal::NodeHashMapPolicy<m  odel::topic_namespace, cluster::topic_table::topic_metadata_item>, model::topic_namespace_hash, model::topic_namespace_eq, std::__1::allocator<std::__1::pair<model::topic_namespace const,   cluster::topic_table::topic_metadata_item> > >::EmplaceDecomposable>(), std::declval<std::__1::pair<model::topic_namespace, cluster::topic_table::topic_metadata_item> >())) absl::lts_20230  802::container_internal::NodeHashMapPolicy<model::topic_namespace, cluster::topic_table::topic_metadata_item>::apply<absl::lts_20230802::container_internal::raw_hash_set<absl::lts_20230802  ::container_internal::NodeHashMapPolicy<model::topic_namespace, cluster::topic_table::topic_metadata_item>, model::topic_namespace_hash, model::topic_namespace_eq, std::__1::allocator<std:  :__1::pair<model::topic_namespace const, cluster::topic_table::topic_metadata_item> > >::EmplaceDecomposable, std::__1::pair<model::topic_namespace, cluster::topic_table::topic_metadata_it  em> >(absl::lts_20230802::container_internal::raw_hash_set<absl::lts_20230802::container_internal::NodeHashMapPolicy<model::topic_namespace, cluster::topic_table::topic_metadata_item>, mod  el::topic_namespace_hash, model::topic_namespace_eq, std::__1::allocator<std::__1::pair<model::topic_namespace const, cluster::topic_table::topic_metadata_item> > >::EmplaceDecomposable&&,   std::__1::pair<model::topic_namespace, cluster::topic_table::topic_metadata_item>&&) at /vectorized/include/absl/container/node_hash_map.h:583                                             
   (inlined by) decltype(absl::lts_20230802::container_internal::NodeHashMapPolicy<model::topic_namespace, cluster::topic_table::topic_metadata_item>::apply(std::forward<absl::lts_20230802::  container_internal::raw_hash_set<absl::lts_20230802::container_internal::NodeHashMapPolicy<model::topic_namespace, cluster::topic_table::topic_metadata_item>, model::topic_namespace_hash,   model::topic_namespace_eq, std::__1::allocator<std::__1::pair<model::topic_namespace const, cluster::topic_table::topic_metadata_item> > >::EmplaceDecomposable>(fp), std::forward<std::__1:  :pair<model::topic_namespace, cluster::topic_table::topic_metadata_item> >(fp0))) absl::lts_20230802::container_internal::hash_policy_traits<absl::lts_20230802::container_internal::NodeHas  hMapPolicy<model::topic_namespace, cluster::topic_table::topic_metadata_item>, void>::apply<absl::lts_20230802::container_internal::raw_hash_set<absl::lts_20230802::container_internal::Nod  eHashMapPolicy<model::topic_namespace, cluster::topic_table::topic_metadata_item>, model::topic_namespace_hash, model::topic_namespace_eq, std::__1::allocator<std::__1::pair<model::topic_n  amespace const, cluster::topic_table::topic_metadata_item> > >::EmplaceDecomposable, std::__1::pair<model::topic_namespace, cluster::topic_table::topic_metadata_item>, absl::lts_20230802::  container_internal::NodeHashMapPolicy<model::topic_namespace, cluster::topic_table::topic_metadata_item> >(absl::lts_20230802::container_internal::raw_hash_set<absl::lts_20230802::containe  r_internal::NodeHashMapPolicy<model::topic_namespace, cluster::topic_table::topic_metadata_item>, model::topic_namespace_hash, model::topic_namespace_eq, std::__1::allocator<std::__1::pair  <model::topic_namespace const, cluster::topic_table::topic_metadata_item> > >::EmplaceDecomposable&&, std::__1::pair<model::topic_namespace, cluster::topic_table::topic_metadata_item>&&) a  t /vectorized/include/absl/container/internal/hash_policy_traits.h:134                                                                                                                      
   (inlined by) std::__1::pair<absl::lts_20230802::container_internal::raw_hash_set<absl::lts_20230802::container_internal::NodeHashMapPolicy<model::topic_namespace, cluster::topic_table::to  pic_metadata_item>, model::topic_namespace_hash, model::topic_namespace_eq, std::__1::allocator<std::__1::pair<model::topic_namespace const, cluster::topic_table::topic_metadata_item> > >:  :iterator, bool> absl::lts_20230802::container_internal::raw_hash_set<absl::lts_20230802::container_internal::NodeHashMapPolicy<model::topic_namespace, cluster::topic_table::topic_metadata  _item>, model::topic_namespace_hash, model::topic_namespace_eq, std::__1::allocator<std::__1::pair<model::topic_namespace const, cluster::topic_table::topic_metadata_item> > >::emplace<std  ::__1::pair<model::topic_namespace, cluster::topic_table::topic_metadata_item>, 0>(std::__1::pair<model::topic_namespace, cluster::topic_table::topic_metadata_item>&&) at /vectorized/inclu  de/absl/container/internal/raw_hash_set.h:2064                                                                                                                                              
   (inlined by) absl::lts_20230802::container_internal::raw_hash_set<absl::lts_20230802::container_internal::NodeHashMapPolicy<model::topic_namespace, cluster::topic_table::topic_metadata_it  em>, model::topic_namespace_hash, model::topic_namespace_eq, std::__1::allocator<std::__1::pair<model::topic_namespace const, cluster::topic_table::topic_metadata_item> > >::insert(std::__  1::pair<model::topic_namespace, cluster::topic_table::topic_metadata_item>&&) at /vectorized/include/absl/container/internal/raw_hash_set.h:1991                                            
   (inlined by) cluster::topic_table::apply(cluster::controller_command<model::topic_namespace, cluster::configuration_with_assignment<cluster::topic_configuration>, (signed char)0, (model::  record_batch_type)6, (cluster::serde_opts)0>, detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true> >) at /var/lib/buildkite-agent/builds/buildkit  e-amd64-builders-i-0a8f97b231595b248-1/redpanda/redpanda/src/v/cluster/topic_table.cc:92                                                                                                    
  {/opt/redpanda/libexec/redpanda} 0x6452db1: seastar::future<std::__1::error_code> cluster::do_apply<cluster::controller_command<model::topic_namespace, cluster::configuration_with_assignme  nt<cluster::topic_configuration>, (signed char)0, (model::record_batch_type)6, (cluster::serde_opts)0> >(unsigned int, cluster::controller_command<model::topic_namespace, cluster::configur  ation_with_assignment<cluster::topic_configuration>, (signed char)0, (model::record_batch_type)6, (cluster::serde_opts)0>, seastar::sharded<cluster::topic_table>&, detail::base_named_type<  long, model::model_offset_type, std::__1::integral_constant<bool, true> >)::'lambda'(cluster::topic_table&)::operator()(cluster::topic_table&) at /var/lib/buildkite-agent/builds/buildkite-  amd64-builders-i-0a8f97b231595b248-1/redpanda/redpanda/src/v/cluster/topic_updates_dispatcher.cc:507                                                                                        
   (inlined by) decltype(std::declval<cluster::controller_command<model::topic_namespace, cluster::configuration_with_assignment<cluster::topic_configuration>, (signed char)0, (model::record  _batch_type)6, (cluster::serde_opts)0> >()(std::declval<cluster::topic_table&>())) std::__1::__invoke[abi:v160004]<seastar::future<std::__1::error_code> cluster::do_apply<cluster::controll  er_command<model::topic_namespace, cluster::configuration_with_assignment<cluster::topic_configuration>, (signed char)0, (model::record_batch_type)6, (cluster::serde_opts)0> >(unsigned int  , cluster::controller_command<model::topic_namespace, cluster::configuration_with_assignment<cluster::topic_configuration>, (signed char)0, (model::record_batch_type)6, (cluster::serde_opt  s)0>, seastar::sharded<cluster::topic_table>&, detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true> >)::'lambda'(cluster::topic_table&), cluster:  :topic_table&>(cluster::controller_command<model::topic_namespace, cluster::configuration_with_assignment<cluster::topic_configuration>, (signed char)0, (model::record_batch_type)6, (clust  er::serde_opts)0>&&, cluster::topic_table&) at /vectorized/llvm/bin/../include/c++/v1/__functional/invoke.h:394                                                                             
   (inlined by) decltype(auto) std::__1::__apply_tuple_impl[abi:v160004]<seastar::future<std::__1::error_code> cluster::do_apply<cluster::controller_command<model::topic_namespace, cluster::  configuration_with_assignment<cluster::topic_configuration>, (signed char)0, (model::record_batch_type)6, (cluster::serde_opts)0> >(unsigned int, cluster::controller_command<model::topic_n  amespace, cluster::configuration_with_assignment<cluster::topic_configuration>, (signed char)0, (model::record_batch_type)6, (cluster::serde_opts)0>, seastar::sharded<cluster::topic_table>  &, detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true> >)::'lambda'(cluster::topic_table&), std::__1::tuple<cluster::topic_table&>, 0ul>(cluster  ::controller_command<model::topic_namespace, cluster::configuration_with_assignment<cluster::topic_configuration>, (signed char)0, (model::record_batch_type)6, (cluster::serde_opts)0>&&, s  td::__1::tuple<cluster::topic_table&>&&, std::__1::__tuple_indices<0ul>) at /vectorized/llvm/bin/../include/c++/v1/tuple:1816                                                               
   (inlined by) decltype(auto) std::__1::apply[abi:v160004]<seastar::future<std::__1::error_code> cluster::do_apply<cluster::controller_command<model::topic_namespace, cluster::configuration  _with_assignment<cluster::topic_configuration>, (signed char)0, (model::record_batch_type)6, (cluster::serde_opts)0> >(unsigned int, cluster::controller_command<model::topic_namespace, clu  ster::configuration_with_assignment<cluster::topic_configuration>, (signed char)0, (model::record_batch_type)6, (cluster::serde_opts)0>, seastar::sharded<cluster::topic_table>&, detail::ba  se_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true> >)::'lambda'(cluster::topic_table&), std::__1::tuple<cluster::topic_table&> >(cluster::controller_comm  and<model::topic_namespace, cluster::configuration_with_assignment<cluster::topic_configuration>, (signed char)0, (model::record_batch_type)6, (cluster::serde_opts)0>&&, std::__1::tuple<cl  uster::topic_table&>&&) at /vectorized/llvm/bin/../include/c++/v1/tuple:1825                                                                                                                
   (inlined by) seastar::future<std::__1::error_code> seastar::sharded<cluster::topic_table>::invoke_on<seastar::future<std::__1::error_code> cluster::do_apply<cluster::controller_command<mo  del::topic_namespace, cluster::configuration_with_assignment<cluster::topic_configuration>, (signed char)0, (model::record_batch_type)6, (cluster::serde_opts)0> >(unsigned int, cluster::co  ntroller_command<model::topic_namespace, cluster::configuration_with_assignment<cluster::topic_configuration>, (signed char)0, (model::record_batch_type)6, (cluster::serde_opts)0>, seastar  ::sharded<cluster::topic_table>&, detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true> >)::'lambda'(cluster::topic_table&), seastar::future<std::  __1::error_code> >(unsigned int, seastar::smp_submit_to_options, cluster::controller_command<model::topic_namespace, cluster::configuration_with_assignment<cluster::topic_configuration>, (  signed char)0, (model::record_batch_type)6, (cluster::serde_opts)0>&&)::'lambda'()::operator()() at /vectorized/include/seastar/core/sharded.hh:468                                         
   (inlined by) seastar::future<std::__1::error_code> seastar::futurize<seastar::future<std::__1::error_code> >::invoke<seastar::future<std::__1::error_code> seastar::sharded<cluster::topic_  table>::invoke_on<seastar::future<std::__1::error_code> cluster::do_apply<cluster::controller_command<model::topic_namespace, cluster::configuration_with_assignment<cluster::topic_configur  ation>, (signed char)0, (model::record_batch_type)6, (cluster::serde_opts)0> >(unsigned int, cluster::controller_command<model::topic_namespace, cluster::configuration_with_assignment<clus  ter::topic_configuration>, (signed char)0, (model::record_batch_type)6, (cluster::serde_opts)0>, seastar::sharded<cluster::topic_table>&, detail::base_named_type<long, model::model_offset_  type, std::__1::integral_constant<bool, true> >)::'lambda'(cluster::topic_table&), seastar::future<std::__1::error_code> >(unsigned int, seastar::smp_submit_to_options, cluster::controller  _command<model::topic_namespace, cluster::configuration_with_assignment<cluster::topic_configuration>, (signed char)0, (model::record_batch_type)6, (cluster::serde_opts)0>&&)::'lambda'()&>  (cluster::controller_command<model::topic_namespace, cluster::configuration_with_assignment<cluster::topic_configuration>, (signed char)0, (model::record_batch_type)6, (cluster::serde_opts  )0>&&) at /vectorized/include/seastar/core/future.hh:2006                                                                                                                                   
   (inlined by) seastar::smp_message_queue::async_work_item<seastar::future<std::__1::error_code> seastar::sharded<cluster::topic_table>::invoke_on<seastar::future<std::__1::error_code> clus  ter::do_apply<cluster::controller_command<model::topic_namespace, cluster::configuration_with_assignment<cluster::topic_configuration>, (signed char)0, (model::record_batch_type)6, (cluste  r::serde_opts)0> >(unsigned int, cluster::controller_command<model::topic_namespace, cluster::configuration_with_assignment<cluster::topic_configuration>, (signed char)0, (model::record_ba  tch_type)6, (cluster::serde_opts)0>, seastar::sharded<cluster::topic_table>&, detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true> >)::'lambda'(c  luster::topic_table&), seastar::future<std::__1::error_code> >(unsigned int, seastar::smp_submit_to_options, cluster::controller_command<model::topic_namespace, cluster::configuration_with  _assignment<cluster::topic_configuration>, (signed char)0, (model::record_batch_type)6, (cluster::serde_opts)0>&&)::'lambda'()>::run_and_dispose() at /vectorized/include/seastar/core/smp.h  h:263                                                                                                                                                                                       
  {/opt/redpanda/libexec/redpanda} 0x91e27cf: seastar::reactor::run_tasks(seastar::reactor::task_queue&) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:2750         
   (inlined by) seastar::reactor::run_some_tasks() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:3213                                                               
  {/opt/redpanda/libexec/redpanda} 0x91e5f41: seastar::reactor::do_run() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:3397                                         
  {/opt/redpanda/libexec/redpanda} 0x9240975: seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_2::operator()() const at /v/build/v_deps_build/seastar-  prefix/src/seastar/src/core/reactor.cc:4624                                                                                                                                                 
   (inlined by) decltype(std::declval<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_2&>()()) std::__1::__invoke[abi:v160004]<seastar::smp::configur  e(seastar::smp_options const&, seastar::reactor_options const&)::$_2&>(seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_2&) at /vectorized/llvm/bin/  ../include/c++/v1/__functional/invoke.h:394                                                                                                                                                 
   (inlined by) void std::__1::__invoke_void_return_wrapper<void, true>::__call<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_2&>(seastar::smp::con  figure(seastar::smp_options const&, seastar::reactor_options const&)::$_2&) at /vectorized/llvm/bin/../include/c++/v1/__functional/invoke.h:487                                             
   (inlined by) std::__1::__function::__alloc_func<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_2, std::__1::allocator<seastar::smp::configure(sea  star::smp_options const&, seastar::reactor_options const&)::$_2>, void ()>::operator()[abi:v160004]() at /vectorized/llvm/bin/../include/c++/v1/__functional/function.h:185                 
   (inlined by) std::__1::__function::__func<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_2, std::__1::allocator<seastar::smp::configure(seastar::  smp_options const&, seastar::reactor_options const&)::$_2>, void ()>::operator()() at /vectorized/llvm/bin/../include/c++/v1/__functional/function.h:356                                    
  {/opt/redpanda/libexec/redpanda} 0x916fc3f: std::__1::__function::__value_func<void ()>::operator()[abi:v160004]() const at /vectorized/llvm/bin/../include/c++/v1/__functional/function.h:5  10                                                                                                                                                                                          
   (inlined by) std::__1::function<void ()>::operator()() const at /vectorized/llvm/bin/../include/c++/v1/__functional/function.h:1156                                                        
   (inlined by) seastar::posix_thread::start_routine(void*) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/posix.cc:90                                                          
  {/opt/redpanda/lib/libc.so.6} 0x91016: ?? at ??:0                                                                                                                                           
  {/opt/redpanda/lib/libc.so.6} 0x1166cf: ?? at ??:0
github-actions[bot] commented 3 weeks ago

This issue hasn't seen activity in 3 months. If you want to keep it open, post a comment or remove the stale label – otherwise this will be closed in two weeks.

mmaslankaprv commented 2 weeks ago

this is not longer releavant as we do not collect the delta vector