Closed ballard26 closed 2 weeks ago
Switch from std::vector to fragment_vector
At 3GB we should probably think twice about this as even a mitigation. That's a lot of data.
Likely the same issue as the following;
Similar issue from recent testing;
./tests/results/latest/ManyTopicsTest/test_many_topics_throughput/1/RedpandaService-0-140400864584720/ip-172-31-2-236/redpanda.log:3647:WARN 2024-03-07 07:02:07,829 [shard 1:main] seasta r_memory - oversized allocation: 151552 bytes. This is non-fatal, but could lead to latency and/or fragmentation issues. Please report: at
[Backtrace #4]
{/opt/redpanda/libexec/redpanda} 0x94ad7a3: void seastar::backtrace<seastar::current_backtrace_tasklocal()::$_0>(seastar::current_backtrace_tasklocal()::$_0&&) at /v/build/v_deps_build/sea star-prefix/src/seastar/include/seastar/util/backtrace.hh:64
(inlined by) seastar::current_backtrace_tasklocal() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/util/backtrace.cc:98
(inlined by) seastar::current_tasktrace() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/util/backtrace.cc:149
(inlined by) seastar::current_backtrace() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/util/backtrace.cc:182
{/opt/redpanda/libexec/redpanda} 0x91152cc: seastar::memory::cpu_pages::warn_large_allocation(unsigned long) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:839
(inlined by) seastar::memory::cpu_pages::check_large_allocation(unsigned long) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:904
(inlined by) seastar::memory::cpu_pages::allocate_large(unsigned int, bool) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:911
(inlined by) seastar::memory::allocate_large(unsigned long, bool) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:1521
(inlined by) seastar::memory::allocate_slowpath(unsigned long) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:1645
{/opt/redpanda/libexec/redpanda} 0x91225a0: seastar::memory::allocate(unsigned long) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:1658
(inlined by) operator new(unsigned long) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:2355
{/opt/redpanda/libexec/redpanda} 0x3709dc5: void* std::__1::__libcpp_operator_new[abi:v160004]<unsigned long>(unsigned long) at /vectorized/llvm/bin/../include/c++/v1/new:266
(inlined by) std::__1::__libcpp_allocate[abi:v160004](unsigned long, unsigned long) at /vectorized/llvm/bin/../include/c++/v1/new:292
(inlined by) std::__1::allocator<absl::lts_20230802::container_internal::AlignedType<8ul> >::allocate[abi:v160004](unsigned long) at /vectorized/llvm/bin/../include/c++/v1/__memory/alloca tor.h:115
(inlined by) std::__1::allocator_traits<std::__1::allocator<absl::lts_20230802::container_internal::AlignedType<8ul> > >::allocate[abi:v160004](std::__1::allocator<absl::lts_20230802::con tainer_internal::AlignedType<8ul> >&, unsigned long) at /vectorized/llvm/bin/../include/c++/v1/__memory/allocator_traits.h:268
(inlined by) void* absl::lts_20230802::container_internal::Allocate<8ul, std::__1::allocator<char> >(std::__1::allocator<char>*, unsigned long) at /vectorized/include/absl/container/inter nal/container_memory.h:65
(inlined by) void absl::lts_20230802::container_internal::InitializeSlots<std::__1::allocator<char>, 8ul, 8ul>(absl::lts_20230802::container_internal::CommonFields&, std::__1::allocator<c har>) at /vectorized/include/absl/container/internal/raw_hash_set.h:1407
{/opt/redpanda/libexec/redpanda} 0x61112f1: absl::lts_20230802::container_internal::raw_hash_set<absl::lts_20230802::container_internal::NodeHashMapPolicy<model::topic_namespace, cluster:: topic_table::topic_metadata_item>, model::topic_namespace_hash, model::topic_namespace_eq, std::__1::allocator<std::__1::pair<model::topic_namespace const, cluster::topic_table::topic_meta data_item> > >::initialize_slots() at /vectorized/include/absl/container/internal/raw_hash_set.h:2505
(inlined by) absl::lts_20230802::container_internal::raw_hash_set<absl::lts_20230802::container_internal::NodeHashMapPolicy<model::topic_namespace, cluster::topic_table::topic_metadata_it em>, model::topic_namespace_hash, model::topic_namespace_eq, std::__1::allocator<std::__1::pair<model::topic_namespace const, cluster::topic_table::topic_metadata_item> > >::resize(unsigne d long) at /vectorized/include/absl/container/internal/raw_hash_set.h:2515
{/opt/redpanda/libexec/redpanda} 0x61112b0: absl::lts_20230802::container_internal::raw_hash_set<absl::lts_20230802::container_internal::NodeHashMapPolicy<model::topic_namespace, cluster:: topic_table::topic_metadata_item>, model::topic_namespace_hash, model::topic_namespace_eq, std::__1::allocator<std::__1::pair<model::topic_namespace const, cluster::topic_table::topic_meta data_item> > >::prepare_insert(unsigned long) at /vectorized/include/absl/container/internal/raw_hash_set.h:2678
{/opt/redpanda/libexec/redpanda} 0x60cbf09: std::__1::pair<unsigned long, bool> absl::lts_20230802::container_internal::raw_hash_set<absl::lts_20230802::container_internal::NodeHashMapPoli cy<model::topic_namespace, cluster::topic_table::topic_metadata_item>, model::topic_namespace_hash, model::topic_namespace_eq, std::__1::allocator<std::__1::pair<model::topic_namespace con st, cluster::topic_table::topic_metadata_item> > >::find_or_prepare_insert<model::topic_namespace>(model::topic_namespace const&) at /vectorized/include/absl/container/internal/raw_hash_se t.h:2659
(inlined by) std::__1::pair<absl::lts_20230802::container_internal::raw_hash_set<absl::lts_20230802::container_internal::NodeHashMapPolicy<model::topic_namespace, cluster::topic_table::to pic_metadata_item>, model::topic_namespace_hash, model::topic_namespace_eq, std::__1::allocator<std::__1::pair<model::topic_namespace const, cluster::topic_table::topic_metadata_item> > >: :iterator, bool> absl::lts_20230802::container_internal::raw_hash_set<absl::lts_20230802::container_internal::NodeHashMapPolicy<model::topic_namespace, cluster::topic_table::topic_metadata _item>, model::topic_namespace_hash, model::topic_namespace_eq, std::__1::allocator<std::__1::pair<model::topic_namespace const, cluster::topic_table::topic_metadata_item> > >::EmplaceDeco mposable::operator()<model::topic_namespace, std::__1::piecewise_construct_t const&, std::__1::tuple<model::topic_namespace&&>, std::__1::tuple<cluster::topic_table::topic_metadata_item&&> >(model::topic_namespace const&, std::__1::piecewise_construct_t const&, std::__1::tuple<model::topic_namespace&&>&&, std::__1::tuple<cluster::topic_table::topic_metadata_item&&>&&) const at /vectorized/include/absl/container/internal/raw_hash_set.h:2459
(inlined by) decltype(std::declval<absl::lts_20230802::container_internal::raw_hash_set<absl::lts_20230802::container_internal::NodeHashMapPolicy<model::topic_namespace, cluster::topic_ta ble::topic_metadata_item>, model::topic_namespace_hash, model::topic_namespace_eq, std::__1::allocator<std::__1::pair<model::topic_namespace const, cluster::topic_table::topic_metadata_ite m> > >::EmplaceDecomposable>()(std::declval<model::topic_namespace&& const&>(), std::__1::piecewise_construct, std::declval<std::__1::tuple<model::topic_namespace&&> >(), std::declval<std: :__1::tuple<cluster::topic_table::topic_metadata_item&&> >())) absl::lts_20230802::container_internal::memory_internal::DecomposePairImpl<absl::lts_20230802::container_internal::raw_hash_s et<absl::lts_20230802::container_internal::NodeHashMapPolicy<model::topic_namespace, cluster::topic_table::topic_metadata_item>, model::topic_namespace_hash, model::topic_namespace_eq, std ::__1::allocator<std::__1::pair<model::topic_namespace const, cluster::topic_table::topic_metadata_item> > >::EmplaceDecomposable, model::topic_namespace&&, std::__1::tuple<cluster::topic_ table::topic_metadata_item&&> >(absl::lts_20230802::container_internal::raw_hash_set<absl::lts_20230802::container_internal::NodeHashMapPolicy<model::topic_namespace, cluster::topic_table: :topic_metadata_item>, model::topic_namespace_hash, model::topic_namespace_eq, std::__1::allocator<std::__1::pair<model::topic_namespace const, cluster::topic_table::topic_metadata_item> > >::EmplaceDecomposable&&, std::__1::pair<std::__1::tuple<model::topic_namespace&&>, std::__1::tuple<cluster::topic_table::topic_metadata_item&&> >) at /vectorized/include/absl/container/i nternal/container_memory.h:140
(inlined by) decltype(memory_internal::DecomposePairImpl(std::forward<absl::lts_20230802::container_internal::raw_hash_set<absl::lts_20230802::container_internal::NodeHashMapPolicy<model: :topic_namespace, cluster::topic_table::topic_metadata_item>, model::topic_namespace_hash, model::topic_namespace_eq, std::__1::allocator<std::__1::pair<model::topic_namespace const, clust er::topic_table::topic_metadata_item> > >::EmplaceDecomposable>(fp), PairArgs(std::forward<std::__1::pair<model::topic_namespace, cluster::topic_table::topic_metadata_item> >(fp0)))) absl: :lts_20230802::container_internal::DecomposePair<absl::lts_20230802::container_internal::raw_hash_set<absl::lts_20230802::container_internal::NodeHashMapPolicy<model::topic_namespace, clus ter::topic_table::topic_metadata_item>, model::topic_namespace_hash, model::topic_namespace_eq, std::__1::allocator<std::__1::pair<model::topic_namespace const, cluster::topic_table::topic _metadata_item> > >::EmplaceDecomposable, std::__1::pair<model::topic_namespace, cluster::topic_table::topic_metadata_item> >(absl::lts_20230802::container_internal::raw_hash_set<absl::lts _20230802::container_internal::NodeHashMapPolicy<model::topic_namespace, cluster::topic_table::topic_metadata_item>, model::topic_namespace_hash, model::topic_namespace_eq, std::__1::alloc ator<std::__1::pair<model::topic_namespace const, cluster::topic_table::topic_metadata_item> > >::EmplaceDecomposable&&, std::__1::pair<model::topic_namespace, cluster::topic_table::topic_ metadata_item>&&) at /vectorized/include/absl/container/internal/container_memory.h:207
(inlined by) decltype(absl::container_internal::DecomposePair(std::declval<absl::lts_20230802::container_internal::raw_hash_set<absl::lts_20230802::container_internal::NodeHashMapPolicy<m odel::topic_namespace, cluster::topic_table::topic_metadata_item>, model::topic_namespace_hash, model::topic_namespace_eq, std::__1::allocator<std::__1::pair<model::topic_namespace const, cluster::topic_table::topic_metadata_item> > >::EmplaceDecomposable>(), std::declval<std::__1::pair<model::topic_namespace, cluster::topic_table::topic_metadata_item> >())) absl::lts_20230 802::container_internal::NodeHashMapPolicy<model::topic_namespace, cluster::topic_table::topic_metadata_item>::apply<absl::lts_20230802::container_internal::raw_hash_set<absl::lts_20230802 ::container_internal::NodeHashMapPolicy<model::topic_namespace, cluster::topic_table::topic_metadata_item>, model::topic_namespace_hash, model::topic_namespace_eq, std::__1::allocator<std: :__1::pair<model::topic_namespace const, cluster::topic_table::topic_metadata_item> > >::EmplaceDecomposable, std::__1::pair<model::topic_namespace, cluster::topic_table::topic_metadata_it em> >(absl::lts_20230802::container_internal::raw_hash_set<absl::lts_20230802::container_internal::NodeHashMapPolicy<model::topic_namespace, cluster::topic_table::topic_metadata_item>, mod el::topic_namespace_hash, model::topic_namespace_eq, std::__1::allocator<std::__1::pair<model::topic_namespace const, cluster::topic_table::topic_metadata_item> > >::EmplaceDecomposable&&, std::__1::pair<model::topic_namespace, cluster::topic_table::topic_metadata_item>&&) at /vectorized/include/absl/container/node_hash_map.h:583
(inlined by) decltype(absl::lts_20230802::container_internal::NodeHashMapPolicy<model::topic_namespace, cluster::topic_table::topic_metadata_item>::apply(std::forward<absl::lts_20230802:: container_internal::raw_hash_set<absl::lts_20230802::container_internal::NodeHashMapPolicy<model::topic_namespace, cluster::topic_table::topic_metadata_item>, model::topic_namespace_hash, model::topic_namespace_eq, std::__1::allocator<std::__1::pair<model::topic_namespace const, cluster::topic_table::topic_metadata_item> > >::EmplaceDecomposable>(fp), std::forward<std::__1: :pair<model::topic_namespace, cluster::topic_table::topic_metadata_item> >(fp0))) absl::lts_20230802::container_internal::hash_policy_traits<absl::lts_20230802::container_internal::NodeHas hMapPolicy<model::topic_namespace, cluster::topic_table::topic_metadata_item>, void>::apply<absl::lts_20230802::container_internal::raw_hash_set<absl::lts_20230802::container_internal::Nod eHashMapPolicy<model::topic_namespace, cluster::topic_table::topic_metadata_item>, model::topic_namespace_hash, model::topic_namespace_eq, std::__1::allocator<std::__1::pair<model::topic_n amespace const, cluster::topic_table::topic_metadata_item> > >::EmplaceDecomposable, std::__1::pair<model::topic_namespace, cluster::topic_table::topic_metadata_item>, absl::lts_20230802:: container_internal::NodeHashMapPolicy<model::topic_namespace, cluster::topic_table::topic_metadata_item> >(absl::lts_20230802::container_internal::raw_hash_set<absl::lts_20230802::containe r_internal::NodeHashMapPolicy<model::topic_namespace, cluster::topic_table::topic_metadata_item>, model::topic_namespace_hash, model::topic_namespace_eq, std::__1::allocator<std::__1::pair <model::topic_namespace const, cluster::topic_table::topic_metadata_item> > >::EmplaceDecomposable&&, std::__1::pair<model::topic_namespace, cluster::topic_table::topic_metadata_item>&&) a t /vectorized/include/absl/container/internal/hash_policy_traits.h:134
(inlined by) std::__1::pair<absl::lts_20230802::container_internal::raw_hash_set<absl::lts_20230802::container_internal::NodeHashMapPolicy<model::topic_namespace, cluster::topic_table::to pic_metadata_item>, model::topic_namespace_hash, model::topic_namespace_eq, std::__1::allocator<std::__1::pair<model::topic_namespace const, cluster::topic_table::topic_metadata_item> > >: :iterator, bool> absl::lts_20230802::container_internal::raw_hash_set<absl::lts_20230802::container_internal::NodeHashMapPolicy<model::topic_namespace, cluster::topic_table::topic_metadata _item>, model::topic_namespace_hash, model::topic_namespace_eq, std::__1::allocator<std::__1::pair<model::topic_namespace const, cluster::topic_table::topic_metadata_item> > >::emplace<std ::__1::pair<model::topic_namespace, cluster::topic_table::topic_metadata_item>, 0>(std::__1::pair<model::topic_namespace, cluster::topic_table::topic_metadata_item>&&) at /vectorized/inclu de/absl/container/internal/raw_hash_set.h:2064
(inlined by) absl::lts_20230802::container_internal::raw_hash_set<absl::lts_20230802::container_internal::NodeHashMapPolicy<model::topic_namespace, cluster::topic_table::topic_metadata_it em>, model::topic_namespace_hash, model::topic_namespace_eq, std::__1::allocator<std::__1::pair<model::topic_namespace const, cluster::topic_table::topic_metadata_item> > >::insert(std::__ 1::pair<model::topic_namespace, cluster::topic_table::topic_metadata_item>&&) at /vectorized/include/absl/container/internal/raw_hash_set.h:1991
(inlined by) cluster::topic_table::apply(cluster::controller_command<model::topic_namespace, cluster::configuration_with_assignment<cluster::topic_configuration>, (signed char)0, (model:: record_batch_type)6, (cluster::serde_opts)0>, detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true> >) at /var/lib/buildkite-agent/builds/buildkit e-amd64-builders-i-0a8f97b231595b248-1/redpanda/redpanda/src/v/cluster/topic_table.cc:92
{/opt/redpanda/libexec/redpanda} 0x6452db1: seastar::future<std::__1::error_code> cluster::do_apply<cluster::controller_command<model::topic_namespace, cluster::configuration_with_assignme nt<cluster::topic_configuration>, (signed char)0, (model::record_batch_type)6, (cluster::serde_opts)0> >(unsigned int, cluster::controller_command<model::topic_namespace, cluster::configur ation_with_assignment<cluster::topic_configuration>, (signed char)0, (model::record_batch_type)6, (cluster::serde_opts)0>, seastar::sharded<cluster::topic_table>&, detail::base_named_type< long, model::model_offset_type, std::__1::integral_constant<bool, true> >)::'lambda'(cluster::topic_table&)::operator()(cluster::topic_table&) at /var/lib/buildkite-agent/builds/buildkite- amd64-builders-i-0a8f97b231595b248-1/redpanda/redpanda/src/v/cluster/topic_updates_dispatcher.cc:507
(inlined by) decltype(std::declval<cluster::controller_command<model::topic_namespace, cluster::configuration_with_assignment<cluster::topic_configuration>, (signed char)0, (model::record _batch_type)6, (cluster::serde_opts)0> >()(std::declval<cluster::topic_table&>())) std::__1::__invoke[abi:v160004]<seastar::future<std::__1::error_code> cluster::do_apply<cluster::controll er_command<model::topic_namespace, cluster::configuration_with_assignment<cluster::topic_configuration>, (signed char)0, (model::record_batch_type)6, (cluster::serde_opts)0> >(unsigned int , cluster::controller_command<model::topic_namespace, cluster::configuration_with_assignment<cluster::topic_configuration>, (signed char)0, (model::record_batch_type)6, (cluster::serde_opt s)0>, seastar::sharded<cluster::topic_table>&, detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true> >)::'lambda'(cluster::topic_table&), cluster: :topic_table&>(cluster::controller_command<model::topic_namespace, cluster::configuration_with_assignment<cluster::topic_configuration>, (signed char)0, (model::record_batch_type)6, (clust er::serde_opts)0>&&, cluster::topic_table&) at /vectorized/llvm/bin/../include/c++/v1/__functional/invoke.h:394
(inlined by) decltype(auto) std::__1::__apply_tuple_impl[abi:v160004]<seastar::future<std::__1::error_code> cluster::do_apply<cluster::controller_command<model::topic_namespace, cluster:: configuration_with_assignment<cluster::topic_configuration>, (signed char)0, (model::record_batch_type)6, (cluster::serde_opts)0> >(unsigned int, cluster::controller_command<model::topic_n amespace, cluster::configuration_with_assignment<cluster::topic_configuration>, (signed char)0, (model::record_batch_type)6, (cluster::serde_opts)0>, seastar::sharded<cluster::topic_table> &, detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true> >)::'lambda'(cluster::topic_table&), std::__1::tuple<cluster::topic_table&>, 0ul>(cluster ::controller_command<model::topic_namespace, cluster::configuration_with_assignment<cluster::topic_configuration>, (signed char)0, (model::record_batch_type)6, (cluster::serde_opts)0>&&, s td::__1::tuple<cluster::topic_table&>&&, std::__1::__tuple_indices<0ul>) at /vectorized/llvm/bin/../include/c++/v1/tuple:1816
(inlined by) decltype(auto) std::__1::apply[abi:v160004]<seastar::future<std::__1::error_code> cluster::do_apply<cluster::controller_command<model::topic_namespace, cluster::configuration _with_assignment<cluster::topic_configuration>, (signed char)0, (model::record_batch_type)6, (cluster::serde_opts)0> >(unsigned int, cluster::controller_command<model::topic_namespace, clu ster::configuration_with_assignment<cluster::topic_configuration>, (signed char)0, (model::record_batch_type)6, (cluster::serde_opts)0>, seastar::sharded<cluster::topic_table>&, detail::ba se_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true> >)::'lambda'(cluster::topic_table&), std::__1::tuple<cluster::topic_table&> >(cluster::controller_comm and<model::topic_namespace, cluster::configuration_with_assignment<cluster::topic_configuration>, (signed char)0, (model::record_batch_type)6, (cluster::serde_opts)0>&&, std::__1::tuple<cl uster::topic_table&>&&) at /vectorized/llvm/bin/../include/c++/v1/tuple:1825
(inlined by) seastar::future<std::__1::error_code> seastar::sharded<cluster::topic_table>::invoke_on<seastar::future<std::__1::error_code> cluster::do_apply<cluster::controller_command<mo del::topic_namespace, cluster::configuration_with_assignment<cluster::topic_configuration>, (signed char)0, (model::record_batch_type)6, (cluster::serde_opts)0> >(unsigned int, cluster::co ntroller_command<model::topic_namespace, cluster::configuration_with_assignment<cluster::topic_configuration>, (signed char)0, (model::record_batch_type)6, (cluster::serde_opts)0>, seastar ::sharded<cluster::topic_table>&, detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true> >)::'lambda'(cluster::topic_table&), seastar::future<std:: __1::error_code> >(unsigned int, seastar::smp_submit_to_options, cluster::controller_command<model::topic_namespace, cluster::configuration_with_assignment<cluster::topic_configuration>, ( signed char)0, (model::record_batch_type)6, (cluster::serde_opts)0>&&)::'lambda'()::operator()() at /vectorized/include/seastar/core/sharded.hh:468
(inlined by) seastar::future<std::__1::error_code> seastar::futurize<seastar::future<std::__1::error_code> >::invoke<seastar::future<std::__1::error_code> seastar::sharded<cluster::topic_ table>::invoke_on<seastar::future<std::__1::error_code> cluster::do_apply<cluster::controller_command<model::topic_namespace, cluster::configuration_with_assignment<cluster::topic_configur ation>, (signed char)0, (model::record_batch_type)6, (cluster::serde_opts)0> >(unsigned int, cluster::controller_command<model::topic_namespace, cluster::configuration_with_assignment<clus ter::topic_configuration>, (signed char)0, (model::record_batch_type)6, (cluster::serde_opts)0>, seastar::sharded<cluster::topic_table>&, detail::base_named_type<long, model::model_offset_ type, std::__1::integral_constant<bool, true> >)::'lambda'(cluster::topic_table&), seastar::future<std::__1::error_code> >(unsigned int, seastar::smp_submit_to_options, cluster::controller _command<model::topic_namespace, cluster::configuration_with_assignment<cluster::topic_configuration>, (signed char)0, (model::record_batch_type)6, (cluster::serde_opts)0>&&)::'lambda'()&> (cluster::controller_command<model::topic_namespace, cluster::configuration_with_assignment<cluster::topic_configuration>, (signed char)0, (model::record_batch_type)6, (cluster::serde_opts )0>&&) at /vectorized/include/seastar/core/future.hh:2006
(inlined by) seastar::smp_message_queue::async_work_item<seastar::future<std::__1::error_code> seastar::sharded<cluster::topic_table>::invoke_on<seastar::future<std::__1::error_code> clus ter::do_apply<cluster::controller_command<model::topic_namespace, cluster::configuration_with_assignment<cluster::topic_configuration>, (signed char)0, (model::record_batch_type)6, (cluste r::serde_opts)0> >(unsigned int, cluster::controller_command<model::topic_namespace, cluster::configuration_with_assignment<cluster::topic_configuration>, (signed char)0, (model::record_ba tch_type)6, (cluster::serde_opts)0>, seastar::sharded<cluster::topic_table>&, detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true> >)::'lambda'(c luster::topic_table&), seastar::future<std::__1::error_code> >(unsigned int, seastar::smp_submit_to_options, cluster::controller_command<model::topic_namespace, cluster::configuration_with _assignment<cluster::topic_configuration>, (signed char)0, (model::record_batch_type)6, (cluster::serde_opts)0>&&)::'lambda'()>::run_and_dispose() at /vectorized/include/seastar/core/smp.h h:263
{/opt/redpanda/libexec/redpanda} 0x91e27cf: seastar::reactor::run_tasks(seastar::reactor::task_queue&) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:2750
(inlined by) seastar::reactor::run_some_tasks() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:3213
{/opt/redpanda/libexec/redpanda} 0x91e5f41: seastar::reactor::do_run() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:3397
{/opt/redpanda/libexec/redpanda} 0x9240975: seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_2::operator()() const at /v/build/v_deps_build/seastar- prefix/src/seastar/src/core/reactor.cc:4624
(inlined by) decltype(std::declval<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_2&>()()) std::__1::__invoke[abi:v160004]<seastar::smp::configur e(seastar::smp_options const&, seastar::reactor_options const&)::$_2&>(seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_2&) at /vectorized/llvm/bin/ ../include/c++/v1/__functional/invoke.h:394
(inlined by) void std::__1::__invoke_void_return_wrapper<void, true>::__call<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_2&>(seastar::smp::con figure(seastar::smp_options const&, seastar::reactor_options const&)::$_2&) at /vectorized/llvm/bin/../include/c++/v1/__functional/invoke.h:487
(inlined by) std::__1::__function::__alloc_func<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_2, std::__1::allocator<seastar::smp::configure(sea star::smp_options const&, seastar::reactor_options const&)::$_2>, void ()>::operator()[abi:v160004]() at /vectorized/llvm/bin/../include/c++/v1/__functional/function.h:185
(inlined by) std::__1::__function::__func<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_2, std::__1::allocator<seastar::smp::configure(seastar:: smp_options const&, seastar::reactor_options const&)::$_2>, void ()>::operator()() at /vectorized/llvm/bin/../include/c++/v1/__functional/function.h:356
{/opt/redpanda/libexec/redpanda} 0x916fc3f: std::__1::__function::__value_func<void ()>::operator()[abi:v160004]() const at /vectorized/llvm/bin/../include/c++/v1/__functional/function.h:5 10
(inlined by) std::__1::function<void ()>::operator()() const at /vectorized/llvm/bin/../include/c++/v1/__functional/function.h:1156
(inlined by) seastar::posix_thread::start_routine(void*) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/posix.cc:90
{/opt/redpanda/lib/libc.so.6} 0x91016: ?? at ??:0
{/opt/redpanda/lib/libc.so.6} 0x1166cf: ?? at ??:0
This issue hasn't seen activity in 3 months. If you want to keep it open, post a comment or remove the stale
label – otherwise this will be closed in two weeks.
this is not longer releavant as we do not collect the delta vector
Version & Environment
Redpanda version: (use
rpk version
):What went wrong?
Out of memory while trying to start Redpanda;
The exact backtrace for the allocation can be found in the "Additional information" section below. The gist this that we are trying to allocate 2.75GiB for the
_pending_deltas
vector intopic_table
. This seems to be due to the 429,459update_topic_properties
commands in the controller log;(from @daisukebe)
While replaying the controller log the
topic_table
collects astd::vector
of pending deltas to the table. And for eachupdate_topic_properties
we add a pending delta for each partition in the topic. A bunch of the topics in the cluster have 90 partitions;Mitigations
std::vector
tofragment_vector
Solution
Snapshotting the controller log
Additional information
JIRA Link: CORE-1232