search

redpanda-data / redpanda

Redpanda is a streaming data platform for developers. Kafka API compatible. 10x faster. No ZooKeeper. No JVM!
https://redpanda.com
9.16k stars 561 forks source link

Reactor stalled for 33ms in `controller_backend::fetch_deltas` #10419

Open ballard26 opened 1 year ago

ballard26 commented 1 year ago

In: https://ci-artifacts.dev.vectorized.cloud:443/vtools/7317/0187c1bf-8e3c-401e-bda2-49c5fa727564/vbuild/output/ducktape-release-clang-amd64.tgz

Backtrace:

WARNING:root:decoded: tmp/ducktape/results/2023-04-27--001/LargeControllerSnapshotTest/test_join_restart/100/RedpandaService-0-139902322358064/ip-172-31-15-231/redpanda.log:366:Reactor stall
ed for 33 ms on shard 0. Backtrace:                                                                                                                                                           
[Backtrace #3]                                                                                                                                                                                
{tmp/libexec/redpanda} 0x636072f: void seastar::backtrace<seastar::backtrace_buffer::append_backtrace_oneline()::'lambda'(seastar::frame)>(seastar::backtrace_buffer::append_backtrace_oneline
()::'lambda'(seastar::frame)&&) at /v/build/v_deps_build/seastar-prefix/src/seastar/include/seastar/util/backtrace.hh:60                                                                      
 (inlined by) seastar::backtrace_buffer::append_backtrace_oneline() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:797                                               
 (inlined by) seastar::print_with_backtrace(seastar::backtrace_buffer&, bool) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:816                                     
{tmp/libexec/redpanda} 0x6361b42: seastar::internal::cpu_stall_detector::generate_trace() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:1386                        
 (inlined by) seastar::internal::cpu_stall_detector::maybe_report() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:1123                                              
 (inlined by) seastar::internal::cpu_stall_detector::on_signal() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:1143                                                 
 (inlined by) seastar::reactor::block_notifier(int) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:1423                                                              
{tmp/libexec/redpanda} 0x42abf: ?? at ??:0                                                                                                                                                    
{tmp/libexec/redpanda} 0x62c742f: seastar::memory::small_pool::add_more_objects() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:1300                                 
 (inlined by) seastar::memory::small_pool::allocate() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:1241                                                             
 (inlined by) seastar::memory::cpu_pages::allocate_small(unsigned int) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:855                                             
 (inlined by) seastar::memory::allocate(unsigned long) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:1404                                                            
{tmp/libexec/redpanda} 0x62d6ab1: operator new(unsigned long) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:2087                                                     
{tmp/libexec/redpanda} 0x480d6fc: void* std::__1::__libcpp_operator_new<unsigned long>(unsigned long) at /vectorized/llvm/bin/../include/c++/v1/new:245                                       
 (inlined by) std::__1::__libcpp_allocate(unsigned long, unsigned long) at /vectorized/llvm/bin/../include/c++/v1/new:271                                                                     
 (inlined by) std::__1::allocator<absl::lts_20220623::container_internal::AlignedType<8ul>>::allocate(unsigned long) at /vectorized/llvm/bin/../include/c++/v1/__memory/allocator.h:105       
 (inlined by) std::__1::allocator_traits<std::__1::allocator<absl::lts_20220623::container_internal::AlignedType<8ul>>>::allocate(std::__1::allocator<absl::lts_20220623::container_internal::
AlignedType<8ul>>&, unsigned long) at /vectorized/llvm/bin/../include/c++/v1/__memory/allocator_traits.h:262
 (inlined by) void* absl::lts_20220623::container_internal::Allocate<8ul, std::__1::allocator<std::__1::pair<model::ntp const, std::__1::vector<cluster::controller_backend::delta_metadata, s
td::__1::allocator<cluster::controller_backend::delta_metadata>>>>>(std::__1::allocator<std::__1::pair<model::ntp const, std::__1::vector<cluster::controller_backend::delta_metadata, std::__
1::allocator<cluster::controller_backend::delta_metadata>>>>*, unsigned long) at /vectorized/include/absl/container/internal/container_memory.h:64
 (inlined by) absl::lts_20220623::container_internal::btree<absl::lts_20220623::container_internal::map_params<model::ntp, std::__1::vector<cluster::controller_backend::delta_metadata, std::
__1::allocator<cluster::controller_backend::delta_metadata>>, std::__1::less<model::ntp>, std::__1::allocator<std::__1::pair<model::ntp const, std::__1::vector<cluster::controller_backend::d
elta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>>, 256, false>>::allocate(unsigned long) at /vectorized/include/absl/container/internal/btree.h:1605
 (inlined by) absl::lts_20220623::container_internal::btree<absl::lts_20220623::container_internal::map_params<model::ntp, std::__1::vector<cluster::controller_backend::delta_metadata, std::
__1::allocator<cluster::controller_backend::delta_metadata>>, std::__1::less<model::ntp>, std::__1::allocator<std::__1::pair<model::ntp const, std::__1::vector<cluster::controller_backend::d
elta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>>, 256, false>>::new_leaf_node(absl::lts_20220623::container_internal::btree_node<absl::lts_20220623::contain
er_internal::map_params<model::ntp, std::__1::vector<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>, std::__1::less<model::ntp
>, std::__1::allocator<std::__1::pair<model::ntp const, std::__1::vector<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>>, 256
, false>>*) at /vectorized/include/absl/container/internal/btree.h:1616
 (inlined by) absl::lts_20220623::container_internal::btree<absl::lts_20220623::container_internal::map_params<model::ntp, std::__1::vector<cluster::controller_backend::delta_metadata, std::
__1::allocator<cluster::controller_backend::delta_metadata>>, std::__1::less<model::ntp>, std::__1::allocator<std::__1::pair<model::ntp const, std::__1::vector<cluster::controller_backend::d
elta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>>, 256, false>>::rebalance_or_split(absl::lts_20220623::container_internal::btree_iterator<absl::lts_20220623
::container_internal::btree_node<absl::lts_20220623::container_internal::map_params<model::ntp, std::__1::vector<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::con
troller_backend::delta_metadata>>, std::__1::less<model::ntp>, std::__1::allocator<std::__1::pair<model::ntp const, std::__1::vector<cluster::controller_backend::delta_metadata, std::__1::al
locator<cluster::controller_backend::delta_metadata>>>>, 256, false>>, std::__1::pair<model::ntp const, std::__1::vector<cluster::controller_backend::delta_metadata, std::__1::allocator<clus
ter::controller_backend::delta_metadata>>>&, std::__1::pair<model::ntp const, std::__1::vector<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::d
elta_metadata>>>*>*) at /vectorized/include/absl/container/internal/btree.h:2549
{tmp/libexec/redpanda} 0x47ded62: absl::lts_20220623::container_internal::btree_iterator<absl::lts_20220623::container_internal::btree_node<absl::lts_20220623::container_internal::map_params
<model::ntp, std::__1::vector<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>, std::__1::less<model::ntp>, std::__1::allocator<
std::__1::pair<model::ntp const, std::__1::vector<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>>, 256, false>>, std::__1::pa
ir<model::ntp const, std::__1::vector<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>&, std::__1::pair<model::ntp const, std::
__1::vector<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>*> absl::lts_20220623::container_internal::btree<absl::lts_20220623
::container_internal::map_params<model::ntp, std::__1::vector<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>, std::__1::less<m
odel::ntp>, std::__1::allocator<std::__1::pair<model::ntp const, std::__1::vector<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata
>>>>, 256, false>>::internal_emplace<std::__1::piecewise_construct_t const&, std::__1::tuple<model::ntp const&>, std::__1::tuple<>>(absl::lts_20220623::container_internal::btree_iterator<abs
l::lts_20220623::container_internal::btree_node<absl::lts_20220623::container_internal::map_params<model::ntp, std::__1::vector<cluster::controller_backend::delta_metadata, std::__1::allocat
or<cluster::controller_backend::delta_metadata>>, std::__1::less<model::ntp>, std::__1::allocator<std::__1::pair<model::ntp const, std::__1::vector<cluster::controller_backend::delta_metadat
a, std::__1::allocator<cluster::controller_backend::delta_metadata>>>>, 256, false>>, std::__1::pair<model::ntp const, std::__1::vector<cluster::controller_backend::delta_metadata, std::__1:
:allocator<cluster::controller_backend::delta_metadata>>>&, std::__1::pair<model::ntp const, std::__1::vector<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::contro
ller_backend::delta_metadata>>>*>, std::__1::piecewise_construct_t const&, std::__1::tuple<model::ntp const&>&&, std::__1::tuple<>&&) at /vectorized/include/absl/container/internal/btree.h:2
687
 (inlined by) std::__1::pair<absl::lts_20220623::container_internal::btree_iterator<absl::lts_20220623::container_internal::btree_node<absl::lts_20220623::container_internal::map_params<mode
l::ntp, std::__1::vector<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>, std::__1::less<model::ntp>, std::__1::allocator<std::
__1::pair<model::ntp const, std::__1::vector<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>>, 256, false>>, std::__1::pair<mo
del::ntp const, std::__1::vector<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>&, std::__1::pair<model::ntp const, std::__1::
vector<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>*>, bool> absl::lts_20220623::container_internal::btree<absl::lts_202206
23::container_internal::map_params<model::ntp, std::__1::vector<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>, std::__1::less
<model::ntp>, std::__1::allocator<std::__1::pair<model::ntp const, std::__1::vector<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metada
ta>>>>, 256, false>>::insert_unique<model::ntp, std::__1::piecewise_construct_t const&, std::__1::tuple<model::ntp const&>, std::__1::tuple<>>(model::ntp const&, std::__1::piecewise_construc
t_t const&, std::__1::tuple<model::ntp const&>&&, std::__1::tuple<>&&) at /vectorized/include/absl/container/internal/btree.h:2155
 (inlined by) std::__1::pair<absl::lts_20220623::container_internal::btree_iterator<absl::lts_20220623::container_internal::btree_node<absl::lts_20220623::container_internal::map_params<mode
l::ntp, std::__1::vector<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>, std::__1::less<model::ntp>, std::__1::allocator<std::
__1::pair<model::ntp const, std::__1::vector<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>>, 256, false>>, std::__1::pair<mo
del::ntp const, std::__1::vector<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>&, std::__1::pair<model::ntp const, std::__1::
vector<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>*>, bool> absl::lts_20220623::container_internal::btree_map_container<ab
sl::lts_20220623::container_internal::btree<absl::lts_20220623::container_internal::map_params<model::ntp, std::__1::vector<cluster::controller_backend::delta_metadata, std::__1::allocator<c
luster::controller_backend::delta_metadata>>, std::__1::less<model::ntp>, std::__1::allocator<std::__1::pair<model::ntp const, std::__1::vector<cluster::controller_backend::delta_metadata, s
td::__1::allocator<cluster::controller_backend::delta_metadata>>>>, 256, false>>>::try_emplace_impl<model::ntp const&>(model::ntp const&) at /vectorized/include/absl/container/internal/btree
_container.h:526
 (inlined by) std::__1::pair<absl::lts_20220623::container_internal::btree_iterator<absl::lts_20220623::container_internal::btree_node<absl::lts_20220623::container_internal::map_params<mode
l::ntp, std::__1::vector<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>, std::__1::less<model::ntp>, std::__1::allocator<std::
__1::pair<model::ntp const, std::__1::vector<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>>, 256, false>>, std::__1::pair<mo
del::ntp const, std::__1::vector<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>&, std::__1::pair<model::ntp const, std::__1::
vector<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>*>, bool> absl::lts_20220623::container_internal::btree_map_container<ab
sl::lts_20220623::container_internal::btree<absl::lts_20220623::container_internal::map_params<model::ntp, std::__1::vector<cluster::controller_backend::delta_metadata, std::__1::allocator<c
luster::controller_backend::delta_metadata>>, std::__1::less<model::ntp>, std::__1::allocator<std::__1::pair<model::ntp const, std::__1::vector<cluster::controller_backend::delta_metadata, s
td::__1::allocator<cluster::controller_backend::delta_metadata>>>>, 256, false>>>::try_emplace<model::ntp, 0>(model::ntp const&) at /vectorized/include/absl/container/internal/btree_containe
r.h:462
 (inlined by) std::__1::vector<cluster::controller_backend::delta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>& absl::lts_20220623::container_internal::btree_m
ap_container<absl::lts_20220623::container_internal::btree<absl::lts_20220623::container_internal::map_params<model::ntp, std::__1::vector<cluster::controller_backend::delta_metadata, std::_
_1::allocator<cluster::controller_backend::delta_metadata>>, std::__1::less<model::ntp>, std::__1::allocator<std::__1::pair<model::ntp const, std::__1::vector<cluster::controller_backend::de
lta_metadata, std::__1::allocator<cluster::controller_backend::delta_metadata>>>>, 256, false>>>::operator[]<model::ntp>(model::ntp const&) at /vectorized/include/absl/container/internal/btr
ee_container.h:483
 (inlined by) cluster::controller_backend::fetch_deltas()::$_7::operator()(fragmented_vector<cluster::topic_table_delta, 8192ul>) const::'lambda'()::operator()() at /var/lib/buildkite-agent/
builds/buildkite-amd64-builders-i-05222bd6360c4da24-1/redpanda/redpanda/src/v/cluster/controller_backend.cc:644
 (inlined by) seastar::future<void> seastar::futurize<void>::invoke<cluster::controller_backend::fetch_deltas()::$_7::operator()(fragmented_vector<cluster::topic_table_delta, 8192ul>) const:
:'lambda'()>(cluster::controller_backend::fetch_deltas()::$_7::operator()(fragmented_vector<cluster::topic_table_delta, 8192ul>) const::'lambda'()&&) at /vectorized/include/seastar/core/futu
re.hh:1996
 (inlined by) auto seastar::futurize_invoke<cluster::controller_backend::fetch_deltas()::$_7::operator()(fragmented_vector<cluster::topic_table_delta, 8192ul>) const::'lambda'()>(cluster::co
ntroller_backend::fetch_deltas()::$_7::operator()(fragmented_vector<cluster::topic_table_delta, 8192ul>) const::'lambda'()&&) at /vectorized/include/seastar/core/future.hh:2030
 (inlined by) auto seastar::futurize<std::__1::invoke_result<cluster::controller_backend::fetch_deltas()::$_7::operator()(fragmented_vector<cluster::topic_table_delta, 8192ul>) const::'lambd
a'()>::type>::type seastar::with_semaphore<seastar::named_semaphore_exception_factory, cluster::controller_backend::fetch_deltas()::$_7::operator()(fragmented_vector<cluster::topic_table_del
ta, 8192ul>) const::'lambda'(), std::__1::chrono::steady_clock>(seastar::basic_semaphore<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock>&, unsigned long, cluster:
:controller_backend::fetch_deltas()::$_7::operator()(fragmented_vector<cluster::topic_table_delta, 8192ul>) const::'lambda'()&&)::'lambda'(seastar::named_semaphore_exception_factory)::operat
or()<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock>>(seastar::named_semaphore_exception_factory) at /vectorized/include/seastar/core/sem
aphore.hh:717
{tmp/libexec/redpanda} 0x47dd7a3: seastar::future<void> seastar::futurize<seastar::future<void>>::invoke<seastar::futurize<std::__1::invoke_result<cluster::controller_backend::fetch_deltas()
::$_7::operator()(fragmented_vector<cluster::topic_table_delta, 8192ul>) const::'lambda'()>::type>::type seastar::with_semaphore<seastar::named_semaphore_exception_factory, cluster::controll
er_backend::fetch_deltas()::$_7::operator()(fragmented_vector<cluster::topic_table_delta, 8192ul>) const::'lambda'(), std::__1::chrono::steady_clock>(seastar::basic_semaphore<seastar::named_
semaphore_exception_factory, std::__1::chrono::steady_clock>&, unsigned long, cluster::controller_backend::fetch_deltas()::$_7::operator()(fragmented_vector<cluster::topic_table_delta, 8192u
l>) const::'lambda'()&&)::'lambda'(seastar::named_semaphore_exception_factory), seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock>>(seastar:
:named_semaphore_exception_factory&&, seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock>&&) at /vectorized/include/seastar/core/future.hh:19
99
 (inlined by) cluster::controller_backend::fetch_deltas()::$_7::operator()(fragmented_vector<cluster::topic_table_delta, 8192ul>) const::'lambda'() seastar::future<seastar::semaphore_units<s
eastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock>>::then_impl<seastar::futurize<std::__1::invoke_result<cluster::controller_backend::fetch_deltas()::$_7::operator()(
fragmented_vector<cluster::topic_table_delta, 8192ul>) const::'lambda'()>::type>::type seastar::with_semaphore<seastar::named_semaphore_exception_factory, cluster::controller_backend::fetch_
deltas()::$_7::operator()(fragmented_vector<cluster::topic_table_delta, 8192ul>) const::'lambda'(), std::__1::chrono::steady_clock>(seastar::basic_semaphore<seastar::named_semaphore_exceptio
n_factory, std::__1::chrono::steady_clock>&, unsigned long, cluster::controller_backend::fetch_deltas()::$_7::operator()(fragmented_vector<cluster::topic_table_delta, 8192ul>) const::'lambda
'()&&)::'lambda'(seastar::named_semaphore_exception_factory), seastar::future<void>>(seastar::named_semaphore_exception_factory&&) at /vectorized/include/seastar/core/future.hh:1470
 (inlined by) cluster::controller_backend::fetch_deltas()::$_7::operator()(fragmented_vector<cluster::topic_table_delta, 8192ul>) const::'lambda'() seastar::future<seastar::semaphore_units<s
eastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock>>::then<seastar::futurize<std::__1::invoke_result<cluster::controller_backend::fetch_deltas()::$_7::operator()(fragm
ented_vector<cluster::topic_table_delta, 8192ul>) const::'lambda'()>::type>::type seastar::with_semaphore<seastar::named_semaphore_exception_factory, cluster::controller_backend::fetch_delta
s()::$_7::operator()(fragmented_vector<cluster::topic_table_delta, 8192ul>) const::'lambda'(), std::__1::chrono::steady_clock>(seastar::basic_semaphore<seastar::named_semaphore_exception_fac
tory, std::__1::chrono::steady_clock>&, unsigned long, cluster::controller_backend::fetch_deltas()::$_7::operator()(fragmented_vector<cluster::topic_table_delta, 8192ul>) const::'lambda'()&&
)::'lambda'(seastar::named_semaphore_exception_factory), seastar::future<void>>(seastar::named_semaphore_exception_factory&&) at /vectorized/include/seastar/core/future.hh:1396
 (inlined by) seastar::futurize<std::__1::invoke_result<cluster::controller_backend::fetch_deltas()::$_7::operator()(fragmented_vector<cluster::topic_table_delta, 8192ul>) const::'lambda'()>
::type>::type seastar::with_semaphore<seastar::named_semaphore_exception_factory, cluster::controller_backend::fetch_deltas()::$_7::operator()(fragmented_vector<cluster::topic_table_delta, 8
192ul>) const::'lambda'(), std::__1::chrono::steady_clock>(seastar::basic_semaphore<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock>&, unsigned long, cluster::cont
roller_backend::fetch_deltas()::$_7::operator()(fragmented_vector<cluster::topic_table_delta, 8192ul>) const::'lambda'()&&) at /vectorized/include/seastar/core/semaphore.hh:716
 (inlined by) cluster::controller_backend::fetch_deltas()::$_7::operator()(fragmented_vector<cluster::topic_table_delta, 8192ul>) const at /var/lib/buildkite-agent/builds/buildkite-amd64-bui
lders-i-05222bd6360c4da24-1/redpanda/redpanda/src/v/cluster/controller_backend.cc:640
{tmp/libexec/redpanda} 0x47e17ae: decltype(static_cast<cluster::controller_backend::fetch_deltas()::$_7&>(fp)(static_cast<fragmented_vector<cluster::topic_table_delta, 8192ul>>(fp0))) std::_
_1::__invoke<cluster::controller_backend::fetch_deltas()::$_7&, fragmented_vector<cluster::topic_table_delta, 8192ul>>(cluster::controller_backend::fetch_deltas()::$_7&, fragmented_vector<cl
uster::topic_table_delta, 8192ul>&&) at /vectorized/llvm/bin/../include/c++/v1/type_traits:3640 
 (inlined by) std::__1::invoke_result<cluster::controller_backend::fetch_deltas()::$_7&, fragmented_vector<cluster::topic_table_delta, 8192ul>>::type std::__1::invoke<cluster::controller_bac
kend::fetch_deltas()::$_7&, fragmented_vector<cluster::topic_table_delta, 8192ul>>(cluster::controller_backend::fetch_deltas()::$_7&, fragmented_vector<cluster::topic_table_delta, 8192ul>&&)
 at /vectorized/llvm/bin/../include/c++/v1/__functional/invoke.h:93
 (inlined by) auto seastar::internal::future_invoke<cluster::controller_backend::fetch_deltas()::$_7&, fragmented_vector<cluster::topic_table_delta, 8192ul>>(cluster::controller_backend::fet
ch_deltas()::$_7&, fragmented_vector<cluster::topic_table_delta, 8192ul>&&) at /vectorized/include/seastar/core/future.hh:1155
 (inlined by) seastar::future<void> seastar::future<fragmented_vector<cluster::topic_table_delta, 8192ul>>::then_impl_nrvo<cluster::controller_backend::fetch_deltas()::$_7, seastar::future<v
oid>>(cluster::controller_backend::fetch_deltas()::$_7&&)::'lambda'(seastar::internal::promise_base_with_type<void>&&, cluster::controller_backend::fetch_deltas()::$_7&, seastar::future_stat
e<fragmented_vector<cluster::topic_table_delta, 8192ul>>&&)::operator()(seastar::internal::promise_base_with_type<void>&&, cluster::controller_backend::fetch_deltas()::$_7&, seastar::future_
state<fragmented_vector<cluster::topic_table_delta, 8192ul>>&&) const::'lambda'()::operator()() const at /vectorized/include/seastar/core/future.hh:1455
 (inlined by) void seastar::futurize<seastar::future<void>>::satisfy_with_result_of<seastar::future<void> seastar::future<fragmented_vector<cluster::topic_table_delta, 8192ul>>::then_impl_nr
vo<cluster::controller_backend::fetch_deltas()::$_7, seastar::future<void>>(cluster::controller_backend::fetch_deltas()::$_7&&)::'lambda'(seastar::internal::promise_base_with_type<void>&&, c
luster::controller_backend::fetch_deltas()::$_7&, seastar::future_state<fragmented_vector<cluster::topic_table_delta, 8192ul>>&&)::operator()(seastar::internal::promise_base_with_type<void>&
&, cluster::controller_backend::fetch_deltas()::$_7&, seastar::future_state<fragmented_vector<cluster::topic_table_delta, 8192ul>>&&) const::'lambda'()>(seastar::internal::promise_base_with_
type<void>&&, cluster::controller_backend::fetch_deltas()::$_7&&) at /vectorized/include/seastar/core/future.hh:1984
 (inlined by) seastar::future<void> seastar::future<fragmented_vector<cluster::topic_table_delta, 8192ul>>::then_impl_nrvo<cluster::controller_backend::fetch_deltas()::$_7, seastar::future<v
oid>>(cluster::controller_backend::fetch_deltas()::$_7&&)::'lambda'(seastar::internal::promise_base_with_type<void>&&, cluster::controller_backend::fetch_deltas()::$_7&, seastar::future_stat
e<fragmented_vector<cluster::topic_table_delta, 8192ul>>&&)::operator()(seastar::internal::promise_base_with_type<void>&&, cluster::controller_backend::fetch_deltas()::$_7&, seastar::future_
state<fragmented_vector<cluster::topic_table_delta, 8192ul>>&&) const at /vectorized/include/seastar/core/future.hh:1451
 (inlined by) seastar::continuation<seastar::internal::promise_base_with_type<void>, cluster::controller_backend::fetch_deltas()::$_7, seastar::future<void> seastar::future<fragmented_vector
<cluster::topic_table_delta, 8192ul>>::then_impl_nrvo<cluster::controller_backend::fetch_deltas()::$_7, seastar::future<void>>(cluster::controller_backend::fetch_deltas()::$_7&&)::'lambda'(s
eastar::internal::promise_base_with_type<void>&&, cluster::controller_backend::fetch_deltas()::$_7&, seastar::future_state<fragmented_vector<cluster::topic_table_delta, 8192ul>>&&), fragment
ed_vector<cluster::topic_table_delta, 8192ul>>::run_and_dispose() at /vectorized/include/seastar/core/future.hh:742
{tmp/libexec/redpanda} 0x638204f: seastar::reactor::run_tasks(seastar::reactor::task_queue&) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:2557
 (inlined by) seastar::reactor::run_some_tasks() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:3020
{tmp/libexec/redpanda} 0x6385cc7: seastar::reactor::do_run() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:3189
{tmp/libexec/redpanda} 0x6383099: seastar::reactor::run() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:3072
{tmp/libexec/redpanda} 0x6298a81: seastar::app_template::run_deprecated(int, char**, std::__1::function<void ()>&&) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/app-template.
cc:265
{tmp/libexec/redpanda} 0x6296b9f: seastar::app_template::run(int, char**, std::__1::function<seastar::future<int> ()>&&) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/app-temp
late.cc:156
{tmp/libexec/redpanda} 0x1ff05de: application::run(int, char**) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-05222bd6360c4da24-1/redpanda/redpanda/src/v/redpanda/application
.cc:330
{tmp/libexec/redpanda} 0x66b4889: main at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-05222bd6360c4da24-1/redpanda/redpanda/src/v/redpanda/main.cc:22
{tmp/libexec/redpanda} 0x2d58f: ?? at ??:0
{tmp/libexec/redpanda} 0x2d648: ?? at ??:0
{tmp/libexec/redpanda} 0x1feaaa4: _start at ??:0

JIRA Link: CORE-1291

StephanDollberg commented 1 year ago 
{tmp/libexec/redpanda} 0x62c742f: seastar::memory::small_pool::add_more_objects() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:1300

Sounds like it was busy in the allocator, possibly either finding free pages or reclaiming some?

ballard26 commented 1 year ago

It's possible, any idea how long finding/reclaiming pages could take? Another possibility is the loop here https://github.com/redpanda-data/redpanda/blob/dev/src/v/cluster/controller_backend.cc#L642-L645 as deltas can grow arbitrarily large depending on user input.

StephanDollberg commented 1 year ago

It's possible, any idea how long finding/reclaiming pages could take?

Not entirely sure, haven't looked at that code path all too much yet. However, the reclaimer stuff calls back into redpanda to clear stuff out of the chunk cache, right? So that might come on top of that.

dotnwat commented 1 year ago

Not entirely sure, haven't looked at that code path all too much yet. However, the reclaimer stuff calls back into redpanda to clear stuff out of the chunk cache, right? So that might come on top of that.

yes, that is inline with reclaim. I think the reclaimer is amenable to bailing out early if there were a way to figure out if we were eating up too much time provided we'd met the requested amount of memory being free. the other thing that could be costly is if the allocator is running cross-core memory frees. I'm not 100% sure if that is synchronous as well, but it might be.

github-actions[bot] commented 6 days ago

This issue hasn't seen activity in 3 months. If you want to keep it open, post a comment or remove the stale label – otherwise this will be closed in two weeks.