search

redpanda-data / redpanda

Redpanda is a streaming data platform for developers. Kafka API compatible. 10x faster. No ZooKeeper. No JVM!
https://redpanda.com
9.48k stars 580 forks source link

Reactor stalled for 32ms in `iobuf::copy` #10418

Closed ballard26 closed 2 months ago

ballard26 commented 1 year ago

In: https://ci-artifacts.dev.vectorized.cloud:443/vtools/7317/0187c1bf-8e3c-401e-bda2-49c5fa727564/vbuild/output/ducktape-release-clang-amd64.tgz

Backtrace:

WARNING:root:decoded: tmp/ducktape/results/2023-04-27--001/KgoVerifierTest/test_with_all_type_of_loads/93/RedpandaService-0-139902322781072/ip-172-31-9-218/redpanda.log:1958:Reactor stalled 
for 32 ms on shard 2. Backtrace:                                                                                                                                                              
[Backtrace #0]                                                                                                                                                                                
{tmp/libexec/redpanda} 0x636072f: void seastar::backtrace<seastar::backtrace_buffer::append_backtrace_oneline()::'lambda'(seastar::frame)>(seastar::backtrace_buffer::append_backtrace_oneline
()::'lambda'(seastar::frame)&&) at /v/build/v_deps_build/seastar-prefix/src/seastar/include/seastar/util/backtrace.hh:60                                                                      
 (inlined by) seastar::backtrace_buffer::append_backtrace_oneline() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:797                                               
 (inlined by) seastar::print_with_backtrace(seastar::backtrace_buffer&, bool) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:816                                     
{tmp/libexec/redpanda} 0x6361b42: seastar::internal::cpu_stall_detector::generate_trace() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:1386                        
 (inlined by) seastar::internal::cpu_stall_detector::maybe_report() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:1123                                              
 (inlined by) seastar::internal::cpu_stall_detector::on_signal() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:1143                                                 
 (inlined by) seastar::reactor::block_notifier(int) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:1423                                                              
{tmp/libexec/redpanda} 0x42abf: ?? at ??:0                                                                                                                                                    
{tmp/libexec/redpanda} 0x19266c: ?? at ??:0                                                                                                                                                   
{tmp/libexec/redpanda} 0x61e98a7: std::__1::enable_if<is_same<std::__1::remove_const<char const>::type, char>::value && is_trivially_copy_assignable<char>::value, char*>::type std::__1::__co
py<char const, char>(char const*, char const*, char*) at /vectorized/llvm/bin/../include/c++/v1/__algorithm/copy.h:56                                                                         
 (inlined by) char* std::__1::copy<char const*, char*>(char const*, char const*, char*) at /vectorized/llvm/bin/../include/c++/v1/__algorithm/copy.h:69                                       
 (inlined by) std::__1::enable_if<__is_cpp17_random_access_iterator<char const*>::value, char*>::type std::__1::copy_n<char const*, unsigned long, char*>(char const*, unsigned long, char*) a
t /vectorized/llvm/bin/../include/c++/v1/__algorithm/copy_n.h:61                                                                                                                              
 (inlined by) iobuf_copy(details::io_iterator_consumer&, unsigned long)::$_4::operator()(char const*, unsigned long) const at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-05222
bd6360c4da24-1/redpanda/redpanda/src/v/bytes/iobuf.cc:147
 (inlined by) unsigned long details::io_iterator_consumer::consume<iobuf_copy(details::io_iterator_consumer&, unsigned long)::$_4>(unsigned long, iobuf_copy(details::io_iterator_consumer&, u
nsigned long)::$_4&&) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-05222bd6360c4da24-1/redpanda/redpanda/src/v/bytes/details/io_iterator_consumer.h:137
 (inlined by) iobuf_copy(details::io_iterator_consumer&, unsigned long) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-05222bd6360c4da24-1/redpanda/redpanda/src/v/bytes/iobuf.
cc:145
 (inlined by) iobuf::copy() const at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-05222bd6360c4da24-1/redpanda/redpanda/src/v/bytes/iobuf.cc:133
{tmp/libexec/redpanda} 0x311892b: kafka::read_result::release_data() &&::'lambda'(seastar::foreign_ptr<std::__1::unique_ptr<iobuf, std::__1::default_delete<iobuf>>>&)::operator()(seastar::fo
reign_ptr<std::__1::unique_ptr<iobuf, std::__1::default_delete<iobuf>>>&) const at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-05222bd6360c4da24-1/redpanda/redpanda/src/v/kafk
a/server/handlers/fetch.h:261
 (inlined by) decltype(static_cast<seastar::internal::variant_visitor<kafka::read_result::release_data() &&::'lambda'(std::__1::unique_ptr<iobuf, std::__1::default_delete<iobuf>>&), kafka::r
ead_result::release_data() &&::'lambda'(seastar::foreign_ptr<std::__1::unique_ptr<iobuf, std::__1::default_delete<iobuf>>>&)>>(fp)(static_cast<seastar::foreign_ptr<std::__1::unique_ptr<iobuf
, std::__1::default_delete<iobuf>>>&>(fp0))) std::__1::__invoke_constexpr<seastar::internal::variant_visitor<kafka::read_result::release_data() &&::'lambda'(std::__1::unique_ptr<iobuf, std::
__1::default_delete<iobuf>>&), kafka::read_result::release_data() &&::'lambda'(seastar::foreign_ptr<std::__1::unique_ptr<iobuf, std::__1::default_delete<iobuf>>>&)>, seastar::foreign_ptr<std
::__1::unique_ptr<iobuf, std::__1::default_delete<iobuf>>>&>(seastar::internal::variant_visitor<kafka::read_result::release_data() &&::'lambda'(std::__1::unique_ptr<iobuf, std::__1::default_
delete<iobuf>>&), kafka::read_result::release_data() &&::'lambda'(seastar::foreign_ptr<std::__1::unique_ptr<iobuf, std::__1::default_delete<iobuf>>>&)>&&, seastar::foreign_ptr<std::__1::uniq
ue_ptr<iobuf, std::__1::default_delete<iobuf>>>&) at /vectorized/llvm/bin/../include/c++/v1/type_traits:3648
 (inlined by) decltype(auto) std::__1::__variant_detail::__visitation::__variant::__value_visitor<seastar::internal::variant_visitor<kafka::read_result::release_data() &&::'lambda'(std::__1:
:unique_ptr<iobuf, std::__1::default_delete<iobuf>>&), kafka::read_result::release_data() &&::'lambda'(seastar::foreign_ptr<std::__1::unique_ptr<iobuf, std::__1::default_delete<iobuf>>>&)>>:
:operator()<std::__1::__variant_detail::__alt<1ul, seastar::foreign_ptr<std::__1::unique_ptr<iobuf, std::__1::default_delete<iobuf>>>>&>(std::__1::__variant_detail::__alt<1ul, seastar::forei
gn_ptr<std::__1::unique_ptr<iobuf, std::__1::default_delete<iobuf>>>>&) const at /vectorized/llvm/bin/../include/c++/v1/variant:656
 (inlined by) decltype(static_cast<std::__1::__variant_detail::__visitation::__variant::__value_visitor<seastar::internal::variant_visitor<kafka::read_result::release_data() &&::'lambda'(std
::__1::unique_ptr<iobuf, std::__1::default_delete<iobuf>>&), kafka::read_result::release_data() &&::'lambda'(seastar::foreign_ptr<std::__1::unique_ptr<iobuf, std::__1::default_delete<iobuf>>
>&)>>>(fp)(static_cast<std::__1::__variant_detail::__alt<1ul, seastar::foreign_ptr<std::__1::unique_ptr<iobuf, std::__1::default_delete<iobuf>>>>&>(fp0))) std::__1::__invoke_constexpr<std::_
_1::__variant_detail::__visitation::__variant::__value_visitor<seastar::internal::variant_visitor<kafka::read_result::release_data() &&::'lambda'(std::__1::unique_ptr<iobuf, std::__1::defaul
t_delete<iobuf>>&), kafka::read_result::release_data() &&::'lambda'(seastar::foreign_ptr<std::__1::unique_ptr<iobuf, std::__1::default_delete<iobuf>>>&)>>, std::__1::__variant_detail::__alt<
1ul, seastar::foreign_ptr<std::__1::unique_ptr<iobuf, std::__1::default_delete<iobuf>>>>&>(std::__1::__variant_detail::__visitation::__variant::__value_visitor<seastar::internal::variant_vis
itor<kafka::read_result::release_data() &&::'lambda'(std::__1::unique_ptr<iobuf, std::__1::default_delete<iobuf>>&), kafka::read_result::release_data() &&::'lambda'(seastar::foreign_ptr<std:
:__1::unique_ptr<iobuf, std::__1::default_delete<iobuf>>>&)>>&&, std::__1::__variant_detail::__alt<1ul, seastar::foreign_ptr<std::__1::unique_ptr<iobuf, std::__1::default_delete<iobuf>>>>&) 
at /vectorized/llvm/bin/../include/c++/v1/type_traits:3648
 (inlined by) decltype(auto) std::__1::__variant_detail::__visitation::__base::__dispatcher<1ul>::__dispatch<std::__1::__variant_detail::__visitation::__variant::__value_visitor<seastar::int
ernal::variant_visitor<kafka::read_result::release_data() &&::'lambda'(std::__1::unique_ptr<iobuf, std::__1::default_delete<iobuf>>&), kafka::read_result::release_data() &&::'lambda'(seastar
::foreign_ptr<std::__1::unique_ptr<iobuf, std::__1::default_delete<iobuf>>>&)>>&&, std::__1::__variant_detail::__base<(std::__1::__variant_detail::_Trait)1, std::__1::unique_ptr<iobuf, std::
__1::default_delete<iobuf>>, seastar::foreign_ptr<std::__1::unique_ptr<iobuf, std::__1::default_delete<iobuf>>>>&>(std::__1::__variant_detail::__visitation::__variant::__value_visitor<seasta
r::internal::variant_visitor<kafka::read_result::release_data() &&::'lambda'(std::__1::unique_ptr<iobuf, std::__1::default_delete<iobuf>>&), kafka::read_result::release_data() &&::'lambda'(s
eastar::foreign_ptr<std::__1::unique_ptr<iobuf, std::__1::default_delete<iobuf>>>&)>>&&, std::__1::__variant_detail::__base<(std::__1::__variant_detail::_Trait)1, std::__1::unique_ptr<iobuf,
 std::__1::default_delete<iobuf>>, seastar::foreign_ptr<std::__1::unique_ptr<iobuf, std::__1::default_delete<iobuf>>>>&) at /vectorized/llvm/bin/../include/c++/v1/variant:536
{tmp/libexec/redpanda} 0x30fa2b4: decltype(auto) std::__1::__variant_detail::__visitation::__base::__visit_alt<std::__1::__variant_detail::__visitation::__variant::__value_visitor<seastar::i
nternal::variant_visitor<kafka::read_result::release_data() &&::'lambda'(std::__1::unique_ptr<iobuf, std::__1::default_delete<iobuf>>&), kafka::read_result::release_data() &&::'lambda'(seast
ar::foreign_ptr<std::__1::unique_ptr<iobuf, std::__1::default_delete<iobuf>>>&)>>, std::__1::__variant_detail::__impl<std::__1::unique_ptr<iobuf, std::__1::default_delete<iobuf>>, seastar::f
oreign_ptr<std::__1::unique_ptr<iobuf, std::__1::default_delete<iobuf>>>>&>(std::__1::__variant_detail::__visitation::__variant::__value_visitor<seastar::internal::variant_visitor<kafka::rea
d_result::release_data() &&::'lambda'(std::__1::unique_ptr<iobuf, std::__1::default_delete<iobuf>>&), kafka::read_result::release_data() &&::'lambda'(seastar::foreign_ptr<std::__1::unique_pt
r<iobuf, std::__1::default_delete<iobuf>>>&)>>&&, std::__1::__variant_detail::__impl<std::__1::unique_ptr<iobuf, std::__1::default_delete<iobuf>>, seastar::foreign_ptr<std::__1::unique_ptr<i
obuf, std::__1::default_delete<iobuf>>>>&) at /vectorized/llvm/bin/../include/c++/v1/variant:499
 (inlined by) decltype(auto) std::__1::__variant_detail::__visitation::__variant::__visit_alt<std::__1::__variant_detail::__visitation::__variant::__value_visitor<seastar::internal::variant_
visitor<kafka::read_result::release_data() &&::'lambda'(std::__1::unique_ptr<iobuf, std::__1::default_delete<iobuf>>&), kafka::read_result::release_data() &&::'lambda'(seastar::foreign_ptr<s
td::__1::unique_ptr<iobuf, std::__1::default_delete<iobuf>>>&)>>, std::__1::variant<std::__1::unique_ptr<iobuf, std::__1::default_delete<iobuf>>, seastar::foreign_ptr<std::__1::unique_ptr<io
buf, std::__1::default_delete<iobuf>>>>&>(std::__1::__variant_detail::__visitation::__variant::__value_visitor<seastar::internal::variant_visitor<kafka::read_result::release_data() &&::'lamb
da'(std::__1::unique_ptr<iobuf, std::__1::default_delete<iobuf>>&), kafka::read_result::release_data() &&::'lambda'(seastar::foreign_ptr<std::__1::unique_ptr<iobuf, std::__1::default_delete<
iobuf>>>&)>>&&, std::__1::variant<std::__1::unique_ptr<iobuf, std::__1::default_delete<iobuf>>, seastar::foreign_ptr<std::__1::unique_ptr<iobuf, std::__1::default_delete<iobuf>>>>&) at /vect
orized/llvm/bin/../include/c++/v1/variant:606
 (inlined by) decltype(auto) std::__1::__variant_detail::__visitation::__variant::__visit_value<seastar::internal::variant_visitor<kafka::read_result::release_data() &&::'lambda'(std::__1::u
nique_ptr<iobuf, std::__1::default_delete<iobuf>>&), kafka::read_result::release_data() &&::'lambda'(seastar::foreign_ptr<std::__1::unique_ptr<iobuf, std::__1::default_delete<iobuf>>>&)>, st
d::__1::variant<std::__1::unique_ptr<iobuf, std::__1::default_delete<iobuf>>, seastar::foreign_ptr<std::__1::unique_ptr<iobuf, std::__1::default_delete<iobuf>>>>&>(seastar::internal::variant
_visitor<kafka::read_result::release_data() &&::'lambda'(std::__1::unique_ptr<iobuf, std::__1::default_delete<iobuf>>&), kafka::read_result::release_data() &&::'lambda'(seastar::foreign_ptr<
std::__1::unique_ptr<iobuf, std::__1::default_delete<iobuf>>>&)>&&, std::__1::variant<std::__1::unique_ptr<iobuf, std::__1::default_delete<iobuf>>, seastar::foreign_ptr<std::__1::unique_ptr<
iobuf, std::__1::default_delete<iobuf>>>>&) at /vectorized/llvm/bin/../include/c++/v1/variant:625
 (inlined by) decltype(auto) std::__1::visit<seastar::internal::variant_visitor<kafka::read_result::release_data() &&::'lambda'(std::__1::unique_ptr<iobuf, std::__1::default_delete<iobuf>>&)
, kafka::read_result::release_data() &&::'lambda'(seastar::foreign_ptr<std::__1::unique_ptr<iobuf, std::__1::default_delete<iobuf>>>&)>, std::__1::variant<std::__1::unique_ptr<iobuf, std::__
1::default_delete<iobuf>>, seastar::foreign_ptr<std::__1::unique_ptr<iobuf, std::__1::default_delete<iobuf>>>>&, void>(seastar::internal::variant_visitor<kafka::read_result::release_data() &
&::'lambda'(std::__1::unique_ptr<iobuf, std::__1::default_delete<iobuf>>&), kafka::read_result::release_data() &&::'lambda'(seastar::foreign_ptr<std::__1::unique_ptr<iobuf, std::__1::default
_delete<iobuf>>>&)>&&, std::__1::variant<std::__1::unique_ptr<iobuf, std::__1::default_delete<iobuf>>, seastar::foreign_ptr<std::__1::unique_ptr<iobuf, std::__1::default_delete<iobuf>>>>&) a
t /vectorized/llvm/bin/../include/c++/v1/variant:1698
 (inlined by) auto seastar::visit<std::__1::variant<std::__1::unique_ptr<iobuf, std::__1::default_delete<iobuf>>, seastar::foreign_ptr<std::__1::unique_ptr<iobuf, std::__1::default_delete<io
buf>>>>&, kafka::read_result::release_data() &&::'lambda'(std::__1::unique_ptr<iobuf, std::__1::default_delete<iobuf>>&), kafka::read_result::release_data() &&::'lambda'(seastar::foreign_ptr
<std::__1::unique_ptr<iobuf, std::__1::default_delete<iobuf>>>&)>(std::__1::variant<std::__1::unique_ptr<iobuf, std::__1::default_delete<iobuf>>, seastar::foreign_ptr<std::__1::unique_ptr<io
buf, std::__1::default_delete<iobuf>>>>&, kafka::read_result::release_data() &&::'lambda'(std::__1::unique_ptr<iobuf, std::__1::default_delete<iobuf>>&)&&, kafka::read_result::release_data()
 &&::'lambda'(seastar::foreign_ptr<std::__1::unique_ptr<iobuf, std::__1::default_delete<iobuf>>>&)&&) at /vectorized/include/seastar/util/variant_utils.hh:71
 (inlined by) kafka::read_result::release_data() && at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-05222bd6360c4da24-1/redpanda/redpanda/src/v/kafka/server/handlers/fetch.h:25
7
 (inlined by) kafka::fill_fetch_responses(kafka::op_context&, std::__1::vector<kafka::read_result, std::__1::allocator<kafka::read_result>>, std::__1::vector<kafka::op_context::response_plac
eholder*, std::__1::allocator<kafka::op_context::response_placeholder*>>, std::__1::vector<std::__1::unique_ptr<hdr_hist::measurement, std::__1::default_delete<hdr_hist::measurement>>, std::
__1::allocator<std::__1::unique_ptr<hdr_hist::measurement, std::__1::default_delete<hdr_hist::measurement>>>>) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-05222bd6360c4da24
-1/redpanda/redpanda/src/v/kafka/server/handlers/fetch.cc:289
 (inlined by) kafka::handle_shard_fetch(unsigned int, kafka::op_context&, kafka::shard_fetch)::$_3::operator()(std::__1::vector<kafka::read_result, std::__1::allocator<kafka::read_re[1/1983]
at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-05222bd6360c4da24-1/redpanda/redpanda/src/v/kafka/server/handlers/fetch.cc:392
{tmp/libexec/redpanda} 0x30fba89: decltype(static_cast<kafka::handle_shard_fetch(unsigned int, kafka::op_context&, kafka::shard_fetch)::$_3&>(fp)(static_cast<std::__1::vector<kafka::read_res
ult, std::__1::allocator<kafka::read_result>>>(fp0))) std::__1::__invoke<kafka::handle_shard_fetch(unsigned int, kafka::op_context&, kafka::shard_fetch)::$_3&, std::__1::vector<kafka::read_r
esult, std::__1::allocator<kafka::read_result>>>(kafka::handle_shard_fetch(unsigned int, kafka::op_context&, kafka::shard_fetch)::$_3&, std::__1::vector<kafka::read_result, std::__1::allocat
or<kafka::read_result>>&&) at /vectorized/llvm/bin/../include/c++/v1/type_traits:3640
 (inlined by) std::__1::invoke_result<kafka::handle_shard_fetch(unsigned int, kafka::op_context&, kafka::shard_fetch)::$_3&, std::__1::vector<kafka::read_result, std::__1::allocator<kafka::r
ead_result>>>::type std::__1::invoke<kafka::handle_shard_fetch(unsigned int, kafka::op_context&, kafka::shard_fetch)::$_3&, std::__1::vector<kafka::read_result, std::__1::allocator<kafka::re
ad_result>>>(kafka::handle_shard_fetch(unsigned int, kafka::op_context&, kafka::shard_fetch)::$_3&, std::__1::vector<kafka::read_result, std::__1::allocator<kafka::read_result>>&&) at /vecto
rized/llvm/bin/../include/c++/v1/__functional/invoke.h:93
 (inlined by) auto seastar::internal::future_invoke<kafka::handle_shard_fetch(unsigned int, kafka::op_context&, kafka::shard_fetch)::$_3&, std::__1::vector<kafka::read_result, std::__1::allo
cator<kafka::read_result>>>(kafka::handle_shard_fetch(unsigned int, kafka::op_context&, kafka::shard_fetch)::$_3&, std::__1::vector<kafka::read_result, std::__1::allocator<kafka::read_result
>>&&) at /vectorized/include/seastar/core/future.hh:1155
 (inlined by) seastar::future<void> seastar::future<std::__1::vector<kafka::read_result, std::__1::allocator<kafka::read_result>>>::then_impl_nrvo<kafka::handle_shard_fetch(unsigned int, kaf
ka::op_context&, kafka::shard_fetch)::$_3, seastar::future<void>>(kafka::handle_shard_fetch(unsigned int, kafka::op_context&, kafka::shard_fetch)::$_3&&)::'lambda'(seastar::internal::promise
_base_with_type<void>&&, kafka::handle_shard_fetch(unsigned int, kafka::op_context&, kafka::shard_fetch)::$_3&, seastar::future_state<std::__1::vector<kafka::read_result, std::__1::allocator
<kafka::read_result>>>&&)::operator()(seastar::internal::promise_base_with_type<void>&&, kafka::handle_shard_fetch(unsigned int, kafka::op_context&, kafka::shard_fetch)::$_3&, seastar::futur
e_state<std::__1::vector<kafka::read_result, std::__1::allocator<kafka::read_result>>>&&) const::'lambda'()::operator()() const at /vectorized/include/seastar/core/future.hh:1455
 (inlined by) void seastar::futurize<void>::satisfy_with_result_of<seastar::future<void> seastar::future<std::__1::vector<kafka::read_result, std::__1::allocator<kafka::read_result>>>::then_
impl_nrvo<kafka::handle_shard_fetch(unsigned int, kafka::op_context&, kafka::shard_fetch)::$_3, seastar::future<void>>(kafka::handle_shard_fetch(unsigned int, kafka::op_context&, kafka::shar
d_fetch)::$_3&&)::'lambda'(seastar::internal::promise_base_with_type<void>&&, kafka::handle_shard_fetch(unsigned int, kafka::op_context&, kafka::shard_fetch)::$_3&, seastar::future_state<std
::__1::vector<kafka::read_result, std::__1::allocator<kafka::read_result>>>&&)::operator()(seastar::internal::promise_base_with_type<void>&&, kafka::handle_shard_fetch(unsigned int, kafka::o
p_context&, kafka::shard_fetch)::$_3&, seastar::future_state<std::__1::vector<kafka::read_result, std::__1::allocator<kafka::read_result>>>&&) const::'lambda'()>(seastar::internal::promise_b
ase_with_type<void>&&, kafka::handle_shard_fetch(unsigned int, kafka::op_context&, kafka::shard_fetch)::$_3&&) at /vectorized/include/seastar/core/future.hh:1981
 (inlined by) seastar::future<void> seastar::future<std::__1::vector<kafka::read_result, std::__1::allocator<kafka::read_result>>>::then_impl_nrvo<kafka::handle_shard_fetch(unsigned int, kaf
ka::op_context&, kafka::shard_fetch)::$_3, seastar::future<void>>(kafka::handle_shard_fetch(unsigned int, kafka::op_context&, kafka::shard_fetch)::$_3&&)::'lambda'(seastar::internal::promise
_base_with_type<void>&&, kafka::handle_shard_fetch(unsigned int, kafka::op_context&, kafka::shard_fetch)::$_3&, seastar::future_state<std::__1::vector<kafka::read_result, std::__1::allocator
<kafka::read_result>>>&&)::operator()(seastar::internal::promise_base_with_type<void>&&, kafka::handle_shard_fetch(unsigned int, kafka::op_context&, kafka::shard_fetch)::$_3&, seastar::futur
e_state<std::__1::vector<kafka::read_result, std::__1::allocator<kafka::read_result>>>&&) const at /vectorized/include/seastar/core/future.hh:1451
 (inlined by) seastar::continuation<seastar::internal::promise_base_with_type<void>, kafka::handle_shard_fetch(unsigned int, kafka::op_context&, kafka::shard_fetch)::$_3, seastar::future<voi
d> seastar::future<std::__1::vector<kafka::read_result, std::__1::allocator<kafka::read_result>>>::then_impl_nrvo<kafka::handle_shard_fetch(unsigned int, kafka::op_context&, kafka::shard_fet
ch)::$_3, seastar::future<void>>(kafka::handle_shard_fetch(unsigned int, kafka::op_context&, kafka::shard_fetch)::$_3&&)::'lambda'(seastar::internal::promise_base_with_type<void>&&, kafka::h
andle_shard_fetch(unsigned int, kafka::op_context&, kafka::shard_fetch)::$_3&, seastar::future_state<std::__1::vector<kafka::read_result, std::__1::allocator<kafka::read_result>>>&&), std::_
_1::vector<kafka::read_result, std::__1::allocator<kafka::read_result>>>::run_and_dispose() at /vectorized/include/seastar/core/future.hh:742
{tmp/libexec/redpanda} 0x638204f: seastar::reactor::run_tasks(seastar::reactor::task_queue&) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:2557
 (inlined by) seastar::reactor::run_some_tasks() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:3020
{tmp/libexec/redpanda} 0x6385cc7: seastar::reactor::do_run() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:3189
{tmp/libexec/redpanda} 0x63d62e5: seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_107::operator()() const at /v/build/v_deps_build/seastar-prefix/src
/seastar/src/core/reactor.cc:4399
 (inlined by) decltype(static_cast<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_107&>(fp)()) std::__1::__invoke<seastar::smp::configure(seastar::s
mp_options const&, seastar::reactor_options const&)::$_107&>(seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_107&) at /vectorized/llvm/bin/../include
/c++/v1/type_traits:3640
 (inlined by) void std::__1::__invoke_void_return_wrapper<void, true>::__call<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_107&>(seastar::smp::con
figure(seastar::smp_options const&, seastar::reactor_options const&)::$_107&) at /vectorized/llvm/bin/../include/c++/v1/__functional/invoke.h:61
 (inlined by) std::__1::__function::__alloc_func<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_107, std::__1::allocator<seastar::smp::configure(sea
star::smp_options const&, seastar::reactor_options const&)::$_107>, void ()>::operator()() at /vectorized/llvm/bin/../include/c++/v1/__functional/function.h:180
 (inlined by) std::__1::__function::__func<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_107, std::__1::allocator<seastar::smp::configure(seastar::
smp_options const&, seastar::reactor_options const&)::$_107>, void ()>::operator()() at /vectorized/llvm/bin/../include/c++/v1/__functional/function.h:354
{tmp/libexec/redpanda} 0x631cfdf: std::__1::__function::__value_func<void ()>::operator()() const at /vectorized/llvm/bin/../include/c++/v1/__functional/function.h:507
 (inlined by) std::__1::function<void ()>::operator()() const at /vectorized/llvm/bin/../include/c++/v1/__functional/function.h:1184
 (inlined by) seastar::posix_thread::start_routine(void*) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/posix.cc:73
{tmp/libexec/redpanda} 0x91016: ?? at ??:0
{tmp/libexec/redpanda} 0x1166cf: ?? at ??:0

JIRA Link: CORE-1290

ballard26 commented 1 year ago

Also see the same stall from the produce path;

INFO:root:got tmp/ducktape/results/2023-04-27--001/KgoVerifierWithSiTestSmallSegments/test_si_with_timeboxed/cloud_storage_type=CloudStorageType.S3/97/RedpandaService-0-139902322358064/ip-17
2-31-14-47/redpanda.log:5216:Reactor stalled for 34 ms on shard 2. Backtrace: 0x636072f 0x6361b42 0x42abf 0x1926cb 0x61e9cc4 0x60fec21 0x5bb5fe4 0x5bb80d1 0x59cb298 0x311f4ba 0x2e2e647 0x2fb
ff5e 0x2fc07f2 0x31729c1 0x317d1e6 0x638204f 0x6385cc7 0x63d62e5 0x631cfdf 0x91016 0x1166cf log line                                                                                          
WARNING:root:for url: https://ci-artifacts.dev.vectorized.cloud:443/vtools/7317/0187c1bf-8e3c-401e-bda2-49c5fa727564/vbuild/output/ducktape-release-clang-amd64.tgz   
WARNING:root:decoded: tmp/ducktape/results/2023-04-27--001/KgoVerifierWithSiTestSmallSegments/test_si_with_timeboxed/cloud_storage_type=CloudStorageType.S3/97/RedpandaService-0-1399023223580
64/ip-172-31-14-47/redpanda.log:5216:Reactor stalled for 34 ms on shard 2. Backtrace:                                                                                                         
[Backtrace #1]                                                                                                                                                                                
{tmp/libexec/redpanda} 0x636072f: void seastar::backtrace<seastar::backtrace_buffer::append_backtrace_oneline()::'lambda'(seastar::frame)>(seastar::backtrace_buffer::append_backtrace_oneline
()::'lambda'(seastar::frame)&&) at /v/build/v_deps_build/seastar-prefix/src/seastar/include/seastar/util/backtrace.hh:60                                                                      
 (inlined by) seastar::backtrace_buffer::append_backtrace_oneline() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:797                                               
 (inlined by) seastar::print_with_backtrace(seastar::backtrace_buffer&, bool) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:816                                     
{tmp/libexec/redpanda} 0x6361b42: seastar::internal::cpu_stall_detector::generate_trace() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:1386                        
 (inlined by) seastar::internal::cpu_stall_detector::maybe_report() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:1123                                              
 (inlined by) seastar::internal::cpu_stall_detector::on_signal() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:1143                                                 
 (inlined by) seastar::reactor::block_notifier(int) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:1423                                                              
{tmp/libexec/redpanda} 0x42abf: ?? at ??:0                                                                                                                                                    
{tmp/libexec/redpanda} 0x1926cb: ?? at ??:0                                                                                                                                                   
{tmp/libexec/redpanda} 0x61e9cc4: std::__1::enable_if<is_same<std::__1::remove_const<char const>::type, char>::value && is_trivially_copy_assignable<char>::value, char*>::type std::__1::__co
py<char const, char>(char const*, char const*, char*) at /vectorized/llvm/bin/../include/c++/v1/__algorithm/copy.h:56                                                                         
 (inlined by) char* std::__1::copy<char const*, char*>(char const*, char const*, char*) at /vectorized/llvm/bin/../include/c++/v1/__algorithm/copy.h:69                                       
 (inlined by) std::__1::enable_if<__is_cpp17_random_access_iterator<char const*>::value, char*>::type std::__1::copy_n<char const*, unsigned long, char*>(char const*, unsigned long, char*) a
t /vectorized/llvm/bin/../include/c++/v1/__algorithm/copy_n.h:61                                                                                                                              
 (inlined by) iobuf_copy(details::io_iterator_consumer&, unsigned long)::$_4::operator()(char const*, unsigned long) const at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-05222
bd6360c4da24-1/redpanda/redpanda/src/v/bytes/iobuf.cc:147                                                                                                                                     
 (inlined by) unsigned long details::io_iterator_consumer::consume<iobuf_copy(details::io_iterator_consumer&, unsigned long)::$_4>(unsigned long, iobuf_copy(details::io_iterator_consumer&, u
nsigned long)::$_4&&) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-05222bd6360c4da24-1/redpanda/redpanda/src/v/bytes/details/io_iterator_consumer.h:137                      
 (inlined by) iobuf_copy(details::io_iterator_consumer&, unsigned long) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-05222bd6360c4da24-1/redpanda/redpanda/src/v/bytes/iobuf.
cc:145                                                                                                                                                                                        
{tmp/libexec/redpanda} 0x60fec21: iobuf_parser_base::copy(unsigned long) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-05222bd6360c4da24-1/redpanda/redpanda/src/v/bytes/iobuf
_parser.h:100                                                                                                                                                                                 
 (inlined by) model::parse_one_record_copy_from_buffer(iobuf_const_parser&)::$_1::operator()(iobuf_const_parser&, long) const at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-05
222bd6360c4da24-1/redpanda/redpanda/src/v/model/record_utils.cc:174                                                                                                                           
 (inlined by) model::record model::do_parse_one_record_from_buffer<iobuf_const_parser, model::parse_one_record_copy_from_buffer(iobuf_const_parser&)::$_1>(iobuf_const_parser&, int, signed ch
ar, model::parse_one_record_copy_from_buffer(iobuf_const_parser&)::$_1) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-05222bd6360c4da24-1/redpanda/redpanda/src/v/model/record
_utils.cc:132   
 (inlined by) model::parse_one_record_copy_from_buffer(iobuf_const_parser&) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-05222bd6360c4da24-1/redpanda/redpanda/src/v/model/re
cord_utils.cc:172                                                                                                                                                                             
{tmp/libexec/redpanda} 0x5bb5fe4: void model::record_batch::for_each_record<kafka::kafka_batch_adapter::adapt(iobuf&&)::$_2>(kafka::kafka_batch_adapter::adapt(iobuf&&)::$_2) const at /var/li
b/buildkite-agent/builds/buildkite-amd64-builders-i-05222bd6360c4da24-1/redpanda/redpanda/src/v/model/record.h:676                                                                            
 (inlined by) kafka::kafka_batch_adapter::adapt(iobuf&&) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-05222bd6360c4da24-1/redpanda/redpanda/src/v/kafka/protocol/kafka_batch_
adapter.cc:189                                                                                                                                                                                
{tmp/libexec/redpanda} 0x5bb80d1: kafka::kafka_batch_adapter::adapt_with_version(iobuf, detail::base_named_type<short, kafka::kafka_api_version, std::__1::integral_constant<bool, true>>) at 
/var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-05222bd6360c4da24-1/redpanda/redpanda/src/v/kafka/protocol/kafka_batch_adapter.cc:279                                              
{tmp/libexec/redpanda} 0x59cb298: kafka::produce_request_record_data::produce_request_record_data(std::__1::optional<iobuf>&&, detail::base_named_type<short, kafka::kafka_api_version, std::_
_1::integral_constant<bool, true>>) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-05222bd6360c4da24-1/redpanda/redpanda/src/v/kafka/protocol/kafka_batch_adapter.h:88         
 (inlined by) kafka::produce_request_data::decode(kafka::protocol::decoder&, detail::base_named_type<short, kafka::kafka_api_version, std::__1::integral_constant<bool, true>>)::$_157::operat
or()(kafka::protocol::decoder&) const::'lambda'(kafka::protocol::decoder&)::operator()(kafka::protocol::decoder&) const at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-05222bd6
360c4da24-1/redpanda/redpanda/vbuild/release/clang/src/v/kafka/protocol/schemata/protocol.cc:6645                                                                                             
 (inlined by) std::__1::vector<kafka::partition_produce_data, std::__1::allocator<kafka::partition_produce_data>> kafka::protocol::decoder::do_read_array<kafka::produce_request_data::decode(
kafka::protocol::decoder&, detail::base_named_type<short, kafka::kafka_api_version, std::__1::integral_constant<bool, true>>)::$_157::operator()(kafka::protocol::decoder&) const::'lambda'(ka
fka::protocol::decoder&), kafka::partition_produce_data>(int, kafka::produce_request_data::decode(kafka::protocol::decoder&, detail::base_named_type<short, kafka::kafka_api_version, std::__1
::integral_constant<bool, true>>)::$_157::operator()(kafka::protocol::decoder&) const::'lambda'(kafka::protocol::decoder&)&&) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-05
222bd6360c4da24-1/redpanda/redpanda/src/v/kafka/protocol/wire.h:285                                                                                                                           
 (inlined by) std::__1::vector<kafka::partition_produce_data, std::__1::allocator<kafka::partition_produce_data>> kafka::protocol::decoder::read_array<kafka::produce_request_data::decode(kaf
ka::protocol::decoder&, detail::base_named_type<short, kafka::kafka_api_version, std::__1::integral_constant<bool, true>>)::$_157::operator()(kafka::protocol::decoder&) const::'lambda'(kafka
::protocol::decoder&), kafka::partition_produce_data>(kafka::produce_request_data::decode(kafka::protocol::decoder&, detail::base_named_type<short, kafka::kafka_api_version, std::__1::integr
al_constant<bool, true>>)::$_157::operator()(kafka::protocol::decoder&) const::'lambda'(kafka::protocol::decoder&)&&) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-05222bd636
0c4da24-1/redpanda/redpanda/src/v/kafka/protocol/wire.h:186
 (inlined by) kafka::produce_request_data::decode(kafka::protocol::decoder&, detail::base_named_type<short, kafka::kafka_api_version, std::__1::integral_constant<bool, true>>)::$_157::operat
or()(kafka::protocol::decoder&) const at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-05222bd6360c4da24-1/redpanda/redpanda/vbuild/release/clang/src/v/kafka/protocol/schemata/p
rotocol.cc:6638
 (inlined by) std::__1::vector<kafka::topic_produce_data, std::__1::allocator<kafka::topic_produce_data>> kafka::protocol::decoder::do_read_array<kafka::produce_request_data::decode(kafka::p
rotocol::decoder&, detail::base_named_type<short, kafka::kafka_api_version, std::__1::integral_constant<bool, true>>)::$_157, kafka::topic_produce_data>(int, kafka::produce_request_data::dec
ode(kafka::protocol::decoder&, detail::base_named_type<short, kafka::kafka_api_version, std::__1::integral_constant<bool, true>>)::$_157&&) at /var/lib/buildkite-agent/builds/buildkite-amd64
-builders-i-05222bd6360c4da24-1/redpanda/redpanda/src/v/kafka/protocol/wire.h:285
 (inlined by) std::__1::vector<kafka::topic_produce_data, std::__1::allocator<kafka::topic_produce_data>> kafka::protocol::decoder::read_array<kafka::produce_request_data::decode(kafka::prot
ocol::decoder&, detail::base_named_type<short, kafka::kafka_api_version, std::__1::integral_constant<bool, true>>)::$_157, kafka::topic_produce_data>(kafka::produce_request_data::decode(kafk
a::protocol::decoder&, detail::base_named_type<short, kafka::kafka_api_version, std::__1::integral_constant<bool, true>>)::$_157&&) at /var/lib/buildkite-agent/builds/buildkite-amd64-builder
s-i-05222bd6360c4da24-1/redpanda/redpanda/src/v/kafka/protocol/wire.h:186
 (inlined by) kafka::produce_request_data::decode(kafka::protocol::decoder&, detail::base_named_type<short, kafka::kafka_api_version, std::__1::integral_constant<bool, true>>) at /var/lib/bu
ildkite-agent/builds/buildkite-amd64-builders-i-05222bd6360c4da24-1/redpanda/redpanda/vbuild/release/clang/src/v/kafka/protocol/schemata/protocol.cc:6634
{tmp/libexec/redpanda} 0x311f4ba: kafka::produce_request::decode(kafka::protocol::decoder&, detail::base_named_type<short, kafka::kafka_api_version, std::__1::integral_constant<bool, true>>)
 at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-05222bd6360c4da24-1/redpanda/redpanda/src/v/kafka/protocol/produce.h:56
 (inlined by) kafka::handler_template<kafka::produce_api, (short)0, (short)7, kafka::process_result_stages, &kafka::default_estimate_adaptor(unsigned long, kafka::connection_context&)>::hand
le(kafka::request_context, seastar::smp_service_group) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-05222bd6360c4da24-1/redpanda/redpanda/src/v/kafka/server/handlers/produce
.cc:531
{tmp/libexec/redpanda} 0x2e2e647: kafka::handler_base<true>::handle(kafka::request_context&&, seastar::smp_service_group) const at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-
05222bd6360c4da24-1/redpanda/redpanda/src/v/kafka/server/handlers/handler_interface.cc:85
{tmp/libexec/redpanda} 0x2fbff5e: kafka::process_generic(kafka::handler_interface const*, kafka::request_context&&, seastar::smp_service_group, kafka::session_resources const&) at /var/lib/b
uildkite-agent/builds/buildkite-amd64-builders-i-05222bd6360c4da24-1/redpanda/redpanda/src/v/kafka/server/requests.cc:110
{tmp/libexec/redpanda} 0x2fc07f2: kafka::process_request(kafka::request_context&&, seastar::smp_service_group, kafka::session_resources const&) at /var/lib/buildkite-agent/builds/buildkite-a
md64-builders-i-05222bd6360c4da24-1/redpanda/redpanda/src/v/kafka/server/requests.cc:286
{tmp/libexec/redpanda} 0x31729c1: kafka::connection_context::dispatch_method_once(kafka::request_header, unsigned long)::$_2::operator()(kafka::session_resources)::'lambda'(iobuf)::operator(
)(iobuf) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-05222bd6360c4da24-1/redpanda/redpanda/src/v/kafka/server/connection_context.cc:374
{tmp/libexec/redpanda} 0x317d1e6: decltype(static_cast<kafka::connection_context::dispatch_method_once(kafka::request_header, unsigned long)::$_2::operator()(kafka::session_resources)::'lamb
da'(iobuf)&>(fp)(static_cast<iobuf>(fp0))) std::__1::__invoke<kafka::connection_context::dispatch_method_once(kafka::request_header, unsigned long)::$_2::operator()(kafka::session_resources)
::'lambda'(iobuf)&, iobuf>(kafka::connection_context::dispatch_method_once(kafka::request_header, unsigned long)::$_2::operator()(kafka::session_resources)::'lambda'(iobuf)&, iobuf&&) at /ve
ctorized/llvm/bin/../include/c++/v1/type_traits:3640
 (inlined by) std::__1::invoke_result<kafka::connection_context::dispatch_method_once(kafka::request_header, unsigned long)::$_2::operator()(kafka::session_resources)::'lambda'(iobuf)&, iobu
f>::type std::__1::invoke<kafka::connection_context::dispatch_method_once(kafka::request_header, unsigned long)::$_2::operator()(kafka::session_resources)::'lambda'(iobuf)&, iobuf>(kafka::co
nnection_context::dispatch_method_once(kafka::request_header, unsigned long)::$_2::operator()(kafka::session_resources)::'lambda'(iobuf)&, iobuf&&) at /vectorized/llvm/bin/../include/c++/v1/
__functional/invoke.h:93
 (inlined by) auto seastar::internal::future_invoke<kafka::connection_context::dispatch_method_once(kafka::request_header, unsigned long)::$_2::operator()(kafka::session_resources)::'lambda'
(iobuf)&, iobuf>(kafka::connection_context::dispatch_method_once(kafka::request_header, unsigned long)::$_2::operator()(kafka::session_resources)::'lambda'(iobuf)&, iobuf&&) at /vectorized/i
nclude/seastar/core/future.hh:1155
 (inlined by) seastar::future<void> seastar::future<iobuf>::then_impl_nrvo<kafka::connection_context::dispatch_method_once(kafka::request_header, unsigned long)::$_2::operator()(kafka::sessi
on_resources)::'lambda'(iobuf), seastar::future<void>>(kafka::connection_context::dispatch_method_once(kafka::request_header, unsigned long)::$_2::operator()(kafka::session_resources)::'lamb
da'(iobuf)&&)::'lambda'(seastar::internal::promise_base_with_type<void>&&, kafka::connection_context::dispatch_method_once(kafka::request_header, unsigned long)::$_2::operator()(kafka::sessi
on_resources)::'lambda'(iobuf)&, seastar::future_state<iobuf>&&)::operator()(seastar::internal::promise_base_with_type<void>&&, kafka::connection_context::dispatch_method_once(kafka::request
_header, unsigned long)::$_2::operator()(kafka::session_resources)::'lambda'(iobuf)&, seastar::future_state<iobuf>&&) const::'lambda'()::operator()() const at /vectorized/include/seastar/cor
e/future.hh:1455
 (inlined by) void seastar::futurize<seastar::future<void>>::satisfy_with_result_of<seastar::future<void> seastar::future<iobuf>::then_impl_nrvo<kafka::connection_context::dispatch_method_on
ce(kafka::request_header, unsigned long)::$_2::operator()(kafka::session_resources)::'lambda'(iobuf), seastar::future<void>>(kafka::connection_context::dispatch_method_once(kafka::request_he
ader, unsigned long)::$_2::operator()(kafka::session_resources)::'lambda'(iobuf)&&)::'lambda'(seastar::internal::promise_base_with_type<void>&&, kafka::connection_context::dispatch_method_on
ce(kafka::request_header, unsigned long)::$_2::operator()(kafka::session_resources)::'lambda'(iobuf)&, seastar::future_state<iobuf>&&)::operator()(seastar::internal::promise_base_with_type<v
oid>&&, kafka::connection_context::dispatch_method_once(kafka::request_header, unsigned long)::$_2::operator()(kafka::session_resources)::'lambda'(iobuf)&, seastar::future_state<iobuf>&&) co
nst::'lambda'()>(seastar::internal::promise_base_with_type<void>&&, kafka::connection_context::dispatch_method_once(kafka::request_header, unsigned long)::$_2::operator()(kafka::session_reso
urces)::'lambda'(iobuf)&&) at /vectorized/include/seastar/core/future.hh:1984
 (inlined by) seastar::future<void> seastar::future<iobuf>::then_impl_nrvo<kafka::connection_context::dispatch_method_once(kafka::request_header, unsigned long)::$_2::operator()(kafka::sessi
on_resources)::'lambda'(iobuf), seastar::future<void>>(kafka::connection_context::dispatch_method_once(kafka::request_header, unsigned long)::$_2::operator()(kafka::session_resources)::'lamb
da'(iobuf)&&)::'lambda'(seastar::internal::promise_base_with_type<void>&&, kafka::connection_context::dispatch_method_once(kafka::request_header, unsigned long)::$_2::operator()(kafka::sessi
on_resources)::'lambda'(iobuf)&, seastar::future_state<iobuf>&&)::operator()(seastar::internal::promise_base_with_type<void>&&, kafka::connection_context::dispatch_method_once(kafka::request
_header, unsigned long)::$_2::operator()(kafka::session_resources)::'lambda'(iobuf)&, seastar::future_state<iobuf>&&) const at /vectorized/include/seastar/core/future.hh:1451
 (inlined by) seastar::continuation<seastar::internal::promise_base_with_type<void>, kafka::connection_context::dispatch_method_once(kafka::request_header, unsigned long)::$_2::operator()(ka
fka::session_resources)::'lambda'(iobuf), seastar::future<void> seastar::future<iobuf>::then_impl_nrvo<kafka::connection_context::dispatch_method_once(kafka::request_header, unsigned long)::
$_2::operator()(kafka::session_resources)::'lambda'(iobuf), seastar::future<void>>(kafka::connection_context::dispatch_method_once(kafka::request_header, unsigned long)::$_2::operator()(kafk
a::session_resources)::'lambda'(iobuf)&&)::'lambda'(seastar::internal::promise_base_with_type<void>&&, kafka::connection_context::dispatch_method_once(kafka::request_header, unsigned long)::
$_2::operator()(kafka::session_resources)::'lambda'(iobuf)&, seastar::future_state<iobuf>&&), iobuf>::run_and_dispose() at /vectorized/include/seastar/core/future.hh:742
{tmp/libexec/redpanda} 0x638204f: seastar::reactor::run_tasks(seastar::reactor::task_queue&) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:2557
 (inlined by) seastar::reactor::run_some_tasks() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:3020
{tmp/libexec/redpanda} 0x6385cc7: seastar::reactor::do_run() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:3189
{tmp/libexec/redpanda} 0x63d62e5: seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_107::operator()() const at /v/build/v_deps_build/seastar-prefix/src
/seastar/src/core/reactor.cc:4399
 (inlined by) decltype(static_cast<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_107&>(fp)()) std::__1::__invoke<seastar::smp::configure(seastar::s
mp_options const&, seastar::reactor_options const&)::$_107&>(seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_107&) at /vectorized/llvm/bin/../include
/c++/v1/type_traits:3640
 (inlined by) void std::__1::__invoke_void_return_wrapper<void, true>::__call<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_107&>(seastar::smp::con
figure(seastar::smp_options const&, seastar::reactor_options const&)::$_107&) at /vectorized/llvm/bin/../include/c++/v1/__functional/invoke.h:61
 (inlined by) std::__1::__function::__alloc_func<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_107, std::__1::allocator<seastar::smp::configure(sea
star::smp_options const&, seastar::reactor_options const&)::$_107>, void ()>::operator()() at /vectorized/llvm/bin/../include/c++/v1/__functional/function.h:180
 (inlined by) std::__1::__function::__func<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_107, std::__1::allocator<seastar::smp::configure(seastar::
smp_options const&, seastar::reactor_options const&)::$_107>, void ()>::operator()() at /vectorized/llvm/bin/../include/c++/v1/__functional/function.h:354
{tmp/libexec/redpanda} 0x631cfdf: std::__1::__function::__value_func<void ()>::operator()() const at /vectorized/llvm/bin/../include/c++/v1/__functional/function.h:507
 (inlined by) std::__1::function<void ()>::operator()() const at /vectorized/llvm/bin/../include/c++/v1/__functional/function.h:1184
 (inlined by) seastar::posix_thread::start_routine(void*) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/posix.cc:73
{tmp/libexec/redpanda} 0x91016: ?? at ??:0
{tmp/libexec/redpanda} 0x1166cf: ?? at ??:0
ballard26 commented 1 year ago

Also from raft append_entries;

WARNING:root:for url: https://ci-artifacts.dev.vectorized.cloud:443/vtools/7317/0187c1bf-8e3c-401e-bda2-49c5fa727564/vbuild/output/ducktape-release-clang-amd64.tgz   
WARNING:root:decoded: tmp/ducktape/results/2023-04-27--001/HighThroughputPartitionMovementTest/test_interrupting_partition_movement_under_load/replication_factor=3/288/RedpandaService-0-1399
02299938656/ip-172-31-15-231/redpanda.log:1047:Reactor stalled for 36 ms on shard 2. Backtrace:                                                                                               
[Backtrace #2]                                                                                                                                                                                
{tmp/libexec/redpanda} 0x636072f: void seastar::backtrace<seastar::backtrace_buffer::append_backtrace_oneline()::'lambda'(seastar::frame)>(seastar::backtrace_buffer::append_backtrace_oneline
()::'lambda'(seastar::frame)&&) at /v/build/v_deps_build/seastar-prefix/src/seastar/include/seastar/util/backtrace.hh:60                                                                      
 (inlined by) seastar::backtrace_buffer::append_backtrace_oneline() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:797                                               
 (inlined by) seastar::print_with_backtrace(seastar::backtrace_buffer&, bool) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:816                                     
{tmp/libexec/redpanda} 0x6361b42: seastar::internal::cpu_stall_detector::generate_trace() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:1386                        
 (inlined by) seastar::internal::cpu_stall_detector::maybe_report() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:1123                                              
 (inlined by) seastar::internal::cpu_stall_detector::on_signal() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:1143                                                 
 (inlined by) seastar::reactor::block_notifier(int) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:1423                                                              
{tmp/libexec/redpanda} 0x42abf: ?? at ??:0                                                                                                                                                    
{tmp/libexec/redpanda} 0x1926cb: ?? at ??:0                                                                                                                                                   
{tmp/libexec/redpanda} 0x61e9cc4: std::__1::enable_if<is_same<std::__1::remove_const<char const>::type, char>::value && is_trivially_copy_assignable<char>::value, char*>::type std::__1::__co
py<char const, char>(char const*, char const*, char*) at /vectorized/llvm/bin/../include/c++/v1/__algorithm/copy.h:56                                                                         
 (inlined by) char* std::__1::copy<char const*, char*>(char const*, char const*, char*) at /vectorized/llvm/bin/../include/c++/v1/__algorithm/copy.h:69                                       
 (inlined by) std::__1::enable_if<__is_cpp17_random_access_iterator<char const*>::value, char*>::type std::__1::copy_n<char const*, unsigned long, char*>(char const*, unsigned long, char*) a
t /vectorized/llvm/bin/../include/c++/v1/__algorithm/copy_n.h:61                                                                                                                              
 (inlined by) iobuf_copy(details::io_iterator_consumer&, unsigned long)::$_4::operator()(char const*, unsigned long) const at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-05222
bd6360c4da24-1/redpanda/redpanda/src/v/bytes/iobuf.cc:147                                                                                                                                     
 (inlined by) unsigned long details::io_iterator_consumer::consume<iobuf_copy(details::io_iterator_consumer&, unsigned long)::$_4>(unsigned long, iobuf_copy(details::io_iterator_consumer&, u
nsigned long)::$_4&&) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-05222bd6360c4da24-1/redpanda/redpanda/src/v/bytes/details/io_iterator_consumer.h:137                      
 (inlined by) iobuf_copy(details::io_iterator_consumer&, unsigned long) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-05222bd6360c4da24-1/redpanda/redpanda/src/v/bytes/iobuf$
cc:145                                                                                                                                                                                        
{tmp/libexec/redpanda} 0x60fec21: iobuf_parser_base::copy(unsigned long) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-05222bd6360c4da24-1/redpanda/redpanda/src/v/bytes/iobuf
_parser.h:100                                                                                                                                                                                 
 (inlined by) model::parse_one_record_copy_from_buffer(iobuf_const_parser&)::$_1::operator()(iobuf_const_parser&, long) const at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-05
222bd6360c4da24-1/redpanda/redpanda/src/v/model/record_utils.cc:174                                                                                                                           
 (inlined by) model::record model::do_parse_one_record_from_buffer<iobuf_const_parser, model::parse_one_record_copy_from_buffer(iobuf_const_parser&)::$_1>(iobuf_const_parser&, int, signed ch
ar, model::parse_one_record_copy_from_buffer(iobuf_const_parser&)::$_1) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-05222bd6360c4da24-1/redpanda/redpanda/src/v/model/record
_utils.cc:132                                                                                                                                                                                 
 (inlined by) model::parse_one_record_copy_from_buffer(iobuf_const_parser&) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-05222bd6360c4da24-1/redpanda/redpanda/src/v/model/re
cord_utils.cc:172                                                                                                                                                                             
{tmp/libexec/redpanda} 0x6118c50: void model::record_batch::for_each_record<reflection::adl<model::record_batch>::to(iobuf&, model::record_batch&&)::$_0>(reflection::adl<model::record_batch>
::to(iobuf&, model::record_batch&&)::$_0) const at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-05222bd6360c4da24-1/redpanda/redpanda/src/v/model/record.h:676                  
 (inlined by) reflection::adl<model::record_batch>::to(iobuf&, model::record_batch&&) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-05222bd6360c4da24-1/redpanda/redpanda/src/
v/model/adl_serde.cc:241                                                                                                                                                                      
{tmp/libexec/redpanda} 0x279cacb: void reflection::serialize<model::record_batch>(iobuf&, model::record_batch&&) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-05222bd6360c4da
24-1/redpanda/redpanda/src/v/reflection/adl.h:224                                                                                                                                             
 (inlined by) raft::append_entries_request::serde_async_write(iobuf&) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-05222bd6360c4da24-1/redpanda/redpanda/src/v/raft/types.cc:
604                                                                                                                                                                                           
{tmp/libexec/redpanda} 0x287e1d1: seastar::future<void> serde::write_async<raft::append_entries_request>(iobuf&, raft::append_entries_request)::'lambda'(raft::append_entries_request&)::opera
tor()(raft::append_entries_request&) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-05222bd6360c4da24-1/redpanda/redpanda/src/v/serde/serde.h:1024                             
 (inlined by) decltype(static_cast<raft::append_entries_request>(fp)(static_cast<raft::append_entries_request&>(fp0))) std::__1::__invoke_constexpr<seastar::future<void> serde::write_async<r
aft::append_entries_request>(iobuf&, raft::append_entries_request)::'lambda'(raft::append_entries_request&)&, raft::append_entries_request&>(raft::append_entries_request&&, raft::append_entr
ies_request&) at /vectorized/llvm/bin/../include/c++/v1/type_traits:3648
 (inlined by) decltype(auto) std::__1::__apply_tuple_impl<seastar::future<void> serde::write_async<raft::append_entries_request>(iobuf&, raft::append_entries_request)::'lambda'(raft::append_
entries_request&)&, std::__1::tuple<raft::append_entries_request>&, 0ul>(raft::append_entries_request&&, std::__1::tuple<raft::append_entries_request>&, std::__1::__tuple_indices<0ul>) at /v
ectorized/llvm/bin/../include/c++/v1/tuple:1595 
 (inlined by) decltype(auto) std::__1::apply<seastar::future<void> serde::write_async<raft::append_entries_request>(iobuf&, raft::append_entries_request)::'lambda'(raft::append_entries_reque
st&)&, std::__1::tuple<raft::append_entries_request>&>(raft::append_entries_request&&, std::__1::tuple<raft::append_entries_request>&) at /vectorized/llvm/bin/../include/c++/v1/tuple:1604
 (inlined by) auto seastar::internal::do_with_impl<raft::append_entries_request, seastar::future<void> serde::write_async<raft::append_entries_request>(iobuf&, raft::append_entries_request):
:'lambda'(raft::append_entries_request&)>(raft::append_entries_request&&, seastar::future<void> serde::write_async<raft::append_entries_request>(iobuf&, raft::append_entries_request)::'lambd
a'(raft::append_entries_request&)&&) at /vectorized/include/seastar/core/do_with.hh:96
 (inlined by) seastar::future<void> seastar::futurize<seastar::future<void>>::invoke<seastar::future<void> (*&)(raft::append_entries_request&&, seastar::future<void> serde::write_async<raft:
:append_entries_request>(iobuf&, raft::append_entries_request)::'lambda'(raft::append_entries_request&)&&), raft::append_entries_request, seastar::future<void> serde::write_async<raft::appen
d_entries_request>(iobuf&, raft::append_entries_request)::'lambda'(raft::append_entries_request&)>(raft::append_entries_request&&, raft::append_entries_request&&, seastar::future<void> serde
::write_async<raft::append_entries_request>(iobuf&, raft::append_entries_request)::'lambda'(raft::append_entries_request&)&&) at /vectorized/include/seastar/core/future.hh:1999
 (inlined by) auto seastar::futurize_invoke<seastar::future<void> (*&)(raft::append_entries_request&&, seastar::future<void> serde::write_async<raft::append_entries_request>(iobuf&, raft::ap
pend_entries_request)::'lambda'(raft::append_entries_request&)&&), raft::append_entries_request, seastar::future<void> serde::write_async<raft::append_entries_request>(iobuf&, raft::append_e
ntries_request)::'lambda'(raft::append_entries_request&)>(raft::append_entries_request&&, raft::append_entries_request&&, seastar::future<void> serde::write_async<raft::append_entries_reques
t>(iobuf&, raft::append_entries_request)::'lambda'(raft::append_entries_request&)&&) at /vectorized/include/seastar/core/future.hh:2030
 (inlined by) auto seastar::do_with<raft::append_entries_request, seastar::future<void> serde::write_async<raft::append_entries_request>(iobuf&, raft::append_entries_request)::'lambda'(raft:
:append_entries_request&)>(raft::append_entries_request&&, seastar::future<void> serde::write_async<raft::append_entries_request>(iobuf&, raft::append_entries_request)::'lambda'(raft::append
_entries_request&)&&) at /vectorized/include/seastar/core/do_with.hh:131
 (inlined by) seastar::future<void> serde::write_async<raft::append_entries_request>(iobuf&, raft::append_entries_request) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-05222
bd6360c4da24-1/redpanda/redpanda/src/v/serde/serde.h:1017
{tmp/libexec/redpanda} 0x287c68c: seastar::future<rpc::transport_version> rpc::encode_for_version<raft::append_entries_request>(iobuf&, raft::append_entries_request, rpc::transport_version):
:'lambda'(raft::append_entries_request&)::operator()(raft::append_entries_request&) const at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-05222bd6360c4da24-1/redpanda/redpanda/
src/v/rpc/parse_utils.h:146
 (inlined by) decltype(static_cast<raft::append_entries_request>(fp)(static_cast<raft::append_entries_request&>(fp0))) std::__1::__invoke_constexpr<seastar::future<rpc::transport_version> rp
c::encode_for_version<raft::append_entries_request>(iobuf&, raft::append_entries_request, rpc::transport_version)::'lambda'(raft::append_entries_request&)&, raft::append_entries_request&>(ra
ft::append_entries_request&&, raft::append_entries_request&) at /vectorized/llvm/bin/../include/c++/v1/type_traits:3648
 (inlined by) decltype(auto) std::__1::__apply_tuple_impl<seastar::future<rpc::transport_version> rpc::encode_for_version<raft::append_entries_request>(iobuf&, raft::append_entries_request, 
rpc::transport_version)::'lambda'(raft::append_entries_request&)&, std::__1::tuple<raft::append_entries_request>&, 0ul>(raft::append_entries_request&&, std::__1::tuple<raft::append_entries_r
equest>&, std::__1::__tuple_indices<0ul>) at /vectorized/llvm/bin/../include/c++/v1/tuple:1595
 (inlined by) decltype(auto) std::__1::apply<seastar::future<rpc::transport_version> rpc::encode_for_version<raft::append_entries_request>(iobuf&, raft::append_entries_request, rpc::transpor
t_version)::'lambda'(raft::append_entries_request&)&, std::__1::tuple<raft::append_entries_request>&>(raft::append_entries_request&&, std::__1::tuple<raft::append_entries_request>&) at /vect
orized/llvm/bin/../include/c++/v1/tuple:1604
 (inlined by) auto seastar::internal::do_with_impl<raft::append_entries_request, seastar::future<rpc::transport_version> rpc::encode_for_version<raft::append_entries_request>(iobuf&, raft::a
ppend_entries_request, rpc::transport_version)::'lambda'(raft::append_entries_request&)>(raft::append_entries_request&&, seastar::future<rpc::transport_version> rpc::encode_for_version<raft:
:append_entries_request>(iobuf&, raft::append_entries_request, rpc::transport_version)::'lambda'(raft::append_entries_request&)&&) at /vectorized/include/seastar/core/do_with.hh:96
 (inlined by) seastar::future<rpc::transport_version> seastar::futurize<seastar::future<rpc::transport_version>>::invoke<seastar::future<rpc::transport_version> (*&)(raft::append_entries_req
uest&&, seastar::future<rpc::transport_version> rpc::encode_for_version<raft::append_entries_request>(iobuf&, raft::append_entries_request, rpc::transport_version)::'lambda'(raft::append_ent
ries_request&)&&), raft::append_entries_request, seastar::future<rpc::transport_version> rpc::encode_for_version<raft::append_entries_request>(iobuf&, raft::append_entries_request, rpc::tran
sport_version)::'lambda'(raft::append_entries_request&)>(raft::append_entries_request&&, raft::append_entries_request&&, seastar::future<rpc::transport_version> rpc::encode_for_version<raft:
:append_entries_request>(iobuf&, raft::append_entries_request, rpc::transport_version)::'lambda'(raft::append_entries_request&)&&) at /vectorized/include/seastar/core/future.hh:1999
 (inlined by) auto seastar::futurize_invoke<seastar::future<rpc::transport_version> (*&)(raft::append_entries_request&&, seastar::future<rpc::transport_version> rpc::encode_for_version<raft:
:append_entries_request>(iobuf&, raft::append_entries_request, rpc::transport_version)::'lambda'(raft::append_entries_request&)&&), raft::append_entries_request, seastar::future<rpc::transpo
rt_version> rpc::encode_for_version<raft::append_entries_request>(iobuf&, raft::append_entries_request, rpc::transport_version)::'lambda'(raft::append_entries_request&)>(raft::append_entries
_request&&, raft::append_entries_request&&, seastar::future<rpc::transport_version> rpc::encode_for_version<raft::append_entries_request>(iobuf&, raft::append_entries_request, rpc::transport
_version)::'lambda'(raft::append_entries_request&)&&) at /vectorized/include/seastar/core/future.hh:2030
 (inlined by) auto seastar::do_with<raft::append_entries_request, seastar::future<rpc::transport_version> rpc::encode_for_version<raft::append_entries_request>(iobuf&, raft::append_entries_r
equest, rpc::transport_version)::'lambda'(raft::append_entries_request&)>(raft::append_entries_request&&, seastar::future<rpc::transport_version> rpc::encode_for_version<raft::append_entries
_request>(iobuf&, raft::append_entries_request, rpc::transport_version)::'lambda'(raft::append_entries_request&)&&) at /vectorized/include/seastar/core/do_with.hh:131
 (inlined by) seastar::future<rpc::transport_version> rpc::encode_for_version<raft::append_entries_request>(iobuf&, raft::append_entries_request, rpc::transport_version) at /var/lib/buildkit
e-agent/builds/buildkite-amd64-builders-i-05222bd6360c4da24-1/redpanda/redpanda/src/v/rpc/parse_utils.h:145
 (inlined by) seastar::future<boost::outcome_v2::basic_result<rpc::result_context<raft::append_entries_reply>, std::__1::error_code, std::__1::conditional<false, boost::outcome_v2::policy::t
erminate, std::__1::conditional<true, boost::outcome_v2::policy::error_code_throw_as_system_error<rpc::result_context<raft::append_entries_reply>, std::__1::error_code, void>, std::__1::cond
itional<true, boost::outcome_v2::policy::exception_ptr_rethrow<rpc::result_context<raft::append_entries_reply>, std::__1::error_code, void>, boost::outcome_v2::policy::fail_to_compile_observ
ers>::type>::type>::type>> rpc::transport::send_typed_versioned<raft::append_entries_request, raft::append_entries_reply>(raft::append_entries_request, rpc::method_info, rpc::client_opts, rp
c::transport_version) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-05222bd6360c4da24-1/redpanda/redpanda/src/v/rpc/transport.h:345
{tmp/libexec/redpanda} 0x2864387: seastar::future<boost::outcome_v2::basic_result<rpc::client_context<raft::append_entries_reply>, std::__1::error_code, std::__1::conditional<false, boost::o
utcome_v2::policy::terminate, std::__1::conditional<true, boost::outcome_v2::policy::error_code_throw_as_system_error<rpc::client_context<raft::append_entries_reply>, std::__1::error_code, v
oid>, std::__1::conditional<true, boost::outcome_v2::policy::exception_ptr_rethrow<rpc::client_context<raft::append_entries_reply>, std::__1::error_code, void>, boost::outcome_v2::policy::fa
il_to_compile_observers>::type>::type>::type>> rpc::transport::send_typed<raft::append_entries_request, raft::append_entries_reply>(raft::append_entries_request, rpc::method_info, rpc::clien
t_opts) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-05222bd6360c4da24-1/redpanda/redpanda/src/v/rpc/transport.h:316
 (inlined by) raft::raftgen_client_protocol::append_entries(raft::append_entries_request&&, rpc::client_opts) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-05222bd6360c4da24-
1/redpanda/redpanda/vbuild/release/clang/src/v/raft/raftgen_service.h:321
 (inlined by) raft::rpc_client_protocol::append_entries(detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true>>, raft::append_entries_request&&, rpc:
:client_opts)::$_1::operator()(raft::raftgen_client_protocol) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-05222bd6360c4da24-1/redpanda/redpanda/src/v/raft/rpc_client_protoc
ol.cc:46
 (inlined by) auto rpc::connection_cache::with_node_client<raft::raftgen_client_protocol, raft::rpc_client_protocol::append_entries(detail::base_named_type<int, model::node_id_model_type, st
d::__1::integral_constant<bool, true>>, raft::append_entries_request&&, rpc::client_opts)::$_1>(detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true
>>, unsigned int, detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true>>, rpc::timeout_spec, raft::rpc_client_protocol::append_entries(detail::base_
named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true>>, raft::append_entries_request&&, rpc::client_opts)::$_1&&)::'lambda'(rpc::connection_cache&)::operator()(r
pc::connection_cache&)::'lambda'(boost::outcome_v2::basic_result<rpc::transport*, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<rpc::transport*, std::__1:
:error_code, void>>)::operator()(boost::outcome_v2::basic_result<rpc::transport*, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<rpc::transport*, std::__1:
:error_code, void>>) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-05222bd6360c4da24-1/redpanda/redpanda/src/v/rpc/connection_cache.h:102
{tmp/libexec/redpanda} 0x2863806: seastar::future<boost::outcome_v2::basic_result<raft::append_entries_reply, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_erro
r<raft::append_entries_reply, std::__1::error_code, void>>> seastar::futurize<seastar::future<boost::outcome_v2::basic_result<raft::append_entries_reply, std::__1::error_code, boost::outcome
_v2::policy::error_code_throw_as_system_error<raft::append_entries_reply, std::__1::error_code, void>>>>::invoke<auto rpc::connection_cache::with_node_client<raft::raftgen_client_protocol, r
aft::rpc_client_protocol::append_entries(detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true>>, raft::append_entries_request&&, rpc::client_opts)::
$_1>(detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true>>, unsigned int, detail::base_named_type<int, model::node_id_model_type, std::__1::integra
l_constant<bool, true>>, rpc::timeout_spec, raft::rpc_client_protocol::append_entries(detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true>>, raft::
append_entries_request&&, rpc::client_opts)::$_1&&)::'lambda'(rpc::connection_cache&)::operator()(rpc::connection_cache&)::'lambda'(boost::outcome_v2::basic_result<rpc::transport*, std::__1:
:error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<rpc::transport*, std::__1::error_code, void>>), boost::outcome_v2::basic_result<rpc::transport*, std::__1::error_code
, boost::outcome_v2::policy::error_code_throw_as_system_error<rpc::transport*, std::__1::error_code, void>>>(auto rpc::connection_cache::with_node_client<raft::raftgen_client_protocol, raft:
:rpc_client_protocol::append_entries(detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true>>, raft::append_entries_request&&, rpc::client_opts)::$_1>
(detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true>>, unsigned int, detail::base_named_type<int, model::node_id_model_type, std::__1::integral_co
nstant<bool, true>>, rpc::timeout_spec, raft::rpc_client_protocol::append_entries(detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true>>, raft::appe
nd_entries_request&&, rpc::client_opts)::$_1&&)::'lambda'(rpc::connection_cache&)::operator()(rpc::connection_cache&)::'lambda'(boost::outcome_v2::basic_result<rpc::transport*, std::__1::err
or_code, boost::outcome_v2::policy::error_code_throw_as_system_error<rpc::transport*, std::__1::error_code, void>>)&&, boost::outcome_v2::basic_result<rpc::transport*, std::__1::error_code, 
boost::outcome_v2::policy::error_code_throw_as_system_error<rpc::transport*, std::__1::error_code, void>>&&) at /vectorized/include/seastar/core/future.hh:1999
 (inlined by) raft::rpc_client_protocol::append_entries(detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true>>, raft::append_entries_request&&, rpc:
:client_opts)::$_1 seastar::future<boost::outcome_v2::basic_result<rpc::transport*, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<rpc::transport*, std::__
1::error_code, void>>>::then_impl<auto rpc::connection_cache::with_node_client<raft::raftgen_client_protocol, raft::rpc_client_protocol::append_entries(detail::base_named_type<int, model::no
de_id_model_type, std::__1::integral_constant<bool, true>>, raft::append_entries_request&&, rpc::client_opts)::$_1>(detail::base_named_type<int, model::node_id_model_type, std::__1::integral
_constant<bool, true>>, unsigned int, detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true>>, rpc::timeout_spec, raft::rpc_client_protocol::append_e
ntries(detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true>>, raft::append_entries_request&&, rpc::client_opts)::$_1&&)::'lambda'(rpc::connection_c
ache&)::operator()(rpc::connection_cache&)::'lambda'(boost::outcome_v2::basic_result<rpc::transport*, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<rpc::t
ransport*, std::__1::error_code, void>>), seastar::future<boost::outcome_v2::basic_result<raft::append_entries_reply, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_sys
tem_error<raft::append_entries_reply, std::__1::error_code, void>>>>(auto rpc::connection_cache::with_node_client<raft::raftgen_client_protocol, raft::rpc_client_protocol::append_entries(det
ail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true>>, raft::append_entries_request&&, rpc::client_opts)::$_1>(detail::base_named_type<int, model::nod
e_id_model_type, std::__1::integral_constant<bool, true>>, unsigned int, detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true>>, rpc::timeout_spec, 
raft::rpc_client_protocol::append_entries(detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true>>, raft::append_entries_request&&, rpc::client_opts):
:$_1&&)::'lambda'(rpc::connection_cache&)::operator()(rpc::connection_cache&)::'lambda'(boost::outcome_v2::basic_result<rpc::transport*, std::__1::error_code, boost::outcome_v2::policy::erro
r_code_throw_as_system_error<rpc::transport*, std::__1::error_code, void>>)&&) at /vectorized/include/seastar/core/future.hh:1470
....truncated...
dotnwat commented 1 year ago

:fire:

travisdowns commented 1 year ago

Brandom did you have any idea if we stall due the copy of a single large iobuf (this seems harder to fix, the "obvious" fix is to make a suspendable copy method but maybe this is not great), or that we process many iobufs without suspending (a bit nicer, means we can move the suspend up to a higher level)?

StephanDollberg commented 1 year ago

or that we process many iobufs without suspending

(not Brandon but)

I had a quick look the other day and I think this is in something like batch.for_each_record which is not async so I expect this to be fixable.

But I will confirm.

StephanDollberg commented 1 year ago

Yes as mentioned above the stall is in batch.for_each_record which is not async. There is the alternative async model::for_each_record. However, this is in some non-async adl method called from (async) raft code.

There is an outstanding PR which adds "async adl" support - https://github.com/redpanda-data/redpanda/pull/5129

@jcsp anything blocking that PR other than I assume general business?

travisdowns commented 1 year ago

However, this is in some non-async adl method

So a single invocation of that adl method seems to have taken more than 32 ms? Or is it itself called in a non-async loop?

Basically we don't want to push the async down "too far" to avoid impacting performance.

StephanDollberg commented 1 year ago

So a single invocation of that adl method seems to have taken more than 32 ms? Or is it itself called in a non-async loop?

The later basically. For each record in the batch the actual serialisation is performed without yielding.

jcsp commented 1 year ago

@jcsp anything blocking that PR other than I assume general business?

@StephanDollberg Nope, nothing blocking it: if you have time to rebase that on dev, please feel free to force push to my branch.

StephanDollberg commented 1 year ago

I revisited the three backtraces from above, they are in fact all a bit different:

1) I think this is quite early on the fetch path and we are copying buffers from another core or something. So this one might be harder to get rid off. 2) Also makes use of for_each_record but on another non-async code path 3) The one described earlier. John's PR is now merged so we can now replace use async_adl

ballard26 commented 1 year ago

Additional information that may help with this issue;

I have been copying the associated user-backtrace for these stalls. However, I haven't been copying the kernel-callstack(which I'll be doing from now on).

While I was running a CDT test I came upon another backtrace similar to the one above. And it had an associated kernel callstack that may shed light on why we're stalling for so long. Looks like we may be getting unlikely and hitting a page boundary during the copy?

Log message:

tests/results/2023-05-23--004/PartitionBalancerScaleTest/test_partition_balancer_with_many_partitions/type=many_partitions/4/RedpandaService-0-140125166615568/ip-172-31-5-203/redpanda.log:50
956:Reactor stalled for 34 ms on shard 3. Backtrace: 0x6ac78b0 0x6ac8e92 0x42abf 0x1923a4 0x6943218 0x314ee4a 0x312e2c1 0x312fa43 0x6aead1f 0x6aeeaa7 0x6b431f8 0x6a80a32 0x91016 0x1166cf    
tests/results/2023-05-23--004/PartitionBalancerScaleTest/test_partition_balancer_with_many_partitions/type=many_partitions/4/RedpandaService-0-140125166615568/ip-172-31-5-203/redpanda.log-50
957-kernel callstack: 0xffffffffffffff80 0xffffffff93178937 0xffffffff92d66737 0xffffffff92d67aab 0xffffffff92d6860b 0xffffffff92d8fa59 0xffffffff92db2dea 0xffffffff92d3e95b 0xffffffff92d3ea
20 0xffffffff92aa8c0a 0xffffffff93910171 0xffffffff93a00b77

Kernel callstack:

clear_page_erms+0x7
get_page_from_freelist+0x307
__alloc_pages+0x18b
__folio_alloc+0x1b
vma_alloc_folio+0x2b9
do_huge_pmd_anonymous_page+0xda
__handle_mm_fault+0x71b
xt_recseq+0xffffffff8f93ea

User backtrace:

void seastar::backtrace<seastar::backtrace_buffer::append_backtrace_oneline()::{lambda(seastar::frame)#1}>(seastar::backtrace_buffer::append_backtrace_oneline()::{lambda(seastar::f[108/1955]
) at /v/build/v_deps_build/seastar-prefix/src/seastar/include/seastar/util/backtrace.hh:60
 (inlined by) seastar::backtrace_buffer::append_backtrace_oneline() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:797
 (inlined by) seastar::print_with_backtrace(seastar::backtrace_buffer&, bool) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:816
seastar::internal::cpu_stall_detector::generate_trace() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:1386
 (inlined by) seastar::internal::cpu_stall_detector::maybe_report() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:1123
 (inlined by) seastar::internal::cpu_stall_detector::on_signal() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:1143
 (inlined by) seastar::reactor::block_notifier(int) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:1423
?? ??:0
?? ??:0
std::__1::enable_if<is_same<std::__1::remove_const<char const>::type, char>::value&&is_trivially_copy_assignable<char>::value, char*>::type std::__1::__copy<char const, char>(char const*, ch
ar const*, char*) at /vectorized/llvm/bin/../include/c++/v1/__algorithm/copy.h:56
 (inlined by) char* std::__1::copy<char const*, char*>(char const*, char const*, char*) at /vectorized/llvm/bin/../include/c++/v1/__algorithm/copy.h:69
 (inlined by) std::__1::enable_if<__is_cpp17_random_access_iterator<char const*>::value, char*>::type std::__1::copy_n<char const*, unsigned long, char*>(char const*, unsigned long, char*) a
t /vectorized/llvm/bin/../include/c++/v1/__algorithm/copy_n.h:61
 (inlined by) operator() at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-061ddaf65761b2b10-1/redpanda/redpanda/src/v/bytes/iobuf.cc:147
 (inlined by) unsigned long details::io_iterator_consumer::consume<iobuf_copy(details::io_iterator_consumer&, unsigned long)::$_4>(unsigned long, iobuf_copy(details::io_iterator_consumer&, u
nsigned long)::$_4&&) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-061ddaf65761b2b10-1/redpanda/redpanda/src/v/bytes/details/io_iterator_consumer.h:137
 (inlined by) iobuf_copy(details::io_iterator_consumer&, unsigned long) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-061ddaf65761b2b10-1/redpanda/redpanda/src/v/bytes/iobuf.
cc:145
 (inlined by) iobuf::copy() const at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-061ddaf65761b2b10-1/redpanda/redpanda/src/v/bytes/iobuf.cc:133
operator() at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-061ddaf65761b2b10-1/redpanda/redpanda/src/v/kafka/server/handlers/fetch.h:273
 (inlined by) decltype ((static_cast<seastar::internal::variant_visitor<kafka::read_result::release_data() &&::{lambda(std::__1::unique_ptr<iobuf, std::__1::default_delete<iobuf> >&)#1}, kaf
ka::read_result::release_data() &&::{lambda(seastar::foreign_ptr<std::__1::unique_ptr<iobuf, std::__1::default_delete<iobuf> > >&)#1}>>({parm#1}))(static_cast<seastar::foreign_ptr<std::__1::
unique_ptr<iobuf, std::__1::default_delete<iobuf> > >&>({parm#2}))) std::__1::__invoke_constexpr<seastar::internal::variant_visitor<kafka::read_result::release_data() &&::{lambda(std::__1::u
nique_ptr<iobuf, std::__1::default_delete<iobuf> >&)#1}, kafka::read_result::release_data() &&::{lambda(seastar::foreign_ptr<std::__1::unique_ptr<iobuf, std::__1::default_delete<iobuf> > >&)
#1}>, seastar::foreign_ptr<std::__1::unique_ptr<iobuf, std::__1::default_delete<iobuf> > >&>(seastar::internal::variant_visitor<kafka::read_result::release_data() &&::{lambda(std::__1::uniqu
e_ptr<iobuf, std::__1::default_delete<iobuf> >&)#1}, kafka::read_result::release_data() &&::{lambda(seastar::foreign_ptr<std::__1::unique_ptr<iobuf, std::__1::default_delete<iobuf> > >&)#1}>
&&, seastar::foreign_ptr<std::__1::unique_ptr<iobuf, std::__1::default_delete<iobuf> > >&) at /vectorized/llvm/bin/../include/c++/v1/type_traits:3648
 (inlined by) decltype(auto) std::__1::__variant_detail::__visitation::__variant::__value_visitor<seastar::internal::variant_visitor<kafka::read_result::release_data() &&::{lambda(std::__1::
unique_ptr<iobuf, std::__1::default_delete<iobuf> >&)#1}, kafka::read_result::release_data() &&::{lambda(seastar::foreign_ptr<std::__1::unique_ptr<iobuf, std::__1::default_delete<iobuf> > >&
)#1}> >::operator()<std::__1::__variant_detail::__alt<1ul, seastar::foreign_ptr<std::__1::unique_ptr<iobuf, std::__1::default_delete<iobuf> > > >&>(std::__1::__variant_detail::__alt<1ul, sea
star::foreign_ptr<std::__1::unique_ptr<iobuf, std::__1::default_delete<iobuf> > > >&) const at /vectorized/llvm/bin/../include/c++/v1/variant:656
 (inlined by) decltype ((static_cast<std::__1::__variant_detail::__visitation::__variant::__value_visitor<seastar::internal::variant_visitor<kafka::read_result::release_data() &&::{lambda(st
d::__1::unique_ptr<iobuf, std::__1::default_delete<iobuf> >&)#1}, kafka::read_result::release_data() &&::{lambda(seastar::foreign_ptr<std::__1::unique_ptr<iobuf, std::__1::default_delete<iob
uf> > >&)#1}> >>({parm#1}))(static_cast<std::__1::__variant_detail::__alt<1ul, seastar::foreign_ptr<std::__1::unique_ptr<iobuf, std::__1::default_delete<iobuf> > > >&>({parm#2}))) std::__1::
__invoke_constexpr<std::__1::__variant_detail::__visitation::__variant::__value_visitor<seastar::internal::variant_visitor<kafka::read_result::release_data() &&::{lambda(std::__1::unique_ptr
<iobuf, std::__1::default_delete<iobuf> >&)#1}, kafka::read_result::release_data() &&::{lambda(seastar::foreign_ptr<std::__1::unique_ptr<iobuf, std::__1::default_delete<iobuf> > >&)#1}> >, s
td::__1::__variant_detail::__alt<1ul, seastar::foreign_ptr<std::__1::unique_ptr<iobuf, std::__1::default_delete<iobuf> > > >&>(std::__1::__variant_detail::__visitation::__variant::__value_vi
sitor<seastar::internal::variant_visitor<kafka::read_result::release_data() &&::{lambda(std::__1::unique_ptr<iobuf, std::__1::default_delete<iobuf> >&)#1}, kafka::read_result::release_data()
 &&::{lambda(seastar::foreign_ptr<std::__1::unique_ptr<iobuf, std::__1::default_delete<iobuf> > >&)#1}> >&&, std::__1::__variant_detail::__alt<1ul, seastar::foreign_ptr<std::__1::unique_ptr<
iobuf, std::__1::default_delete<iobuf> > > >&) at /vectorized/llvm/bin/../include/c++/v1/type_traits:3648
 (inlined by) decltype(auto) std::__1::__variant_detail::__visitation::__base::__dispatcher<1ul>::__dispatch<std::__1::__variant_detail::__visitation::__variant::__value_visitor<seastar::int
ernal::variant_visitor<kafka::read_result::release_data() &&::{lambda(std::__1::unique_ptr<iobuf, std::__1::default_delete<iobuf> >&)#1}, kafka::read_result::release_data() &&::{lambda(seast
ar::foreign_ptr<std::__1::unique_ptr<iobuf, std::__1::default_delete<iobuf> > >&)#1}> >&&, std::__1::__variant_detail::__base<(std::__1::__variant_detail::_Trait)1, std::__1::unique_ptr<iobu
f, std::__1::default_delete<iobuf> >, seastar::foreign_ptr<std::__1::unique_ptr<iobuf, std::__1::default_delete<iobuf> > > >&>(std::__1::__variant_detail::__visitation::__variant::__value_vi
sitor<seastar::internal::variant_visitor<kafka::read_result::release_data() &&::{lambda(std::__1::unique_ptr<iobuf, std::__1::default_delete<iobuf> >&)#1}, kafka::read_result::release_data()
 &&::{lambda(seastar::foreign_ptr<std::__1::unique_ptr<iobuf, std::__1::default_delete<iobuf> > >&)#1}> >&&, std::__1::__variant_detail::__base<(std::__1::__variant_detail::_Trait)1, std::__
1::unique_ptr<iobuf, std::__1::default_delete<iobuf> >, seastar::foreign_ptr<std::__1::unique_ptr<iobuf, std::__1::default_delete<iobuf> > > >&) at /vectorized/llvm/bin/../include/c++/v1/var
iant:536
decltype(auto) std::__1::__variant_detail::__visitation::__base::__visit_alt<std::__1::__variant_detail::__visitation::__variant::__value_visitor<seastar::internal::variant_visitor<kafka::re
ad_result::release_data() &&::{lambda(std::__1::unique_ptr<iobuf, std::__1::default_delete<iobuf> >&)#1}, kafka::read_result::release_data() &&::{lambda(seastar::foreign_ptr<std::__1::unique
_ptr<iobuf, std::__1::default_delete<iobuf> > >&)#1}> >, std::__1::__variant_detail::__impl<std::__1::unique_ptr<iobuf, std::__1::default_delete<iobuf> >, seastar::foreign_ptr<std::__1::uniq
ue_ptr<iobuf, std::__1::default_delete<iobuf> > > >&>(std::__1::__variant_detail::__visitation::__variant::__value_visitor<seastar::internal::variant_visitor<kafka::read_result::release_data
() &&::{lambda(std::__1::unique_ptr<iobuf, std::__1::default_delete<iobuf> >&)#1}, kafka::read_result::release_data() &&::{lambda(seastar::foreign_ptr<std::__1::unique_ptr<iobuf, std::__1::d
efault_delete<iobuf> > >&)#1}> >&&, std::__1::__variant_detail::__impl<std::__1::unique_ptr<iobuf, std::__1::default_delete<iobuf> >, seastar::foreign_ptr<std::__1::unique_ptr<iobuf, std::__
1::default_delete<iobuf> > > >&) at /vectorized/llvm/bin/../include/c++/v1/variant:499
 (inlined by) decltype(auto) std::__1::__variant_detail::__visitation::__variant::__visit_alt<std::__1::__variant_detail::__visitation::__variant::__value_visitor<seastar::internal::variant_
visitor<kafka::read_result::release_data() &&::{lambda(std::__1::unique_ptr<iobuf, std::__1::default_delete<iobuf> >&)#1}, kafka::read_result::release_data() &&::{lambda(seastar::foreign_ptr
<std::__1::unique_ptr<iobuf, std::__1::default_delete<iobuf> > >&)#1}> >, std::__1::variant<std::__1::unique_ptr<iobuf, std::__1::default_delete<iobuf> >, seastar::foreign_ptr<std::__1::uniq
ue_ptr<iobuf, std::__1::default_delete<iobuf> > > >&>(std::__1::__variant_detail::__visitation::__variant::__value_visitor<seastar::internal::variant_visitor<kafka::read_result::release_data
() &&::{lambda(std::__1::unique_ptr<iobuf, std::__1::default_delete<iobuf> >&)#1}, kafka::read_result::release_data() &&::{lambda(seastar::foreign_ptr<std::__1::unique_ptr<iobuf, std::__1::d
efault_delete<iobuf> > >&)#1}> >&&, std::__1::variant<std::__1::unique_ptr<iobuf, std::__1::default_delete<iobuf> >, seastar::foreign_ptr<std::__1::unique_ptr<iobuf, std::__1::default_delete
<iobuf> > > >&) at /vectorized/llvm/bin/../include/c++/v1/variant:606
 (inlined by) decltype(auto) std::__1::__variant_detail::__visitation::__variant::__visit_value<seastar::internal::variant_visitor<kafka::read_result::release_data() &&::{lambda(std::__1::un
ique_ptr<iobuf, std::__1::default_delete<iobuf> >&)#1}, kafka::read_result::release_data() &&::{lambda(seastar::foreign_ptr<std::__1::unique_ptr<iobuf, std::__1::default_delete<iobuf> > >&)#
1}>, std::__1::variant<std::__1::unique_ptr<iobuf, std::__1::default_delete<iobuf> >, seastar::foreign_ptr<std::__1::unique_ptr<iobuf, std::__1::default_delete<iobuf> > > >&>(seastar::intern
al::variant_visitor<kafka::read_result::release_data() &&::{lambda(std::__1::unique_ptr<iobuf, std::__1::default_delete<iobuf> >&)#1}, kafka::read_result::release_data() &&::{lambda(seastar:
:foreign_ptr<std::__1::unique_ptr<iobuf, std::__1::default_delete<iobuf> > >&)#1}>&&, std::__1::variant<std::__1::unique_ptr<iobuf, std::__1::default_delete<iobuf> >, seastar::foreign_ptr<st
d::__1::unique_ptr<iobuf, std::__1::default_delete<iobuf> > > >&) at /vectorized/llvm/bin/../include/c++/v1/variant:625
 (inlined by) decltype(auto) std::__1::visit<seastar::internal::variant_visitor<kafka::read_result::release_data() &&::{lambda(std::__1::unique_ptr<iobuf, std::__1::default_delete<iobuf> >&)
#1}, kafka::read_result::release_data() &&::{lambda(seastar::foreign_ptr<std::__1::unique_ptr<iobuf, std::__1::default_delete<iobuf> > >&)#1}>, std::__1::variant<std::__1::unique_ptr<iobuf, 
std::__1::default_delete<iobuf> >, seastar::foreign_ptr<std::__1::unique_ptr<iobuf, std::__1::default_delete<iobuf> > > >&, void>(seastar::internal::variant_visitor<kafka::read_result::relea
se_data() &&::{lambda(std::__1::unique_ptr<iobuf, std::__1::default_delete<iobuf> >&)#1}, kafka::read_result::release_data() &&::{lambda(seastar::foreign_ptr<std::__1::unique_ptr<iobuf, std:
:__1::default_delete<iobuf> > >&)#1}>&&, std::__1::variant<std::__1::unique_ptr<iobuf, std::__1::default_delete<iobuf> >, seastar::foreign_ptr<std::__1::unique_ptr<iobuf, std::__1::default_d
elete<iobuf> > > >&) at /vectorized/llvm/bin/../include/c++/v1/variant:1698
 (inlined by) auto seastar::visit<std::__1::variant<std::__1::unique_ptr<iobuf, std::__1::default_delete<iobuf> >, seastar::foreign_ptr<std::__1::unique_ptr<iobuf, std::__1::default_delete<i
obuf> > > >&, kafka::read_result::release_data() &&::{lambda(std::__1::unique_ptr<iobuf, std::__1::default_delete<iobuf> >&)#1}, kafka::read_result::release_data() &&::{lambda(seastar::forei
gn_ptr<std::__1::unique_ptr<iobuf, std::__1::default_delete<iobuf> > >&)#1}>(std::__1::variant<std::__1::unique_ptr<iobuf, std::__1::default_delete<iobuf> >, seastar::foreign_ptr<std::__1::u
nique_ptr<iobuf, std::__1::default_delete<iobuf> > > >&, kafka::read_result::release_data() &&::{lambda(std::__1::unique_ptr<iobuf, std::__1::default_delete<iobuf> >&)#1}&&, kafka::read_resu
lt::release_data() &&::{lambda(seastar::foreign_ptr<std::__1::unique_ptr<iobuf, std::__1::default_delete<iobuf> > >&)#1}&&) at /vectorized/include/seastar/util/variant_utils.hh:71
 (inlined by) kafka::read_result::release_data() && at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-061ddaf65761b2b10-1/redpanda/redpanda/src/v/kafka/server/handlers/fetch.h:26
9
 (inlined by) kafka::fill_fetch_responses(kafka::op_context&, std::__1::vector<kafka::read_result, std::__1::allocator<kafka::read_result> >, std::__1::vector<kafka::op_context::response_pla
ceholder*, std::__1::allocator<kafka::op_context::response_placeholder*> >, std::__1::chrono::time_point<std::__1::chrono::steady_clock, std::__1::chrono::duration<long long, std::__1::ratio
<1l, 1000000000l> > >) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-061ddaf65761b2b10-1/redpanda/redpanda/src/v/kafka/server/handlers/fetch.cc:332
 (inlined by) operator() at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-061ddaf65761b2b10-1/redpanda/redpanda/src/v/kafka/server/handlers/fetch.cc:453
decltype ((static_cast<kafka::handle_shard_fetch(unsigned int, kafka::op_context&, kafka::shard_fetch)::$_3&>({parm#1}))(static_cast<std::__1::vector<kafka::read_result, std::__1::allocator<
kafka::read_result> >>({parm#2}))) std::__1::__invoke<kafka::handle_shard_fetch(unsigned int, kafka::op_context&, kafka::shard_fetch)::$_3&, std::__1::vector<kafka::read_result, std::__1::al
locator<kafka::read_result> > >(kafka::handle_shard_fetch(unsigned int, kafka::op_context&, kafka::shard_fetch)::$_3&, std::__1::vector<kafka::read_result, std::__1::allocator<kafka::read_re
sult> >&&) at /vectorized/llvm/bin/../include/c++/v1/type_traits:3640
 (inlined by) std::__1::invoke_result<kafka::handle_shard_fetch(unsigned int, kafka::op_context&, kafka::shard_fetch)::$_3&, std::__1::vector<kafka::read_result, std::__1::allocator<kafka::r
ead_result> > >::type std::__1::invoke<kafka::handle_shard_fetch(unsigned int, kafka::op_context&, kafka::shard_fetch)::$_3&, std::__1::vector<kafka::read_result, std::__1::allocator<kafka::
read_result> > >(kafka::handle_shard_fetch(unsigned int, kafka::op_context&, kafka::shard_fetch)::$_3&, std::__1::vector<kafka::read_result, std::__1::allocator<kafka::read_result> >&&) at /
vectorized/llvm/bin/../include/c++/v1/__functional/invoke.h:93
 (inlined by) auto seastar::internal::future_invoke<kafka::handle_shard_fetch(unsigned int, kafka::op_context&, kafka::shard_fetch)::$_3&, std::__1::vector<kafka::read_result, std::__1::allo
cator<kafka::read_result> > >(kafka::handle_shard_fetch(unsigned int, kafka::op_context&, kafka::shard_fetch)::$_3&, std::__1::vector<kafka::read_result, std::__1::allocator<kafka::read_resu
lt> >&&) at /vectorized/include/seastar/core/future.hh:1155
 (inlined by) operator() at /vectorized/include/seastar/core/future.hh:1455
 (inlined by) void seastar::futurize<void>::satisfy_with_result_of<seastar::future<std::__1::vector<kafka::read_result, std::__1::allocator<kafka::read_result> > >::then_impl_nrvo<kafka::han
dle_shard_fetch(unsigned int, kafka::op_context&, kafka::shard_fetch)::$_3, seastar::future<void> >(kafka::handle_shard_fetch(unsigned int, kafka::op_context&, kafka::shard_fetch)::$_3&&)::{
lambda(seastar::internal::promise_base_with_type<void>&&, kafka::handle_shard_fetch(unsigned int, kafka::op_context&, kafka::shard_fetch)::$_3&, seastar::future_state<std::__1::vector<kafka:
:read_result, std::__1::allocator<kafka::read_result> > >&&)#1}::operator()(seastar::internal::promise_base_with_type<void>&&, kafka::handle_shard_fetch(unsigned int, kafka::op_context&, kaf
ka::shard_fetch)::$_3&, seastar::future_state<std::__1::vector<kafka::read_result, std::__1::allocator<kafka::read_result> > >&&) const::{lambda()#1}>(seastar::internal::promise_base_with_ty
pe<void>&&, kafka::handle_shard_fetch(unsigned int, kafka::op_context&, kafka::shard_fetch)::$_3&&) at /vectorized/include/seastar/core/future.hh:1981                                        
 (inlined by) operator() at /vectorized/include/seastar/core/future.hh:1451                                                                                                                   
 (inlined by) seastar::continuation<seastar::internal::promise_base_with_type<void>, kafka::handle_shard_fetch(unsigned int, kafka::op_context&, kafka::shard_fetch)::$_3, seastar::future<std
::__1::vector<kafka::read_result, std::__1::allocator<kafka::read_result> > >::then_impl_nrvo<kafka::handle_shard_fetch(unsigned int, kafka::op_context&, kafka::shard_fetch)::$_3, seastar::f
uture<void> >(kafka::handle_shard_fetch(unsigned int, kafka::op_context&, kafka::shard_fetch)::$_3&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, kafka::handle_shard_fetch(un
signed int, kafka::op_context&, kafka::shard_fetch)::$_3&, seastar::future_state<std::__1::vector<kafka::read_result, std::__1::allocator<kafka::read_result> > >&&)#1}, std::__1::vector<kafk
a::read_result, std::__1::allocator<kafka::read_result> > >::run_and_dispose() at /vectorized/include/seastar/core/future.hh:742
seastar::reactor::run_tasks(seastar::reactor::task_queue&) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:2557
 (inlined by) seastar::reactor::run_some_tasks() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:3020
seastar::reactor::do_run() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:3189
operator() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:4399
 (inlined by) decltype ((static_cast<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_107&>({parm#1}))()) std::__1::__invoke<seastar::smp::configure(s
eastar::smp_options const&, seastar::reactor_options const&)::$_107&>(seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_107&) at /vectorized/llvm/bin/.
./include/c++/v1/type_traits:3640
 (inlined by) void std::__1::__invoke_void_return_wrapper<void, true>::__call<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_107&>(seastar::smp::con
figure(seastar::smp_options const&, seastar::reactor_options const&)::$_107&) at /vectorized/llvm/bin/../include/c++/v1/__functional/invoke.h:61
 (inlined by) std::__1::__function::__alloc_func<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_107, std::__1::allocator<seastar::smp::configure(sea
star::smp_options const&, seastar::reactor_options const&)::$_107>, void ()>::operator()() at /vectorized/llvm/bin/../include/c++/v1/__functional/function.h:180
 (inlined by) std::__1::__function::__func<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_107, std::__1::allocator<seastar::smp::configure(seastar::
smp_options const&, seastar::reactor_options const&)::$_107>, void ()>::operator()() at /vectorized/llvm/bin/../include/c++/v1/__functional/function.h:354
std::__1::__function::__value_func<void ()>::operator()() const at /vectorized/llvm/bin/../include/c++/v1/__functional/function.h:507
 (inlined by) std::__1::function<void ()>::operator()() const at /vectorized/llvm/bin/../include/c++/v1/__functional/function.h:1184
 (inlined by) seastar::posix_thread::start_routine(void*) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/posix.cc:73
?? ??:0
?? ??:0
ballard26 commented 1 year ago

Though I don't see how allocating and clearing a 2MB page could account for 34ms. There is some benchmarks here related to clearing a page https://lwn.net/Articles/873492/ which put it at 250us to clear a page. Not sure of the time to allocate one. If we were compacting existing pages to make the allocation we could see a few ms of latency (see https://github.com/torvalds/linux/commit/facdaa917c4d ). However, I don't see any evidence of that happening.

travisdowns commented 1 year ago

compacting existing pages to make the allocation we could see a few ms of latency (see https://github.com/torvalds/linux/commit/facdaa917c4d ). However, I don't see any evidence of that happening.

Yeah the big slowdowns are usually related to compaction in order to be able to allocate a contiguous 2MB page, but if you don't see evidence of that in the stack trace...

Other possibility is that we are just taking several of these faults and they add up to 34ms rather than one fault taking 34ms alone.

dotnwat commented 1 year ago

is this something that locking memory (i think seastar also goes ahead and touches every page too) and/or mbind (if these are numa nodes) would help with?

ballard26 commented 1 year ago

is this something that locking memory (i think seastar also goes ahead and touches every page too) and/or mbind (if these are numa nodes) would help with?

Touching every page would prevent one of these faults from occurring in the middle of a request early on.

jcsp commented 1 year ago

For searchability: KgoVerifierWithSiTestSmallSegments is now TieredStorageIoStressTest

github-actions[bot] commented 3 months ago

This issue hasn't seen activity in 3 months. If you want to keep it open, post a comment or remove the stale label – otherwise this will be closed in two weeks.

github-actions[bot] commented 2 months ago

This issue was closed due to lack of activity. Feel free to reopen if it's still relevant.