search

redpanda-data / redpanda

Redpanda is a streaming data platform for developers. Kafka API compatible. 10x faster. No ZooKeeper. No JVM!
https://redpanda.com
9.65k stars 589 forks source link

CI Failure (Crash post chunk hydration attempt) in `ShadowIndexingWhileBusyTest.test_create_or_delete_topics_while_busy` #11285

Closed VladLazar closed 1 year ago

VladLazar commented 1 year ago

https://buildkite.com/redpanda/vtools/builds/7944#01888fbc-d4c8-4453-b4df-12771fbef0f0

VladLazar commented 1 year ago

Backtrace decoded with ea8ae5505:

[Backtrace #0]                                                                                                                                                                                                                                                                  
seastar::metrics::label_instance::label_instance<char*>(seastar::basic_sstring<char, unsigned int, 15u, true> const&, char*) at /home/vlad/Work/dev/redpanda/vbuild/release/clang/v_deps_build/seastar-prefix/src/seastar/include/seastar/core/metrics.hh:173                   
 (inlined by) seastar::metrics::label_instance seastar::metrics::label::operator()<char*>(char*) const at /home/vlad/Work/dev/redpanda/vbuild/release/clang/v_deps_build/seastar-prefix/src/seastar/include/seastar/core/metrics.hh:237                                          (inlined by) seastar::smp_message_queue::start(unsigned int) at /home/vlad/Work/dev/redpanda/vbuild/release/clang/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:3622                                                                                   
std::__1::pair<std::__1::__wrap_iter<seastar::smp_message_queue::work_item* const*>, seastar::smp_message_queue::work_item**> std::__1::__uninitialized_copy[abi:v160004]<seastar::smp_message_queue::work_item*, std::__1::__wrap_iter<seastar::smp_message_queue::work_item* const*>, std::__1::__wrap_iter<seastar::smp_message_queue::work_item* const*>, seastar::smp_message_queue::work_item**, std::__1::__unreachable_sentinel>(std::__1::__wrap_iter<seastar::smp_message_queue::work_item* const*>, std::__1::__wrap_iter<seastar::smp_message_queue:
:work_item* const*>, seastar::smp_message_queue::work_item**, std::__1::__unreachable_sentinel) at /home/vlad/Work/dev/redpanda/vbuild/llvm/install/bin/../include/c++/v1/__memory/uninitialized_algorithms.h:64                                                                
 (inlined by) seastar::smp_message_queue::work_item** std::__1::uninitialized_copy[abi:v160004]<std::__1::__wrap_iter<seastar::smp_message_queue::work_item* const*>, seastar::smp_message_queue::work_item**>(std::__1::__wrap_iter<seastar::smp_message_queue::work_item* const*>, std::__1::__wrap_iter<seastar::smp_message_queue::work_item* const*>, seastar::smp_message_queue::work_item**) at /home/vlad/Work/dev/redpanda/vbuild/llvm/install/bin/../include/c++/v1/__memory/uninitialized_algorithms.h:81
 (inlined by) std::__1::__wrap_iter<seastar::smp_message_queue::work_item* const*> boost::lockfree::detail::ringbuffer_base<seastar::smp_message_queue::work_item*>::push<std::__1::__wrap_iter<seastar::smp_message_queue::work_item* const*>>(std::__1::__wrap_iter<seastar::smp_message_queue::work_item* const*>, std::__1::__wrap_iter<seastar::smp_message_queue::work_item* const*>, seastar::smp_message_queue::work_item**, unsigned long) at /home/vlad/Work/dev/redpanda/vbuild/release/clang/rp_deps_install/include/boost/lockfree/spsc_queue.hpp:149                                                                                                                                                                                                                                                                               (inlined by) std::__1::__wrap_iter<seastar::smp_message_queue::work_item* const*> boost::lockfree::detail::compile_time_sized_ringbuffer<seastar::smp_message_queue::work_item*, 128ul>::push<std::__1::__wrap_iter<seastar::smp_message_queue::work_item* const*>>(std::__1::_
_wrap_iter<seastar::smp_message_queue::work_item* const*>, std::__1::__wrap_iter<seastar::smp_message_queue::work_item* const*>) at /home/vlad/Work/dev/redpanda/vbuild/release/clang/rp_deps_install/include/boost/lockfree/spsc_queue.hpp:501                                  (inlined by) std::__1::__wrap_iter<seastar::smp_message_queue::work_item* const*> boost::lockfree::spsc_queue<seastar::smp_message_queue::work_item*, boost::lockfree::capacity<128ul>>::push<std::__1::__wrap_iter<seastar::smp_message_queue::work_item* const*>>(std::__1::__wrap_iter<seastar::smp_message_queue::work_item* const*>, std::__1::__wrap_iter<seastar::smp_message_queue::work_item* const*>) at /home/vlad/Work/dev/redpanda/vbuild/release/clang/rp_deps_install/include/boost/lockfree/spsc_queue.hpp:903                                  (inlined by) seastar::smp_message_queue::flush_response_batch() at /home/vlad/Work/dev/redpanda/vbuild/release/clang/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:3514                                                                                           (inlined by) seastar::smp_message_queue::respond(seastar::smp_message_queue::work_item*) at /home/vlad/Work/dev/redpanda/vbuild/release/clang/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:3506
 (inlined by) auto seastar::smp_message_queue::async_work_item<seastar::scheduling_group_key_create(seastar::scheduling_group_key_config)::$_1>::run_and_dispose()::'lambda'(auto)::operator()<seastar::future<void>>(auto) const at /home/vlad/Work/dev/redpanda/vbuild/release/clang/v_deps_build/seastar-prefix/src/seastar/include/seastar/core/smp.hh:246                                                                                                                                                                                                   (inlined by) seastar::futurize<auto>::type seastar::future<void>::then_wrapped_nrvo<void, seastar::smp_message_queue::async_work_item<seastar::scheduling_group_key_create(seastar::scheduling_group_key_config)::$_1>::run_and_dispose()::'lambda'(auto)>(seastar::smp_message_queue::async_work_item<seastar::scheduling_group_key_create(seastar::scheduling_group_key_config)::$_1>::run_and_dispose()::'lambda'(auto)&&)::'lambda'(seastar::internal::promise_base_with_type<void>&&, seastar::smp_message_queue::async_work_item<seastar::scheduling_grou
p_key_create(seastar::scheduling_group_key_config)::$_1>::run_and_dispose()::'lambda'(auto)&, seastar::future_state<seastar::internal::monostate>&&)::operator()(seastar::internal::promise_base_with_type<void>&&, seastar::smp_message_queue::async_work_item<seastar::scheduling_group_key_create(seastar::scheduling_group_key_config)::$_1>::run_and_dispose()::'lambda'(auto)&, seastar::future_state<seastar::internal::monostate>&&) const::'lambda'()::operator()() const at /home/vlad/Work/dev/redpanda/vbuild/release/clang/v_deps_build/seastar-prefix/src/seastar/include/seastar/core/future.hh:1536                                                                                                                                                                                                                              (inlined by) void seastar::futurize<void>::satisfy_with_result_of<seastar::futurize<auto>::type seastar::future<void>::then_wrapped_nrvo<void, seastar::smp_message_queue::async_work_item<seastar::scheduling_group_key_create(seastar::scheduling_group_key_config)::$_1>::ru
n_and_dispose()::'lambda'(auto)>(seastar::smp_message_queue::async_work_item<seastar::scheduling_group_key_create(seastar::scheduling_group_key_config)::$_1>::run_and_dispose()::'lambda'(auto)&&)::'lambda'(seastar::internal::promise_base_with_type<void>&&, seastar::smp_me
ssage_queue::async_work_item<seastar::scheduling_group_key_create(seastar::scheduling_group_key_config)::$_1>::run_and_dispose()::'lambda'(auto)&, seastar::future_state<seastar::internal::monostate>&&)::operator()(seastar::internal::promise_base_with_type<void>&&, seastar
::smp_message_queue::async_work_item<seastar::scheduling_group_key_create(seastar::scheduling_group_key_config)::$_1>::run_and_dispose()::'lambda'(auto)&, seastar::future_state<seastar::internal::monostate>&&) const::'lambda'()>(seastar::internal::promise_base_with_type<v
oid>&&, auto&&) at /home/vlad/Work/dev/redpanda/vbuild/release/clang/v_deps_build/seastar-prefix/src/seastar/include/seastar/core/future.hh:1981                                                                                                                                 (inlined by) seastar::futurize<auto>::type seastar::future<void>::then_wrapped_nrvo<void, seastar::smp_message_queue::async_work_item<seastar::scheduling_group_key_create(seastar::scheduling_group_key_config)::$_1>::run_and_dispose()::'lambda'(auto)>(seastar::smp_message
_queue::async_work_item<seastar::scheduling_group_key_create(seastar::scheduling_group_key_config)::$_1>::run_and_dispose()::'lambda'(auto)&&)::'lambda'(seastar::internal::promise_base_with_type<void>&&, seastar::smp_message_queue::async_work_item<seastar::scheduling_group_key_create(seastar::scheduling_group_key_config)::$_1>::run_and_dispose()::'lambda'(auto)&, seastar::future_state<seastar::internal::monostate>&&)::operator()(seastar::internal::promise_base_with_type<void>&&, seastar::smp_message_queue::async_work_item<seastar::schedul
ing_group_key_create(seastar::scheduling_group_key_config)::$_1>::run_and_dispose()::'lambda'(auto)&, seastar::future_state<seastar::internal::monostate>&&) const at /home/vlad/Work/dev/redpanda/vbuild/release/clang/v_deps_build/seastar-prefix/src/seastar/include/seastar/core/future.hh:1535                                                                                                                                                                                                                                                             
 (inlined by) seastar::continuation<seastar::internal::promise_base_with_type<void>, seastar::smp_message_queue::async_work_item<seastar::scheduling_group_key_create(seastar::scheduling_group_key_config)::$_1>::run_and_dispose()::'lambda'(auto), seastar::futurize<auto>::type seastar::future<void>::then_wrapped_nrvo<void, seastar::smp_message_queue::async_work_item<seastar::scheduling_group_key_create(seastar::scheduling_group_key_config)::$_1>::run_and_dispose()::'lambda'(auto)>(seastar::smp_message_queue::async_work_item<seastar::scheduling_group_key_create(seastar::scheduling_group_key_config)::$_1>::run_and_dispose()::'lambda'(auto)&&)::'lambda'(seastar::internal::promise_base_with_type<void>&&, seastar::smp_message_queue::async_work_item<seastar::scheduling_group_key_create(seastar::scheduling_group_k
ey_config)::$_1>::run_and_dispose()::'lambda'(auto)&, seastar::future_state<seastar::internal::monostate>&&), void>::run_and_dispose() at /home/vlad/Work/dev/redpanda/vbuild/release/clang/v_deps_build/seastar-prefix/src/seastar/include/seastar/core/future.hh:742
/opt/redpanda/lib/libc.so.6: ELF 64-bit LSB shared object, x86-64, version 1 (GNU/Linux), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=6e7b96dfb83f0bdcb6a410469b82f86415e5ada3, for GNU/Linux 3.2.0, stripped

__cxxabiv1::__vmi_class_type_info::search_below_dst(__cxxabiv1::__dynamic_cast_info*, void const*, int, bool) const at cxa_thread_atexit.cpp:0
__cxa_demangle at ??:0
__cxxabiv1::__free_with_fallback(void*) at cxa_thread_atexit.cpp:0
__cxxabiv1::__aligned_free_with_fallback(void*) at cxa_thread_atexit.cpp:0
/opt/redpanda/lib/libc++.so.1: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped

std::__1::__throw_future_error(std::__1::future_errc) at int128_builtins.cpp:0
seastar::internal::promise_base* std::__1::exchange[abi:v160004]<seastar::internal::promise_base*, std::nullptr_t>(seastar::internal::promise_base*&, std::nullptr_t&&) at /home/vlad/Work/dev/redpanda/vbuild/llvm/install/bin/../include/c++/v1/__utility/exchange.h:31
 (inlined by) seastar::internal::future_base::detach_promise() at /home/vlad/Work/dev/redpanda/vbuild/release/clang/rp_deps_install/include/seastar/core/future.hh:1115
 (inlined by) seastar::internal::future_base::clear() at /home/vlad/Work/dev/redpanda/vbuild/release/clang/rp_deps_install/include/seastar/core/future.hh:1104
 (inlined by) seastar::internal::future_base::~future_base() at /home/vlad/Work/dev/redpanda/vbuild/release/clang/rp_deps_install/include/seastar/core/future.hh:1109
 (inlined by) seastar::future<void>::~future() at /home/vlad/Work/dev/redpanda/vbuild/release/clang/rp_deps_install/include/seastar/core/scheduling.hh:38
 (inlined by) seastar::internal::awaiter<true, void>::~awaiter() at /home/vlad/Work/dev/redpanda/vbuild/release/clang/rp_deps_install/include/seastar/core/coroutine.hh:157
 (inlined by) cloud_storage_clients::client_pool::stop() at /home/vlad/Work/dev/redpanda/src/v/cloud_storage_clients/client_pool.cc:40
void seastar::logger::log<std::exception_ptr>(seastar::log_level, seastar::logger::format_info, std::exception_ptr&&) at /home/vlad/Work/dev/redpanda/vbuild/release/clang/v_deps_build/seastar-prefix/src/seastar/include/seastar/util/log.hh:233
 (inlined by) void seastar::logger::error<std::exception_ptr>(seastar::logger::format_info, std::exception_ptr&&) at /home/vlad/Work/dev/redpanda/vbuild/release/clang/v_deps_build/seastar-prefix/src/seastar/include/seastar/util/log.hh:328
 (inlined by) seastar::install_signal_handler_stack()::$_0::operator()() at /home/vlad/Work/dev/redpanda/vbuild/release/clang/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:847
 (inlined by) seastar::deferred_action<seastar::install_signal_handler_stack()::$_0>::~deferred_action() at /home/vlad/Work/dev/redpanda/vbuild/release/clang/v_deps_build/seastar-prefix/src/seastar/include/seastar/util/defer.hh:63
 (inlined by) seastar::reactor::do_run() at /home/vlad/Work/dev/redpanda/vbuild/release/clang/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:3261
std::__1::deque<seastar::smp_message_queue::work_item*, std::__1::allocator<seastar::smp_message_queue::work_item*>>::__size[abi:v160004]() const at /home/vlad/Work/dev/redpanda/vbuild/llvm/install/bin/../include/c++/v1/deque:681
 (inlined by) std::__1::deque<seastar::smp_message_queue::work_item*, std::__1::allocator<seastar::smp_message_queue::work_item*>>::size[abi:v160004]() const at /home/vlad/Work/dev/redpanda/vbuild/llvm/install/bin/../include/c++/v1/deque:678
 (inlined by) std::__1::deque<seastar::smp_message_queue::work_item*, std::__1::allocator<seastar::smp_message_queue::work_item*>>::end[abi:v160004]() const at /home/vlad/Work/dev/redpanda/vbuild/llvm/install/bin/../include/c++/v1/deque:645
 (inlined by) std::__1::deque<seastar::smp_message_queue::work_item*, std::__1::allocator<seastar::smp_message_queue::work_item*>>::cend[abi:v160004]() const at /home/vlad/Work/dev/redpanda/vbuild/llvm/install/bin/../include/c++/v1/deque:668
 (inlined by) seastar::smp_message_queue::move_pending() at /home/vlad/Work/dev/redpanda/vbuild/release/clang/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:3462
std::__1::vector<seastar::smp_message_queue::work_item*, std::__1::allocator<seastar::smp_message_queue::work_item*>>::size[abi:v160004]() const at /home/vlad/Work/dev/redpanda/vbuild/llvm/install/bin/../include/c++/v1/vector:546
 (inlined by) void std::__1::vector<seastar::smp_message_queue::work_item*, std::__1::allocator<seastar::smp_message_queue::work_item*>>::__push_back_slow_path<seastar::smp_message_queue::work_item* const&>(seastar::smp_message_queue::work_item* const&) at /home/vlad/Work/dev/redpanda/vbuild/llvm/install/bin/../include/c++/v1/vector:1538
 (inlined by) std::__1::vector<seastar::smp_message_queue::work_item*, std::__1::allocator<seastar::smp_message_queue::work_item*>>::push_back[abi:v160004](seastar::smp_message_queue::work_item* const&) at /home/vlad/Work/dev/redpanda/vbuild/llvm/install/bin/../include/c++/v1/vector:1556
 (inlined by) seastar::smp_message_queue::respond(seastar::smp_message_queue::work_item*) at /home/vlad/Work/dev/redpanda/vbuild/release/clang/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:3504
 (inlined by) auto seastar::smp_message_queue::async_work_item<seastar::destroy_scheduling_group(seastar::scheduling_group)::$_1>::run_and_dispose()::'lambda'(auto)::operator()<seastar::future<void>>(auto) const at /home/vlad/Work/dev/redpanda/vbuild/release/clang/v_deps_build/seastar-prefix/src/seastar/include/seastar/core/smp.hh:246
 (inlined by) seastar::futurize<auto>::type seastar::future<void>::then_wrapped_nrvo<void, seastar::smp_message_queue::async_work_item<seastar::destroy_scheduling_group(seastar::scheduling_group)::$_1>::run_and_dispose()::'lambda'(auto)>(seastar::smp_message_queue::async_work_item<seastar::destroy_scheduling_group(seastar::scheduling_group)::$_1>::run_and_dispose()::'lambda'(auto)&&)::'lambda'(seastar::internal::promise_base_with_type<void>&&, seastar::smp_message_queue::async_work_item<seastar::destroy_scheduling_group(seastar::scheduling_group)::$_1>::run_and_dispose()::'lambda'(auto)&, seastar::future_state<seastar::internal::monostate>&&)::operator()(seastar::internal::promise_base_with_type<void>&&, seastar::smp_message_queue::async_work_item<seastar::destroy_scheduling_group(seastar::scheduling_group)::$_1>::run_and_dispose()::'lambda'(auto)&, seastar::future_state<seastar::internal::monostate>&&) const::'lambda'()::operator()() const at /home/vlad/Work/dev/redpanda/vbuild/release/clang/v_deps_build/seastar-prefix/src/seastar/include/seastar/core/future.hh:1536
 (inlined by) void seastar::futurize<void>::satisfy_with_result_of<seastar::futurize<auto>::type seastar::future<void>::then_wrapped_nrvo<void, seastar::smp_message_queue::async_work_item<seastar::destroy_scheduling_group(seastar::scheduling_group)::$_1>::run_and_dispose()::'lambda'(auto)>(seastar::smp_message_queue::async_work_item<seastar::destroy_scheduling_group(seastar::scheduling_group)::$_1>::run_and_dispose()::'lambda'(auto)&&)::'lambda'(seastar::internal::promise_base_with_type<void>&&, seastar::smp_message_queue::async_work_item<seastar::destroy_scheduling_group(seastar::scheduling_group)::$_1>::run_and_dispose()::'lambda'(auto)&, seastar::future_state<seastar::internal::monostate>&&)::operator()(seastar::internal::promise_base_with_type<void>&&, seastar::smp_message_queue::async_work_item<seastar::destroy_scheduling_group(seastar::scheduling_group)::$_1>::run_and_dispose()::'lambda'(auto)&, seastar::future_state<seastar::internal::monostate>&&) const::'lambda'()>(seastar::internal::promise_base_with_type<void>&&, auto&&) at /home/vlad/Work/dev/redpanda/vbuild/release/clang/v_deps_build/seastar-prefix/src/seastar/include/seastar/core/future.hh:1981
 (inlined by) seastar::futurize<auto>::type seastar::future<void>::then_wrapped_nrvo<void, seastar::smp_message_queue::async_work_item<seastar::destroy_scheduling_group(seastar::scheduling_group)::$_1>::run_and_dispose()::'lambda'(auto)>(seastar::smp_message_queue::async_work_item<seastar::destroy_scheduling_group(seastar::scheduling_group)::$_1>::run_and_dispose()::'lambda'(auto)&&)::'lambda'(seastar::internal::promise_base_with_type<void>&&, seastar::smp_message_queue::async_work_item<seastar::destroy_scheduling_group(seastar::scheduling_group)::$_1>::run_and_dispose()::'lambda'(auto)&, seastar::future_state<seastar::internal::monostate>&&)::operator()(seastar::internal::promise_base_with_type<void>&&, seastar::smp_message_queue::async_work_item<seastar::destroy_scheduling_group(seastar::scheduling_group)::$_1>::run_and_dispose()::'lambda'(auto)&, seastar::future_state<seastar::internal::monostate>&&) const at /home/vlad/Work/dev/redpanda/vbuild/release/clang/v_deps_build/seastar-prefix/src/seastar/include/seastar/core/future.hh:1535
 (inlined by) seastar::continuation<seastar::internal::promise_base_with_type<void>, seastar::smp_message_queue::async_work_item<seastar::destroy_scheduling_group(seastar::scheduling_group)::$_1>::run_and_dispose()::'lambda'(auto), seastar::futurize<auto>::type seastar::future<void>::then_wrapped_nrvo<void, seastar::smp_message_queue::async_work_item<seastar::destroy_scheduling_group(seastar::scheduling_group)::$_1>::run_and_dispose()::'lambda'(auto)>(seastar::smp_message_queue::async_work_item<seastar::destroy_scheduling_group(seastar::scheduling_group)::$_1>::run_and_dispose()::'lambda'(auto)&&)::'lambda'(seastar::internal::promise_base_with_type<void>&&, seastar::smp_message_queue::async_work_item<seastar::destroy_scheduling_group(seastar::scheduling_group)::$_1>::run_and_dispose()::'lambda'(auto)&, seastar::future_state<seastar::internal::monostate>&&), void>::run_and_dispose() at /home/vlad/Work/dev/redpanda/vbuild/release/clang/v_deps_build/seastar-prefix/src/seastar/include/seastar/core/future.hh:742
std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>::__set_short_size[abi:v160004](unsigned long) at /home/vlad/Work/dev/redpanda/vbuild/llvm/install/bin/../include/c++/v1/string:1752
 (inlined by) std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>::__init(unsigned long, char) at /home/vlad/Work/dev/redpanda/vbuild/llvm/install/bin/../include/c++/v1/string:2206
 (inlined by) std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>::basic_string[abi:v160004](unsigned long, char) at /home/vlad/Work/dev/redpanda/vbuild/llvm/install/bin/../include/c++/v1/string:910
 (inlined by) std::__1::ostreambuf_iterator<char, std::__1::char_traits<char>> std::__1::__pad_and_output[abi:v160004]<char, std::__1::char_traits<char>>(std::__1::ostreambuf_iterator<char, std::__1::char_traits<char>>, char const*, char const*, char const*, std::__1::ios_base&, char) at /home/vlad/Work/dev/redpanda/vbuild/llvm/install/bin/../include/c++/v1/locale:1438
 (inlined by) std::__1::basic_ostream<char, std::__1::char_traits<char>>& std::__1::__put_character_sequence[abi:v160004]<char, std::__1::char_traits<char>>(std::__1::basic_ostream<char, std::__1::char_traits<char>>&, char const*, unsigned long) at /home/vlad/Work/dev/redpanda/vbuild/llvm/install/bin/../include/c++/v1/ostream:762
 (inlined by) std::__1::basic_ostream<char, std::__1::char_traits<char>>& std::__1::operator<<[abi:v160004]<std::__1::char_traits<char>>(std::__1::basic_ostream<char, std::__1::char_traits<char>>&, char const*) at /home/vlad/Work/dev/redpanda/vbuild/llvm/install/bin/../include/c++/v1/ostream:901
 (inlined by) seastar::prometheus::add_name(std::__1::basic_ostream<char, std::__1::char_traits<char>>&, seastar::basic_sstring<char, unsigned int, 15u, true> const&, std::__1::map<seastar::basic_sstring<char, unsigned int, 15u, true>, seastar::basic_sstring<char, unsigned int, 15u, true>, std::__1::less<seastar::basic_sstring<char, unsigned int, 15u, true>>, std::__1::allocator<std::__1::pair<seastar::basic_sstring<char, unsigned int, 15u, true> const, seastar::basic_sstring<char, unsigned int, 15u, true>>>> const&, seastar::prometheus::config const&) at /home/vlad/Work/dev/redpanda/vbuild/release/clang/v_deps_build/seastar-prefix/src/seastar/src/core/prometheus.cc:74
?? at ??:0
?? at ??:0
dotnwat commented 1 year ago

pinging @BenPope as he recently was looking into metrics and some other crash?

BenPope commented 1 year ago

This is the one I looked at recently: https://github.com/redpanda-data/redpanda/pull/11155

BenPope commented 1 year ago

At a glance, I would try this:

diff --git a/src/v/cloud_storage_clients/client_pool.cc b/src/v/cloud_storage_clients/client_pool.cc
index d998448508..b6a417884d 100644
--- a/src/v/cloud_storage_clients/client_pool.cc
+++ b/src/v/cloud_storage_clients/client_pool.cc
@@ -44,6 +44,7 @@ ss::future<> client_pool::stop() {
     }

     vlog(pool_log.info, "Stopped client pool");
+    _probe = nullptr;
 }

 void client_pool::shutdown_connections() {
VladLazar commented 1 year ago

Hmm. I am very confused by this back-trace. The logs don't line up with client_pool::stop() and I don't see why it would be called there. The crash happened after a failed chunk hydration and there's some potential UB around there (see https://github.com/redpanda-data/redpanda/pull/11292). Makes me think we threw a rubbish exception.

VladLazar commented 1 year ago

For the sake of clarity, we're not sure what caused the crash (info logs only on the test and the backtrace is inconclusive). Inspecting the code, a couple issues surfaced and got/are being fixed in: https://github.com/redpanda-data/redpanda/pull/11288 and https://github.com/redpanda-data/redpanda/pull/11292

VladLazar commented 1 year ago

Found another older instance here: https://buildkite.com/redpanda/vtools/builds/7910#0188804a-c980-4ca8-9797-6afa79dd52f5

VladLazar commented 1 year ago

A similar crash occurred on June 9th: https://buildkite.com/redpanda/vtools/builds/8014#01889f30-3f1a-4fb2-88df-d882e9e4d0ec

[Backtrace #0]                                                                                                                                                                                                                                                                  
std::__1::vector<seastar::metrics::label_instance, std::__1::allocator<seastar::metrics::label_instance>>::vector[abi:v160004](std::initializer_list<seastar::metrics::label_instance>) at /home/vlad/Work/dev/redpanda/vbuild/llvm/install/bin/../include/c++/v1/vector:672    
 (inlined by) seastar::smp_message_queue::start(unsigned int) at /home/vlad/Work/dev/redpanda/vbuild/release/clang/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:3626                                                                                             
seastar::future_state_base::any::available() const at /home/vlad/Work/dev/redpanda/vbuild/release/clang/v_deps_build/seastar-prefix/src/seastar/include/seastar/core/future.hh:441                                                                                              
 (inlined by) seastar::future_state_base::available() const at /home/vlad/Work/dev/redpanda/vbuild/release/clang/v_deps_build/seastar-prefix/src/seastar/include/seastar/core/future.hh:518                                                                                     
 (inlined by) seastar::future_state<seastar::internal::monostate>::take() && at /home/vlad/Work/dev/redpanda/vbuild/release/clang/v_deps_build/seastar-prefix/src/seastar/include/seastar/core/future.hh:648                                                                    
 (inlined by) seastar::future<void>::get() at /home/vlad/Work/dev/redpanda/vbuild/release/clang/v_deps_build/seastar-prefix/src/seastar/include/seastar/core/future.hh:1326                                                                                                     
 (inlined by) auto seastar::smp_message_queue::async_work_item<seastar::destroy_scheduling_group(seastar::scheduling_group)::$_1>::run_and_dispose()::'lambda'(auto)::operator()<seastar::future<void>>(auto) const at /home/vlad/Work/dev/redpanda/vbuild/release/clang/v_deps_
build/seastar-prefix/src/seastar/include/seastar/core/smp.hh:244                                                                                                                                                                                                                
 (inlined by) seastar::future<void> seastar::futurize<void>::invoke<seastar::smp_message_queue::async_work_item<seastar::destroy_scheduling_group(seastar::scheduling_group)::$_1>::run_and_dispose()::'lambda'(auto), seastar::future<void>>(auto&&, seastar::future<void>&&) a
t /home/vlad/Work/dev/redpanda/vbuild/release/clang/v_deps_build/seastar-prefix/src/seastar/include/seastar/core/future.hh:1996                                                                                                                                                 
 (inlined by) seastar::futurize<void>::type seastar::future<void>::then_wrapped_common<true, void, seastar::smp_message_queue::async_work_item<seastar::destroy_scheduling_group(seastar::scheduling_group)::$_1>::run_and_dispose()::'lambda'(auto)>(seastar::smp_message_queue
::async_work_item<seastar::destroy_scheduling_group(seastar::scheduling_group)::$_1>::run_and_dispose()::'lambda'(auto)&&) at /home/vlad/Work/dev/redpanda/vbuild/release/clang/v_deps_build/seastar-prefix/src/seastar/include/seastar/core/future.hh:1553                     
 (inlined by) seastar::futurize<void>::type seastar::future<void>::then_wrapped_maybe_erase<true, void, seastar::smp_message_queue::async_work_item<seastar::destroy_scheduling_group(seastar::scheduling_group)::$_1>::run_and_dispose()::'lambda'(auto)>(seastar::smp_message_
queue::async_work_item<seastar::destroy_scheduling_group(seastar::scheduling_group)::$_1>::run_and_dispose()::'lambda'(auto)&&) at /home/vlad/Work/dev/redpanda/vbuild/release/clang/v_deps_build/seastar-prefix/src/seastar/include/seastar/core/future.hh:1512                
 (inlined by) seastar::futurize<void>::type seastar::future<void>::then_wrapped<seastar::smp_message_queue::async_work_item<seastar::destroy_scheduling_group(seastar::scheduling_group)::$_1>::run_and_dispose()::'lambda'(auto), void>(auto&&) && at /home/vlad/Work/dev/redpa
nda/vbuild/release/clang/v_deps_build/seastar-prefix/src/seastar/include/seastar/core/future.hh:1503                                                                                                                                                                            
 (inlined by) seastar::smp_message_queue::async_work_item<seastar::destroy_scheduling_group(seastar::scheduling_group)::$_1>::run_and_dispose() at /home/vlad/Work/dev/redpanda/vbuild/release/clang/v_deps_build/seastar-prefix/src/seastar/include/seastar/core/smp.hh:240    
/opt/redpanda/lib/libc.so.6: ELF 64-bit LSB shared object, x86-64, version 1 (GNU/Linux), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=6e7b96dfb83f0bdcb6a410469b82f86415e5ada3, for GNU/Linux 3.2.0, stripped                                    

?? at ??:0                                                                                                                                                                                                                                                                      
/opt/redpanda/lib/libc++abi.so.1: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped                                                                                                                                                      

__cxxabiv1::__vmi_class_type_info::search_above_dst(__cxxabiv1::__dynamic_cast_info*, void const*, void const*, int, bool) const at cxa_thread_atexit.cpp:0                                                                                                                     
__cxxabiv1::__class_type_info::search_below_dst(__cxxabiv1::__dynamic_cast_info*, void const*, int, bool) const at cxa_thread_atexit.cpp:0                                                                                                                                      
_fini at cxa_thread_atexit.cpp:0                                                                                                                                                                                                                                                
_fini at cxa_thread_atexit.cpp:0                                                                                                                                                                                                                                                
/opt/redpanda/lib/libgcc_s.so.1: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=d14e4e492003556b25e1a4572b3c14bbecc0f48e, stripped                                                                                                   

_Unwind_RaiseException at ??:0                                                                                                                                                                                                                                                  
seastar::deleter::deleter(seastar::deleter&&) at /home/vlad/Work/dev/redpanda/vbuild/release/clang/v_deps_build/seastar-prefix/src/seastar/include/seastar/core/deleter.hh:61                                                                                                   
 (inlined by) seastar::deleter::operator=(seastar::deleter&&) at /home/vlad/Work/dev/redpanda/vbuild/release/clang/v_deps_build/seastar-prefix/src/seastar/include/seastar/core/deleter.hh:133                                                                                  
 (inlined by) seastar::temporary_buffer<char>::operator=(seastar::temporary_buffer<char>&&) at /home/vlad/Work/dev/redpanda/vbuild/release/clang/v_deps_build/seastar-prefix/src/seastar/include/seastar/core/temporary_buffer.hh:114                                           
 (inlined by) seastar::file_data_sink_impl::do_put(unsigned long, seastar::temporary_buffer<char>) at /home/vlad/Work/dev/redpanda/vbuild/release/clang/v_deps_build/seastar-prefix/src/seastar/src/core/fstream.cc:425                                                         
 (inlined by) seastar::file_data_sink_impl::put(seastar::temporary_buffer<char>) at /home/vlad/Work/dev/redpanda/vbuild/release/clang/v_deps_build/seastar-prefix/src/seastar/src/core/fstream.cc:373                                                                           
_fini at cxa_thread_atexit.cpp:0                                                                                                                                                                                                                                                
/opt/redpanda/lib/libc++.so.1: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped

std::__1::__throw_future_error(std::__1::future_errc) at int128_builtins.cpp:0                                                                                                                                                                                                  
seastar::memory::cpu_pages::drain_cross_cpu_freelist() at /home/vlad/Work/dev/redpanda/vbuild/release/clang/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:919                                                                                                      
cloud_storage::remote_partition::try_erase(seastar::abort_source&) (.resume) at /home/vlad/Work/dev/redpanda/src/v/cloud_storage/remote_partition.cc:0                                                                                                                          
seastar::basic_sstring<unsigned char, unsigned int, 31u, false>::~basic_sstring() at /home/vlad/Work/dev/redpanda/vbuild/release/clang/rp_deps_install/include/seastar/core/sstring.hh:0                                                                                        
 (inlined by) std::__1::pair<detail::base_named_type<unsigned int, kafka::tag_id_type, std::__1::integral_constant<bool, true>>, seastar::basic_sstring<unsigned char, unsigned int, 31u, false>>::~pair() at /home/vlad/Work/dev/redpanda/vbuild/llvm/install/bin/../include/c+
+/v1/__utility/pair.h:63                                                                                                                                                                                                                                                        
 (inlined by) void std::__1::__destroy_at[abi:v160004]<std::__1::pair<detail::base_named_type<unsigned int, kafka::tag_id_type, std::__1::integral_constant<bool, true>>, seastar::basic_sstring<unsigned char, unsigned int, 31u, false>>, 0>(std::__1::pair<detail::base_named
_type<unsigned int, kafka::tag_id_type, std::__1::integral_constant<bool, true>>, seastar::basic_sstring<unsigned char, unsigned int, 31u, false>>*) at /home/vlad/Work/dev/redpanda/vbuild/llvm/install/bin/../include/c++/v1/__memory/construct_at.h:66                       
 (inlined by) void std::__1::destroy_at[abi:v160004]<std::__1::pair<detail::base_named_type<unsigned int, kafka::tag_id_type, std::__1::integral_constant<bool, true>>, seastar::basic_sstring<unsigned char, unsigned int, 31u, false>>, 0>(std::__1::pair<detail::base_named_t
ype<unsigned int, kafka::tag_id_type, std::__1::integral_constant<bool, true>>, seastar::basic_sstring<unsigned char, unsigned int, 31u, false>>*) at /home/vlad/Work/dev/redpanda/vbuild/llvm/install/bin/../include/c++/v1/__memory/construct_at.h:101                        
 (inlined by) void std::__1::allocator_traits<std::__1::allocator<std::__1::pair<detail::base_named_type<unsigned int, kafka::tag_id_type, std::__1::integral_constant<bool, true>> const, seastar::basic_sstring<unsigned char, unsigned int, 31u, false>>>>::destroy[abi:v1600
04]<std::__1::pair<detail::base_named_type<unsigned int, kafka::tag_id_type, std::__1::integral_constant<bool, true>>, seastar::basic_sstring<unsigned char, unsigned int, 31u, false>>, void, void>(std::__1::allocator<std::__1::pair<detail::base_named_type<unsigned int, ka
fka::tag_id_type, std::__1::integral_constant<bool, true>> const, seastar::basic_sstring<unsigned char, unsigned int, 31u, false>>>&, std::__1::pair<detail::base_named_type<unsigned int, kafka::tag_id_type, std::__1::integral_constant<bool, true>>, seastar::basic_sstring<
unsigned char, unsigned int, 31u, false>>*) at /home/vlad/Work/dev/redpanda/vbuild/llvm/install/bin/../include/c++/v1/__memory/allocator_traits.h:323                                                                                                                           
 (inlined by) void absl::lts_20220623::container_internal::map_slot_policy<detail::base_named_type<unsigned int, kafka::tag_id_type, std::__1::integral_constant<bool, true>>, seastar::basic_sstring<unsigned char, unsigned int, 31u, false>>::destroy<std::__1::allocator<std
::__1::pair<detail::base_named_type<unsigned int, kafka::tag_id_type, std::__1::integral_constant<bool, true>> const, seastar::basic_sstring<unsigned char, unsigned int, 31u, false>>>>(std::__1::allocator<std::__1::pair<detail::base_named_type<unsigned int, kafka::tag_id_
type, std::__1::integral_constant<bool, true>> const, seastar::basic_sstring<unsigned char, unsigned int, 31u, false>>>*, absl::lts_20220623::container_internal::map_slot_type<detail::base_named_type<unsigned int, kafka::tag_id_type, std::__1::integral_constant<bool, true
>>, seastar::basic_sstring<unsigned char, unsigned int, 31u, false>>*) at /home/vlad/Work/dev/redpanda/vbuild/release/clang/rp_deps_install/include/absl/container/internal/container_memory.h:417                                                                              
 (inlined by) absl::lts_20220623::container_internal::common_params<detail::base_named_type<unsigned int, kafka::tag_id_type, std::__1::integral_constant<bool, true>>, std::__1::less<detail::base_named_type<unsigned int, kafka::tag_id_type, std::__1::integral_constant<boo
l, true>>>, std::__1::allocator<std::__1::pair<detail::base_named_type<unsigned int, kafka::tag_id_type, std::__1::integral_constant<bool, true>> const, seastar::basic_sstring<unsigned char, unsigned int, 31u, false>>>, 256, false, true, absl::lts_20220623::container_inte
rnal::map_slot_policy<detail::base_named_type<unsigned int, kafka::tag_id_type, std::__1::integral_constant<bool, true>>, seastar::basic_sstring<unsigned char, unsigned int, 31u, false>>>::destroy(std::__1::allocator<std::__1::pair<detail::base_named_type<unsigned int, ka
fka::tag_id_type, std::__1::integral_constant<bool, true>> const, seastar::basic_sstring<unsigned char, unsigned int, 31u, false>>>*, absl::lts_20220623::container_internal::map_slot_type<detail::base_named_type<unsigned int, kafka::tag_id_type, std::__1::integral_constan
t<bool, true>>, seastar::basic_sstring<unsigned char, unsigned int, 31u, false>>*) at /home/vlad/Work/dev/redpanda/vbuild/release/clang/rp_deps_install/include/absl/container/internal/btree.h:460                                                                             
 (inlined by) absl::lts_20220623::container_internal::btree_node<absl::lts_20220623::container_internal::map_params<detail::base_named_type<unsigned int, kafka::tag_id_type, std::__1::integral_constant<bool, true>>, seastar::basic_sstring<unsigned char, unsigned int, 31u,
 false>, std::__1::less<detail::base_named_type<unsigned int, kafka::tag_id_type, std::__1::integral_constant<bool, true>>>, std::__1::allocator<std::__1::pair<detail::base_named_type<unsigned int, kafka::tag_id_type, std::__1::integral_constant<bool, true>> const, seasta
r::basic_sstring<unsigned char, unsigned int, 31u, false>>>, 256, false>>::value_destroy_n(unsigned char, unsigned char, std::__1::allocator<std::__1::pair<detail::base_named_type<unsigned int, kafka::tag_id_type, std::__1::integral_constant<bool, true>> const, seastar::b
asic_sstring<unsigned char, unsigned int, 31u, false>>>*) at /home/vlad/Work/dev/redpanda/vbuild/release/clang/rp_deps_install/include/absl/container/internal/btree.h:983                                                                                                      
 (inlined by) absl::lts_20220623::container_internal::btree_node<absl::lts_20220623::container_internal::map_params<detail::base_named_type<unsigned int, kafka::tag_id_type, std::__1::integral_constant<bool, true>>, seastar::basic_sstring<unsigned char, unsigned int, 31u,
 false>, std::__1::less<detail::base_named_type<unsigned int, kafka::tag_id_type, std::__1::integral_constant<bool, true>>>, std::__1::allocator<std::__1::pair<detail::base_named_type<unsigned int, kafka::tag_id_type, std::__1::integral_constant<bool, true>> const, seasta
r::basic_sstring<unsigned char, unsigned int, 31u, false>>>, 256, false>>::clear_and_delete(absl::lts_20220623::container_internal::btree_node<absl::lts_20220623::container_internal::map_params<detail::base_named_type<unsigned int, kafka::tag_id_type, std::__1::integral_c
onstant<bool, true>>, seastar::basic_sstring<unsigned char, unsigned int, 31u, false>, std::__1::less<detail::base_named_type<unsigned int, kafka::tag_id_type, std::__1::integral_constant<bool, true>>>, std::__1::allocator<std::__1::pair<detail::base_named_type<unsigned i
nt, kafka::tag_id_type, std::__1::integral_constant<bool, true>> const, seastar::basic_sstring<unsigned char, unsigned int, 31u, false>>>, 256, false>>*, std::__1::allocator<std::__1::pair<detail::base_named_type<unsigned int, kafka::tag_id_type, std::__1::integral_consta
nt<bool, true>> const, seastar::basic_sstring<unsigned char, unsigned int, 31u, false>>>*) at /home/vlad/Work/dev/redpanda/vbuild/release/clang/rp_deps_install/include/absl/container/internal/btree.h:1962                                                                    
 (inlined by) absl::lts_20220623::container_internal::btree<absl::lts_20220623::container_internal::map_params<detail::base_named_type<unsigned int, kafka::tag_id_type, std::__1::integral_constant<bool, true>>, seastar::basic_sstring<unsigned char, unsigned int, 31u, fals
e>, std::__1::less<detail::base_named_type<unsigned int, kafka::tag_id_type, std::__1::integral_constant<bool, true>>>, std::__1::allocator<std::__1::pair<detail::base_named_type<unsigned int, kafka::tag_id_type, std::__1::integral_constant<bool, true>> const, seastar::ba
sic_sstring<unsigned char, unsigned int, 31u, false>>>, 256, false>>::clear() at /home/vlad/Work/dev/redpanda/vbuild/release/clang/rp_deps_install/include/absl/container/internal/btree.h:2425                                                                                 
 (inlined by) absl::lts_20220623::container_internal::btree<absl::lts_20220623::container_internal::map_params<detail::base_named_type<unsigned int, kafka::tag_id_type, std::__1::integral_constant<bool, true>>, seastar::basic_sstring<unsigned char, unsigned int, 31u, fals
e>, std::__1::less<detail::base_named_type<unsigned int, kafka::tag_id_type, std::__1::integral_constant<bool, true>>>, std::__1::allocator<std::__1::pair<detail::base_named_type<unsigned int, kafka::tag_id_type, std::__1::integral_constant<bool, true>> const, seastar::ba
sic_sstring<unsigned char, unsigned int, 31u, false>>>, 256, false>>::operator=(absl::lts_20220623::container_internal::btree<absl::lts_20220623::container_internal::map_params<detail::base_named_type<unsigned int, kafka::tag_id_type, std::__1::integral_constant<bool, tru
e>>, seastar::basic_sstring<unsigned char, unsigned int, 31u, false>, std::__1::less<detail::base_named_type<unsigned int, kafka::tag_id_type, std::__1::integral_constant<bool, true>>>, std::__1::allocator<std::__1::pair<detail::base_named_type<unsigned int, kafka::tag_id
_type, std::__1::integral_constant<bool, true>> const, seastar::basic_sstring<unsigned char, unsigned int, 31u, false>>>, 256, false>>&&) at /home/vlad/Work/dev/redpanda/vbuild/release/clang/rp_deps_install/include/absl/container/internal/btree.h:2266
(inlined by) absl::lts_20220623::container_internal::btree<absl::lts_20220623::container_internal::map_params<detail::base_named_type<unsigned int, kafka::tag_id_type, std::__1::integral_constant<bool, true>>, seastar::basic_sstring<unsigned char, unsigned int, 31u, fals
e>, std::__1::less<detail::base_named_type<unsigned int, kafka::tag_id_type, std::__1::integral_constant<bool, true>>>, std::__1::allocator<std::__1::pair<detail::base_named_type<unsigned int, kafka::tag_id_type, std::__1::integral_constant<bool, true>> const, seastar::ba
sic_sstring<unsigned char, unsigned int, 31u, false>>>, 256, false>>::operator=(absl::lts_20220623::container_internal::btree<absl::lts_20220623::container_internal::map_params<detail::base_named_type<unsigned int, kafka::tag_id_type, std::__1::integral_constant<bool, tru
e>>, seastar::basic_sstring<unsigned char, unsigned int, 31u, false>, std::__1::less<detail::base_named_type<unsigned int, kafka::tag_id_type, std::__1::integral_constant<bool, true>>>, std::__1::allocator<std::__1::pair<detail::base_named_type<unsigned int, kafka::tag_id
_type, std::__1::integral_constant<bool, true>> const, seastar::basic_sstring<unsigned char, unsigned int, 31u, false>>>, 256, false>>&&) at /home/vlad/Work/dev/redpanda/vbuild/release/clang/rp_deps_install/include/absl/container/internal/btree.h:2266
 (inlined by) absl::lts_20220623::container_internal::btree_container<absl::lts_20220623::container_internal::btree<absl::lts_20220623::container_internal::map_params<detail::base_named_type<unsigned int, kafka::tag_id_type, std::__1::integral_constant<bool, true>>, seast
ar::basic_sstring<unsigned char, unsigned int, 31u, false>, std::__1::less<detail::base_named_type<unsigned int, kafka::tag_id_type, std::__1::integral_constant<bool, true>>>, std::__1::allocator<std::__1::pair<detail::base_named_type<unsigned int, kafka::tag_id_type, std
::__1::integral_constant<bool, true>> const, seastar::basic_sstring<unsigned char, unsigned int, 31u, false>>>, 256, false>>>::operator=(absl::lts_20220623::container_internal::btree_container<absl::lts_20220623::container_internal::btree<absl::lts_20220623::container_int
ernal::map_params<detail::base_named_type<unsigned int, kafka::tag_id_type, std::__1::integral_constant<bool, true>>, seastar::basic_sstring<unsigned char, unsigned int, 31u, false>, std::__1::less<detail::base_named_type<unsigned int, kafka::tag_id_type, std::__1::integr
al_constant<bool, true>>>, std::__1::allocator<std::__1::pair<detail::base_named_type<unsigned int, kafka::tag_id_type, std::__1::integral_constant<bool, true>> const, seastar::basic_sstring<unsigned char, unsigned int, 31u, false>>>, 256, false>>>&&) at /home/vlad/Work/d
ev/redpanda/vbuild/release/clang/rp_deps_install/include/absl/container/internal/btree_container.h:90                                                                                                                                                                           
 (inlined by) absl::lts_20220623::container_internal::btree_set_container<absl::lts_20220623::container_internal::btree<absl::lts_20220623::container_internal::map_params<detail::base_named_type<unsigned int, kafka::tag_id_type, std::__1::integral_constant<bool, true>>, s
eastar::basic_sstring<unsigned char, unsigned int, 31u, false>, std::__1::less<detail::base_named_type<unsigned int, kafka::tag_id_type, std::__1::integral_constant<bool, true>>>, std::__1::allocator<std::__1::pair<detail::base_named_type<unsigned int, kafka::tag_id_type,
 std::__1::integral_constant<bool, true>> const, seastar::basic_sstring<unsigned char, unsigned int, 31u, false>>>, 256, false>>>::operator=(absl::lts_20220623::container_internal::btree_set_container<absl::lts_20220623::container_internal::btree<absl::lts_20220623::conta
iner_internal::map_params<detail::base_named_type<unsigned int, kafka::tag_id_type, std::__1::integral_constant<bool, true>>, seastar::basic_sstring<unsigned char, unsigned int, 31u, false>, std::__1::less<detail::base_named_type<unsigned int, kafka::tag_id_type, std::__1
::integral_constant<bool, true>>>, std::__1::allocator<std::__1::pair<detail::base_named_type<unsigned int, kafka::tag_id_type, std::__1::integral_constant<bool, true>> const, seastar::basic_sstring<unsigned char, unsigned int, 31u, false>>>, 256, false>>>&&) at /home/vla
d/Work/dev/redpanda/vbuild/release/clang/rp_deps_install/include/absl/container/internal/btree_container.h:238                                                                                                                                                                  
 (inlined by) absl::lts_20220623::container_internal::btree_map_container<absl::lts_20220623::container_internal::btree<absl::lts_20220623::container_internal::map_params<detail::base_named_type<unsigned int, kafka::tag_id_type, std::__1::integral_constant<bool, true>>, s
eastar::basic_sstring<unsigned char, unsigned int, 31u, false>, std::__1::less<detail::base_named_type<unsigned int, kafka::tag_id_type, std::__1::integral_constant<bool, true>>>, std::__1::allocator<std::__1::pair<detail::base_named_type<unsigned int, kafka::tag_id_type,
 std::__1::integral_constant<bool, true>> const, seastar::basic_sstring<unsigned char, unsigned int, 31u, false>>>, 256, false>>>::operator=(absl::lts_20220623::container_internal::btree_map_container<absl::lts_20220623::container_internal::btree<absl::lts_20220623::conta
iner_internal::map_params<detail::base_named_type<unsigned int, kafka::tag_id_type, std::__1::integral_constant<bool, true>>, seastar::basic_sstring<unsigned char, unsigned int, 31u, false>, std::__1::less<detail::base_named_type<unsigned int, kafka::tag_id_type, std::__1
::integral_constant<bool, true>>>, std::__1::allocator<std::__1::pair<detail::base_named_type<unsigned int, kafka::tag_id_type, std::__1::integral_constant<bool, true>> const, seastar::basic_sstring<unsigned char, unsigned int, 31u, false>>>, 256, false>>>&&) at /home/vla
d/Work/dev/redpanda/vbuild/release/clang/rp_deps_install/include/absl/container/internal/btree_container.h:397                                                                                                                                                                  
 (inlined by) absl::lts_20220623::btree_map<detail::base_named_type<unsigned int, kafka::tag_id_type, std::__1::integral_constant<bool, true>>, seastar::basic_sstring<unsigned char, unsigned int, 31u, false>, std::__1::less<detail::base_named_type<unsigned int, kafka::tag
_id_type, std::__1::integral_constant<bool, true>>>, std::__1::allocator<std::__1::pair<detail::base_named_type<unsigned int, kafka::tag_id_type, std::__1::integral_constant<bool, true>> const, seastar::basic_sstring<unsigned char, unsigned int, 31u, false>>>>::operator=(
absl::lts_20220623::btree_map<detail::base_named_type<unsigned int, kafka::tag_id_type, std::__1::integral_constant<bool, true>>, seastar::basic_sstring<unsigned char, unsigned int, 31u, false>, std::__1::less<detail::base_named_type<unsigned int, kafka::tag_id_type, std:
:__1::integral_constant<bool, true>>>, std::__1::allocator<std::__1::pair<detail::base_named_type<unsigned int, kafka::tag_id_type, std::__1::integral_constant<bool, true>> const, seastar::basic_sstring<unsigned char, unsigned int, 31u, false>>>>&&) at /home/vlad/Work/dev
/redpanda/vbuild/release/clang/rp_deps_install/include/absl/container/btree_map.h:84                                                                                                                                                                                            
 (inlined by) detail::base_named_type<absl::lts_20220623::btree_map<detail::base_named_type<unsigned int, kafka::tag_id_type, std::__1::integral_constant<bool, true>>, seastar::basic_sstring<unsigned char, unsigned int, 31u, false>, std::__1::less<detail::base_named_type<
unsigned int, kafka::tag_id_type, std::__1::integral_constant<bool, true>>>, std::__1::allocator<std::__1::pair<detail::base_named_type<unsigned int, kafka::tag_id_type, std::__1::integral_constant<bool, true>> const, seastar::basic_sstring<unsigned char, unsigned int, 31
u, false>>>>, kafka::tagged_fields_type, std::__1::integral_constant<bool, false>>::operator=(detail::base_named_type<absl::lts_20220623::btree_map<detail::base_named_type<unsigned int, kafka::tag_id_type, std::__1::integral_constant<bool, true>>, seastar::basic_sstring<u
nsigned char, unsigned int, 31u, false>, std::__1::less<detail::base_named_type<unsigned int, kafka::tag_id_type, std::__1::integral_constant<bool, true>>>, std::__1::allocator<std::__1::pair<detail::base_named_type<unsigned int, kafka::tag_id_type, std::__1::integral_con
stant<bool, true>> const, seastar::basic_sstring<unsigned char, unsigned int, 31u, false>>>>, kafka::tagged_fields_type, std::__1::integral_constant<bool, false>>&&) at /home/vlad/Work/dev/redpanda/src/v/utils/named_type.h:141
 (inlined by) kafka::join_group_request_data::operator=(kafka::join_group_request_data&&) at /home/vlad/Work/dev/redpanda/vbuild/release/clang/src/v/kafka/protocol/schemata/join_group_request.h:62                                                                            
 (inlined by) kafka::client::consumer::join()::$_1::operator()() const at /home/vlad/Work/dev/redpanda/src/v/kafka/client/consumer.cc:161                                                                                                                                       
 (inlined by) seastar::future<std::__1::invoke_result<kafka::client::consumer::join()::$_1>::type::api_type::response_type> kafka::client::consumer::req_res<kafka::client::consumer::join()::$_1>(kafka::client::consumer::join()::$_1)::'lambda'()::operator()() at /home/vlad
/Work/dev/redpanda/src/v/kafka/client/consumer.h:111                                                                                                                                                                                                                            
 (inlined by) seastar::future<kafka::join_group_response> seastar::futurize<seastar::future<kafka::join_group_response>>::invoke<seastar::future<std::__1::invoke_result<kafka::client::consumer::join()::$_1>::type::api_type::response_type> kafka::client::consumer::req_res<
kafka::client::consumer::join()::$_1>(kafka::client::consumer::join()::$_1)::'lambda'()>(kafka::client::consumer::join()::$_1&&) at /home/vlad/Work/dev/redpanda/vbuild/release/clang/rp_deps_install/include/seastar/core/future.hh:1999                                       
 (inlined by) auto seastar::futurize_invoke<seastar::future<std::__1::invoke_result<kafka::client::consumer::join()::$_1>::type::api_type::response_type> kafka::client::consumer::req_res<kafka::client::consumer::join()::$_1>(kafka::client::consumer::join()::$_1)::'lambda'
()>(kafka::client::consumer::join()::$_1&&) at /home/vlad/Work/dev/redpanda/vbuild/release/clang/rp_deps_install/include/seastar/core/future.hh:2030                                                                                                                            
 (inlined by) auto seastar::internal::invoke_func_with_gate<seastar::future<std::__1::invoke_result<kafka::client::consumer::join()::$_1>::type::api_type::response_type> kafka::client::consumer::req_res<kafka::client::consumer::join()::$_1>(kafka::client::consumer::join()
::$_1)::'lambda'()>(seastar::gate&, kafka::client::consumer::join()::$_1&&) at /home/vlad/Work/dev/redpanda/vbuild/release/clang/rp_deps_install/include/seastar/core/gate.hh:273                                                                                               
 (inlined by) auto seastar::try_with_gate<seastar::future<std::__1::invoke_result<kafka::client::consumer::join()::$_1>::type::api_type::response_type> kafka::client::consumer::req_res<kafka::client::consumer::join()::$_1>(kafka::client::consumer::join()::$_1)::'lambda'()
>(seastar::gate&, kafka::client::consumer::join()::$_1&&) at /home/vlad/Work/dev/redpanda/vbuild/release/clang/rp_deps_install/include/seastar/core/gate.hh:313
(inlined by) seastar::future<std::__1::invoke_result<kafka::client::consumer::join()::$_1>::type::api_type::response_type> kafka::client::consumer::req_res<kafka::client::consumer::join()::$_1>(kafka::client::consumer::join()::$_1) at /home/vlad/Work/(1 results) [62/208]
afka/client/consumer.h:110                                                                                                                                                                                                                                                      
boost::system::error_code::error_code(int, boost::system::error_category const&) at /home/vlad/Work/dev/redpanda/vbuild/release/clang/rp_deps_install/include/boost/system/detail/error_code.hpp:129                                                                            
 (inlined by) boost::thread_exception::thread_exception(int, char const*) at /home/vlad/Work/dev/redpanda/vbuild/release/clang/rp_deps_install/include/boost/thread/exceptions.hpp:51                                                                                           
 (inlined by) boost::lock_error::lock_error(int, char const*) at /home/vlad/Work/dev/redpanda/vbuild/release/clang/rp_deps_install/include/boost/thread/exceptions.hpp:108                                                                                                      
 (inlined by) boost::mutex::lock() at /home/vlad/Work/dev/redpanda/vbuild/release/clang/rp_deps_install/include/boost/thread/pthread/mutex.hpp:64                                                                                                                               
 (inlined by) boost::unique_lock<boost::mutex>::lock() at /home/vlad/Work/dev/redpanda/vbuild/release/clang/rp_deps_install/include/boost/thread/lock_types.hpp:346                                                                                                             
 (inlined by) boost::unique_lock<boost::mutex>::unique_lock(boost::mutex&) at /home/vlad/Work/dev/redpanda/vbuild/release/clang/rp_deps_install/include/boost/thread/lock_types.hpp:124                                                                                         
 (inlined by) boost::barrier::wait() at /home/vlad/Work/dev/redpanda/vbuild/release/clang/rp_deps_install/include/boost/thread/barrier.hpp:220                                                                                                                                  
 (inlined by) seastar::smp::arrive_at_event_loop_end() at /home/vlad/Work/dev/redpanda/vbuild/release/clang/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:3862                                                                                                    
seastar::semaphore_units<seastar::named_semaphore_exception_factory, seastar::lowres_clock>::semaphore_units(seastar::semaphore_units<seastar::named_semaphore_exception_factory, seastar::lowres_clock>&&) at /home/vlad/Work/dev/redpanda/vbuild/release/clang/v_deps_build/se
astar-prefix/src/seastar/include/seastar/core/semaphore.hh:475                                                                                                                                                                                                                  
 (inlined by) _ZN7seastar8internal26uninitialized_wrapper_baseINS_15semaphore_unitsINS_33named_semaphore_exception_factoryENS_12lowres_clockEEELb0EE17uninitialized_setIJS5_EEENSt3__19enable_ifIXntsr3stdE9is_same_vINS8_5tupleIJDpu11__remove_cvIT_EEEENSA_IJNSA_IJS5_EEEEEEEE
vE4typeEDpOSB_ at /home/vlad/Work/dev/redpanda/vbuild/release/clang/v_deps_build/seastar-prefix/src/seastar/include/seastar/core/future.hh:296
 (inlined by) seastar::future_state<seastar::semaphore_units<seastar::named_semaphore_exception_factory, seastar::lowres_clock>>::move_it(seastar::future_state<seastar::semaphore_units<seastar::named_semaphore_exception_factory, seastar::lowres_clock>>&&) at /home/vlad/Wo
rk/dev/redpanda/vbuild/release/clang/v_deps_build/seastar-prefix/src/seastar/include/seastar/core/future.hh:586
 (inlined by) seastar::future_state<seastar::semaphore_units<seastar::named_semaphore_exception_factory, seastar::lowres_clock>>::future_state(seastar::future_state<seastar::semaphore_units<seastar::named_semaphore_exception_factory, seastar::lowres_clock>>&&) at /home/vl
ad/Work/dev/redpanda/vbuild/release/clang/v_deps_build/seastar-prefix/src/seastar/include/seastar/core/future.hh:593
 (inlined by) seastar::future<seastar::semaphore_units<seastar::named_semaphore_exception_factory, seastar::lowres_clock>>::future(seastar::future<seastar::semaphore_units<seastar::named_semaphore_exception_factory, seastar::lowres_clock>>&&) at /home/vlad/Work/dev/redpan
da/vbuild/release/clang/v_deps_build/seastar-prefix/src/seastar/include/seastar/core/future.hh:1305
 (inlined by) seastar::future<void> seastar::futurize<void>::invoke<seastar::smp_message_queue::submit_item(unsigned int, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>, std::__1::unique_ptr<sea
star::smp_message_queue::work_item, std::__1::default_delete<seastar::smp_message_queue::work_item>>)::$_0, seastar::future<seastar::semaphore_units<seastar::named_semaphore_exception_factory, seastar::lowres_clock>>>(seastar::smp_message_queue::submit_item(unsigned int, 
std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>, std::__1::unique_ptr<seastar::smp_message_queue::work_item, std::__1::default_delete<seastar::smp_message_queue::work_item>>)::$_0&&, seastar::fut
ure<seastar::semaphore_units<seastar::named_semaphore_exception_factory, seastar::lowres_clock>>&&) at /home/vlad/Work/dev/redpanda/vbuild/release/clang/v_deps_build/seastar-prefix/src/seastar/include/seastar/core/future.hh:1996
 (inlined by) seastar::futurize<void>::type seastar::future<seastar::semaphore_units<seastar::named_semaphore_exception_factory, seastar::lowres_clock>>::then_wrapped_common<true, void, seastar::smp_message_queue::submit_item(unsigned int, std::__1::chrono::time_point<sea
star::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>, std::__1::unique_ptr<seastar::smp_message_queue::work_item, std::__1::default_delete<seastar::smp_message_queue::work_item>>)::$_0>(seastar::smp_message_queue::submit_item(unsign
ed int, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>, std::__1::unique_ptr<seastar::smp_message_queue::work_item, std::__1::default_delete<seastar::smp_message_queue::work_item>>)::$_0&&) at /
home/vlad/Work/dev/redpanda/vbuild/release/clang/v_deps_build/seastar-prefix/src/seastar/include/seastar/core/future.hh:1553
 (inlined by) seastar::futurize<void>::type seastar::future<seastar::semaphore_units<seastar::named_semaphore_exception_factory, seastar::lowres_clock>>::then_wrapped_maybe_erase<true, void, seastar::smp_message_queue::submit_item(unsigned int, std::__1::chrono::time_poin
t<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>, std::__1::unique_ptr<seastar::smp_message_queue::work_item, std::__1::default_delete<seastar::smp_message_queue::work_item>>)::$_0>(seastar::smp_message_queue::submit_item(u
nsigned int, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>, std::__1::unique_ptr<seastar::smp_message_queue::work_item, std::__1::default_delete<seastar::smp_message_queue::work_item>>)::$_0&&)
 at /home/vlad/Work/dev/redpanda/vbuild/release/clang/v_deps_build/seastar-prefix/src/seastar/include/seastar/core/future.hh:1512
 (inlined by) seastar::futurize<void>::type seastar::future<seastar::semaphore_units<seastar::named_semaphore_exception_factory, seastar::lowres_clock>>::then_wrapped<seastar::smp_message_queue::submit_item(unsigned int, std::__1::chrono::time_point<seastar::lowres_clock,
 std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>, std::__1::unique_ptr<seastar::smp_message_queue::work_item, std::__1::default_delete<seastar::smp_message_queue::work_item>>)::$_0, void>(seastar::smp_message_queue::submit_item(unsigned int, std::
__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>, std::__1::unique_ptr<seastar::smp_message_queue::work_item, std::__1::default_delete<seastar::smp_message_queue::work_item>>)::$_0&&) && at /home/vlad/
Work/dev/redpanda/vbuild/release/clang/v_deps_build/seastar-prefix/src/seastar/include/seastar/core/future.hh:1503
 (inlined by) seastar::smp_message_queue::submit_item(unsigned int, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>, std::__1::unique_ptr<seastar::smp_message_queue::work_item, std::__1::default_
delete<seastar::smp_message_queue::work_item>>) at /home/vlad/Work/dev/redpanda/vbuild/release/clang/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:3486
seastar::future_state_base::any::move_it(seastar::future_state_base::any&&) at /home/vlad/Work/dev/redpanda/vbuild/release/clang/v_deps_build/seastar-prefix/src/seastar/include/seastar/core/future.hh:0
 (inlined by) seastar::future_state_base::any::any(seastar::future_state_base::any&&) at /home/vlad/Work/dev/redpanda/vbuild/release/clang/v_deps_build/seastar-prefix/src/seastar/include/seastar/core/future.hh:480
 (inlined by) seastar::future_state_base::future_state_base(seastar::future_state_base&&) at /home/vlad/Work/dev/redpanda/vbuild/release/clang/v_deps_build/seastar-prefix/src/seastar/include/seastar/core/future.hh:500
 (inlined by) seastar::future_state<seastar::internal::monostate>::future_state(seastar::future_state<seastar::internal::monostate>&&) at /home/vlad/Work/dev/redpanda/vbuild/release/clang/v_deps_build/seastar-prefix/src/seastar/include/seastar/core/future.hh:592
 (inlined by) seastar::future<void>::future(seastar::future<void>&&) at /home/vlad/Work/dev/redpanda/vbuild/release/clang/v_deps_build/seastar-prefix/src/seastar/include/seastar/core/future.hh:1305
 (inlined by) seastar::future<void> seastar::futurize<void>::invoke<seastar::smp_message_queue::async_work_item<seastar::rename_scheduling_group(seastar::scheduling_group, seastar::basic_sstring<char, unsigned int, 15u, true>)::$_0>::run_and_dispose()::'lambda'(auto), sea
star::future<void>>(auto&&, seastar::future<void>&&) at /home/vlad/Work/dev/redpanda/vbuild/release/clang/v_deps_build/seastar-prefix/src/seastar/include/seastar/core/future.hh:1996
 (inlined by) seastar::futurize<void>::type seastar::future<void>::then_wrapped_common<true, void, seastar::smp_message_queue::async_work_item<seastar::rename_scheduling_group(seastar::scheduling_group, seastar::basic_sstring<char, unsigned int, 15u, true>)::$_0>::run_and
_dispose()::'lambda'(auto)>(seastar::smp_message_queue::async_work_item<seastar::rename_scheduling_group(seastar::scheduling_group, seastar::basic_sstring<char, unsigned int, 15u, true>)::$_0>::run_and_dispose()::'lambda'(auto)&&) at /home/vlad/Work/dev/redpanda/vbuild/re
lease/clang/v_deps_build/seastar-prefix/src/seastar/include/seastar/core/future.hh:1553
 (inlined by) seastar::futurize<void>::type seastar::future<void>::then_wrapped_maybe_erase<true, void, seastar::smp_message_queue::async_work_item<seastar::rename_scheduling_group(seastar::scheduling_group, seastar::basic_sstring<char, unsigned int, 15u, true>)::$_0>::ru
n_and_dispose()::'lambda'(auto)>(seastar::smp_message_queue::async_work_item<seastar::rename_scheduling_group(seastar::scheduling_group, seastar::basic_sstring<char, unsigned int, 15u, true>)::$_0>::run_and_dispose()::'lambda'(auto)&&) at /home/vlad/Work/dev/redpanda/vbui
ld/release/clang/v_deps_build/seastar-prefix/src/seastar/include/seastar/core/future.hh:1512
 (inlined by) seastar::futurize<void>::type seastar::future<void>::then_wrapped<seastar::smp_message_queue::async_work_item<seastar::rename_scheduling_group(seastar::scheduling_group, seastar::basic_sstring<char, unsigned int, 15u, true>)::$_0>::run_and_dispose()::'lambda
'(auto), void>(auto&&) && at /home/vlad/Work/dev/redpanda/vbuild/release/clang/v_deps_build/seastar-prefix/src/seastar/include/seastar/core/future.hh:1503
 (inlined by) seastar::smp_message_queue::async_work_item<seastar::rename_scheduling_group(seastar::scheduling_group, seastar::basic_sstring<char, unsigned int, 15u, true>)::$_0>::run_and_dispose() at /home/vlad/Work/dev/redpanda/vbuild/release/clang/v_deps_build/seastar-
prefix/src/seastar/include/seastar/core/smp.hh:240
std::__1::basic_streambuf<char, std::__1::char_traits<char>>::sputn[abi:v160004](char const*, long) at /home/vlad/Work/dev/redpanda/vbuild/llvm/install/bin/../include/c++/v1/streambuf:232
 (inlined by) std::__1::ostreambuf_iterator<char, std::__1::char_traits<char>> std::__1::__pad_and_output[abi:v160004]<char, std::__1::char_traits<char>>(std::__1::ostreambuf_iterator<char, std::__1::char_traits<char>>, char const*, char const*, char const*, std::__1::ios
_base&, char) at /home/vlad/Work/dev/redpanda/vbuild/llvm/install/bin/../include/c++/v1/locale:1430
 (inlined by) std::__1::basic_ostream<char, std::__1::char_traits<char>>& std::__1::__put_character_sequence[abi:v160004]<char, std::__1::char_traits<char>>(std::__1::basic_ostream<char, std::__1::char_traits<char>>&, char const*, unsigned long) at /home/vlad/Work/dev/red
panda/vbuild/llvm/install/bin/../include/c++/v1/ostream:762
 (inlined by) std::__1::basic_ostream<char, std::__1::char_traits<char>>& std::__1::operator<<[abi:v160004]<std::__1::char_traits<char>>(std::__1::basic_ostream<char, std::__1::char_traits<char>>&, char) at /home/vlad/Work/dev/redpanda/vbuild/llvm/install/bin/../include/c
++/v1/ostream:827
 (inlined by) seastar::prometheus::add_name(std::__1::basic_ostream<char, std::__1::char_traits<char>>&, seastar::basic_sstring<char, unsigned int, 15u, true> const&, std::__1::map<seastar::basic_sstring<char, unsigned int, 15u, true>, seastar::basic_sstring<char, unsigne
d int, 15u, true>, std::__1::less<seastar::basic_sstring<char, unsigned int, 15u, true>>, std::__1::allocator<std::__1::pair<seastar::basic_sstring<char, unsigned int, 15u, true> const, seastar::basic_sstring<char, unsigned int, 15u, true>>>> const&, seastar::prometheus::
config const&) at /home/vlad/Work/dev/redpanda/vbuild/release/clang/v_deps_build/seastar-prefix/src/seastar/src/core/prometheus.cc:85
?? at ??:0
?? at ??:0
piyushredpanda commented 1 year ago

Maybe, abhijat's PR will do the trick, @VladLazar ? Or did it seem like an entirely new thing?

VladLazar commented 1 year ago

It's very likely the same thing. Yeah, we should merge Abhijat's pr.