search

redpanda-data / redpanda

Redpanda is a streaming data platform for developers. Kafka API compatible. 10x faster. No ZooKeeper. No JVM!
https://redpanda.com
9.15k stars 558 forks source link

Redpanda reports oversized allocation when trying to create topics near the partition limit #16521

Closed savex closed 4 months ago

savex commented 4 months ago

Version & Environment

Manual CDT run using 10 i3en.xlarge nodes on AWS.EC2

Redpanda version: (use rpk version):

Version:     v0.0.0-20240207git0afd0bd
Git ref:     0afd0bd1ab
Build date:  2024-02-07T01:09:41Z
OS/Arch:     linux/amd64
Go version:  go1.21.3

Kafka python client

ubuntu@ip-172-31-52-58:~$ pip3 list | grep kafka
confluent-kafka          2.2.0
kafka-python             2.0.2
kafkatest                3.2.0

What went wrong?

When creating large topic number (11950) on 9 node redpanda cluster test failed with BadLogLines:

[WARNING - 2024-02-07 16:32:34,171 - redpanda - raise_on_bad_logs - lineno:1401]: Ignoring oversized allocation, 393216 is less than the max allowable allocation size of 409600 bytes
[WARNING - 2024-02-07 16:32:34,173 - redpanda - raise_on_bad_logs - lineno:1408]: [test_topic_swarm] Unexpected log line on ip-172-31-3-143: WARN  2024-02-07 16:32:03,579 [shard 0:main] seastar_memory - oversized allocation: 1638400 bytes. This is non-fatal, but could lead to latency and/or fragmentation issues. Please report: at 0x92f4ef3 0x8f5ca1c 0x8f69cf0 0x46d4772 0x434915a 0x9029f1f 0x902d691 0x902a826 0x8f299a0 0x8f27d98 0x2c8c776 0x934af99 /opt/redpanda/lib/libc.so.6+0x2d58f /opt/redpanda/lib/libc.so.6+0x2d648 0x2c84fa4
[WARNING - 2024-02-07 16:32:34,184 - redpanda - raise_on_bad_logs - lineno:1401]: Ignoring oversized allocation, 401408 is less than the max allowable allocation size of 409600 bytes

What should have happened instead?

Redpanda able to create topics up to the limit and/or report something like "No more topics creation possible, limit reached"

How to reproduce the issue?

Ducktape cmd line is

ducktape --cluster=ducktape.cluster.json.JsonCluster --cluster-file=cluster.json --globals=globals.json --max-parallel=1 --repeat=1 --test-runner-timeout=86400000 rptest/scale_tests/many_partitions_test.py::ManyPartitionsTest.test_topic_swarm
  1. Either use script from PR or run test_topic_swarm test while selecting number of topics near the limit of the cluster abilities: (<rp_nodes_cluster> * <vcpu_count> * 1000) / <replica_count>. Example for i3en.xlarge: (941000)/3 = 12000
  2. Run the test and check BadLogLines errors at the end

Additional information

Backtrace decoding for one of the logs. cmd: python3 ~/seastar-addr2line.py -v -a /usr/bin/llvm-addr2line-14 -e /opt/redpanda/libexec/redpanda -f ~/tests/results/2024-02-07--001/ManyPartitionsTest/test_topic_swarm/1/RedpandaService-0-140592323297680/ip-172-31-3-143/redpanda.log

/opt/redpanda/libexec/redpanda: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /opt/redpanda/lib/ld.so, for GNU/Linux 3.2.0, BuildID[sha1]=754f98daf43987fb40970d066dd369a9023d4dfa, stripped

INFO  2024-02-07 16:31:56,116 [shard 0:main] cluster - members_manager.cc:375 - applying finish_reallocations_cmd, offset: 65, node id: 5
WARN  2024-02-07 16:32:03,567 [shard 0:main] seastar_memory - oversized allocation: 393216 bytes. This is non-fatal, but could lead to latency and/or fragmentation issues. Please report: at
[Backtrace #0]
{/opt/redpanda/libexec/redpanda} 0x92f4ef3: void seastar::backtrace<seastar::current_backtrace_tasklocal()::$_0>(seastar::current_backtrace_tasklocal()::$_0&&) at /v/build/v_deps_build/seastar-prefix/src/seastar/include/seastar/util/backtrace.hh:64
 (inlined by) seastar::current_backtrace_tasklocal() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/util/backtrace.cc:98
 (inlined by) seastar::current_tasktrace() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/util/backtrace.cc:149
 (inlined by) seastar::current_backtrace() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/util/backtrace.cc:182
{/opt/redpanda/libexec/redpanda} 0x8f5ca1c: seastar::memory::cpu_pages::warn_large_allocation(unsigned long) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:839
 (inlined by) seastar::memory::cpu_pages::check_large_allocation(unsigned long) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:904
 (inlined by) seastar::memory::cpu_pages::allocate_large(unsigned int, bool) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:911
 (inlined by) seastar::memory::allocate_large(unsigned long, bool) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:1521
 (inlined by) seastar::memory::allocate_slowpath(unsigned long) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:1645
{/opt/redpanda/libexec/redpanda} 0x8f69cf0: seastar::memory::allocate(unsigned long) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:1658
 (inlined by) operator new(unsigned long) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:2355
{/opt/redpanda/libexec/redpanda} 0x799e1fd: void* std::__1::__libcpp_operator_new[abi:v160004]<unsigned long>(unsigned long) at /vectorized/llvm/bin/../include/c++/v1/new:266
 (inlined by) std::__1::__libcpp_allocate[abi:v160004](unsigned long, unsigned long) at /vectorized/llvm/bin/../include/c++/v1/new:292
 (inlined by) std::__1::allocator<kafka::creatable_topic>::allocate[abi:v160004](unsigned long) at /vectorized/llvm/bin/../include/c++/v1/__memory/allocator.h:115
 (inlined by) std::__1::__allocation_result<std::__1::allocator_traits<std::__1::allocator<kafka::creatable_topic> >::pointer> std::__1::__allocate_at_least[abi:v160004]<std::__1::allocator<kafka::creatable_topic> >(std::__1::allocator<kafka::creatable_topic>&, unsigned long) at /vectorized/llvm/bin/../include/c++/v1/__memory/allocate_at_least.h:55
 (inlined by) std::__1::__split_buffer<kafka::creatable_topic, std::__1::allocator<kafka::creatable_topic>&>::__split_buffer(unsigned long, unsigned long, std::__1::allocator<kafka::creatable_topic>&) at /vectorized/llvm/bin/../include/c++/v1/__split_buffer:323
 (inlined by) std::__1::vector<kafka::creatable_topic, std::__1::allocator<kafka::creatable_topic> >::reserve(unsigned long) at /vectorized/llvm/bin/../include/c++/v1/vector:1503
{/opt/redpanda/libexec/redpanda} 0x77cc0f1: std::__1::vector<kafka::creatable_topic> kafka::protocol::decoder::do_read_array<std::__1::vector, kafka::create_topics_request_data::decode_standard(kafka::protocol::decoder&, detail::base_named_type<short, kafka::kafka_api_version, std::__1::integral_constant<bool, true> >)::$_0, kafka::creatable_topic>(int, kafka::create_topics_request_data::decode_standard(kafka::protocol::decoder&, detail::base_named_type<short, kafka::kafka_api_version, std::__1::integral_constant<bool, true> >)::$_0&&) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-0fad785cdf3e3a542-1/redpanda/redpanda/src/v/kafka/protocol/wire.h:314
 (inlined by) std::__1::vector<kafka::creatable_topic> kafka::protocol::decoder::read_array<std::__1::vector, kafka::create_topics_request_data::decode_standard(kafka::protocol::decoder&, detail::base_named_type<short, kafka::kafka_api_version, std::__1::integral_constant<bool, true> >)::$_0, kafka::creatable_topic>(kafka::create_topics_request_data::decode_standard(kafka::protocol::decoder&, detail::base_named_type<short, kafka::kafka_api_version, std::__1::integral_constant<bool, true> >)::$_0&&) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-0fad785cdf3e3a542-1/redpanda/redpanda/src/v/kafka/protocol/wire.h:204
{/opt/redpanda/libexec/redpanda} 0x77cb5f7: kafka::create_topics_request_data::decode_standard(kafka::protocol::decoder&, detail::base_named_type<short, kafka::kafka_api_version, std::__1::integral_constant<bool, true> >) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-0fad785cdf3e3a542-1/redpanda/redpanda/vbuild/release/clang/src/v/kafka/protocol/schemata/protocol.cc:4814
{/opt/redpanda/libexec/redpanda} 0x46bbfca: kafka::create_topics_request::decode(kafka::protocol::decoder&, detail::base_named_type<short, kafka::kafka_api_version, std::__1::integral_constant<bool, true> >) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-0fad785cdf3e3a542-1/redpanda/redpanda/src/v/kafka/protocol/create_topics.h:34
 (inlined by) kafka::handler_template<kafka::create_topics_api, (short)0, (short)6, seastar::future<seastar::foreign_ptr<std::__1::unique_ptr<kafka::response, std::__1::default_delete<kafka::response> > > >, &(kafka::default_estimate_adaptor(unsigned long, kafka::connection_context&))>::handle(kafka::request_context, seastar::smp_service_group) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-0fad785cdf3e3a542-1/redpanda/redpanda/src/v/kafka/server/handlers/create_topics.cc:147
{/opt/redpanda/libexec/redpanda} 0x43dbe76: kafka::handler_base<false>::handle(kafka::request_context&&, seastar::smp_service_group) const at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-0fad785cdf3e3a542-1/redpanda/redpanda/src/v/kafka/server/handlers/handler_interface.cc:88
{/opt/redpanda/libexec/redpanda} 0x444631e: kafka::process_generic(kafka::handler_interface const*, kafka::request_context&&, seastar::smp_service_group, kafka::session_resources const&) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-0fad785cdf3e3a542-1/redpanda/redpanda/src/v/kafka/server/requests.cc:112
 (inlined by) kafka::process_request(kafka::request_context&&, seastar::smp_service_group, kafka::session_resources const&) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-0fad785cdf3e3a542-1/redpanda/redpanda/src/v/kafka/server/requests.cc:339
{/opt/redpanda/libexec/redpanda} 0x4425c5f: kafka::connection_context::dispatch_method_once(kafka::request_header, unsigned long) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-0fad785cdf3e3a542-1/redpanda/redpanda/src/v/kafka/server/connection_context.cc:597
{/opt/redpanda/libexec/redpanda} 0x2dd666a: std::__1::coroutine_handle<seastar::internal::coroutine_traits_base<void>::promise_type>::resume[abi:v160004]() const at /vectorized/llvm/bin/../include/c++/v1/__coroutine/coroutine_handle.h:169
 (inlined by) seastar::internal::coroutine_traits_base<void>::promise_type::run_and_dispose() at /vectorized/include/seastar/core/coroutine.hh:125
{/opt/redpanda/libexec/redpanda} 0x9029f1f: seastar::reactor::run_tasks(seastar::reactor::task_queue&) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:2750
 (inlined by) seastar::reactor::run_some_tasks() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:3213
{/opt/redpanda/libexec/redpanda} 0x902d691: seastar::reactor::do_run() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:3397
{/opt/redpanda/libexec/redpanda} 0x902a826: seastar::reactor::run() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:3265
{/opt/redpanda/libexec/redpanda} 0x8f299a0: seastar::app_template::run_deprecated(int, char**, std::__1::function<void ()>&&) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/app-template.cc:276
{/opt/redpanda/libexec/redpanda} 0x8f27d98: seastar::app_template::run(int, char**, std::__1::function<seastar::future<int> ()>&&) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/app-template.cc:167
{/opt/redpanda/libexec/redpanda} 0x2c8c776: application::run(int, char**) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-0fad785cdf3e3a542-1/redpanda/redpanda/src/v/redpanda/application.cc:416
{/opt/redpanda/libexec/redpanda} 0x934af99: main at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-0fad785cdf3e3a542-1/redpanda/redpanda/src/v/redpanda/main.cc:22
/opt/redpanda/lib/libc.so.6: ELF 64-bit LSB shared object, x86-64, version 1 (GNU/Linux), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=6e7b96dfb83f0bdcb6a410469b82f86415e5ada3, for GNU/Linux 3.2.0, stripped

{/opt/redpanda/lib/libc.so.6} 0x2d58f: ?? at ??:0
{/opt/redpanda/lib/libc.so.6} 0x2d648: __libc_start_main at ??:0
{/opt/redpanda/libexec/redpanda} 0x2c84fa4: _start at ??:0

   seastar::internal::coroutine_traits_base<void>::promise_type
WARN  2024-02-07 16:32:03,579 [shard 0:main] seastar_memory - oversized allocation: 1638400 bytes. This is non-fatal, but could lead to latency and/or fragmentation issues. Please report: at
[Backtrace #1]
{/opt/redpanda/libexec/redpanda} 0x92f4ef3: void seastar::backtrace<seastar::current_backtrace_tasklocal()::$_0>(seastar::current_backtrace_tasklocal()::$_0&&) at /v/build/v_deps_build/seastar-prefix/src/seastar/include/seastar/util/backtrace.hh:64
 (inlined by) seastar::current_backtrace_tasklocal() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/util/backtrace.cc:98
 (inlined by) seastar::current_tasktrace() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/util/backtrace.cc:149
 (inlined by) seastar::current_backtrace() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/util/backtrace.cc:182
{/opt/redpanda/libexec/redpanda} 0x8f5ca1c: seastar::memory::cpu_pages::warn_large_allocation(unsigned long) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:839
 (inlined by) seastar::memory::cpu_pages::check_large_allocation(unsigned long) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:904
 (inlined by) seastar::memory::cpu_pages::allocate_large(unsigned int, bool) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:911
 (inlined by) seastar::memory::allocate_large(unsigned long, bool) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:1521
 (inlined by) seastar::memory::allocate_slowpath(unsigned long) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:1645
{/opt/redpanda/libexec/redpanda} 0x8f69cf0: seastar::memory::allocate(unsigned long) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:1658
 (inlined by) operator new(unsigned long) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:2355
{/opt/redpanda/libexec/redpanda} 0x46d4772: void* std::__1::__libcpp_operator_new[abi:v160004]<unsigned long>(unsigned long) at /vectorized/llvm/bin/../include/c++/v1/new:266
 (inlined by) std::__1::__libcpp_allocate[abi:v160004](unsigned long, unsigned long) at /vectorized/llvm/bin/../include/c++/v1/new:292
 (inlined by) std::__1::allocator<cluster::custom_assignable_topic_configuration>::allocate[abi:v160004](unsigned long) at /vectorized/llvm/bin/../include/c++/v1/__memory/allocator.h:115
 (inlined by) std::__1::__allocation_result<std::__1::allocator_traits<std::__1::allocator<cluster::custom_assignable_topic_configuration> >::pointer> std::__1::__allocate_at_least[abi:v160004]<std::__1::allocator<cluster::custom_assignable_topic_configuration> >(std::__1::allocator<cluster::custom_assignable_topic_configuration>&, unsigned long) at /vectorized/llvm/bin/../include/c++/v1/__memory/allocate_at_least.h:55
 (inlined by) std::__1::__split_buffer<cluster::custom_assignable_topic_configuration, std::__1::allocator<cluster::custom_assignable_topic_configuration>&>::__split_buffer(unsigned long, unsigned long, std::__1::allocator<cluster::custom_assignable_topic_configuration>&) at /vectorized/llvm/bin/../include/c++/v1/__split_buffer:323
 (inlined by) std::__1::vector<cluster::custom_assignable_topic_configuration, std::__1::allocator<cluster::custom_assignable_topic_configuration> >::reserve(unsigned long) at /vectorized/llvm/bin/../include/c++/v1/vector:1503
 (inlined by) std::__1::vector<decltype(to_cluster_type(*(fp))), std::__1::allocator<decltype(to_cluster_type(*(fp)))> > kafka::to_cluster_type<std::__1::__wrap_iter<kafka::creatable_topic*> >(std::__1::__wrap_iter<kafka::creatable_topic*>, std::__1::__wrap_iter<kafka::creatable_topic*>) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-0fad785cdf3e3a542-1/redpanda/redpanda/src/v/kafka/server/handlers/topics/topic_utils.h:127
 (inlined by) kafka::handler_template<kafka::create_topics_api, (short)0, (short)6, seastar::future<seastar::foreign_ptr<std::__1::unique_ptr<kafka::response, std::__1::default_delete<kafka::response> > > >, &(kafka::default_estimate_adaptor(unsigned long, kafka::connection_context&))>::handle(kafka::request_context, seastar::smp_service_group) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-0fad785cdf3e3a542-1/redpanda/redpanda/src/v/kafka/server/handlers/create_topics.cc:320
{/opt/redpanda/libexec/redpanda} 0x434915a: std::__1::coroutine_handle<seastar::internal::coroutine_traits_base<seastar::foreign_ptr<std::__1::unique_ptr<kafka::response, std::__1::default_delete<kafka::response> > > >::promise_type>::resume[abi:v160004]() const at /vectorized/llvm/bin/../include/c++/v1/__coroutine/coroutine_handle.h:169
 (inlined by) seastar::internal::coroutine_traits_base<seastar::foreign_ptr<std::__1::unique_ptr<kafka::response, std::__1::default_delete<kafka::response> > > >::promise_type::run_and_dispose() at /vectorized/include/seastar/core/coroutine.hh:83
{/opt/redpanda/libexec/redpanda} 0x9029f1f: seastar::reactor::run_tasks(seastar::reactor::task_queue&) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:2750
 (inlined by) seastar::reactor::run_some_tasks() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:3213
{/opt/redpanda/libexec/redpanda} 0x902d691: seastar::reactor::do_run() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:3397
{/opt/redpanda/libexec/redpanda} 0x902a826: seastar::reactor::run() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:3265
{/opt/redpanda/libexec/redpanda} 0x8f299a0: seastar::app_template::run_deprecated(int, char**, std::__1::function<void ()>&&) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/app-template.cc:276
{/opt/redpanda/libexec/redpanda} 0x8f27d98: seastar::app_template::run(int, char**, std::__1::function<seastar::future<int> ()>&&) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/app-template.cc:167
{/opt/redpanda/libexec/redpanda} 0x2c8c776: application::run(int, char**) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-0fad785cdf3e3a542-1/redpanda/redpanda/src/v/redpanda/application.cc:416
{/opt/redpanda/libexec/redpanda} 0x934af99: main at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-0fad785cdf3e3a542-1/redpanda/redpanda/src/v/redpanda/main.cc:22
{/opt/redpanda/lib/libc.so.6} 0x2d58f: ?? at ??:0
{/opt/redpanda/lib/libc.so.6} 0x2d648: __libc_start_main at ??:0
{/opt/redpanda/libexec/redpanda} 0x2c84fa4: _start at ??:0

INFO  2024-02-07 16:32:35,076 [shard 2:main] cluster - controller_backend.cc:395 - Stopping Controller Backend...
Reactor stalled for 65 ms on shard 0. Backtrace:
[Backtrace #2]
{/opt/redpanda/libexec/redpanda} 0x90056ef: void seastar::backtrace<seastar::backtrace_buffer::append_backtrace_oneline()::'lambda'(seastar::frame)>(seastar::backtrace_buffer::append_backtrace_oneline()::'lambda'(seastar::frame)&&) at /v/build/v_deps_build/seastar-prefix/src/seastar/include/seastar/util/backtrace.hh:64
 (inlined by) seastar::backtrace_buffer::append_backtrace_oneline() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:838
 (inlined by) seastar::print_with_backtrace(seastar::backtrace_buffer&, bool) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:857
{/opt/redpanda/libexec/redpanda} 0x9006e4d: seastar::internal::cpu_stall_detector::generate_trace() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:1539
 (inlined by) seastar::internal::cpu_stall_detector::maybe_report() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:1376
 (inlined by) seastar::internal::cpu_stall_detector::on_signal() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:1396
 (inlined by) seastar::reactor::block_notifier(int) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:1597
{/opt/redpanda/libexec/redpanda} 0x42abf: ?? at ??:0
{/opt/redpanda/libexec/redpanda} 0x8f9e09f: std::__1::__tree<std::__1::__value_type<seastar::basic_sstring<char, unsigned int, 15u, true>, seastar::basic_sstring<char, unsigned int, 15u, true> >, std::__1::__map_value_compare<seastar::basic_sstring<char, unsigned int, 15u, true>, std::__1::__value_type<seastar::basic_sstring<char, unsigned int, 15u, true>, seastar::basic_sstring<char, unsigned int, 15u, true> >, std::__1::less<seastar::basic_sstring<char, unsigned int, 15u, true> >, true>, std::__1::allocator<std::__1::__value_type<seastar::basic_sstring<char, unsigned int, 15u, true>, seastar::basic_sstring<char, unsigned int, 15u, true> > > >::begin[abi:v160004]() const at /vectorized/llvm/bin/../include/c++/v1/__tree:1137
 (inlined by) std::__1::map<seastar::basic_sstring<char, unsigned int, 15u, true>, seastar::basic_sstring<char, unsigned int, 15u, true>, std::__1::less<seastar::basic_sstring<char, unsigned int, 15u, true> >, std::__1::allocator<std::__1::pair<seastar::basic_sstring<char, unsigned int, 15u, true> const, seastar::basic_sstring<char, unsigned int, 15u, true> > > >::begin[abi:v160004]() const at /vectorized/llvm/bin/../include/c++/v1/map:1169
 (inlined by) bool std::__1::operator<[abi:v160004]<seastar::basic_sstring<char, unsigned int, 15u, true>, seastar::basic_sstring<char, unsigned int, 15u, true>, std::__1::less<seastar::basic_sstring<char, unsigned int, 15u, true> >, std::__1::allocator<std::__1::pair<seastar::basic_sstring<char, unsigned int, 15u, true> const, seastar::basic_sstring<char, unsigned int, 15u, true> > > >(std::__1::map<seastar::basic_sstring<char, unsigned int, 15u, true>, seastar::basic_sstring<char, unsigned int, 15u, true>, std::__1::less<seastar::basic_sstring<char, unsigned int, 15u, true> >, std::__1::allocator<std::__1::pair<seastar::basic_sstring<char, unsigned int, 15u, true> const, seastar::basic_sstring<char, unsigned int, 15u, true> > > > const&, std::__1::map<seastar::basic_sstring<char, unsigned int, 15u, true>, seastar::basic_sstring<char, unsigned int, 15u, true>, std::__1::less<seastar::basic_sstring<char, unsigned int, 15u, true> >, std::__1::allocator<std::__1::pair<seastar::basic_sstring<char, unsigned int, 15u, true> const, seastar::basic_sstring<char, unsigned int, 15u, true> > > > const&) at /vectorized/llvm/bin/../include/c++/v1/map:1686
 (inlined by) std::__1::less<std::__1::map<seastar::basic_sstring<char, unsigned int, 15u, true>, seastar::basic_sstring<char, unsigned int, 15u, true>, std::__1::less<seastar::basic_sstring<char, unsigned int, 15u, true> >, std::__1::allocator<std::__1::pair<seastar::basic_sstring<char, unsigned int, 15u, true> const, seastar::basic_sstring<char, unsigned int, 15u, true> > > > >::operator()[abi:v160004](std::__1::map<seastar::basic_sstring<char, unsigned int, 15u, true>, seastar::basic_sstring<char, unsigned int, 15u, true>, std::__1::less<seastar::basic_sstring<char, unsigned int, 15u, true> >, std::__1::allocator<std::__1::pair<seastar::basic_sstring<char, unsigned int, 15u, true> const, seastar::basic_sstring<char, unsigned int, 15u, true> > > > const&, std::__1::map<seastar::basic_sstring<char, unsigned int, 15u, true>, seastar::basic_sstring<char, unsigned int, 15u, true>, std::__1::less<seastar::basic_sstring<char, unsigned int, 15u, true> >, std::__1::allocator<std::__1::pair<seastar::basic_sstring<char, unsigned int, 15u, true> const, seastar::basic_sstring<char, unsigned int, 15u, true> > > > const&) const at /vectorized/llvm/bin/../include/c++/v1/__functional/operations.h:384
 (inlined by) std::__1::__map_value_compare<std::__1::map<seastar::basic_sstring<char, unsigned int, 15u, true>, seastar::basic_sstring<char, unsigned int, 15u, true>, std::__1::less<seastar::basic_sstring<char, unsigned int, 15u, true> >, std::__1::allocator<std::__1::pair<seastar::basic_sstring<char, unsigned int, 15u, true> const, seastar::basic_sstring<char, unsigned int, 15u, true> > > >, std::__1::__value_type<std::__1::map<seastar::basic_sstring<char, unsigned int, 15u, true>, seastar::basic_sstring<char, unsigned int, 15u, true>, std::__1::less<seastar::basic_sstring<char, unsigned int, 15u, true> >, std::__1::allocator<std::__1::pair<seastar::basic_sstring<char, unsigned int, 15u, true> const, seastar::basic_sstring<char, unsigned int, 15u, true> > > >, seastar::shared_ptr<seastar::metrics::impl::registered_metric> >, std::__1::less<std::__1::map<seastar::basic_sstring<char, unsigned int, 15u, true>, seastar::basic_sstring<char, unsigned int, 15u, true>, std::__1::less<seastar::basic_sstring<char, unsigned int, 15u, true> >, std::__1::allocator<std::__1::pair<seastar::basic_sstring<char, unsigned int, 15u, true> const, seastar::basic_sstring<char, unsigned int, 15u, true> > > > >, true>::operator()[abi:v160004](std::__1::__value_type<std::__1::map<seastar::basic_sstring<char, unsigned int, 15u, true>, seastar::basic_sstring<char, unsigned int, 15u, true>, std::__1::less<seastar::basic_sstring<char, unsigned int, 15u, true> >, std::__1::allocator<std::__1::pair<seastar::basic_sstring<char, unsigned int, 15u, true> const, seastar::basic_sstring<char, unsigned int, 15u, true> > > >, seastar::shared_ptr<seastar::metrics::impl::registered_metric> > const&, std::__1::map<seastar::basic_sstring<char, unsigned int, 15u, true>, seastar::basic_sstring<char, unsigned int, 15u, true>, std::__1::less<seastar::basic_sstring<char, unsigned int, 15u, true> >, std::__1::allocator<std::__1::pair<seastar::basic_sstring<char, unsigned int, 15u, true> const, seastar::basic_sstring<char, unsigned int, 15u, true> > > > const&) const at /vectorized/llvm/bin/../include/c++/v1/map:593
 (inlined by) std::__1::__tree_iterator<std::__1::__value_type<std::__1::map<seastar::basic_sstring<char, unsigned int, 15u, true>, seastar::basic_sstring<char, unsigned int, 15u, true>, std::__1::less<seastar::basic_sstring<char, unsigned int, 15u, true> >, std::__1::allocator<std::__1::pair<seastar::basic_sstring<char, unsigned int, 15u, true> const, seastar::basic_sstring<char, unsigned int, 15u, true> > > >, seastar::shared_ptr<seastar::metrics::impl::registered_metric> >, std::__1::__tree_node<std::__1::__value_type<std::__1::map<seastar::basic_sstring<char, unsigned int, 15u, true>, seastar::basic_sstring<char, unsigned int, 15u, true>, std::__1::less<seastar::basic_sstring<char, unsigned int, 15u, true> >, std::__1::allocator<std::__1::pair<seastar::basic_sstring<char, unsigned int, 15u, true> const, seastar::basic_sstring<char, unsigned int, 15u, true> > > >, seastar::shared_ptr<seastar::metrics::impl::registered_metric> >, void*>*, long> std::__1::__tree<std::__1::__value_type<std::__1::map<seastar::basic_sstring<char, unsigned int, 15u, true>, seastar::basic_sstring<char, unsigned int, 15u, true>, std::__1::less<seastar::basic_sstring<char, unsigned int, 15u, true> >, std::__1::allocator<std::__1::pair<seastar::basic_sstring<char, unsigned int, 15u, true> const, seastar::basic_sstring<char, unsigned int, 15u, true> > > >, seastar::shared_ptr<seastar::metrics::impl::registered_metric> >, std::__1::__map_value_compare<std::__1::map<seastar::basic_sstring<char, unsigned int, 15u, true>, seastar::basic_sstring<char, unsigned int, 15u, true>, std::__1::less<seastar::basic_sstring<char, unsigned int, 15u, true> >, std::__1::allocator<std::__1::pair<seastar::basic_sstring<char, unsigned int, 15u, true> const, seastar::basic_sstring<char, unsigned int, 15u, true> > > >, std::__1::__value_type<std::__1::map<seastar::basic_sstring<char, unsigned int, 15u, true>, seastar::basic_sstring<char, unsigned int, 15u, true>, std::__1::less<seastar::basic_sstring<char, unsigned int, 15u, true> >, std::__1::allocator<std::__1::pair<seastar::basic_sstring<char, unsigned int, 15u, true> const, seastar::basic_sstring<char, unsigned int, 15u, true> > > >, seastar::shared_ptr<seastar::metrics::impl::registered_metric> >, std::__1::less<std::__1::map<seastar::basic_sstring<char, unsigned int, 15u, true>, seastar::basic_sstring<char, unsigned int, 15u, true>, std::__1::less<seastar::basic_sstring<char, unsigned int, 15u, true> >, std::__1::allocator<std::__1::pair<seastar::basic_sstring<char, unsigned int, 15u, true> const, seastar::basic_sstring<char, unsigned int, 15u, true> > > > >, true>, std::__1::allocator<std::__1::__value_type<std::__1::map<seastar::basic_sstring<char, unsigned int, 15u, true>, seastar::basic_sstring<char, unsigned int, 15u, true>, std::__1::less<seastar::basic_sstring<char, unsigned int, 15u, true> >, std::__1::allocator<std::__1::pair<seastar::basic_sstring<char, unsigned int, 15u, true> const, seastar::basic_sstring<char, unsigned int, 15u, true> > > >, seastar::shared_ptr<seastar::metrics::impl::registered_metric> > > >::__lower_bound<std::__1::map<seastar::basic_sstring<char, unsigned int, 15u, true>, seastar::basic_sstring<char, unsigned int, 15u, true>, std::__1::less<seastar::basic_sstring<char, unsigned int, 15u, true> >, std::__1::allocator<std::__1::pair<seastar::basic_sstring<char, unsigned int, 15u, true> const, seastar::basic_sstring<char, unsigned int, 15u, true> > > > >(std::__1::map<seastar::basic_sstring<char, unsigned int, 15u, true>, seastar::basic_sstring<char, unsigned int, 15u, true>, std::__1::less<seastar::basic_sstring<char, unsigned int, 15u, true> >, std::__1::allocator<std::__1::pair<seastar::basic_sstring<char, unsigned int, 15u, true> const, seastar::basic_sstring<char, unsigned int, 15u, true> > > > const&, std::__1::__tree_node<std::__1::__value_type<std::__1::map<seastar::basic_sstring<char, unsigned int, 15u, true>, seastar::basic_sstring<char, unsigned int, 15u, true>, std::__1::less<seastar::basic_sstring<char, unsigned int, 15u, true> >, std::__1::allocator<std::__1::pair<seastar::basic_sstring<char, unsigned int, 15u, true> const, seastar::basic_sstring<char, unsigned int, 15u, true> > > >, seastar::shared_ptr<seastar::metrics::impl::registered_metric> >, void*>*, std::__1::__tree_end_node<std::__1::__tree_node_base<void*>*>*) at /vectorized/llvm/bin/../include/c++/v1/__tree:2549
 (inlined by) std::__1::__tree_iterator<std::__1::__value_type<std::__1::map<seastar::basic_sstring<char, unsigned int, 15u, true>, seastar::basic_sstring<char, unsigned int, 15u, true>, std::__1::less<seastar::basic_sstring<char, unsigned int, 15u, true> >, std::__1::allocator<std::__1::pair<seastar::basic_sstring<char, unsigned int, 15u, true> const, seastar::basic_sstring<char, unsigned int, 15u, true> > > >, seastar::shared_ptr<seastar::metrics::impl::registered_metric> >, std::__1::__tree_node<std::__1::__value_type<std::__1::map<seastar::basic_sstring<char, unsigned int, 15u, true>, seastar::basic_sstring<char, unsigned int, 15u, true>, std::__1::less<seastar::basic_sstring<char, unsigned int, 15u, true> >, std::__1::allocator<std::__1::pair<seastar::basic_sstring<char, unsigned int, 15u, true> const, seastar::basic_sstring<char, unsigned int, 15u, true> > > >, seastar::shared_ptr<seastar::metrics::impl::registered_metric> >, void*>*, long> std::__1::__tree<std::__1::__value_type<std::__1::map<seastar::basic_sstring<char, unsigned int, 15u, true>, seastar::basic_sstring<char, unsigned int, 15u, true>, std::__1::less<seastar::basic_sstring<char, unsigned int, 15u, true> >, std::__1::allocator<std::__1::pair<seastar::basic_sstring<char, unsigned int, 15u, true> const, seastar::basic_sstring<char, unsigned int, 15u, true> > > >, seastar::shared_ptr<seastar::metrics::impl::registered_metric> >, std::__1::__map_value_compare<std::__1::map<seastar::basic_sstring<char, unsigned int, 15u, true>, seastar::basic_sstring<char, unsigned int, 15u, true>, std::__1::less<seastar::basic_sstring<char, unsigned int, 15u, true> >, std::__1::allocator<std::__1::pair<seastar::basic_sstring<char, unsigned int, 15u, true> const, seastar::basic_sstring<char, unsigned int, 15u, true> > > >, std::__1::__value_type<std::__1::map<seastar::basic_sstring<char, unsigned int, 15u, true>, seastar::basic_sstring<char, unsigned int, 15u, true>, std::__1::less<seastar::basic_sstring<char, unsigned int, 15u, true> >, std::__1::allocator<std::__1::pair<seastar::basic_sstring<char, unsigned int, 15u, true> const, seastar::basic_sstring<char, unsigned int, 15u, true> > > >, seastar::shared_ptr<seastar::metrics::impl::registered_metric> >, std::__1::less<std::__1::map<seastar::basic_sstring<char, unsigned int, 15u, true>, seastar::basic_sstring<char, unsigned int, 15u, true>, std::__1::less<seastar::basic_sstring<char, unsigned int, 15u, true> >, std::__1::allocator<std::__1::pair<seastar::basic_sstring<char, unsigned int, 15u, true> const, seastar::basic_sstring<char, unsigned int, 15u, true> > > > >, true>, std::__1::allocator<std::__1::__value_type<std::__1::map<seastar::basic_sstring<char, unsigned int, 15u, true>, seastar::basic_sstring<char, unsigned int, 15u, true>, std::__1::less<seastar::basic_sstring<char, unsigned int, 15u, true> >, std::__1::allocator<std::__1::pair<seastar::basic_sstring<char, unsigned int, 15u, true> const, seastar::basic_sstring<char, unsigned int, 15u, true> > > >, seastar::shared_ptr<seastar::metrics::impl::registered_metric> > > >::find<std::__1::map<seastar::basic_sstring<char, unsigned int, 15u, true>, seastar::basic_sstring<char, unsigned int, 15u, true>, std::__1::less<seastar::basic_sstring<char, unsigned int, 15u, true> >, std::__1::allocator<std::__1::pair<seastar::basic_sstring<char, unsigned int, 15u, true> const, seastar::basic_sstring<char, unsigned int, 15u, true> > > > >(std::__1::map<seastar::basic_sstring<char, unsigned int, 15u, true>, seastar::basic_sstring<char, unsigned int, 15u, true>, std::__1::less<seastar::basic_sstring<char, unsigned int, 15u, true> >, std::__1::allocator<std::__1::pair<seastar::basic_sstring<char, unsigned int, 15u, true> const, seastar::basic_sstring<char, unsigned int, 15u, true> > > > const&) at /vectorized/llvm/bin/../include/c++/v1/__tree:2478
 (inlined by) std::__1::map<std::__1::map<seastar::basic_sstring<char, unsigned int, 15u, true>, seastar::basic_sstring<char, unsigned int, 15u, true>, std::__1::less<seastar::basic_sstring<char, unsigned int, 15u, true> >, std::__1::allocator<std::__1::pair<seastar::basic_sstring<char, unsigned int, 15u, true> const, seastar::basic_sstring<char, unsigned int, 15u, true> > > >, seastar::shared_ptr<seastar::metrics::impl::registered_metric>, std::__1::less<std::__1::map<seastar::basic_sstring<char, unsigned int, 15u, true>, seastar::basic_sstring<char, unsigned int, 15u, true>, std::__1::less<seastar::basic_sstring<char, unsigned int, 15u, true> >, std::__1::allocator<std::__1::pair<seastar::basic_sstring<char, unsigned int, 15u, true> const, seastar::basic_sstring<char, unsigned int, 15u, true> > > > >, std::__1::allocator<std::__1::pair<std::__1::map<seastar::basic_sstring<char, unsigned int, 15u, true>, seastar::basic_sstring<char, unsigned int, 15u, true>, std::__1::less<seastar::basic_sstring<char, unsigned int, 15u, true> >, std::__1::allocator<std::__1::pair<seastar::basic_sstring<char, unsigned int, 15u, true> const, seastar::basic_sstring<char, unsigned int, 15u, true> > > > const, seastar::shared_ptr<seastar::metrics::impl::registered_metric> > > >::find[abi:v160004](std::__1::map<seastar::basic_sstring<char, unsigned int, 15u, true>, seastar::basic_sstring<char, unsigned int, 15u, true>, std::__1::less<seastar::basic_sstring<char, unsigned int, 15u, true> >, std::__1::allocator<std::__1::pair<seastar::basic_sstring<char, unsigned int, 15u, true> const, seastar::basic_sstring<char, unsigned int, 15u, true> > > > const&) at /vectorized/llvm/bin/../include/c++/v1/map:1452
 (inlined by) seastar::metrics::impl::metric_family::find(std::__1::map<seastar::basic_sstring<char, unsigned int, 15u, true>, seastar::basic_sstring<char, unsigned int, 15u, true>, std::__1::less<seastar::basic_sstring<char, unsigned int, 15u, true> >, std::__1::allocator<std::__1::pair<seastar::basic_sstring<char, unsigned int, 15u, true> const, seastar::basic_sstring<char, unsigned int, 15u, true> > > > const&) at /v/build/v_deps_build/seastar-prefix/src/seastar/include/seastar/core/metrics_api.hh:300
 (inlined by) seastar::metrics::impl::impl::remove_registration(seastar::metrics::impl::metric_id const&) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/metrics.cc:371
{/opt/redpanda/libexec/redpanda} 0x8f94536: seastar::metrics::impl::unregister_metric(seastar::metrics::impl::metric_id const&, int) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/metrics.cc:414
 (inlined by) seastar::metrics::impl::metric_groups_impl::~metric_groups_impl() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/metrics.cc:302
 (inlined by) seastar::metrics::impl::metric_groups_impl::~metric_groups_impl() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/metrics.cc:300
{/opt/redpanda/libexec/redpanda} 0x6447f97: std::__1::pair<model::topic_namespace const, seastar::metrics::metric_groups>::~pair() at /vectorized/include/boost/container/detail/std_fwd.hpp:39
 (inlined by) void std::__1::__destroy_at[abi:v160004]<std::__1::pair<model::topic_namespace const, seastar::metrics::metric_groups>, 0>(std::__1::pair<model::topic_namespace const, seastar::metrics::metric_groups>*) at /vectorized/llvm/bin/../include/c++/v1/__memory/construct_at.h:66
 (inlined by) void std::__1::destroy_at[abi:v160004]<std::__1::pair<model::topic_namespace const, seastar::metrics::metric_groups>, 0>(std::__1::pair<model::topic_namespace const, seastar::metrics::metric_groups>*) at /vectorized/llvm/bin/../include/c++/v1/__memory/construct_at.h:101
 (inlined by) void std::__1::allocator_traits<std::__1::allocator<std::__1::pair<model::topic_namespace const, seastar::metrics::metric_groups> > >::destroy[abi:v160004]<std::__1::pair<model::topic_namespace const, seastar::metrics::metric_groups>, void, void>(std::__1::allocator<std::__1::pair<model::topic_namespace const, seastar::metrics::metric_groups> >&, std::__1::pair<model::topic_namespace const, seastar::metrics::metric_groups>*) at /vectorized/llvm/bin/../include/c++/v1/__memory/allocator_traits.h:323
 (inlined by) void absl::lts_20230802::container_internal::map_slot_policy<model::topic_namespace, seastar::metrics::metric_groups>::destroy<std::__1::allocator<std::__1::pair<model::topic_namespace const, seastar::metrics::metric_groups> > >(std::__1::allocator<std::__1::pair<model::topic_namespace const, seastar::metrics::metric_groups> >*, absl::lts_20230802::container_internal::map_slot_type<model::topic_namespace, seastar::metrics::metric_groups>*) at /vectorized/include/absl/container/internal/container_memory.h:421
 (inlined by) void absl::lts_20230802::container_internal::FlatHashMapPolicy<model::topic_namespace, seastar::metrics::metric_groups>::destroy<std::__1::allocator<std::__1::pair<model::topic_namespace const, seastar::metrics::metric_groups> > >(std::__1::allocator<std::__1::pair<model::topic_namespace const, seastar::metrics::metric_groups> >*, absl::lts_20230802::container_internal::map_slot_type<model::topic_namespace, seastar::metrics::metric_groups>*) at /vectorized/include/absl/container/flat_hash_map.h:578
 (inlined by) void absl::lts_20230802::container_internal::common_policy_traits<absl::lts_20230802::container_internal::FlatHashMapPolicy<model::topic_namespace, seastar::metrics::metric_groups>, void>::destroy<std::__1::allocator<std::__1::pair<model::topic_namespace const, seastar::metrics::metric_groups> > >(std::__1::allocator<std::__1::pair<model::topic_namespace const, seastar::metrics::metric_groups> >*, absl::lts_20230802::container_internal::map_slot_type<model::topic_namespace, seastar::metrics::metric_groups>*) at /vectorized/include/absl/container/internal/common_policy_traits.h:50
 (inlined by) absl::lts_20230802::container_internal::raw_hash_set<absl::lts_20230802::container_internal::FlatHashMapPolicy<model::topic_namespace, seastar::metrics::metric_groups>, absl::lts_20230802::hash_internal::Hash<model::topic_namespace>, std::__1::equal_to<model::topic_namespace>, std::__1::allocator<std::__1::pair<model::topic_namespace const, seastar::metrics::metric_groups> > >::destroy_slots() at /vectorized/include/absl/container/internal/raw_hash_set.h:1946
 (inlined by) absl::lts_20230802::container_internal::raw_hash_set<absl::lts_20230802::container_internal::FlatHashMapPolicy<model::topic_namespace, seastar::metrics::metric_groups>, absl::lts_20230802::hash_internal::Hash<model::topic_namespace>, std::__1::equal_to<model::topic_namespace>, std::__1::allocator<std::__1::pair<model::topic_namespace const, seastar::metrics::metric_groups> > >::~raw_hash_set() at /vectorized/include/absl/container/internal/raw_hash_set.h:1885
 (inlined by) cluster::topic_table_probe::~topic_table_probe() at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-0fad785cdf3e3a542-1/redpanda/redpanda/src/v/cluster/topic_table_probe.h:20
 (inlined by) cluster::topic_table::~topic_table() at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-0fad785cdf3e3a542-1/redpanda/redpanda/src/v/cluster/topic_table.h:78
{/opt/redpanda/libexec/redpanda} 0x6447dba: seastar::shared_ptr_count_for<cluster::topic_table>::~shared_ptr_count_for() at /vectorized/include/seastar/core/shared_ptr.hh:492
 (inlined by) seastar::shared_ptr_count_for<cluster::topic_table>::~shared_ptr_count_for() at /vectorized/include/seastar/core/shared_ptr.hh:492
{/opt/redpanda/libexec/redpanda} 0x644b157: seastar::shared_ptr<cluster::topic_table>::~shared_ptr() at /vectorized/include/seastar/core/shared_ptr.hh:570
 (inlined by) seastar::shared_ptr<cluster::topic_table>::operator=(seastar::shared_ptr<cluster::topic_table>&&) at /vectorized/include/seastar/core/shared_ptr.hh:583
 (inlined by) seastar::shared_ptr<cluster::topic_table>::operator=(std::nullptr_t) at /vectorized/include/seastar/core/shared_ptr.hh:589
 (inlined by) seastar::sharded<cluster::topic_table>::stop()::'lambda'(seastar::future<void>)::operator()(seastar::future<void>) const::'lambda'(unsigned int)::operator()(unsigned int) const::'lambda'()::operator()() const at /vectorized/include/seastar/core/sharded.hh:735
 (inlined by) seastar::future<void> seastar::futurize<seastar::future<void> >::invoke<seastar::sharded<cluster::topic_table>::stop()::'lambda'(seastar::future<void>)::operator()(seastar::future<void>) const::'lambda'(unsigned int)::operator()(unsigned int) const::'lambda'()&>(seastar::sharded<cluster::topic_table>::stop()::'lambda'(seastar::future<void>)::operator()(seastar::future<void>) const::'lambda'(unsigned int)::operator()(unsigned int) const::'lambda'()&) at /vectorized/include/seastar/core/future.hh:2006
 (inlined by) seastar::futurize<std::__1::invoke_result<seastar::sharded<cluster::topic_table>::stop()::'lambda'(seastar::future<void>)::operator()(seastar::future<void>) const::'lambda'(unsigned int)::operator()(unsigned int) const::'lambda'()>::type>::type seastar::smp::submit_to<seastar::sharded<cluster::topic_table>::stop()::'lambda'(seastar::future<void>)::operator()(seastar::future<void>) const::'lambda'(unsigned int)::operator()(unsigned int) const::'lambda'()>(unsigned int, seastar::smp_submit_to_options, seastar::sharded<cluster::topic_table>::stop()::'lambda'(seastar::future<void>)::operator()(seastar::future<void>) const::'lambda'(unsigned int)::operator()(unsigned int) const::'lambda'()&&) at /vectorized/include/seastar/core/smp.hh:385
 (inlined by) seastar::futurize<std::__1::invoke_result<seastar::sharded<cluster::topic_table>::stop()::'lambda'(seastar::future<void>)::operator()(seastar::future<void>) const::'lambda'(unsigned int)::operator()(unsigned int) const::'lambda'()>::type>::type seastar::smp::submit_to<seastar::sharded<cluster::topic_table>::stop()::'lambda'(seastar::future<void>)::operator()(seastar::future<void>) const::'lambda'(unsigned int)::operator()(unsigned int) const::'lambda'()>(unsigned int, seastar::sharded<cluster::topic_table>::stop()::'lambda'(seastar::future<void>)::operator()(seastar::future<void>) const::'lambda'(unsigned int)::operator()(unsigned int) const::'lambda'()&&) at /vectorized/include/seastar/core/smp.hh:412
 (inlined by) seastar::sharded<cluster::topic_table>::stop()::'lambda'(seastar::future<void>)::operator()(seastar::future<void>) const::'lambda'(unsigned int)::operator()(unsigned int) const at /vectorized/include/seastar/core/sharded.hh:731
 (inlined by) decltype(std::declval<seastar::sharded<cluster::topic_table>::stop()::'lambda'(seastar::future<void>)::operator()(seastar::future<void>) const::'lambda'(unsigned int)&>()(std::declval<unsigned int>())) std::__1::__invoke[abi:v160004]<seastar::sharded<cluster::topic_table>::stop()::'lambda'(seastar::future<void>)::operator()(seastar::future<void>) const::'lambda'(unsigned int)&, unsigned int>(seastar::sharded<cluster::topic_table>::stop()::'lambda'(seastar::future<void>)::operator()(seastar::future<void>) const::'lambda'(unsigned int)&, unsigned int&&) at /vectorized/llvm/bin/../include/c++/v1/__functional/invoke.h:394
 (inlined by) seastar::future<void> std::__1::__invoke_void_return_wrapper<seastar::future<void>, false>::__call<seastar::sharded<cluster::topic_table>::stop()::'lambda'(seastar::future<void>)::operator()(seastar::future<void>) const::'lambda'(unsigned int)&, unsigned int>(seastar::sharded<cluster::topic_table>::stop()::'lambda'(seastar::future<void>)::operator()(seastar::future<void>) const::'lambda'(unsigned int)&, unsigned int&&) at /vectorized/llvm/bin/../include/c++/v1/__functional/invoke.h:478
 (inlined by) std::__1::__function::__alloc_func<seastar::sharded<cluster::topic_table>::stop()::'lambda'(seastar::future<void>)::operator()(seastar::future<void>) const::'lambda'(unsigned int), std::__1::allocator<seastar::sharded<cluster::topic_table>::stop()::'lambda'(seastar::future<void>)::operator()(seastar::future<void>) const::'lambda'(unsigned int)>, seastar::future<void> (unsigned int)>::operator()[abi:v160004](unsigned int&&) at /vectorized/llvm/bin/../include/c++/v1/__functional/function.h:185
 (inlined by) std::__1::__function::__func<seastar::sharded<cluster::topic_table>::stop()::'lambda'(seastar::future<void>)::operator()(seastar::future<void>) const::'lambda'(unsigned int), std::__1::allocator<seastar::sharded<cluster::topic_table>::stop()::'lambda'(seastar::future<void>)::operator()(seastar::future<void>) const::'lambda'(unsigned int)>, seastar::future<void> (unsigned int)>::operator()(unsigned int&&) at /vectorized/llvm/bin/../include/c++/v1/__functional/function.h:356
{/opt/redpanda/libexec/redpanda} 0x91055ef: std::__1::__function::__value_func<seastar::future<void> (unsigned int)>::operator()[abi:v160004](unsigned int&&) const at /vectorized/llvm/bin/../include/c++/v1/__functional/function.h:510
 (inlined by) std::__1::function<seastar::future<void> (unsigned int)>::operator()(unsigned int) const at /vectorized/llvm/bin/../include/c++/v1/__functional/function.h:1156
 (inlined by) seastar::future<void> seastar::futurize<seastar::future<void> >::invoke<std::__1::function<seastar::future<void> (unsigned int)>, unsigned int>(std::__1::function<seastar::future<void> (unsigned int)>&&, unsigned int&&) at /v/build/v_deps_build/seastar-prefix/src/seastar/include/seastar/core/future.hh:2006
 (inlined by) auto seastar::futurize_invoke<std::__1::function<seastar::future<void> (unsigned int)>, unsigned int>(std::__1::function<seastar::future<void> (unsigned int)>&&, unsigned int&&) at /v/build/v_deps_build/seastar-prefix/src/seastar/include/seastar/core/future.hh:2037
 (inlined by) seastar::future<void> seastar::parallel_for_each<boost::range_detail::integer_iterator<unsigned int>, boost::range_detail::integer_iterator<unsigned int>, std::__1::function<seastar::future<void> (unsigned int)> >(boost::range_detail::integer_iterator<unsigned int>, boost::range_detail::integer_iterator<unsigned int>, std::__1::function<seastar::future<void> (unsigned int)>&&) at /v/build/v_deps_build/seastar-prefix/src/seastar/include/seastar/core/loop.hh:572
 (inlined by) seastar::future<void> seastar::internal::parallel_for_each_impl<boost::integer_range<unsigned int>, std::__1::function<seastar::future<void> (unsigned int)> >(boost::integer_range<unsigned int>&&, std::__1::function<seastar::future<void> (unsigned int)>&&) at /v/build/v_deps_build/seastar-prefix/src/seastar/include/seastar/core/loop.hh:625
 (inlined by) seastar::future<void> seastar::futurize<seastar::future<void> >::invoke<seastar::future<void> (*&)(boost::integer_range<unsigned int>&&, std::__1::function<seastar::future<void> (unsigned int)>&&), boost::integer_range<unsigned int>, std::__1::function<seastar::future<void> (unsigned int)> >(seastar::future<void> (*&)(boost::integer_range<unsigned int>&&, std::__1::function<seastar::future<void> (unsigned int)>&&), boost::integer_range<unsigned int>&&, std::__1::function<seastar::future<void> (unsigned int)>&&) at /v/build/v_deps_build/seastar-prefix/src/seastar/include/seastar/core/future.hh:2006
 (inlined by) auto seastar::futurize_invoke<seastar::future<void> (*&)(boost::integer_range<unsigned int>&&, std::__1::function<seastar::future<void> (unsigned int)>&&), boost::integer_range<unsigned int>, std::__1::function<seastar::future<void> (unsigned int)> >(seastar::future<void> (*&)(boost::integer_range<unsigned int>&&, std::__1::function<seastar::future<void> (unsigned int)>&&), boost::integer_range<unsigned int>&&, std::__1::function<seastar::future<void> (unsigned int)>&&) at /v/build/v_deps_build/seastar-prefix/src/seastar/include/seastar/core/future.hh:2037
 (inlined by) seastar::future<void> seastar::parallel_for_each<boost::integer_range<unsigned int>, std::__1::function<seastar::future<void> (unsigned int)> >(boost::integer_range<unsigned int>&&, std::__1::function<seastar::future<void> (unsigned int)>&&) at /v/build/v_deps_build/seastar-prefix/src/seastar/include/seastar/core/loop.hh:640
 (inlined by) seastar::internal::sharded_parallel_for_each(unsigned int, std::__1::function<seastar::future<void> (unsigned int)>) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/sharded.cc:42
{/opt/redpanda/libexec/redpanda} 0x644a8f4: seastar::sharded<cluster::topic_table>::sharded_parallel_for_each(std::__1::function<seastar::future<void> (unsigned int)>) at /vectorized/include/seastar/core/sharded.hh:207
 (inlined by) seastar::sharded<cluster::topic_table>::stop()::'lambda'(seastar::future<void>)::operator()(seastar::future<void>) const at /vectorized/include/seastar/core/sharded.hh:730
{/opt/redpanda/libexec/redpanda} 0x644c6b6: seastar::futurize<seastar::future<void> >::type seastar::future<void>::then_wrapped_nrvo<seastar::future<void>, seastar::sharded<cluster::topic_table>::stop()::'lambda'(seastar::future<void>)>(seastar::sharded<cluster::topic_table>::stop()::'lambda'(seastar::future<void>)&&)::'lambda'(seastar::internal::promise_base_with_type<void>&&, seastar::sharded<cluster::topic_table>::stop()::'lambda'(seastar::future<void>)&, seastar::future_state<seastar::internal::monostate>&&)::operator()(seastar::internal::promise_base_with_type<void>&&, seastar::sharded<cluster::topic_table>::stop()::'lambda'(seastar::future<void>)&, seastar::future_state<seastar::internal::monostate>&&) const::'lambda'()::operator()() const at /vectorized/include/seastar/core/future.hh:1539
 (inlined by) void seastar::futurize<seastar::future<void> >::satisfy_with_result_of<seastar::futurize<seastar::future<void> >::type seastar::future<void>::then_wrapped_nrvo<seastar::future<void>, seastar::sharded<cluster::topic_table>::stop()::'lambda'(seastar::future<void>)>(seastar::sharded<cluster::topic_table>::stop()::'lambda'(seastar::future<void>)&&)::'lambda'(seastar::internal::promise_base_with_type<void>&&, seastar::sharded<cluster::topic_table>::stop()::'lambda'(seastar::future<void>)&, seastar::future_state<seastar::internal::monostate>&&)::operator()(seastar::internal::promise_base_with_type<void>&&, seastar::sharded<cluster::topic_table>::stop()::'lambda'(seastar::future<void>)&, seastar::future_state<seastar::internal::monostate>&&) const::'lambda'()>(seastar::internal::promise_base_with_type<void>&&, seastar::future<void>&&) at /vectorized/include/seastar/core/future.hh:1991
 (inlined by) seastar::futurize<seastar::future<void> >::type seastar::future<void>::then_wrapped_nrvo<seastar::future<void>, seastar::sharded<cluster::topic_table>::stop()::'lambda'(seastar::future<void>)>(seastar::sharded<cluster::topic_table>::stop()::'lambda'(seastar::future<void>)&&)::'lambda'(seastar::internal::promise_base_with_type<void>&&, seastar::sharded<cluster::topic_table>::stop()::'lambda'(seastar::future<void>)&, seastar::future_state<seastar::internal::monostate>&&)::operator()(seastar::internal::promise_base_with_type<void>&&, seastar::sharded<cluster::topic_table>::stop()::'lambda'(seastar::future<void>)&, seastar::future_state<seastar::internal::monostate>&&) const at /vectorized/include/seastar/core/future.hh:1538
 (inlined by) seastar::continuation<seastar::internal::promise_base_with_type<void>, seastar::sharded<cluster::topic_table>::stop()::'lambda'(seastar::future<void>), seastar::futurize<seastar::future<void> >::type seastar::future<void>::then_wrapped_nrvo<seastar::future<void>, seastar::sharded<cluster::topic_table>::stop()::'lambda'(seastar::future<void>)>(seastar::sharded<cluster::topic_table>::stop()::'lambda'(seastar::future<void>)&&)::'lambda'(seastar::internal::promise_base_with_type<void>&&, seastar::sharded<cluster::topic_table>::stop()::'lambda'(seastar::future<void>)&, seastar::future_state<seastar::internal::monostate>&&), void>::run_and_dispose() at /vectorized/include/seastar/core/future.hh:741
{/opt/redpanda/libexec/redpanda} 0x9029f1f: seastar::reactor::run_tasks(seastar::reactor::task_queue&) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:2750
 (inlined by) seastar::reactor::run_some_tasks() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:3213
{/opt/redpanda/libexec/redpanda} 0x902d691: seastar::reactor::do_run() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:3397
{/opt/redpanda/libexec/redpanda} 0x902a826: seastar::reactor::run() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:3265
{/opt/redpanda/libexec/redpanda} 0x8f299a0: seastar::app_template::run_deprecated(int, char**, std::__1::function<void ()>&&) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/app-template.cc:276
{/opt/redpanda/libexec/redpanda} 0x8f27d98: seastar::app_template::run(int, char**, std::__1::function<seastar::future<int> ()>&&) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/app-template.cc:167
{/opt/redpanda/libexec/redpanda} 0x2c8c776: application::run(int, char**) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-0fad785cdf3e3a542-1/redpanda/redpanda/src/v/redpanda/application.cc:416
{/opt/redpanda/libexec/redpanda} 0x934af99: main at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-0fad785cdf3e3a542-1/redpanda/redpanda/src/v/redpanda/main.cc:22
{/opt/redpanda/libexec/redpanda} 0x2d58f: ?? at ??:0
{/opt/redpanda/libexec/redpanda} 0x2d648: ?? at ??:0
{/opt/redpanda/libexec/redpanda} 0x2c84fa4: _start at ??:0
savex commented 4 months ago

cc: @ballard26, @travisdowns, @piyushredpanda