cluster: assertion deallocating partition from core

[jcsp]

v21.10.1

ERROR 2021-12-21 05:51:35,176 [shard 0] assert - ../../../src/v/cluster/scheduling/allocation_node.cc:44 @deallocate: failed to log message: fmt='Assert failure: ({}:{}) '_allocated_partitions > allocation_capacity{0} && _weights[core] > 0' unable to deallocate partition from core {} at node {}': fmt::v7::format_error (cannot switch from automatic to manual argument indexing)

redpanda.log

Full log bundle on internal slack.

JIRA Link: CORE-808

[jcsp]

Evidently the code for printing the assertion message is also having an error, let's fix that at the same time

[dotnwat]

@rystsov, @jcsp Have you seen this since you originally added the issue / added to the Jepson report milestone?

[jcsp]

I have not, but I'm guessing that Kyle would have stopped mentioning it once we acked it with the issue, so unless we fixed it I'd presume it's still a real issue.

[mmaslankaprv]

We haven't seen this error for ~5 months. We have a test node_operations_fuzzy_test which exercise this code path.

[jcsp]

But didn't node_operations_fuzzy_test also pass on 21.10.1, where we know the issue is present? It seems like the check for this should be to re-run the Jepsen test that triggered the issue

[travisdowns]

Here's the decoded backtrace for the instance in the first post of this issue:

{opt/redpanda/libexec/redpanda} 0x3529f74: void seastar::backtrace<seastar::current_backtrace_tasklocal()::$_3>(seastar::current_backtrace_tasklocal()::$_3&&) at /v/build/v_deps_build/seastar-prefix/src/seastar/include/seastar/util/backtrace.hh:59
 (inlined by) seastar::current_backtrace_tasklocal() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/util/backtrace.cc:86
 (inlined by) seastar::current_tasktrace() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/util/backtrace.cc:137
 (inlined by) seastar::current_backtrace() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/util/backtrace.cc:170
{opt/redpanda/libexec/redpanda} 0x2b03065: cluster::allocation_node::deallocate(unsigned int) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-0684f8b909f3f330a-1/vectorized/redpanda/vbuild/release/clang/../../../src/v/cluster/scheduling/allocation_node.cc:40
{opt/redpanda/libexec/redpanda} 0x27d08da: cluster::partition_allocator::update_allocation_state(std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> > const&, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> > const&) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-0684f8b909f3f330a-1/vectorized/redpanda/vbuild/release/clang/../../../src/v/cluster/scheduling/partition_allocator.cc:229
{opt/redpanda/libexec/redpanda} 0x28a0634: cluster::topic_updates_dispatcher::reallocate_partition(std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> > const&, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> > const&) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-0684f8b909f3f330a-1/vectorized/redpanda/vbuild/release/clang/../../../src/v/cluster/topic_updates_dispatcher.cc:159
 (inlined by) auto cluster::topic_updates_dispatcher::apply_update(model::record_batch)::$_0::operator()<std::__1::variant<cluster::controller_command<model::topic_namespace, cluster::topic_configuration_assignment, (signed char)0, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, model::topic_namespace, (signed char)1, (model::record_batch_type)6>, cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)2, (model::record_batch_type)6>, cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)3, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, cluster::incremental_topic_updates, (signed char)4, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, cluster::create_partititions_configuration_assignment, (signed char)5, (model::record_batch_type)6>, cluster::controller_command<cluster::non_replicable_topic, signed char, (signed char)6, (model::record_batch_type)6> > >(std::__1::variant<cluster::controller_command<model::topic_namespace, cluster::topic_configuration_assignment, (signed char)0, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, model::topic_namespace, (signed char)1, (model::record_batch_type)6>, cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)2, (model::record_batch_type)6>, cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)3, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, cluster::incremental_topic_updates, (signed char)4, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, cluster::create_partititions_configuration_assignment, (signed char)5, (model::record_batch_type)6>, cluster::controller_command<cluster::non_replicable_topic, signed char, (signed char)6, (model::record_batch_type)6> >) const::'lambda'(cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)2, (model::record_batch_type)6>)::operator()(cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)2, (model::record_batch_type)6>) const::'lambda'(std::__1::error_code)::operator()(std::__1::error_code) const at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-0684f8b909f3f330a-1/vectorized/redpanda/vbuild/release/clang/../../../src/v/cluster/topic_updates_dispatcher.cc:79
 (inlined by) decltype(std::__1::forward<std::__1::variant<cluster::controller_command<model::topic_namespace, cluster::topic_configuration_assignment, (signed char)0, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, model::topic_namespace, (signed char)1, (model::record_batch_type)6>, cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)2, (model::record_batch_type)6>, cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)3, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, cluster::incremental_topic_updates, (signed char)4, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, cluster::create_partititions_configuration_assignment, (signed char)5, (model::record_batch_type)6>, cluster::controller_command<cluster::non_replicable_topic, signed char, (signed char)6, (model::record_batch_type)6> > >(fp)(std::__1::forward<std::__1::error_code>(fp0))) std::__1::__invoke<auto cluster::topic_updates_dispatcher::apply_update(model::record_batch)::$_0::operator()<std::__1::variant<cluster::controller_command<model::topic_namespace, cluster::topic_configuration_assignment, (signed char)0, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, model::topic_namespace, (signed char)1, (model::record_batch_type)6>, cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)2, (model::record_batch_type)6>, cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)3, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, cluster::incremental_topic_updates, (signed char)4, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, cluster::create_partititions_configuration_assignment, (signed char)5, (model::record_batch_type)6>, cluster::controller_command<cluster::non_replicable_topic, signed char, (signed char)6, (model::record_batch_type)6> > >(std::__1::variant<cluster::controller_command<model::topic_namespace, cluster::topic_configuration_assignment, (signed char)0, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, model::topic_namespace, (signed char)1, (model::record_batch_type)6>, cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)2, (model::record_batch_type)6>, cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)3, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, cluster::incremental_topic_updates, (signed char)4, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, cluster::create_partititions_configuration_assignment, (signed char)5, (model::record_batch_type)6>, cluster::controller_command<cluster::non_replicable_topic, signed char, (signed char)6, (model::record_batch_type)6> >) const::'lambda'(cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)2, (model::record_batch_type)6>)::operator()(cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)2, (model::record_batch_type)6>) const::'lambda'(std::__1::error_code)&, std::__1::error_code>(std::__1::variant<cluster::controller_command<model::topic_namespace, cluster::topic_configuration_assignment, (signed char)0, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, model::topic_namespace, (signed char)1, (model::record_batch_type)6>, cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)2, (model::record_batch_type)6>, cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)3, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, cluster::incremental_topic_updates, (signed char)4, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, cluster::create_partititions_configuration_assignment, (signed char)5, (model::record_batch_type)6>, cluster::controller_command<cluster::non_replicable_topic, signed char, (signed char)6, (model::record_batch_type)6> >&&, std::__1::error_code&&) at /vectorized/llvm/bin/../include/c++/v1/type_traits:3694
 (inlined by) std::__1::invoke_result<std::__1::variant<cluster::controller_command<model::topic_namespace, cluster::topic_configuration_assignment, (signed char)0, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, model::topic_namespace, (signed char)1, (model::record_batch_type)6>, cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)2, (model::record_batch_type)6>, cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)3, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, cluster::incremental_topic_updates, (signed char)4, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, cluster::create_partititions_configuration_assignment, (signed char)5, (model::record_batch_type)6>, cluster::controller_command<cluster::non_replicable_topic, signed char, (signed char)6, (model::record_batch_type)6> >, std::__1::error_code>::type std::__1::invoke<auto cluster::topic_updates_dispatcher::apply_update(model::record_batch)::$_0::operator()<std::__1::variant<cluster::controller_command<model::topic_namespace, cluster::topic_configuration_assignment, (signed char)0, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, model::topic_namespace, (signed char)1, (model::record_batch_type)6>, cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)2, (model::record_batch_type)6>, cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)3, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, cluster::incremental_topic_updates, (signed char)4, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, cluster::create_partititions_configuration_assignment, (signed char)5, (model::record_batch_type)6>, cluster::controller_command<cluster::non_replicable_topic, signed char, (signed char)6, (model::record_batch_type)6> > >(std::__1::variant<cluster::controller_command<model::topic_namespace, cluster::topic_configuration_assignment, (signed char)0, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, model::topic_namespace, (signed char)1, (model::record_batch_type)6>, cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)2, (model::record_batch_type)6>, cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)3, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, cluster::incremental_topic_updates, (signed char)4, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, cluster::create_partititions_configuration_assignment, (signed char)5, (model::record_batch_type)6>, cluster::controller_command<cluster::non_replicable_topic, signed char, (signed char)6, (model::record_batch_type)6> >) const::'lambda'(cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)2, (model::record_batch_type)6>)::operator()(cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)2, (model::record_batch_type)6>) const::'lambda'(std::__1::error_code)&, std::__1::error_code>(std::__1::variant<cluster::controller_command<model::topic_namespace, cluster::topic_configuration_assignment, (signed char)0, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, model::topic_namespace, (signed char)1, (model::record_batch_type)6>, cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)2, (model::record_batch_type)6>, cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)3, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, cluster::incremental_topic_updates, (signed char)4, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, cluster::create_partititions_configuration_assignment, (signed char)5, (model::record_batch_type)6>, cluster::controller_command<cluster::non_replicable_topic, signed char, (signed char)6, (model::record_batch_type)6> >&&, std::__1::error_code&&) at /vectorized/llvm/bin/../include/c++/v1/functional:2989
 (inlined by) auto seastar::internal::future_invoke<auto cluster::topic_updates_dispatcher::apply_update(model::record_batch)::$_0::operator()<std::__1::variant<cluster::controller_command<model::topic_namespace, cluster::topic_configuration_assignment, (signed char)0, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, model::topic_namespace, (signed char)1, (model::record_batch_type)6>, cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)2, (model::record_batch_type)6>, cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)3, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, cluster::incremental_topic_updates, (signed char)4, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, cluster::create_partititions_configuration_assignment, (signed char)5, (model::record_batch_type)6>, cluster::controller_command<cluster::non_replicable_topic, signed char, (signed char)6, (model::record_batch_type)6> > >(std::__1::variant<cluster::controller_command<model::topic_namespace, cluster::topic_configuration_assignment, (signed char)0, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, model::topic_namespace, (signed char)1, (model::record_batch_type)6>, cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)2, (model::record_batch_type)6>, cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)3, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, cluster::incremental_topic_updates, (signed char)4, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, cluster::create_partititions_configuration_assignment, (signed char)5, (model::record_batch_type)6>, cluster::controller_command<cluster::non_replicable_topic, signed char, (signed char)6, (model::record_batch_type)6> >) const::'lambda'(cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)2, (model::record_batch_type)6>)::operator()(cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)2, (model::record_batch_type)6>) const::'lambda'(std::__1::error_code)&, std::__1::error_code>(std::__1::variant<cluster::controller_command<model::topic_namespace, cluster::topic_configuration_assignment, (signed char)0, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, model::topic_namespace, (signed char)1, (model::record_batch_type)6>, cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)2, (model::record_batch_type)6>, cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)3, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, cluster::incremental_topic_updates, (signed char)4, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, cluster::create_partititions_configuration_assignment, (signed char)5, (model::record_batch_type)6>, cluster::controller_command<cluster::non_replicable_topic, signed char, (signed char)6, (model::record_batch_type)6> >&&, std::__1::error_code&&) at /vectorized/include/seastar/core/future.hh:1211
 (inlined by) seastar::future<std::__1::error_code> seastar::future<std::__1::error_code>::then_impl_nrvo<auto cluster::topic_updates_dispatcher::apply_update(model::record_batch)::$_0::operator()<std::__1::variant<cluster::controller_command<model::topic_namespace, cluster::topic_configuration_assignment, (signed char)0, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, model::topic_namespace, (signed char)1, (model::record_batch_type)6>, cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)2, (model::record_batch_type)6>, cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)3, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, cluster::incremental_topic_updates, (signed char)4, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, cluster::create_partititions_configuration_assignment, (signed char)5, (model::record_batch_type)6>, cluster::controller_command<cluster::non_replicable_topic, signed char, (signed char)6, (model::record_batch_type)6> > >(std::__1::variant<cluster::controller_command<model::topic_namespace, cluster::topic_configuration_assignment, (signed char)0, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, model::topic_namespace, (signed char)1, (model::record_batch_type)6>, cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)2, (model::record_batch_type)6>, cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)3, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, cluster::incremental_topic_updates, (signed char)4, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, cluster::create_partititions_configuration_assignment, (signed char)5, (model::record_batch_type)6>, cluster::controller_command<cluster::non_replicable_topic, signed char, (signed char)6, (model::record_batch_type)6> >) const::'lambda'(cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)2, (model::record_batch_type)6>)::operator()(cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)2, (model::record_batch_type)6>) const::'lambda'(std::__1::error_code), seastar::future<std::__1::error_code> >(std::__1::variant<cluster::controller_command<model::topic_namespace, cluster::topic_configuration_assignment, (signed char)0, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, model::topic_namespace, (signed char)1, (model::record_batch_type)6>, cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)2, (model::record_batch_type)6>, cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)3, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, cluster::incremental_topic_updates, (signed char)4, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, cluster::create_partititions_configuration_assignment, (signed char)5, (model::record_batch_type)6>, cluster::controller_command<cluster::non_replicable_topic, signed char, (signed char)6, (model::record_batch_type)6> >&&)::'lambda'(seastar::internal::promise_base_with_type<std::__1::error_code>&&, auto cluster::topic_updates_dispatcher::apply_update(model::record_batch)::$_0::operator()<std::__1::variant<cluster::controller_command<model::topic_namespace, cluster::topic_configuration_assignment, (signed char)0, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, model::topic_namespace, (signed char)1, (model::record_batch_type)6>, cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)2, (model::record_batch_type)6>, cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)3, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, cluster::incremental_topic_updates, (signed char)4, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, cluster::create_partititions_configuration_assignment, (signed char)5, (model::record_batch_type)6>, cluster::controller_command<cluster::non_replicable_topic, signed char, (signed char)6, (model::record_batch_type)6> > >(std::__1::variant<cluster::controller_command<model::topic_namespace, cluster::topic_configuration_assignment, (signed char)0, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, model::topic_namespace, (signed char)1, (model::record_batch_type)6>, cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)2, (model::record_batch_type)6>, cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)3, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, cluster::incremental_topic_updates, (signed char)4, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, cluster::create_partititions_configuration_assignment, (signed char)5, (model::record_batch_type)6>, cluster::controller_command<cluster::non_replicable_topic, signed char, (signed char)6, (model::record_batch_type)6> >) const::'lambda'(cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)2, (model::record_batch_type)6>)::operator()(cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)2, (model::record_batch_type)6>) const::'lambda'(std::__1::error_code)&, seastar::future_state<std::__1::error_code>&&)::operator()(seastar::internal::promise_base_with_type<std::__1::error_code>&&, auto cluster::topic_updates_dispatcher::apply_update(model::record_batch)::$_0::operator()<std::__1::variant<cluster::controller_command<model::topic_namespace, cluster::topic_configuration_assignment, (signed char)0, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, model::topic_namespace, (signed char)1, (model::record_batch_type)6>, cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)2, (model::record_batch_type)6>, cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)3, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, cluster::incremental_topic_updates, (signed char)4, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, cluster::create_partititions_configuration_assignment, (signed char)5, (model::record_batch_type)6>, cluster::controller_command<cluster::non_replicable_topic, signed char, (signed char)6, (model::record_batch_type)6> > >(std::__1::variant<cluster::controller_command<model::topic_namespace, cluster::topic_configuration_assignment, (signed char)0, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, model::topic_namespace, (signed char)1, (model::record_batch_type)6>, cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)2, (model::record_batch_type)6>, cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)3, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, cluster::incremental_topic_updates, (signed char)4, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, cluster::create_partititions_configuration_assignment, (signed char)5, (model::record_batch_type)6>, cluster::controller_command<cluster::non_replicable_topic, signed char, (signed char)6, (model::record_batch_type)6> >) const::'lambda'(cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)2, (model::record_batch_type)6>)::operator()(cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)2, (model::record_batch_type)6>) const::'lambda'(std::__1::error_code)&, seastar::future_state<std::__1::error_code>&&) const::'lambda'()::operator()() const at /vectorized/include/seastar/core/future.hh:1582
 (inlined by) void seastar::futurize<std::__1::error_code>::satisfy_with_result_of<seastar::future<std::__1::error_code> seastar::future<std::__1::error_code>::then_impl_nrvo<auto cluster::topic_updates_dispatcher::apply_update(model::record_batch)::$_0::operator()<std::__1::variant<cluster::controller_command<model::topic_namespace, cluster::topic_configuration_assignment, (signed char)0, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, model::topic_namespace, (signed char)1, (model::record_batch_type)6>, cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)2, (model::record_batch_type)6>, cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)3, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, cluster::incremental_topic_updates, (signed char)4, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, cluster::create_partititions_configuration_assignment, (signed char)5, (model::record_batch_type)6>, cluster::controller_command<cluster::non_replicable_topic, signed char, (signed char)6, (model::record_batch_type)6> > >(std::__1::variant<cluster::controller_command<model::topic_namespace, cluster::topic_configuration_assignment, (signed char)0, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, model::topic_namespace, (signed char)1, (model::record_batch_type)6>, cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)2, (model::record_batch_type)6>, cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)3, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, cluster::incremental_topic_updates, (signed char)4, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, cluster::create_partititions_configuration_assignment, (signed char)5, (model::record_batch_type)6>, cluster::controller_command<cluster::non_replicable_topic, signed char, (signed char)6, (model::record_batch_type)6> >) const::'lambda'(cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)2, (model::record_batch_type)6>)::operator()(cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)2, (model::record_batch_type)6>) const::'lambda'(std::__1::error_code), seastar::future<std::__1::error_code> >(std::__1::variant<cluster::controller_command<model::topic_namespace, cluster::topic_configuration_assignment, (signed char)0, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, model::topic_namespace, (signed char)1, (model::record_batch_type)6>, cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)2, (model::record_batch_type)6>, cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)3, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, cluster::incremental_topic_updates, (signed char)4, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, cluster::create_partititions_configuration_assignment, (signed char)5, (model::record_batch_type)6>, cluster::controller_command<cluster::non_replicable_topic, signed char, (signed char)6, (model::record_batch_type)6> >&&)::'lambda'(seastar::internal::promise_base_with_type<std::__1::error_code>&&, auto cluster::topic_updates_dispatcher::apply_update(model::record_batch)::$_0::operator()<std::__1::variant<cluster::controller_command<model::topic_namespace, cluster::topic_configuration_assignment, (signed char)0, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, model::topic_namespace, (signed char)1, (model::record_batch_type)6>, cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)2, (model::record_batch_type)6>, cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)3, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, cluster::incremental_topic_updates, (signed char)4, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, cluster::create_partititions_configuration_assignment, (signed char)5, (model::record_batch_type)6>, cluster::controller_command<cluster::non_replicable_topic, signed char, (signed char)6, (model::record_batch_type)6> > >(std::__1::variant<cluster::controller_command<model::topic_namespace, cluster::topic_configuration_assignment, (signed char)0, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, model::topic_namespace, (signed char)1, (model::record_batch_type)6>, cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)2, (model::record_batch_type)6>, cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)3, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, cluster::incremental_topic_updates, (signed char)4, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, cluster::create_partititions_configuration_assignment, (signed char)5, (model::record_batch_type)6>, cluster::controller_command<cluster::non_replicable_topic, signed char, (signed char)6, (model::record_batch_type)6> >) const::'lambda'(cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)2, (model::record_batch_type)6>)::operator()(cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)2, (model::record_batch_type)6>) const::'lambda'(std::__1::error_code)&, seastar::future_state<std::__1::error_code>&&)::operator()(seastar::internal::promise_base_with_type<std::__1::error_code>&&, auto cluster::topic_updates_dispatcher::apply_update(model::record_batch)::$_0::operator()<std::__1::variant<cluster::controller_command<model::topic_namespace, cluster::topic_configuration_assignment, (signed char)0, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, model::topic_namespace, (signed char)1, (model::record_batch_type)6>, cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)2, (model::record_batch_type)6>, cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)3, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, cluster::incremental_topic_updates, (signed char)4, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, cluster::create_partititions_configuration_assignment, (signed char)5, (model::record_batch_type)6>, cluster::controller_command<cluster::non_replicable_topic, signed char, (signed char)6, (model::record_batch_type)6> > >(std::__1::variant<cluster::controller_command<model::topic_namespace, cluster::topic_configuration_assignment, (signed char)0, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, model::topic_namespace, (signed char)1, (model::record_batch_type)6>, cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)2, (model::record_batch_type)6>, cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)3, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, cluster::incremental_topic_updates, (signed char)4, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, cluster::create_partititions_configuration_assignment, (signed char)5, (model::record_batch_type)6>, cluster::controller_command<cluster::non_replicable_topic, signed char, (signed char)6, (model::record_batch_type)6> >) const::'lambda'(cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)2, (model::record_batch_type)6>)::operator()(cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)2, (model::record_batch_type)6>) const::'lambda'(std::__1::error_code)&, seastar::future_state<std::__1::error_code>&&) const::'lambda'()>(seastar::internal::promise_base_with_type<std::__1::error_code>&&, std::__1::variant<cluster::controller_command<model::topic_namespace, cluster::topic_configuration_assignment, (signed char)0, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, model::topic_namespace, (signed char)1, (model::record_batch_type)6>, cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)2, (model::record_batch_type)6>, cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)3, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, cluster::incremental_topic_updates, (signed char)4, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, cluster::create_partititions_configuration_assignment, (signed char)5, (model::record_batch_type)6>, cluster::controller_command<cluster::non_replicable_topic, signed char, (signed char)6, (model::record_batch_type)6> >&&) at /vectorized/include/seastar/core/future.hh:2122
 (inlined by) seastar::future<std::__1::error_code> seastar::future<std::__1::error_code>::then_impl_nrvo<auto cluster::topic_updates_dispatcher::apply_update(model::record_batch)::$_0::operator()<std::__1::variant<cluster::controller_command<model::topic_namespace, cluster::topic_configuration_assignment, (signed char)0, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, model::topic_namespace, (signed char)1, (model::record_batch_type)6>, cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)2, (model::record_batch_type)6>, cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)3, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, cluster::incremental_topic_updates, (signed char)4, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, cluster::create_partititions_configuration_assignment, (signed char)5, (model::record_batch_type)6>, cluster::controller_command<cluster::non_replicable_topic, signed char, (signed char)6, (model::record_batch_type)6> > >(std::__1::variant<cluster::controller_command<model::topic_namespace, cluster::topic_configuration_assignment, (signed char)0, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, model::topic_namespace, (signed char)1, (model::record_batch_type)6>, cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)2, (model::record_batch_type)6>, cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)3, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, cluster::incremental_topic_updates, (signed char)4, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, cluster::create_partititions_configuration_assignment, (signed char)5, (model::record_batch_type)6>, cluster::controller_command<cluster::non_replicable_topic, signed char, (signed char)6, (model::record_batch_type)6> >) const::'lambda'(cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)2, (model::record_batch_type)6>)::operator()(cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)2, (model::record_batch_type)6>) const::'lambda'(std::__1::error_code), seastar::future<std::__1::error_code> >(std::__1::variant<cluster::controller_command<model::topic_namespace, cluster::topic_configuration_assignment, (signed char)0, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, model::topic_namespace, (signed char)1, (model::record_batch_type)6>, cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)2, (model::record_batch_type)6>, cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)3, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, cluster::incremental_topic_updates, (signed char)4, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, cluster::create_partititions_configuration_assignment, (signed char)5, (model::record_batch_type)6>, cluster::controller_command<cluster::non_replicable_topic, signed char, (signed char)6, (model::record_batch_type)6> >&&)::'lambda'(seastar::internal::promise_base_with_type<std::__1::error_code>&&, auto cluster::topic_updates_dispatcher::apply_update(model::record_batch)::$_0::operator()<std::__1::variant<cluster::controller_command<model::topic_namespace, cluster::topic_configuration_assignment, (signed char)0, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, model::topic_namespace, (signed char)1, (model::record_batch_type)6>, cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)2, (model::record_batch_type)6>, cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)3, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, cluster::incremental_topic_updates, (signed char)4, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, cluster::create_partititions_configuration_assignment, (signed char)5, (model::record_batch_type)6>, cluster::controller_command<cluster::non_replicable_topic, signed char, (signed char)6, (model::record_batch_type)6> > >(std::__1::variant<cluster::controller_command<model::topic_namespace, cluster::topic_configuration_assignment, (signed char)0, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, model::topic_namespace, (signed char)1, (model::record_batch_type)6>, cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)2, (model::record_batch_type)6>, cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)3, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, cluster::incremental_topic_updates, (signed char)4, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, cluster::create_partititions_configuration_assignment, (signed char)5, (model::record_batch_type)6>, cluster::controller_command<cluster::non_replicable_topic, signed char, (signed char)6, (model::record_batch_type)6> >) const::'lambda'(cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)2, (model::record_batch_type)6>)::operator()(cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)2, (model::record_batch_type)6>) const::'lambda'(std::__1::error_code)&, seastar::future_state<std::__1::error_code>&&)::operator()(seastar::internal::promise_base_with_type<std::__1::error_code>&&, auto cluster::topic_updates_dispatcher::apply_update(model::record_batch)::$_0::operator()<std::__1::variant<cluster::controller_command<model::topic_namespace, cluster::topic_configuration_assignment, (signed char)0, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, model::topic_namespace, (signed char)1, (model::record_batch_type)6>, cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)2, (model::record_batch_type)6>, cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)3, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, cluster::incremental_topic_updates, (signed char)4, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, cluster::create_partititions_configuration_assignment, (signed char)5, (model::record_batch_type)6>, cluster::controller_command<cluster::non_replicable_topic, signed char, (signed char)6, (model::record_batch_type)6> > >(std::__1::variant<cluster::controller_command<model::topic_namespace, cluster::topic_configuration_assignment, (signed char)0, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, model::topic_namespace, (signed char)1, (model::record_batch_type)6>, cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)2, (model::record_batch_type)6>, cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)3, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, cluster::incremental_topic_updates, (signed char)4, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, cluster::create_partititions_configuration_assignment, (signed char)5, (model::record_batch_type)6>, cluster::controller_command<cluster::non_replicable_topic, signed char, (signed char)6, (model::record_batch_type)6> >) const::'lambda'(cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)2, (model::record_batch_type)6>)::operator()(cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)2, (model::record_batch_type)6>) const::'lambda'(std::__1::error_code)&, seastar::future_state<std::__1::error_code>&&) const at /vectorized/include/seastar/core/future.hh:1575
 (inlined by) seastar::continuation<seastar::internal::promise_base_with_type<std::__1::error_code>, auto cluster::topic_updates_dispatcher::apply_update(model::record_batch)::$_0::operator()<std::__1::variant<cluster::controller_command<model::topic_namespace, cluster::topic_configuration_assignment, (signed char)0, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, model::topic_namespace, (signed char)1, (model::record_batch_type)6>, cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)2, (model::record_batch_type)6>, cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)3, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, cluster::incremental_topic_updates, (signed char)4, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, cluster::create_partititions_configuration_assignment, (signed char)5, (model::record_batch_type)6>, cluster::controller_command<cluster::non_replicable_topic, signed char, (signed char)6, (model::record_batch_type)6> > >(std::__1::variant<cluster::controller_command<model::topic_namespace, cluster::topic_configuration_assignment, (signed char)0, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, model::topic_namespace, (signed char)1, (model::record_batch_type)6>, cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)2, (model::record_batch_type)6>, cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)3, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, cluster::incremental_topic_updates, (signed char)4, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, cluster::create_partititions_configuration_assignment, (signed char)5, (model::record_batch_type)6>, cluster::controller_command<cluster::non_replicable_topic, signed char, (signed char)6, (model::record_batch_type)6> >) const::'lambda'(cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)2, (model::record_batch_type)6>)::operator()(cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)2, (model::record_batch_type)6>) const::'lambda'(std::__1::error_code), seastar::future<std::__1::error_code> seastar::future<std::__1::error_code>::then_impl_nrvo<auto cluster::topic_updates_dispatcher::apply_update(model::record_batch)::$_0::operator()<std::__1::variant<cluster::controller_command<model::topic_namespace, cluster::topic_configuration_assignment, (signed char)0, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, model::topic_namespace, (signed char)1, (model::record_batch_type)6>, cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)2, (model::record_batch_type)6>, cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)3, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, cluster::incremental_topic_updates, (signed char)4, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, cluster::create_partititions_configuration_assignment, (signed char)5, (model::record_batch_type)6>, cluster::controller_command<cluster::non_replicable_topic, signed char, (signed char)6, (model::record_batch_type)6> > >(std::__1::variant<cluster::controller_command<model::topic_namespace, cluster::topic_configuration_assignment, (signed char)0, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, model::topic_namespace, (signed char)1, (model::record_batch_type)6>, cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)2, (model::record_batch_type)6>, cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)3, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, cluster::incremental_topic_updates, (signed char)4, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, cluster::create_partititions_configuration_assignment, (signed char)5, (model::record_batch_type)6>, cluster::controller_command<cluster::non_replicable_topic, signed char, (signed char)6, (model::record_batch_type)6> >) const::'lambda'(cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)2, (model::record_batch_type)6>)::operator()(cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)2, (model::record_batch_type)6>) const::'lambda'(std::__1::error_code), seastar::future<std::__1::error_code> >(std::__1::variant<cluster::controller_command<model::topic_namespace, cluster::topic_configuration_assignment, (signed char)0, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, model::topic_namespace, (signed char)1, (model::record_batch_type)6>, cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)2, (model::record_batch_type)6>, cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)3, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, cluster::incremental_topic_updates, (signed char)4, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, cluster::create_partititions_configuration_assignment, (signed char)5, (model::record_batch_type)6>, cluster::controller_command<cluster::non_replicable_topic, signed char, (signed char)6, (model::record_batch_type)6> >&&)::'lambda'(seastar::internal::promise_base_with_type<std::__1::error_code>&&, auto cluster::topic_updates_dispatcher::apply_update(model::record_batch)::$_0::operator()<std::__1::variant<cluster::controller_command<model::topic_namespace, cluster::topic_configuration_assignment, (signed char)0, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, model::topic_namespace, (signed char)1, (model::record_batch_type)6>, cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)2, (model::record_batch_type)6>, cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)3, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, cluster::incremental_topic_updates, (signed char)4, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, cluster::create_partititions_configuration_assignment, (signed char)5, (model::record_batch_type)6>, cluster::controller_command<cluster::non_replicable_topic, signed char, (signed char)6, (model::record_batch_type)6> > >(std::__1::variant<cluster::controller_command<model::topic_namespace, cluster::topic_configuration_assignment, (signed char)0, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, model::topic_namespace, (signed char)1, (model::record_batch_type)6>, cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)2, (model::record_batch_type)6>, cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)3, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, cluster::incremental_topic_updates, (signed char)4, (model::record_batch_type)6>, cluster::controller_command<model::topic_namespace, cluster::create_partititions_configuration_assignment, (signed char)5, (model::record_batch_type)6>, cluster::controller_command<cluster::non_replicable_topic, signed char, (signed char)6, (model::record_batch_type)6> >) const::'lambda'(cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)2, (model::record_batch_type)6>)::operator()(cluster::controller_command<model::ntp, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard> >, (signed char)2, (model::record_batch_type)6>) const::'lambda'(std::__1::error_code)&, seastar::future_state<std::__1::error_code>&&), std::__1::error_code>::run_and_dispose() at /vectorized/include/seastar/core/future.hh:767

It may or may not be the same as #5558. The problem in 5558 is that we are manipulating the partiton_allocator state on shard 0 concurrently from other shards: a race condition. In that case, the assert itself fired on one of these other (non-0) shards, so it is pretty clear what is going on.

In this case we are applying updates from the controller STM to the allocator and this happens on shard 0, so it is different in that sense. However, it is entirely possible that the race in 5558 does cause this and it just happens that shard 0 gets the failure in this case, i.e., an earlier race caused the allocator to end up with 1 fewer partitions in its weights counter/allocated counter but that goes unnoticed until later we go below 0 when some partitions are moved. It's a shame the assert was broken so we can't see that detail.

[dotnwat]

It may or may not be the same as https://github.com/redpanda-data/redpanda/issues/5558. The problem in 5558 is that we are manipulating the partiton_allocator state on shard 0 concurrently from other shards:

yikes. glad you caught that.