The whole cluster collapsed after returning a node to a cluster with fewer cores

[ermakov-oleg]

Version & Environment

Redpanda version: (use rpk version):

rpk version
v21.11.2 (rev f58e69b)

What went wrong?

The whole cluster collapsed after returning a node to a cluster with fewer cores.

Logs on leader:

INFO  2022-02-01 10:37:20,811 [shard 0] cluster - members_manager.cc:453 - Processing node '4' join request
ERROR 2022-02-01 10:37:20,895 [shard 0] assert - Assert failure: (../../../src/v/cluster/scheduling/allocation_node.cc:71) 'core_count >= cpus()' decreasing node core count is not supported, current core count 6 > requested core count 1
ERROR 2022-02-01 10:37:20,895 [shard 0] assert - Backtrace below:
0x3a3f3a4 0x2f429cb 0x2f3d014 0x2af95ca 0x2afbe89 0x2afa843 0x2c4d41c 0x2c4c235 0x2c4b6d9 0x174b77a 0x1749e19 0x37ba9f4 0x37bdeb7 0x37bb295 0x370d0b9 0x370b150 0x116ee94 0x3b01d3c /opt/redpanda/lib/libc.so.6+0x27b74 0x116bced
   --------
   seastar::continuation<seastar::internal::promise_base_with_type<void>, raft::state_machine::apply()::$_5, seastar::future<void> seastar::future<model::record_batch_reader>::then_impl_nrvo<raft::state_machine::apply()::$_5, seastar::future<void> >(raft::state_machine::apply()::$_5&&)::'lambda'(seastar::internal::promise_base_with_type<void>&&, raft::state_machine::apply()::$_5&, seastar::future_state<model::record_batch_reader>&&), model::record_batch_reader>
   --------
   seastar::continuation<seastar::internal::promise_base_with_type<void>, seastar::future<void> seastar::future<void>::handle_exception<raft::state_machine::apply()::$_6>(raft::state_machine::apply()::$_6&&)::'lambda'(raft::state_machine::apply()::$_6&&), seastar::futurize<raft::state_machine::apply()::$_6>::type seastar::future<void>::then_wrapped_nrvo<seastar::future<void>, seastar::future<void> seastar::future<void>::handle_exception<raft::state_machine::apply()::$_6>(raft::state_machine::apply()::$_6&&)::'lambda'(raft::state_machine::apply()::$_6&&)>(seastar::future<void> seastar::future<void>::handle_exception<raft::state_machine::apply()::$_6>(raft::state_machine::apply()::$_6&&)::'lambda'(raft::state_machine::apply()::$_6&&)&&)::'lambda'(seastar::internal::promise_base_with_type<void>&&, seastar::future<void> seastar::future<void>::handle_exception<raft::state_machine::apply()::$_6>(raft::state_machine::apply()::$_6&&)::'lambda'(raft::state_machine::apply()::$_6&&)&, seastar::future_state<seastar::internal::monostate>&&), void>
   --------
   seastar::internal::do_until_state<raft::state_machine::start()::$_0::operator()() const::'lambda'(), raft::state_machine::start()::$_0::operator()() const::'lambda0'()>
   --------
   seastar::continuation<seastar::internal::promise_base_with_type<void>, seastar::future<void>::finally_body<auto seastar::internal::invoke_func_with_gate<raft::state_machine::start()::$_0>(seastar::gate&, raft::state_machine::start()::$_0&&)::'lambda'(), false>, seastar::futurize<raft::state_machine::start()::$_0>::type seastar::future<void>::then_wrapped_nrvo<seastar::future<void>, seastar::future<void>::finally_body<auto seastar::internal::invoke_func_with_gate<raft::state_machine::start()::$_0>(seastar::gate&, raft::state_machine::start()::$_0&&)::'lambda'(), false> >(seastar::future<void>::finally_body<auto seastar::internal::invoke_func_with_gate<raft::state_machine::start()::$_0>(seastar::gate&, raft::state_machine::start()::$_0&&)::'lambda'(), false>&&)::'lambda'(seastar::internal::promise_base_with_type<void>&&, seastar::future<void>::finally_body<auto seastar::internal::invoke_func_with_gate<raft::state_machine::start()::$_0>(seastar::gate&, raft::state_machine::start()::$_0&&)::'lambda'(), false>&, seastar::future_state<seastar::internal::monostate>&&), void>

As a result, the cluster was left without a leader.

After restarting the node, the logs contain the following entries:

health_monitor_backend.cc:256 - error refreshing cluster health state - Currently there is no leader controller elected in the cluster

The leader of the cluster was never chosen ...

What should have happened instead?

Add a check for cpu count when entering a node into a cluster, or allow to change node configuration in a cluster.

How to reproduce the issue?

1. Create cluster with 5 nodes
2. Decommission one node
3. Change cpu count on a node
4. Revert node to the cluster

JIRA Link: CORE-826

[dotnwat]

@mmaslankaprv do you think this is a regression or an edge case? we handle (ie I think reject) restarting nodes with a reduced number of cores, but this issue looks like the cluster is holding onto previous state for a decommissioned node.