raft: `Exceptional future ignored: seastar::semaphore_timed_out`

[dotnwat]

cluster-0 redpanda INFO redpanda::main - application.cc:160 - Redpanda v21.11.13 - cd5a271896c89457306e8da7fc15882cc23ad15b-dirty

INFO  2022-04-22 06:36:41,592 [shard 3] raft - [group_id:10, {kafka/atomic-state-Active-ActProcVal-Endplates/0}] consensus.cc:145 - [heartbeats_majority] Stepping down as leader in term 44, dirty offset 37718
INFO  2022-04-22 06:36:45,741 [shard 0] cluster - health_monitor_backend.cc:263 - timed out when refreshing cluster health state, falling back to previous cluster health snapshot
INFO  2022-04-22 06:36:45,742 [shard 0] cluster - metadata_dissemination_service.cc:337 - unable to retrieve cluster health report - Timeout occurred while processing request
INFO  2022-04-22 06:36:50,111 [shard 0] cluster - leader_balancer.cc:92 - Leader balancer: controller leadership lost
INFO  2022-04-22 06:36:50,778 [shard 0] raft - [group_id:7, {kafka_internal/id_allocator/0}] consensus.cc:1444 - Received vote request with larger term from node {id: {1}, revision: {30}}, received 88, current 87
WARN  2022-04-22 06:36:50,839 [shard 0] seastar - Exceptional future ignored: seastar::semaphore_timed_out (Semaphore timedout), backtrace: 0x3b5f5f4 0x384b812 0x170e516 0x38dac94 0x38de157 0x38db535 0x382d359 0x382b3f0 0x11c8454 0x3c21f8c /opt/redpanda/lib/libc.so.6+0x27b74 0x11c522d
   --------
   seastar::continuation<seastar::internal::promise_base_with_type<void>, raft::prevote_stm::dispatch_prevote(raft::vnode)::$_0::operator()() const::'lambda'(seastar::future<boost::outcome_v2::basic_result<raft::vote_reply, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<raft::vote_reply, std::__1::error_code, void> > >), seastar::futurize<seastar::future<void> >::type seastar::future<boost::outcome_v2::basic_result<raft::vote_reply, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<raft::vote_reply, std::__1::error_code, void> > >::then_wrapped_nrvo<seastar::future<void>, raft::prevote_stm::dispatch_prevote(raft::vnode)::$_0::operator()() const::'lambda'(seastar::future<boost::outcome_v2::basic_result<raft::vote_reply, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<raft::vote_reply, std::__1::error_code, void> > >)>(raft::prevote_stm::dispatch_prevote(raft::vnode)::$_0::operator()() const::'lambda'(seastar::future<boost::outcome_v2::basic_result<raft::vote_reply, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<raft::vote_reply, std::__1::error_code, void> > >)&&)::'lambda'(seastar::internal::promise_base_with_type<void>&&, raft::prevote_stm::dispatch_prevote(raft::vnode)::$_0::operator()() const::'lambda'(seastar::future<boost::outcome_v2::basic_result<raft::vote_reply, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<raft::vote_reply, std::__1::error_code, void> > >)&, seastar::future_state<boost::outcome_v2::basic_result<raft::vote_reply, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<raft::vote_reply, std::__1::error_code, void> > >&&), boost::outcome_v2::basic_result<raft::vote_reply, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<raft::vote_reply, std::__1::error_code, void> > >
   --------
   seastar::continuation<seastar::internal::promise_base_with_type<void>, seastar::future<void>::finally_body<auto seastar::internal::invoke_func_with_gate<raft::prevote_stm::dispatch_prevote(raft::vnode)::$_0>(seastar::gate&, raft::prevote_stm::dispatch_prevote(raft::vnode)::$_0&&)::'lambda'(), false>, seastar::futurize<raft::prevote_stm::dispatch_prevote(raft::vnode)::$_0>::type seastar::future<void>::then_wrapped_nrvo<seastar::future<void>, seastar::future<void>::finally_body<auto seastar::internal::invoke_func_with_gate<raft::prevote_stm::dispatch_prevote(raft::vnode)::$_0>(seastar::gate&, raft::prevote_stm::dispatch_prevote(raft::vnode)::$_0&&)::'lambda'(), false> >(seastar::future<void>::finally_body<auto seastar::internal::invoke_func_with_gate<raft::prevote_stm::dispatch_prevote(raft::vnode)::$_0>(seastar::gate&, raft::prevote_stm::dispatch_prevote(raft::vnode)::$_0&&)::'lambda'(), false>&&)::'lambda'(seastar::internal::promise_base_with_type<void>&&, seastar::future<void>::finally_body<auto seastar::internal::invoke_func_with_gate<raft::prevote_stm::dispatch_prevote(raft::vnode)::$_0>(seastar::gate&, raft::prevote_stm::dispatch_prevote(raft::vnode)::$_0&&)::'lambda'(), false>&, seastar::future_state<seastar::internal::monostate>&&), void>
INFO  2022-04-22 06:36:50,860 [shard 0] raft - [group_id:15, {kafka/atomic-state-Active-Recipe-External-All-BasicQuantities/0}] consensus.cc:1444 - Received vote request with larger term from node {id: {1}, revision: {47}}, received 67, current 66
INFO  2022-04-22 06:36:50,889 [shard 0] cluster - health_monitor_backend.cc:263 - timed out when refreshing cluster health state, falling back to previous cluster health snapshot
INFO  2022-04-22 06:36:50,924 [shard 1] raft - [group_id:16, {kafka/atomic-state-Active-Recipe-External-All-MainDrive/0}] consensus.cc:1444 - Received vote request with larger term from node {id: {1}, revision: {48}}, received 55, current 54

JIRA Link: CORE-891

[dotnwat]

@mmaslankaprv look familiar? seems to be coming from prevote stm?

[mmaslankaprv]

i haven't seen that before

[dotnwat]

Duplicate of https://github.com/redpanda-data/redpanda/issues/4787