search

redpanda-data / redpanda

Redpanda is a streaming data platform for developers. Kafka API compatible. 10x faster. No ZooKeeper. No JVM!
https://redpanda.com
9.33k stars 574 forks source link

Exceptional future ignored in prevote_stm #5813

Open Lazin opened 2 years ago

Lazin commented 2 years ago

Version & Environment

Redpanda version: tip of dev

INFO  2022-08-03 15:26:11,610 [shard 0] raft - [group_id:7, {kafka/__consumer_offsets/2}] vote_stm.cc:255 - becoming the leader term:8
INFO  2022-08-03 15:26:11,610 [shard 0] storage - segment.cc:614 - Creating new segment /var/lib/redpanda/data/kafka/__consumer_offsets/2_40/7-8-v1.log
INFO  2022-08-03 15:26:11,610 [shard 0] cluster - ntp: {kafka/__consumer_offsets/2} - archival_metadata_stm.cc:408 - creating snapshot at offset: 6, remote start_offset: -9223372036854775808, last_offset: -9223372036854775808
INFO  2022-08-03 15:26:11,667 [shard 0] raft - [group_id:7, {kafka/__consumer_offsets/2}] vote_stm.cc:270 - became the leader term:8
INFO  2022-08-03 15:26:12,141 [shard 0] raft - [group_id:0, {redpanda/controller/0}] vote_stm.cc:76 - vote reply from {id: {2}, revision: {0}} - {term:{5}, target_node_id{id: {0}, revision: {0}}, vote_granted: 1, log_ok:1}
INFO  2022-08-03 15:26:12,142 [shard 0] raft - [group_id:0, {redpanda/controller/0}] vote_stm.cc:255 - becoming the leader term:5
INFO  2022-08-03 15:26:12,142 [shard 0] cluster - leader_balancer.cc:89 - Leader balancer: controller leadership detected. Starting rebalancer in 30 seconds
INFO  2022-08-03 15:26:12,147 [shard 0] storage - segment.cc:614 - Creating new segment /var/lib/redpanda/data/redpanda/controller/0_0/537-5-v1.log
INFO  2022-08-03 15:26:12,210 [shard 0] raft - [group_id:0, {redpanda/controller/0}] vote_stm.cc:270 - became the leader term:5
INFO  2022-08-03 15:26:12,337 [shard 0] raft - [group_id:9, {kafka/__consumer_offsets/4}] consensus.cc:1567 - Received vote request with larger term from node {id: {2}, revision: {40}}, received 6, current 5
INFO  2022-08-03 15:26:12,337 [shard 0] raft - [group_id:6, {kafka/__consumer_offsets/1}] consensus.cc:1567 - Received vote request with larger term from node {id: {2}, revision: {40}}, received 6, current 5
INFO  2022-08-03 15:26:12,383 [shard 0] storage - segment.cc:614 - Creating new segment /var/lib/redpanda/data/kafka/__consumer_offsets/4_40/5-6-v1.log
INFO  2022-08-03 15:26:12,383 [shard 0] storage - segment.cc:614 - Creating new segment /var/lib/redpanda/data/kafka/__consumer_offsets/1_40/5-6-v1.log
INFO  2022-08-03 15:26:12,383 [shard 0] cluster - ntp: {kafka/__consumer_offsets/4} - archival_metadata_stm.cc:408 - creating snapshot at offset: 4, remote start_offset: -9223372036854775808, last_offset: -9223372036854775808
INFO  2022-08-03 15:26:12,383 [shard 0] cluster - ntp: {kafka/__consumer_offsets/1} - archival_metadata_stm.cc:408 - creating snapshot at offset: 4, remote start_offset: -9223372036854775808, last_offset: -9223372036854775808
INFO  2022-08-03 15:26:12,446 [shard 0] raft - [group_id:4, {kafka_internal/id_allocator/0}] consensus.cc:1567 - Received vote request with larger term from node {id: {2}, revision: {39}}, received 6, current 5
INFO  2022-08-03 15:26:12,869 [shard 0] storage - segment.cc:614 - Creating new segment /var/lib/redpanda/data/kafka_internal/id_allocator/0_39/7-6-v1.log
INFO  2022-08-03 15:26:13,191 [shard 0] raft - [group_id:1, {kafka/__redpanda.cloud.sla_verification/0}] consensus.cc:1567 - Received vote request with larger term from node {id: {2}, revision: {36}}, received 10, current 9
INFO  2022-08-03 15:26:13,321 [shard 0] storage - segment.cc:614 - Creating new segment /var/lib/redpanda/data/kafka/__redpanda.cloud.sla_verification/0_36/449796-10-v1.log
INFO  2022-08-03 15:26:13,322 [shard 0] cluster - ntp: {kafka/__redpanda.cloud.sla_verification/0} - archival_metadata_stm.cc:408 - creating snapshot at offset: 449794, remote start_offset: -9223372036854775808, last_offset: -9223372036854775808
WARN  2022-08-03 15:26:13,606 [shard 0] seastar - Exceptional future ignored: seastar::semaphore_timed_out (Semaphore timedout), backtrace: 0x49b4134 0x46a9942 0x1e509f6 0x476499f 0x4768677 0x4765a49 0x4685ec1 0x4683fdf 0x182ec44 0x4a66140 /opt/redpanda/lib/libc.so.6+0x27b74 0x182bb8d
   --------
   seastar::continuation<seastar::internal::promise_base_with_type<void>, raft::prevote_stm::dispatch_prevote(raft::vnode)::$_1::operator()() const::'lambda'(seastar::future<boost::outcome_v2::basic_result<raft::vote_reply, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<raft::vote_reply, std::__1::error_code, void> > >), seastar::futurize<seastar::future<void> >::type seastar::future<boost::outcome_v2::basic_result<raft::vote_reply, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<raft::vote_reply, std::__1::error_code, void> > >::then_wrapped_nrvo<seastar::future<void>, raft::prevote_stm::dispatch_prevote(raft::vnode)::$_1::operator()() const::'lambda'(seastar::future<boost::outcome_v2::basic_result<raft::vote_reply, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<raft::vote_reply, std::__1::error_code, void> > >)>(raft::prevote_stm::dispatch_prevote(raft::vnode)::$_1::operator()() const::'lambda'(seastar::future<boost::outcome_v2::basic_result<raft::vote_reply, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<raft::vote_reply, std::__1::error_code, void> > >)&&)::'lambda'(seastar::internal::promise_base_with_type<void>&&, raft::prevote_stm::dispatch_prevote(raft::vnode)::$_1::operator()() const::'lambda'(seastar::future<boost::outcome_v2::basic_result<raft::vote_reply, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<raft::vote_reply, std::__1::error_code, void> > >)&, seastar::future_state<boost::outcome_v2::basic_result<raft::vote_reply, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<raft::vote_reply, std::__1::error_code, void> > >&&), boost::outcome_v2::basic_result<raft::vote_reply, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<raft::vote_reply, std::__1::error_code, void> > >
   --------
   seastar::continuation<seastar::internal::promise_base_with_type<void>, seastar::future<void>::finally_body<auto seastar::internal::invoke_func_with_gate<raft::prevote_stm::dispatch_prevote(raft::vnode)::$_1>(seastar::gate&, raft::prevote_stm::dispatch_prevote(raft::vnode)::$_1&&)::'lambda'(), false>, seastar::futurize<raft::prevote_stm::dispatch_prevote(raft::vnode)::$_1>::type seastar::future<void>::then_wrapped_nrvo<seastar::future<void>, seastar::future<void>::finally_body<auto seastar::internal::invoke_func_with_gate<raft::prevote_stm::dispatch_prevote(raft::vnode)::$_1>(seastar::gate&, raft::prevote_stm::dispatch_prevote(raft::vnode)::$_1&&)::'lambda'(), false> >(seastar::future<void>::finally_body<auto seastar::internal::invoke_func_with_gate<raft::prevote_stm::dispatch_prevote(raft::vnode)::$_1>(seastar::gate&, raft::prevote_stm::dispatch_prevote(raft::vnode)::$_1&&)::'lambda'(), false>&&)::'lambda'(seastar::internal::promise_base_with_type<void>&&, seastar::future<void>::finally_body<auto seastar::internal::invoke_func_with_gate<raft::prevote_stm::dispatch_prevote(raft::vnode)::$_1>(seastar::gate&, raft::prevote_stm::dispatch_prevote(raft::vnode)::$_1&&)::'lambda'(), false>&, seastar::future_state<seastar::internal::monostate>&&), void>

Decoded backtrace

[Backtrace #0]
addr2line: DWARF error: invalid or unhandled FORM value: 0x25
seastar::current_backtrace() at backtrace.cc:?
seastar::report_failed_future(seastar::future_state_base::any&&) at future.cc:?
addr2line: DWARF error: could not find variable specification at offset 4b
addr2line: DWARF error: could not find variable specification at offset 814c
addr2line: DWARF error: could not find variable specification at offset 10264
addr2line: DWARF error: could not find variable specification at offset 10425
seastar::future_state_base::any::check_failure() at /home/lazin/.data/rp1-vbuild/release/clang/rp_deps_install/include/seastar/core/future.hh:574
 (inlined by) seastar::future_state<boost::outcome_v2::basic_result<raft::vote_reply, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<raft::vote_reply, std::__1::error_code, void> > >::clear() at /home/lazin/.data/rp1-vbuild/release/clang/rp_deps_install/include/seastar/core/future.hh:616
 (inlined by) ~future_state at /home/lazin/.data/rp1-vbuild/release/clang/rp_deps_install/include/seastar/core/future.hh:621
 (inlined by) ~future at /home/lazin/.data/rp1-vbuild/release/clang/rp_deps_install/include/seastar/core/future.hh:1344
 (inlined by) operator() at /home/lazin/workspace/rp1/src/v/raft/prevote_stm.cc:128
 (inlined by) operator() at /home/lazin/.data/rp1-vbuild/release/clang/rp_deps_install/include/seastar/core/future.hh:1675
 (inlined by) void seastar::futurize<seastar::future<void> >::satisfy_with_result_of<seastar::future<boost::outcome_v2::basic_result<raft::vote_reply, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<raft::vote_reply, std::__1::error_code, void> > >::then_wrapped_nrvo<seastar::future<void>, raft::prevote_stm::dispatch_prevote(raft::vnode)::$_1::operator()() const::{lambda(seastar::future<boost::outcome_v2::basic_result<raft::vote_reply, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<raft::vote_reply, std::__1::error_code, void> > >)#1}>(raft::prevote_stm::dispatch_prevote(raft::vnode)::$_1::operator()() const::{lambda(seastar::future<boost::outcome_v2::basic_result<raft::vote_reply, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<raft::vote_reply, std::__1::error_code, void> > >)#1}&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, {lambda(seastar::future<boost::outcome_v2::basic_result<raft::vote_reply, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<raft::vote_reply, std::__1::error_code, void> > >)#1}&, seastar::future_state<boost::outcome_v2::basic_result<raft::vote_reply, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<raft::vote_reply, std::__1::error_code, void> > >&&)#1}::operator()(seastar::internal::promise_base_with_type<void>, seastar::internal::promise_base_with_type<void>&&, seastar::future_state<boost::outcome_v2::basic_result<raft::vote_reply, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<raft::vote_reply, std::__1::error_code, void> > >) const::{lambda()#1}>(seastar::internal::promise_base_with_type<void>, raft::prevote_stm::dispatch_prevote(raft::vnode)::$_1::operator()() const::{lambda(seastar::future<boost::outcome_v2::basic_result<raft::vote_reply, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<raft::vote_reply, std::__1::error_code, void> > >)#1}&&) at /home/lazin/.data/rp1-vbuild/release/clang/rp_deps_install/include/seastar/core/future.hh:2127
 (inlined by) operator() at /home/lazin/.data/rp1-vbuild/release/clang/rp_deps_install/include/seastar/core/future.hh:1674
 (inlined by) seastar::continuation<seastar::internal::promise_base_with_type<void>, raft::prevote_stm::dispatch_prevote(raft::vnode)::$_1::operator()() const::{lambda(seastar::future<boost::outcome_v2::basic_result<raft::vote_reply, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<raft::vote_reply, std::__1::error_code, void> > >)#1}, seastar::future<boost::outcome_v2::basic_result<raft::vote_reply, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<raft::vote_reply, std::__1::error_code, void> > >::then_wrapped_nrvo<seastar::future<void>, {lambda(seastar::future<boost::outcome_v2::basic_result<raft::vote_reply, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<raft::vote_reply, std::__1::error_code, void> > >)#1}>({lambda(seastar::future<boost::outcome_v2::basic_result<raft::vote_reply, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<raft::vote_reply, std::__1::error_code, void> > >)#1}&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, {lambda(seastar::future<boost::outcome_v2::basic_result<raft::vote_reply, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<raft::vote_reply, std::__1::error_code, void> > >)#1}&, seastar::future_state<boost::outcome_v2::basic_result<raft::vote_reply, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<raft::vote_reply, std::__1::error_code, void> > >&&)#1}, boost::outcome_v2::basic_result<raft::vote_reply, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<raft::vote_reply, std::__1::error_code, void> > >::run_and_dispose() at /home/lazin/.data/rp1-vbuild/release/clang/rp_deps_install/include/seastar/core/future.hh:774
seastar::reactor::run_some_tasks() at reactor.cc:?
seastar::reactor::do_run() at reactor.cc:?
seastar::reactor::run() at reactor.cc:?
seastar::app_template::run_deprecated(int, char**, std::__1::function<void ()>&&) at app-template.cc:?
seastar::app_template::run(int, char**, std::__1::function<seastar::future<int> ()>&&) at app-template.cc:?
addr2line: DWARF error: could not find variable specification at offset 4b
addr2line: DWARF error: could not find variable specification at offset 2367b
addr2line: DWARF error: could not find variable specification at offset 43705
addr2line: DWARF error: could not find variable specification at offset 53acd
addr2line: DWARF error: could not find variable specification at offset 53bfd
addr2line: DWARF error: could not find variable specification at offset 54214
addr2line: DWARF error: could not find variable specification at offset 5439e
addr2line: DWARF error: could not find variable specification at offset 548c9
addr2line: DWARF error: could not find variable specification at offset 54a76
addr2line: DWARF error: could not find variable specification at offset 33dea0
addr2line: DWARF error: could not find variable specification at offset 33dfee
addr2line: DWARF error: could not find variable specification at offset 33e034
addr2line: DWARF error: could not find variable specification at offset 33ebc7
addr2line: DWARF error: could not find variable specification at offset 33ed8c
addr2line: DWARF error: could not find variable specification at offset 43977c
application::run(int, char**) at /home/lazin/workspace/rp1/src/v/redpanda/application.cc:224
main at ??:?
addr2line: '/opt/redpanda/lib/libc.so.6': No such file
/opt/redpanda/lib/libc.so.6 0x27b74 
_start at ??:?

What went wrong?

"Exceptional future ignored" log message in our test cluster.

What should have happened instead?

How to reproduce the issue?

Additional information

JIRA Link: CORE-988

dotnwat commented 2 years ago

@Lazin any more context for this? sha1, cdt, custom cluster, test/no-test?