Open Lazin opened 2 years ago
Redpanda version: tip of dev
INFO 2022-08-03 15:26:11,610 [shard 0] raft - [group_id:7, {kafka/__consumer_offsets/2}] vote_stm.cc:255 - becoming the leader term:8 INFO 2022-08-03 15:26:11,610 [shard 0] storage - segment.cc:614 - Creating new segment /var/lib/redpanda/data/kafka/__consumer_offsets/2_40/7-8-v1.log INFO 2022-08-03 15:26:11,610 [shard 0] cluster - ntp: {kafka/__consumer_offsets/2} - archival_metadata_stm.cc:408 - creating snapshot at offset: 6, remote start_offset: -9223372036854775808, last_offset: -9223372036854775808 INFO 2022-08-03 15:26:11,667 [shard 0] raft - [group_id:7, {kafka/__consumer_offsets/2}] vote_stm.cc:270 - became the leader term:8 INFO 2022-08-03 15:26:12,141 [shard 0] raft - [group_id:0, {redpanda/controller/0}] vote_stm.cc:76 - vote reply from {id: {2}, revision: {0}} - {term:{5}, target_node_id{id: {0}, revision: {0}}, vote_granted: 1, log_ok:1} INFO 2022-08-03 15:26:12,142 [shard 0] raft - [group_id:0, {redpanda/controller/0}] vote_stm.cc:255 - becoming the leader term:5 INFO 2022-08-03 15:26:12,142 [shard 0] cluster - leader_balancer.cc:89 - Leader balancer: controller leadership detected. Starting rebalancer in 30 seconds INFO 2022-08-03 15:26:12,147 [shard 0] storage - segment.cc:614 - Creating new segment /var/lib/redpanda/data/redpanda/controller/0_0/537-5-v1.log INFO 2022-08-03 15:26:12,210 [shard 0] raft - [group_id:0, {redpanda/controller/0}] vote_stm.cc:270 - became the leader term:5 INFO 2022-08-03 15:26:12,337 [shard 0] raft - [group_id:9, {kafka/__consumer_offsets/4}] consensus.cc:1567 - Received vote request with larger term from node {id: {2}, revision: {40}}, received 6, current 5 INFO 2022-08-03 15:26:12,337 [shard 0] raft - [group_id:6, {kafka/__consumer_offsets/1}] consensus.cc:1567 - Received vote request with larger term from node {id: {2}, revision: {40}}, received 6, current 5 INFO 2022-08-03 15:26:12,383 [shard 0] storage - segment.cc:614 - Creating new segment /var/lib/redpanda/data/kafka/__consumer_offsets/4_40/5-6-v1.log INFO 2022-08-03 15:26:12,383 [shard 0] storage - segment.cc:614 - Creating new segment /var/lib/redpanda/data/kafka/__consumer_offsets/1_40/5-6-v1.log INFO 2022-08-03 15:26:12,383 [shard 0] cluster - ntp: {kafka/__consumer_offsets/4} - archival_metadata_stm.cc:408 - creating snapshot at offset: 4, remote start_offset: -9223372036854775808, last_offset: -9223372036854775808 INFO 2022-08-03 15:26:12,383 [shard 0] cluster - ntp: {kafka/__consumer_offsets/1} - archival_metadata_stm.cc:408 - creating snapshot at offset: 4, remote start_offset: -9223372036854775808, last_offset: -9223372036854775808 INFO 2022-08-03 15:26:12,446 [shard 0] raft - [group_id:4, {kafka_internal/id_allocator/0}] consensus.cc:1567 - Received vote request with larger term from node {id: {2}, revision: {39}}, received 6, current 5 INFO 2022-08-03 15:26:12,869 [shard 0] storage - segment.cc:614 - Creating new segment /var/lib/redpanda/data/kafka_internal/id_allocator/0_39/7-6-v1.log INFO 2022-08-03 15:26:13,191 [shard 0] raft - [group_id:1, {kafka/__redpanda.cloud.sla_verification/0}] consensus.cc:1567 - Received vote request with larger term from node {id: {2}, revision: {36}}, received 10, current 9 INFO 2022-08-03 15:26:13,321 [shard 0] storage - segment.cc:614 - Creating new segment /var/lib/redpanda/data/kafka/__redpanda.cloud.sla_verification/0_36/449796-10-v1.log INFO 2022-08-03 15:26:13,322 [shard 0] cluster - ntp: {kafka/__redpanda.cloud.sla_verification/0} - archival_metadata_stm.cc:408 - creating snapshot at offset: 449794, remote start_offset: -9223372036854775808, last_offset: -9223372036854775808 WARN 2022-08-03 15:26:13,606 [shard 0] seastar - Exceptional future ignored: seastar::semaphore_timed_out (Semaphore timedout), backtrace: 0x49b4134 0x46a9942 0x1e509f6 0x476499f 0x4768677 0x4765a49 0x4685ec1 0x4683fdf 0x182ec44 0x4a66140 /opt/redpanda/lib/libc.so.6+0x27b74 0x182bb8d -------- seastar::continuation<seastar::internal::promise_base_with_type<void>, raft::prevote_stm::dispatch_prevote(raft::vnode)::$_1::operator()() const::'lambda'(seastar::future<boost::outcome_v2::basic_result<raft::vote_reply, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<raft::vote_reply, std::__1::error_code, void> > >), seastar::futurize<seastar::future<void> >::type seastar::future<boost::outcome_v2::basic_result<raft::vote_reply, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<raft::vote_reply, std::__1::error_code, void> > >::then_wrapped_nrvo<seastar::future<void>, raft::prevote_stm::dispatch_prevote(raft::vnode)::$_1::operator()() const::'lambda'(seastar::future<boost::outcome_v2::basic_result<raft::vote_reply, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<raft::vote_reply, std::__1::error_code, void> > >)>(raft::prevote_stm::dispatch_prevote(raft::vnode)::$_1::operator()() const::'lambda'(seastar::future<boost::outcome_v2::basic_result<raft::vote_reply, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<raft::vote_reply, std::__1::error_code, void> > >)&&)::'lambda'(seastar::internal::promise_base_with_type<void>&&, raft::prevote_stm::dispatch_prevote(raft::vnode)::$_1::operator()() const::'lambda'(seastar::future<boost::outcome_v2::basic_result<raft::vote_reply, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<raft::vote_reply, std::__1::error_code, void> > >)&, seastar::future_state<boost::outcome_v2::basic_result<raft::vote_reply, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<raft::vote_reply, std::__1::error_code, void> > >&&), boost::outcome_v2::basic_result<raft::vote_reply, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<raft::vote_reply, std::__1::error_code, void> > > -------- seastar::continuation<seastar::internal::promise_base_with_type<void>, seastar::future<void>::finally_body<auto seastar::internal::invoke_func_with_gate<raft::prevote_stm::dispatch_prevote(raft::vnode)::$_1>(seastar::gate&, raft::prevote_stm::dispatch_prevote(raft::vnode)::$_1&&)::'lambda'(), false>, seastar::futurize<raft::prevote_stm::dispatch_prevote(raft::vnode)::$_1>::type seastar::future<void>::then_wrapped_nrvo<seastar::future<void>, seastar::future<void>::finally_body<auto seastar::internal::invoke_func_with_gate<raft::prevote_stm::dispatch_prevote(raft::vnode)::$_1>(seastar::gate&, raft::prevote_stm::dispatch_prevote(raft::vnode)::$_1&&)::'lambda'(), false> >(seastar::future<void>::finally_body<auto seastar::internal::invoke_func_with_gate<raft::prevote_stm::dispatch_prevote(raft::vnode)::$_1>(seastar::gate&, raft::prevote_stm::dispatch_prevote(raft::vnode)::$_1&&)::'lambda'(), false>&&)::'lambda'(seastar::internal::promise_base_with_type<void>&&, seastar::future<void>::finally_body<auto seastar::internal::invoke_func_with_gate<raft::prevote_stm::dispatch_prevote(raft::vnode)::$_1>(seastar::gate&, raft::prevote_stm::dispatch_prevote(raft::vnode)::$_1&&)::'lambda'(), false>&, seastar::future_state<seastar::internal::monostate>&&), void>
Decoded backtrace
[Backtrace #0] addr2line: DWARF error: invalid or unhandled FORM value: 0x25 seastar::current_backtrace() at backtrace.cc:? seastar::report_failed_future(seastar::future_state_base::any&&) at future.cc:? addr2line: DWARF error: could not find variable specification at offset 4b addr2line: DWARF error: could not find variable specification at offset 814c addr2line: DWARF error: could not find variable specification at offset 10264 addr2line: DWARF error: could not find variable specification at offset 10425 seastar::future_state_base::any::check_failure() at /home/lazin/.data/rp1-vbuild/release/clang/rp_deps_install/include/seastar/core/future.hh:574 (inlined by) seastar::future_state<boost::outcome_v2::basic_result<raft::vote_reply, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<raft::vote_reply, std::__1::error_code, void> > >::clear() at /home/lazin/.data/rp1-vbuild/release/clang/rp_deps_install/include/seastar/core/future.hh:616 (inlined by) ~future_state at /home/lazin/.data/rp1-vbuild/release/clang/rp_deps_install/include/seastar/core/future.hh:621 (inlined by) ~future at /home/lazin/.data/rp1-vbuild/release/clang/rp_deps_install/include/seastar/core/future.hh:1344 (inlined by) operator() at /home/lazin/workspace/rp1/src/v/raft/prevote_stm.cc:128 (inlined by) operator() at /home/lazin/.data/rp1-vbuild/release/clang/rp_deps_install/include/seastar/core/future.hh:1675 (inlined by) void seastar::futurize<seastar::future<void> >::satisfy_with_result_of<seastar::future<boost::outcome_v2::basic_result<raft::vote_reply, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<raft::vote_reply, std::__1::error_code, void> > >::then_wrapped_nrvo<seastar::future<void>, raft::prevote_stm::dispatch_prevote(raft::vnode)::$_1::operator()() const::{lambda(seastar::future<boost::outcome_v2::basic_result<raft::vote_reply, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<raft::vote_reply, std::__1::error_code, void> > >)#1}>(raft::prevote_stm::dispatch_prevote(raft::vnode)::$_1::operator()() const::{lambda(seastar::future<boost::outcome_v2::basic_result<raft::vote_reply, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<raft::vote_reply, std::__1::error_code, void> > >)#1}&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, {lambda(seastar::future<boost::outcome_v2::basic_result<raft::vote_reply, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<raft::vote_reply, std::__1::error_code, void> > >)#1}&, seastar::future_state<boost::outcome_v2::basic_result<raft::vote_reply, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<raft::vote_reply, std::__1::error_code, void> > >&&)#1}::operator()(seastar::internal::promise_base_with_type<void>, seastar::internal::promise_base_with_type<void>&&, seastar::future_state<boost::outcome_v2::basic_result<raft::vote_reply, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<raft::vote_reply, std::__1::error_code, void> > >) const::{lambda()#1}>(seastar::internal::promise_base_with_type<void>, raft::prevote_stm::dispatch_prevote(raft::vnode)::$_1::operator()() const::{lambda(seastar::future<boost::outcome_v2::basic_result<raft::vote_reply, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<raft::vote_reply, std::__1::error_code, void> > >)#1}&&) at /home/lazin/.data/rp1-vbuild/release/clang/rp_deps_install/include/seastar/core/future.hh:2127 (inlined by) operator() at /home/lazin/.data/rp1-vbuild/release/clang/rp_deps_install/include/seastar/core/future.hh:1674 (inlined by) seastar::continuation<seastar::internal::promise_base_with_type<void>, raft::prevote_stm::dispatch_prevote(raft::vnode)::$_1::operator()() const::{lambda(seastar::future<boost::outcome_v2::basic_result<raft::vote_reply, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<raft::vote_reply, std::__1::error_code, void> > >)#1}, seastar::future<boost::outcome_v2::basic_result<raft::vote_reply, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<raft::vote_reply, std::__1::error_code, void> > >::then_wrapped_nrvo<seastar::future<void>, {lambda(seastar::future<boost::outcome_v2::basic_result<raft::vote_reply, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<raft::vote_reply, std::__1::error_code, void> > >)#1}>({lambda(seastar::future<boost::outcome_v2::basic_result<raft::vote_reply, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<raft::vote_reply, std::__1::error_code, void> > >)#1}&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, {lambda(seastar::future<boost::outcome_v2::basic_result<raft::vote_reply, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<raft::vote_reply, std::__1::error_code, void> > >)#1}&, seastar::future_state<boost::outcome_v2::basic_result<raft::vote_reply, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<raft::vote_reply, std::__1::error_code, void> > >&&)#1}, boost::outcome_v2::basic_result<raft::vote_reply, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<raft::vote_reply, std::__1::error_code, void> > >::run_and_dispose() at /home/lazin/.data/rp1-vbuild/release/clang/rp_deps_install/include/seastar/core/future.hh:774 seastar::reactor::run_some_tasks() at reactor.cc:? seastar::reactor::do_run() at reactor.cc:? seastar::reactor::run() at reactor.cc:? seastar::app_template::run_deprecated(int, char**, std::__1::function<void ()>&&) at app-template.cc:? seastar::app_template::run(int, char**, std::__1::function<seastar::future<int> ()>&&) at app-template.cc:? addr2line: DWARF error: could not find variable specification at offset 4b addr2line: DWARF error: could not find variable specification at offset 2367b addr2line: DWARF error: could not find variable specification at offset 43705 addr2line: DWARF error: could not find variable specification at offset 53acd addr2line: DWARF error: could not find variable specification at offset 53bfd addr2line: DWARF error: could not find variable specification at offset 54214 addr2line: DWARF error: could not find variable specification at offset 5439e addr2line: DWARF error: could not find variable specification at offset 548c9 addr2line: DWARF error: could not find variable specification at offset 54a76 addr2line: DWARF error: could not find variable specification at offset 33dea0 addr2line: DWARF error: could not find variable specification at offset 33dfee addr2line: DWARF error: could not find variable specification at offset 33e034 addr2line: DWARF error: could not find variable specification at offset 33ebc7 addr2line: DWARF error: could not find variable specification at offset 33ed8c addr2line: DWARF error: could not find variable specification at offset 43977c application::run(int, char**) at /home/lazin/workspace/rp1/src/v/redpanda/application.cc:224 main at ??:? addr2line: '/opt/redpanda/lib/libc.so.6': No such file /opt/redpanda/lib/libc.so.6 0x27b74 _start at ??:?
"Exceptional future ignored" log message in our test cluster.
JIRA Link: CORE-988
@Lazin any more context for this? sha1, cdt, custom cluster, test/no-test?
Version & Environment
Redpanda version: tip of dev
Decoded backtrace
What went wrong?
"Exceptional future ignored" log message in our test cluster.
What should have happened instead?
How to reproduce the issue?
Additional information
JIRA Link: CORE-988