search

sofastack / sofa-jraft

A production-grade java implementation of RAFT consensus algorithm.
https://www.sofastack.tech/projects/sofa-jraft/
Apache License 2.0
3.58k stars 1.15k forks source link

三个节点组成集群，一台A机器一个jraft服务，另一台B机器启动两个jraft服务 #835

Closed SpringLin97 closed 1 year ago

SpringLin97 commented 2 years ago

A机器一个jraft服务（192.168.5.98：8086）， B机器两个jraft服务（192.168.5.98：8084，192.168.5.98：8085）

  1. 两台机器所有服务先关闭
  2. 集群连接成功的启动顺序：首先启动A机器，然后再启动B机器。或者一起启动AB机器。
  3. 启动失败的启动顺序：先启动B机器，B机器内部有两个jraft,集群没有连接成功，出现概率问题，需要等分别几秒，十几秒，二十秒，或者两分钟，就会自动连接成功。如果马上启动A机器，就会马上连接成功。 同时启动两台jraft服务代码, new 两个实例 image 1、这个是启动B机器，同时启动两个jraft, 启动报错，集群没有连接成功 Hibernate: SELECT PARTITION_NAME 'name',PARTITION_DESCRIPTION 'describe' FROM information_schema.PARTITIONS WHERE table_schema=schema() AND table_name=? AND PARTITION_NAME <> 'pmax' ORDER BY PARTITION_DESCRIPTION ASC [2022-05-19 20:41:56,215][INFO][main][com.alipay.sofa.jraft.util.JRaftServiceLoader]: SPI service [com.alipay.sofa.jraft.JRaftServiceFactory - com.alipay.sofa.jraft.core.DefaultJRaftServiceFactory] loading. Hibernate: SELECT PARTITION_NAME 'name',PARTITION_DESCRIPTION 'describe' FROM information_schema.PARTITIONS WHERE table_schema=schema() AND table_name=? AND PARTITION_NAME <> 'pmax' ORDER BY PARTITION_DESCRIPTION ASC [2022-05-19 20:41:56,253][INFO][main][com.alipay.sofa.jraft.util.JRaftServiceLoader]: SPI service [com.alipay.sofa.jraft.rpc.RaftRpcFactory - com.alipay.sofa.jraft.rpc.impl.BoltRaftRpcFactory] loading. Sofa-Middleware-Log SLF4J : Actual binding is of type [ com.alipay.remoting Log4j2 ] [2022-05-19 20:41:56,316][INFO][main][com.alipay.sofa.common.log.ReportUtil]: Sofa-Middleware-Log SLF4J : Actual binding is of type [ com.alipay.remoting Log4j2 ] [2022-05-19 20:41:56,461][INFO][main][com.alipay.sofa.jraft.util.JRaftServiceLoader]: SPI service [com.alipay.sofa.jraft.util.timer.RaftTimerFactory - com.alipay.sofa.jraft.util.timer.DefaultRaftTimerFactory] loading. [2022-05-19 20:41:56,465][INFO][main][com.alipay.sofa.jraft.core.NodeImpl]: The number of active nodes increment to 1. [2022-05-19 20:41:56,621][INFO][JRaft-Closure-Executor-0][com.alipay.sofa.jraft.storage.impl.RocksDBLogStorage]: Truncated prefix logs in data path: /database/jraft/master/data\log from log index 0 to 110, cost 1 ms. [2022-05-19 20:41:56,626][INFO][main][com.alipay.sofa.jraft.core.FSMCallerImpl]: Starts FSMCaller successfully. [2022-05-19 20:41:56,636][INFO][main][com.alipay.sofa.jraft.storage.snapshot.SnapshotExecutorImpl]: Loading snapshot, meta=last_included_index: 109 last_included_term: 389 peers: "192.168.5.98:8084" peers: "192.168.5.98:8085" peers: "192.168.5.98:8086" . [2022-05-19 20:41:56,655][INFO][JRaft-FSMCaller-Disruptor-0][com.alipay.sofa.jraft.core.StateMachineAdapter]: onConfigurationCommitted: 192.168.5.98:8084,192.168.5.98:8085,192.168.5.98:8086. [2022-05-19 20:41:56,658][INFO][JRaft-FSMCaller-Disruptor-0][com.alipay.sofa.jraft.storage.snapshot.SnapshotExecutorImpl]: Node <TEST2/192.168.5.98:8084> onSnapshotLoadDone, last_included_index: 109 last_included_term: 389 peers: "192.168.5.98:8084" peers: "192.168.5.98:8085" peers: "192.168.5.98:8086"

[2022-05-19 20:41:56,658][INFO][main][com.alipay.sofa.jraft.core.NodeImpl]: Node <TEST2/192.168.5.98:8084> target priority value has changed from: 0, to: -1. [2022-05-19 20:41:56,659][INFO][JRaft-Closure-Executor-1][com.alipay.sofa.jraft.storage.impl.RocksDBLogStorage]: Truncated prefix logs in data path: /database/jraft/master/data\log from log index 110 to 110, cost 0 ms. [2022-05-19 20:41:56,716][INFO][main][com.alipay.sofa.jraft.core.NodeImpl]: Node <TEST2/192.168.5.98:8084> init, term=389, lastLogId=LogId [index=109, term=389], conf=192.168.5.98:8084,192.168.5.98:8085,192.168.5.98:8086, oldConf=. [2022-05-19 20:41:56,968][INFO][main][com.alipay.sofa.jraft.RaftGroupService]: Start the RaftGroupService successfully. [2022-05-19 20:41:56,968][INFO][main][com.visec.platform.webstarter.cluster.core.jraft.server.RaftServer]: Started counter server at port:8084 [2022-05-19 20:41:56,969][INFO][main][com.alipay.sofa.jraft.core.NodeImpl]: The number of active nodes increment to 2. [2022-05-19 20:41:57,028][INFO][JRaft-Closure-Executor-2][com.alipay.sofa.jraft.storage.impl.RocksDBLogStorage]: Truncated prefix logs in data path: /database/jraft/backup/data\log from log index 0 to 110, cost 1 ms. [2022-05-19 20:41:57,030][INFO][main][com.alipay.sofa.jraft.core.FSMCallerImpl]: Starts FSMCaller successfully. [2022-05-19 20:41:57,031][INFO][main][com.alipay.sofa.jraft.storage.snapshot.SnapshotExecutorImpl]: Loading snapshot, meta=last_included_index: 109 last_included_term: 389 peers: "192.168.5.98:8084" peers: "192.168.5.98:8085" peers: "192.168.5.98:8086" . [2022-05-19 20:41:57,033][INFO][JRaft-FSMCaller-Disruptor-0][com.alipay.sofa.jraft.core.StateMachineAdapter]: onConfigurationCommitted: 192.168.5.98:8084,192.168.5.98:8085,192.168.5.98:8086. [2022-05-19 20:41:57,034][INFO][JRaft-FSMCaller-Disruptor-0][com.alipay.sofa.jraft.storage.snapshot.SnapshotExecutorImpl]: Node <TEST2/192.168.5.98:8085> onSnapshotLoadDone, last_included_index: 109 last_included_term: 389 peers: "192.168.5.98:8084" peers: "192.168.5.98:8085" peers: "192.168.5.98:8086"

[2022-05-19 20:41:57,034][INFO][main][com.alipay.sofa.jraft.core.NodeImpl]: Node <TEST2/192.168.5.98:8085> target priority value has changed from: 0, to: -1. [2022-05-19 20:41:57,035][INFO][JRaft-Closure-Executor-3][com.alipay.sofa.jraft.storage.impl.RocksDBLogStorage]: Truncated prefix logs in data path: /database/jraft/backup/data\log from log index 110 to 110, cost 0 ms. [2022-05-19 20:41:57,035][INFO][main][com.alipay.sofa.jraft.core.NodeImpl]: Node <TEST2/192.168.5.98:8085> init, term=389, lastLogId=LogId [index=109, term=389], conf=192.168.5.98:8084,192.168.5.98:8085,192.168.5.98:8086, oldConf=. [2022-05-19 20:41:57,037][INFO][main][com.alipay.sofa.jraft.RaftGroupService]: Start the RaftGroupService successfully. [2022-05-19 20:41:57,038][INFO][main][com.visec.platform.webstarter.cluster.core.jraft.server.RaftServer]: Started counter server at port:8085 Hibernate: update sys_raft_cluster set c_balance_mode=?, c_cluster_mode=?, c_cluster_status=?, c_endpoints=?, c_endpoints_vip=?, c_group_id=?, c_ip=?, c_joined=?, c_license_port=?, c_master=?, c_node=?, c_node2=?, c_other_endpoints_vip=?, c_port=?, c_status=?, c_sync_status=?, c_vip_iface=? where cid=? Hibernate: select sysraftclu0.c_id as c_id1_100, sysraftclu0_.c_balance_mode as c_balanc2_100, sysraftclu0_.c_cluster_mode as c_cluste3_100, sysraftclu0_.c_cluster_status as c_cluste4_100, sysraftclu0_.c_endpoints as c_endpoi5_100, sysraftclu0_.c_endpoints_vip as c_endpoi6_100, sysraftclu0_.c_group_id as c_group_7_100, sysraftclu0_.c_ip as c_ip8_100, sysraftclu0_.c_joined as c_joined9_100, sysraftclu0_.c_license_port as c_licen10_100, sysraftclu0_.c_master as c_maste11_100, sysraftclu0_.c_node as c_node12_100, sysraftclu0_.c_node2 as c_node13_100, sysraftclu0_.c_other_endpoints_vip as c_other14_100, sysraftclu0_.c_port as c_port15_100, sysraftclu0_.c_status as c_statu16_100, sysraftclu0_.c_sync_status as c_sync_17_100, sysraftclu0_.c_vip_iface as c_vip_i18_100 from sys_raftcluster sysraftclu0 where sysraftclu0_.cid=? Hibernate: select sysraftmsg0.c_key as c_key111, sysraftmsg0_.c_cluster_mode as c_cluste211, sysraftmsg0_.c_msg_time as c_msg_ti311, sysraftmsg0_.c_node as c_node411, sysraftmsg0_.c_status as c_status511, sysraftmsg0_.c_update_time as c_update611 from sys_raft_clustermsg sysraftmsg0 where sysraftmsg0_.cstatus=? Hibernate: select syssecurit0.c_serial_number as c_serial118, syssecurit0_.c_alias as c_alias218, syssecurit0_.c_create_time as c_create318, syssecurit0_.c_p12 as c_p418, syssecurit0_.c_remark as c_remark518, syssecurit0_.c_validity as c_validi618 from sys_security_usercert syssecurit0 Hibernate: select syssecurit0_.c_code as c_code115, syssecurit0_.c_desc as c_desc215, syssecurit0_.c_title as c_title315 from sys_security_pretask syssecurit0 where syssecurit0_.ccode in (? , ? , ?) Hibernate: select permission0.c_pre_task_id as c_pre_ta5_141, permission0_.c_id as c_id1_141, permission0_.c_id as c_id1_140, permission0_.c_permission_id as c_permis2_140 from sys_securitypermission permission0 where permission0_.c_pre_taskid in (select syssecurit0.c_code from sys_security_pretask syssecurit0 where syssecurit0_.c_code in (? , ? , ?))

Hibernate: insert into sys_security_user (c_create_time, c_department, c_email, c_identity_id, c_last_login_time, c_lock, c_login_errors, c_nickname, c_password, c_user_pre_task_id, c_pwd_change_time, c_user_cert_id, c_username) values (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) [2022-05-19 20:41:58,071][WARN][main][org.hibernate.engine.jdbc.spi.SqlExceptionHelper]: SQL Error: 1062, SQLState: 23000 [2022-05-19 20:41:58,072][ERROR][main][org.hibernate.engine.jdbc.spi.SqlExceptionHelper]: Duplicate entry 'admin' for key 'sys_security_user.UK_610qqx8bogkimgjuk0cesbl2n' [2022-05-19 20:41:58,104][ERROR][main][com.visec.platform.web.LifeCycleRunner]: 开启模块失败：业务-采集模块 状态：START msg：{"state":false,"data":javax.persistence.PersistenceException: org.hibernate.exception.ConstraintViolationException: could not execute statement}

[2022-05-19 20:41:58,104][INFO][main][com.visec.platform.web.LifeCycleRunner]: 切换html资源映射：old:ResourceHttpRequestHandler [URL [file:G:/project/example/html/]] now:file:G:\project\example\html\ Hibernate: SELECT count() FROM information_schema.TABLES WHERE table_schema=schema() AND table_name=? AND ENGINE='InnoDB' Hibernate: SELECT count() FROM information_schema.TABLES WHERE table_schema=schema() AND table_name=? AND ENGINE='InnoDB' Hibernate: SELECT count() FROM information_schema.PARTITIONS WHERE table_schema=schema() AND table_name=? AND !isnull(PARTITION_METHOD) Hibernate: SELECT count() FROM information_schema.PARTITIONS WHERE table_schema=schema() AND table_name=? AND !isnull(PARTITION_METHOD) [2022-05-19 20:41:58,390][INFO][JRaft-ElectionTimer-<TEST2/192.168.5.98:8085>0][com.alipay.sofa.jraft.core.NodeImpl]: Node <TEST2/192.168.5.98:8085> term 389 start preVote. [2022-05-19 20:41:58,409][INFO][JRaft-ElectionTimer-<TEST2/192.168.5.98:8084>0][com.alipay.sofa.jraft.core.NodeImpl]: Node <TEST2/192.168.5.98:8084> term 389 start preVote. [2022-05-19 20:41:59,410][ERROR][JRaft-ElectionTimer-<TEST2/192.168.5.98:8085>0][com.alipay.sofa.jraft.rpc.impl.AbstractClientService]: Fail to connect 192.168.5.98:8086, remoting exception: com.alipay.remoting.exception.RemotingException: Create connection failed. The address is 192.168.5.98:8086. [2022-05-19 20:41:59,411][WARN][JRaft-ElectionTimer-<TEST2/192.168.5.98:8085>0][com.alipay.sofa.jraft.core.NodeImpl]: Node <TEST2/192.168.5.98:8085> channel init failed, address=192.168.5.98:8086. [2022-05-19 20:41:59,412][ERROR][JRaft-ElectionTimer-<TEST2/192.168.5.98:8084>0][com.alipay.sofa.jraft.rpc.impl.AbstractClientService]: Fail to connect 192.168.5.98:8086, remoting exception: com.alipay.remoting.exception.RemotingException: Create connection failed. The address is 192.168.5.98:8086. [2022-05-19 20:41:59,412][WARN][JRaft-ElectionTimer-<TEST2/192.168.5.98:8084>0][com.alipay.sofa.jraft.core.NodeImpl]: Node <TEST2/192.168.5.98:8084> channel init failed, address=192.168.5.98:8086. [2022-05-19 20:41:59,417][INFO][Bolt-conn-event-executor-13-thread-1][com.alipay.sofa.jraft.rpc.impl.core.ClientServiceConnectionEventProcessor]: Peer 192.168.5.98:8084 is connected [2022-05-19 20:41:59,417][INFO][Bolt-conn-event-executor-5-thread-1][com.alipay.sofa.jraft.rpc.impl.core.ClientServiceConnectionEventProcessor]: Peer 192.168.5.98:8085 is connected [2022-05-19 20:41:59,468][INFO][Bolt-default-executor-6-thread-6][com.alipay.sofa.jraft.core.NodeImpl]: Node <TEST2/192.168.5.98:8084> received PreVoteRequest from 192.168.5.98:8085, term=390, currTerm=389, granted=true, requestLastLogId=LogId [index=109, term=389], lastLogId=LogId [index=109, term=389]. [2022-05-19 20:41:59,468][INFO][Bolt-default-executor-6-thread-5][com.alipay.sofa.jraft.core.NodeImpl]: Node <TEST2/192.168.5.98:8085> received PreVoteRequest from 192.168.5.98:8084, term=390, currTerm=389, granted=true, requestLastLogId=LogId [index=109, term=389], lastLogId=LogId [index=109, term=389]. [2022-05-19 20:41:59,474][INFO][JRaft-RPC-Processor-0][com.alipay.sofa.jraft.core.NodeImpl]: Node <TEST2/192.168.5.98:8084> received PreVoteResponse from 192.168.5.98:8085, term=389, granted=true. [2022-05-19 20:41:59,474][INFO][JRaft-RPC-Processor-0][com.alipay.sofa.jraft.core.NodeImpl]: Node <TEST2/192.168.5.98:8085> received PreVoteResponse from 192.168.5.98:8084, term=389, granted=true. [2022-05-19 20:41:59,474][INFO][JRaft-RPC-Processor-0][com.alipay.sofa.jraft.core.NodeImpl]: Node <TEST2/192.168.5.98:8084> start vote and grant vote self, term=389. [2022-05-19 20:41:59,474][INFO][JRaft-RPC-Processor-0][com.alipay.sofa.jraft.core.NodeImpl]: Node <TEST2/192.168.5.98:8085> start vote and grant vote self, term=389. [2022-05-19 20:42:00,477][ERROR][JRaft-RPC-Processor-0][com.alipay.sofa.jraft.rpc.impl.AbstractClientService]: Fail to connect 192.168.5.98:8086, remoting exception: com.alipay.remoting.exception.RemotingException: Create connection failed. The address is 192.168.5.98:8086. [2022-05-19 20:42:00,477][ERROR][JRaft-RPC-Processor-0][com.alipay.sofa.jraft.rpc.impl.AbstractClientService]: Fail to connect 192.168.5.98:8086, remoting exception: com.alipay.remoting.exception.RemotingException: Create connection failed. The address is 192.168.5.98:8086. [2022-05-19 20:42:00,477][WARN][JRaft-RPC-Processor-0][com.alipay.sofa.jraft.core.NodeImpl]: Node <TEST2/192.168.5.98:8085> channel init failed, address=192.168.5.98:8086. [2022-05-19 20:42:00,477][WARN][JRaft-RPC-Processor-0][com.alipay.sofa.jraft.core.NodeImpl]: Node <TEST2/192.168.5.98:8084> channel init failed, address=192.168.5.98:8086. [2022-05-19 20:42:00,492][WARN][JRaft-RPC-Processor-0][com.alipay.sofa.jraft.util.Utils]: Unable to fsync directory \database\jraft\backup\data\raft_meta on windows. [2022-05-19 20:42:00,492][WARN][JRaft-RPC-Processor-0][com.alipay.sofa.jraft.util.Utils]: Unable to fsync directory \database\jraft\master\data\raft_meta on windows. [2022-05-19 20:42:00,493][INFO][JRaft-RPC-Processor-0][com.alipay.sofa.jraft.storage.impl.LocalRaftMetaStorage]: Save raft meta, path=/database/jraft/backup/data\raft_meta, term=390, votedFor=192.168.5.98:8085, cost time=14 ms [2022-05-19 20:42:00,493][INFO][JRaft-RPC-Processor-0][com.alipay.sofa.jraft.storage.impl.LocalRaftMetaStorage]: Save raft meta, path=/database/jraft/master/data\raft_meta, term=390, votedFor=192.168.5.98:8084, cost time=14 ms [2022-05-19 20:42:00,493][INFO][Bolt-default-executor-6-thread-10][com.alipay.sofa.jraft.core.NodeImpl]: Node <TEST2/192.168.5.98:8085> received RequestVoteRequest from 192.168.5.98:8084, term=390, currTerm=390. [2022-05-19 20:42:00,493][INFO][Bolt-default-executor-6-thread-9][com.alipay.sofa.jraft.core.NodeImpl]: Node <TEST2/192.168.5.98:8084> received RequestVoteRequest from 192.168.5.98:8085, term=390, currTerm=390. [2022-05-19 20:42:00,714][WARN][JRaft-VoteTimer-<TEST2/192.168.5.98:8084>0][com.alipay.sofa.jraft.core.NodeImpl]: Candidate node <TEST2/192.168.5.98:8084> term 390 steps down when election reaching vote timeout: fail to get quorum vote-granted. [2022-05-19 20:42:00,714][INFO][JRaft-VoteTimer-<TEST2/192.168.5.98:8084>0][com.alipay.sofa.jraft.core.NodeImpl]: Node <TEST2/192.168.5.98:8084> term 390 start preVote. [2022-05-19 20:42:01,190][WARN][JRaft-VoteTimer-<TEST2/192.168.5.98:8085>0][com.alipay.sofa.jraft.core.NodeImpl]: Candidate node <TEST2/192.168.5.98:8085> term 390 steps down when election reaching vote timeout: fail to get quorum vote-granted. [2022-05-19 20:42:01,190][INFO][JRaft-VoteTimer-<TEST2/192.168.5.98:8085>0][com.alipay.sofa.jraft.core.NodeImpl]: Node <TEST2/192.168.5.98:8085> term 390 start preVote. [2022-05-19 20:42:01,718][ERROR][JRaft-VoteTimer-<TEST2/192.168.5.98:8084>0][com.alipay.sofa.jraft.rpc.impl.AbstractClientService]: Fail to connect 192.168.5.98:8086, remoting exception: com.alipay.remoting.exception.RemotingException: Create connection failed. The address is 192.168.5.98:8086. [2022-05-19 20:42:01,718][WARN][JRaft-VoteTimer-<TEST2/192.168.5.98:8084>0][com.alipay.sofa.jraft.core.NodeImpl]: Node <TEST2/192.168.5.98:8084> channel init failed, address=192.168.5.98:8086. [2022-05-19 20:42:01,825][INFO][JRaft-ElectionTimer-<TEST2/192.168.5.98:8084>0][com.alipay.sofa.jraft.core.NodeImpl]: Node <TEST2/192.168.5.98:8084> term 390 start preVote. [2022-05-19 20:42:02,193][ERROR][JRaft-VoteTimer-<TEST2/192.168.5.98:8085>0][com.alipay.sofa.jraft.rpc.impl.AbstractClientService]: Fail to connect 192.168.5.98:8086, remoting exception: com.alipay.remoting.exception.RemotingException: Create connection failed. The address is 192.168.5.98:8086. [2022-05-19 20:42:02,193][WARN][JRaft-VoteTimer-<TEST2/192.168.5.98:8085>0][com.alipay.sofa.jraft.core.NodeImpl]: Node <TEST2/192.168.5.98:8085> channel init failed, address=192.168.5.98:8086. [2022-05-19 20:42:02,193][INFO][Bolt-default-executor-6-thread-16][com.alipay.sofa.jraft.core.NodeImpl]: Node <TEST2/192.168.5.98:8085> received PreVoteRequest from 192.168.5.98:8084, term=391, currTerm=390, granted=true, requestLastLogId=LogId [index=109, term=389], lastLogId=LogId [index=109, term=389]. [2022-05-19 20:42:02,310][INFO][JRaft-ElectionTimer-<TEST2/192.168.5.98:8085>0][com.alipay.sofa.jraft.core.NodeImpl]: Node <TEST2/192.168.5.98:8085> term 390 start preVote. [2022-05-19 20:42:02,827][ERROR][JRaft-ElectionTimer-<TEST2/192.168.5.98:8084>0][com.alipay.sofa.jraft.rpc.impl.AbstractClientService]: Fail to connect 192.168.5.98:8086, remoting exception: com.alipay.remoting.exception.RemotingException: Create connection failed. The address is 192.168.5.98:8086. [2022-05-19 20:42:02,827][WARN][JRaft-ElectionTimer-<TEST2/192.168.5.98:8084>0][com.alipay.sofa.jraft.core.NodeImpl]: Node <TEST2/192.168.5.98:8084> channel init failed, address=192.168.5.98:8086. [2022-05-19 20:42:02,828][INFO][Bolt-default-executor-6-thread-1][com.alipay.sofa.jraft.core.NodeImpl]: Node <TEST2/192.168.5.98:8084> received PreVoteRequest from 192.168.5.98:8085, term=391, currTerm=390, granted=true, requestLastLogId=LogId [index=109, term=389], lastLogId=LogId [index=109, term=389]. [2022-05-19 20:42:02,828][INFO][JRaft-RPC-Processor-2][com.alipay.sofa.jraft.core.NodeImpl]: Node <TEST2/192.168.5.98:8084> received PreVoteResponse from 192.168.5.98:8085, term=390, granted=true. [2022-05-19 20:42:02,828][INFO][JRaft-RPC-Processor-2][com.alipay.sofa.jraft.core.NodeImpl]: Node <TEST2/192.168.5.98:8084> start vote and grant vote self, term=390. [2022-05-19 20:42:03,199][ERROR][scheduling-1][com.alipay.sofa.jraft.rpc.impl.AbstractClientService]: Fail to connect 192.168.5.98:8086, remoting exception: com.alipay.remoting.exception.RemotingException: Create connection failed. The address is 192.168.5.98:8086. [2022-05-19 20:42:03,199][ERROR][scheduling-1][com.alipay.sofa.jraft.core.CliServiceImpl]: Fail to connect peer 192.168.5.98:8086 to get leader for group TEST2. java.lang.IllegalStateException: Fail to get leader of group TEST2, Unknown leader, Unknown leader at com.alipay.sofa.jraft.core.CliServiceImpl.getPeers(CliServiceImpl.java:605) at com.alipay.sofa.jraft.core.CliServiceImpl.getAlivePeers(CliServiceImpl.java:503) at com.visec.platform.webstarter.cluster.core.jraft.RaftHandle.getNodeHealthByCluster(RaftHandle.java:119) at com.visec.platform.webstarter.cluster.core.tasks.ScheduledRaftTasks.monitorEndpointHealth(ScheduledRaftTasks.java:334) at com.visec.platform.webstarter.cluster.core.tasks.ScheduledRaftTasks.monitorEndpointStatus(ScheduledRaftTasks.java:260) at com.visec.platform.webstarter.cluster.core.tasks.ScheduledRaftTasks$$FastClassBySpringCGLIB$$a2e2d2aa.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:783) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:753) at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123) at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:388) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:753) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:698) at com.visec.platform.webstarter.cluster.core.tasks.ScheduledRaftTasks$$EnhancerBySpringCGLIB$$943a2a93.monitorEndpointStatus() at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:84) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset$$$capture(FutureTask.java:305) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [2022-05-19 20:42:03,321][ERROR][JRaft-ElectionTimer-<TEST2/192.168.5.98:8085>0][com.alipay.sofa.jraft.rpc.impl.AbstractClientService]: Fail to connect 192.168.5.98:8086, remoting exception: com.alipay.remoting.exception.RemotingException: Create connection failed. The address is 192.168.5.98:8086. [2022-05-19 20:42:03,321][WARN][JRaft-ElectionTimer-<TEST2/192.168.5.98:8085>0][com.alipay.sofa.jraft.core.NodeImpl]: Node <TEST2/192.168.5.98:8085> channel init failed, address=192.168.5.98:8086. [2022-05-19 20:42:03,321][INFO][Bolt-default-executor-6-thread-4][com.alipay.sofa.jraft.core.NodeImpl]: Node <TEST2/192.168.5.98:8085> received PreVoteRequest from 192.168.5.98:8084, term=391, currTerm=390, granted=true, requestLastLogId=LogId [index=109, term=389], lastLogId=LogId [index=109, term=389]. [2022-05-19 20:42:03,321][INFO][JRaft-RPC-Processor-2][com.alipay.sofa.jraft.core.NodeImpl]: Node <TEST2/192.168.5.98:8085> received PreVoteResponse from 192.168.5.98:8084, term=390, granted=true. [2022-05-19 20:42:03,322][INFO][JRaft-RPC-Processor-2][com.alipay.sofa.jraft.core.NodeImpl]: Node <TEST2/192.168.5.98:8085> start vote and grant vote self, term=390. [2022-05-19 20:42:03,831][ERROR][JRaft-RPC-Processor-2][com.alipay.sofa.jraft.rpc.impl.AbstractClientService]: Fail to connect 192.168.5.98:8086, remoting exception: com.alipay.remoting.exception.RemotingException: Create connection failed. The address is 192.168.5.98:8086. [2022-05-19 20:42:03,831][WARN][JRaft-RPC-Processor-2][com.alipay.sofa.jraft.core.NodeImpl]: Node <TEST2/192.168.5.98:8084> channel init failed, address=192.168.5.98:8086. [2022-05-19 20:42:03,834][WARN][JRaft-RPC-Processor-2][com.alipay.sofa.jraft.util.Utils]: Unable to fsync directory \database\jraft\master\data\raft_meta on windows. [2022-05-19 20:42:03,834][INFO][JRaft-RPC-Processor-2][com.alipay.sofa.jraft.storage.impl.LocalRaftMetaStorage]: Save raft meta, path=/database/jraft/master/data\raft_meta, term=391, votedFor=192.168.5.98:8084, cost time=3 ms [2022-05-19 20:42:03,834][INFO][Bolt-default-executor-6-thread-7][com.alipay.sofa.jraft.core.NodeImpl]: Node <TEST2/192.168.5.98:8084> received PreVoteRequest from 192.168.5.98:8085, term=391, currTerm=391, granted=true, requestLastLogId=LogId [index=109, term=389], lastLogId=LogId [index=109, term=389]. [2022-05-19 20:42:03,834][WARN][JRaft-RPC-Processor-3][com.alipay.sofa.jraft.core.NodeImpl]: Node <TEST2/192.168.5.98:8084> received invalid PreVoteResponse from 192.168.5.98:8085, state not in STATE_FOLLOWER but STATE_CANDIDATE. [2022-05-19 20:42:04,323][ERROR][JRaft-RPC-Processor-2][com.alipay.sofa.jraft.rpc.impl.AbstractClientService]: Fail to connect 192.168.5.98:8086, remoting exception: com.alipay.remoting.exception.RemotingException: Create connection failed. The address is 192.168.5.98:8086. [2022-05-19 20:42:04,323][WARN][JRaft-RPC-Processor-2][com.alipay.sofa.jraft.core.NodeImpl]: Node <TEST2/192.168.5.98:8085> channel init failed, address=192.168.5.98:8086. [2022-05-19 20:42:04,324][INFO][Bolt-default-executor-6-thread-13][com.alipay.sofa.jraft.core.NodeImpl]: Node <TEST2/192.168.5.98:8084> received RequestVoteRequest from 192.168.5.98:8085, term=391, currTerm=391. [2022-05-19 20:42:04,325][WARN][JRaft-RPC-Processor-2][com.alipay.sofa.jraft.util.Utils]: Unable to fsync directory \database\jraft\backup\data\raft_meta on windows. [2022-05-19 20:42:04,325][INFO][JRaft-RPC-Processor-2][com.alipay.sofa.jraft.storage.impl.LocalRaftMetaStorage]: Save raft meta, path=/database/jraft/backup/data\raft_meta, term=391, votedFor=192.168.5.98:8085, cost time=3 ms [2022-05-19 20:42:04,326][INFO][Bolt-default-executor-6-thread-10][com.alipay.sofa.jraft.core.NodeImpl]: Node <TEST2/192.168.5.98:8085> received RequestVoteRequest from 192.168.5.98:8084, term=391, currTerm=391. [2022-05-19 20:42:04,326][WARN][JRaft-RPC-Processor-3][com.alipay.sofa.jraft.core.NodeImpl]: Node <TEST2/192.168.5.98:8085> received invalid PreVoteResponse from 192.168.5.98:8084, state not in STATE_FOLLOWER but STATE_CANDIDATE. [2022-05-19 20:42:04,396][WARN][JRaft-VoteTimer-<TEST2/192.168.5.98:8085>0][com.alipay.sofa.jraft.core.NodeImpl]: Candidate node <TEST2/192.168.5.98:8085> term 391 steps down when election reaching vote timeout: fail to get quorum vote-granted. [2022-05-19 20:42:04,396][INFO][JRaft-VoteTimer-<TEST2/192.168.5.98:8085>0][com.alipay.sofa.jraft.core.NodeImpl]: Node <TEST2/192.168.5.98:8085> term 391 start preVote. [2022-05-19 20:42:04,563][WARN][JRaft-VoteTimer-<TEST2/192.168.5.98:8084>0][com.alipay.sofa.jraft.core.NodeImpl]: Candidate node <TEST2/192.168.5.98:8084> term 391 steps down when election reaching vote timeout: fail to get quorum vote-granted. [2022-05-19 20:42:04,563][INFO][JRaft-VoteTimer-<TEST2/192.168.5.98:8084>0][com.alipay.sofa.jraft.core.NodeImpl]: Node <TEST2/192.168.5.98:8084> term 391 start preVote. [2022-05-19 20:42:05,329][ERROR][scheduling-1][com.alipay.sofa.jraft.rpc.impl.AbstractClientService]: Fail to connect 192.168.5.98:8086, remoting exception: com.alipay.remoting.exception.RemotingException: Create connection failed. The address is 192.168.5.98:8086. [2022-05-19 20:42:05,329][ERROR][scheduling-1][com.alipay.sofa.jraft.core.CliServiceImpl]: Fail to connect peer 192.168.5.98:8086 to get leader for group TEST2. [2022-05-19 20:42:05,399][ERROR][JRaft-VoteTimer-<TEST2/192.168.5.98:8085>0][com.alipay.sofa.jraft.rpc.impl.AbstractClientService]: Fail to connect 192.168.5.98:8086, remoting exception: com.alipay.remoting.exception.RemotingException: Create connection failed. The address is 192.168.5.98:8086. [2022-05-19 20:42:05,399][WARN][JRaft-VoteTimer-<TEST2/192.168.5.98:8085>0][com.alipay.sofa.jraft.core.NodeImpl]: Node <TEST2/192.168.5.98:8085> channel init failed, address=192.168.5.98:8086. [2022-05-19 20:42:05,432][INFO][JRaft-ElectionTimer-<TEST2/192.168.5.98:8085>0][com.alipay.sofa.jraft.core.NodeImpl]: Node <TEST2/192.168.5.98:8085> term 391 start preVote. [2022-05-19 20:42:05,567][ERROR][JRaft-VoteTimer-<TEST2/192.168.5.98:8084>0][com.alipay.sofa.jraft.rpc.impl.AbstractClientService]: Fail to connect 192.168.5.98:8086, remoting exception: com.alipay.remoting.exception.RemotingException: Create connection failed. The address is 192.168.5.98:8086. [2022-05-19 20:42:05,567][WARN][JRaft-VoteTimer-<TEST2/192.168.5.98:8084>0][com.alipay.sofa.jraft.core.NodeImpl]: Node <TEST2/192.168.5.98:8084> channel init failed, address=192.168.5.98:8086. [2022-05-19 20:42:05,568][INFO][Bolt-default-executor-6-thread-19][com.alipay.sofa.jraft.core.NodeImpl]: Node <TEST2/192.168.5.98:8084> received PreVoteRequest from 192.168.5.98:8085, term=392, currTerm=391, granted=true, requestLastLogId=LogId [index=109, term=389], lastLogId=LogId [index=109, term=389]. [2022-05-19 20:42:06,026][INFO][JRaft-ElectionTimer-<TEST2/192.168.5.98:8084>0][com.alipay.sofa.jraft.core.NodeImpl]: Node <TEST2/192.168.5.98:8084> term 391 start preVote. [2022-05-19 20:42:06,433][ERROR][JRaft-ElectionTimer-<TEST2/192.168.5.98:8085>0][com.alipay.sofa.jraft.rpc.impl.AbstractClientService]: Fail to connect 192.168.5.98:8086, remoting exception: com.alipay.remoting.exception.RemotingException: Create connection failed. The address is 192.168.5.98:8086. [2022-05-19 20:42:06,434][WARN][JRaft-ElectionTimer-<TEST2/192.168.5.98:8085>0][com.alipay.sofa.jraft.core.NodeImpl]: Node <TEST2/192.168.5.98:8085> channel init failed, address=192.168.5.98:8086. [2022-05-19 20:42:06,434][INFO][Bolt-default-executor-6-thread-16][com.alipay.sofa.jraft.core.NodeImpl]: Node <TEST2/192.168.5.98:8085> received PreVoteRequest from 192.168.5.98:8084, term=392, currTerm=391, granted=true, requestLastLogId=LogId [index=109, term=389], lastLogId=LogId [index=109, term=389]. [2022-05-19 20:42:06,434][INFO][JRaft-RPC-Processor-5][com.alipay.sofa.jraft.core.NodeImpl]: Node <TEST2/192.168.5.98:8085> received PreVoteResponse from 192.168.5.98:8084, term=391, granted=true. [2022-05-19 20:42:06,435][INFO][JRaft-RPC-Processor-5][com.alipay.sofa.jraft.core.NodeImpl]: Node <TEST2/192.168.5.98:8085> start vote and grant vote self, term=391. [2022-05-19 20:42:07,027][ERROR][JRaft-ElectionTimer-<TEST2/192.168.5.98:8084>0][com.alipay.sofa.jraft.rpc.impl.AbstractClientService]: Fail to connect 192.168.5.98:8086, remoting exception: com.alipay.remoting.exception.RemotingException: Create connection failed. The address is 192.168.5.98:8086. [2022-05-19 20:42:07,027][WARN][JRaft-ElectionTimer-<TEST2/192.168.5.98:8084>0][com.alipay.sofa.jraft.core.NodeImpl]: Node <TEST2/192.168.5.98:8084> channel init failed, address=192.168.5.98:8086. [2022-05-19 20:42:07,028][INFO][Bolt-default-executor-6-thread-1][com.alipay.sofa.jraft.core.NodeImpl]: Node <TEST2/192.168.5.98:8084> received PreVoteRequest from 192.168.5.98:8085, term=392, currTerm=391, granted=true, requestLastLogId=LogId [index=109, term=389], lastLogId=LogId [index=109, term=389]. [2022-05-19 20:42:07,028][INFO][JRaft-RPC-Processor-5][com.alipay.sofa.jraft.core.NodeImpl]: Node <TEST2/192.168.5.98:8084> received PreVoteResponse from 192.168.5.98:8085, term=391, granted=true. [2022-05-19 20:42:07,028][INFO][JRaft-RPC-Processor-5][com.alipay.sofa.jraft.core.NodeImpl]: Node <TEST2/192.168.5.98:8084> start vote and grant vote self, term=391. [2022-05-19 20:42:07,437][ERROR][JRaft-RPC-Processor-5][com.alipay.sofa.jraft.rpc.impl.AbstractClientService]: Fail to connect 192.168.5.98:8086, remoting exception: com.alipay.remoting.exception.RemotingException: Create connection failed. The address is 192.168.5.98:8086. [2022-05-19 20:42:07,437][WARN][JRaft-RPC-Processor-5][com.alipay.sofa.jraft.core.NodeImpl]: Node <TEST2/192.168.5.98:8085> channel init failed, address=192.168.5.98:8086. [2022-05-19 20:42:07,440][WARN][JRaft-RPC-Processor-5][com.alipay.sofa.jraft.util.Utils]: Unable to fsync directory \database\jraft\backup\data\raft_meta on windows. [2022-05-19 20:42:07,440][INFO][JRaft-RPC-Processor-5][com.alipay.sofa.jraft.storage.impl.LocalRaftMetaStorage]: Save raft meta, path=/database/jraft/backup/data\raft_meta, term=392, votedFor=192.168.5.98:8085, cost time=3 ms [2022-05-19 20:42:07,440][INFO][Bolt-default-executor-6-thread-4][com.alipay.sofa.jraft.core.NodeImpl]: Node <TEST2/192.168.5.98:8085> received PreVoteRequest from 192.168.5.98:8084, term=392, currTerm=392, granted=true, requestLastLogId=LogId [index=109, term=389], lastLogId=LogId [index=109, term=389]. [2022-05-19 20:42:07,440][WARN][JRaft-RPC-Processor-6][com.alipay.sofa.jraft.core.NodeImpl]: Node <TEST2/192.168.5.98:8085> received invalid PreVoteResponse from 192.168.5.98:8084, state not in STATE_FOLLOWER but STATE_CANDIDATE. [2022-05-19 20:42:07,715][WARN][JRaft-VoteTimer-<TEST2/192.168.5.98:8085>0][com.alipay.sofa.jraft.core.NodeImpl]: Candidate node <TEST2/192.168.5.98:8085> term 392 steps down when election reaching vote timeout: fail to get quorum vote-granted. [2022-05-19 20:42:07,715][INFO][JRaft-VoteTimer-<TEST2/192.168.5.98:8085>0][com.alipay.sofa.jraft.core.NodeImpl]: Node <TEST2/192.168.5.98:8085> term 392 start preVote. [2022-05-19 20:42:08,031][ERROR][JRaft-RPC-Processor-5][com.alipay.sofa.jraft.rpc.impl.AbstractClientService]: Fail to connect 192.168.5.98:8086, remoting exception: com.alipay.remoting.exception.RemotingException: Create connection failed. The address is 192.168.5.98:8086. [2022-05-19 20:42:08,031][WARN][JRaft-RPC-Processor-5][com.alipay.sofa.jraft.core.NodeImpl]: Node <TEST2/192.168.5.98:8084> channel init failed, address=192.168.5.98:8086. [2022-05-19 20:42:08,034][WARN][JRaft-RPC-Processor-5][com.alipay.sofa.jraft.util.Utils]: Unable to fsync directory \database\jraft\master\data\raft_meta on windows. [2022-05-19 20:42:08,034][INFO][JRaft-RPC-Processor-5][com.alipay.sofa.jraft.storage.impl.LocalRaftMetaStorage]: Save raft meta, path=/database/jraft/master/data\raft_meta, term=392, votedFor=192.168.5.98:8084, cost time=3 ms [2022-05-19 20:42:08,034][INFO][Bolt-default-executor-6-thread-5][com.alipay.sofa.jraft.core.NodeImpl]: Node <TEST2/192.168.5.98:8084> received RequestVoteRequest from 192.168.5.98:8085, term=392, currTerm=392. [2022-05-19 20:42:08,034][WARN][JRaft-RPC-Processor-6][com.alipay.sofa.jraft.core.NodeImpl]: Node <TEST2/192.168.5.98:8084> received invalid PreVoteResponse from 192.168.5.98:8085, state not in STATE_FOLLOWER but STATE_CANDIDATE. [2022-05-19 20:42:08,336][WARN][JRaft-VoteTimer-<TEST2/192.168.5.98:8084>0][com.alipay.sofa.jraft.core.NodeImpl]: Candidate node <TEST2/192.168.5.98:8084> term 392 steps down when election reaching vote timeout: fail to get quorum vote-granted. [2022-05-19 20:42:08,336][INFO][JRaft-VoteTimer-<TEST2/192.168.5.98:8084>0][com.alipay.sofa.jraft.core.NodeImpl]: Node <TEST2/192.168.5.98:8084> term 392 start preVote. [2022-05-19 20:42:08,442][ERROR][scheduling-1][com.alipay.sofa.jraft.rpc.impl.AbstractClientService]: Fail to connect 192.168.5.98:8086, remoting exception: com.alipay.remoting.exception.RemotingException: Create connection failed. The address is 192.168.5.98:8086. [2022-05-19 20:42:08,443][ERROR][scheduling-1][com.alipay.sofa.jraft.core.CliServiceImpl]: Fail to connect peer 192.168.5.98:8086 to get leader for group TEST2. [2022-05-19 20:42:08,717][ERROR][JRaft-VoteTimer-<TEST2/192.168.5.98:8085>0][com.alipay.sofa.jraft.rpc.impl.AbstractClientService]: Fail to connect 192.168.5.98:8086, remoting exception: com.alipay.remoting.exception.RemotingException: Create connection failed. The address is 192.168.5.98:8086. [2022-05-19 20:42:08,717][WARN][JRaft-VoteTimer-<TEST2/192.168.5.98:8085>0][com.alipay.sofa.jraft.core.NodeImpl]: Node <TEST2/192.168.5.98:8085> channel init failed, address=192.168.5.98:8086. [2022-05-19 20:42:08,717][INFO][Bolt-default-executor-6-thread-11][com.alipay.sofa.jraft.core.NodeImpl]: Node <TEST2/192.168.5.98:8085> received RequestVoteRequest from 192.168.5.98:8084, term=392, currTerm=392. [2022-05-19 20:42:08,718][WARN][JRaft-RPC-Processor-7][com.alipay.sofa.jraft.core.NodeImpl]: Node <TEST2/192.168.5.98:8085> received invalid RequestVoteResponse from 192.168.5.98:8084, state not in STATE_CANDIDATE but STATE_FOLLOWER. [2022-05-19 20:42:09,338][ERROR][JRaft-VoteTimer-<TEST2/192.168.5.98:8084>0][com.alipay.sofa.jraft.rpc.impl.AbstractClientService]: Fail to connect 192.168.5.98:8086, remoting exception: com.alipay.remoting.exception.RemotingException: Create connection failed. The address is 192.168.5.98:8086. [2022-05-19 20:42:09,338][WARN][JRaft-VoteTimer-<TEST2/192.168.5.98:8084>0][com.alipay.sofa.jraft.core.NodeImpl]: Node <TEST2/192.168.5.98:8084> channel init failed, address=192.168.5.98:8086. [2022-05-19 20:42:09,339][INFO][Bolt-default-executor-6-thread-11][com.alipay.sofa.jraft.core.NodeImpl]: Node <TEST2/192.168.5.98:8084> received PreVoteRequest from 192.168.5.98:8085, term=393, currTerm=392, granted=true, requestLastLogId=LogId [index=109, term=389], lastLogId=LogId [index=109, term=389]. [2022-05-19 20:42:09,339][WARN][JRaft-RPC-Processor-7][com.alipay.sofa.jraft.core.NodeImpl]: Node <TEST2/192.168.5.98:8084> received invalid RequestVoteResponse from 192.168.5.98:8085, state not in STATE_CANDIDATE but STATE_FOLLOWER. [2022-05-19 20:42:09,339][INFO][Bolt-default-executor-6-thread-15][com.alipay.sofa.jraft.core.NodeImpl]: Node <TEST2/192.168.5.98:8085> received PreVoteRequest from 192.168.5.98:8084, term=393, currTerm=392, granted=true, requestLastLogId=LogId [index=109, term=389], lastLogId=LogId [index=109, term=389]. [2022-05-19 20:42:09,341][INFO][JRaft-RPC-Processor-8][com.alipay.sofa.jraft.core.NodeImpl]: Node <TEST2/192.168.5.98:8085> received PreVoteResponse from 192.168.5.98:8084, term=392, granted=true. [2022-05-19 20:42:09,341][INFO][JRaft-RPC-Processor-8][com.alipay.sofa.jraft.core.NodeImpl]: Node <TEST2/192.168.5.98:8084> received PreVoteResponse from 192.168.5.98:8085, term=392, granted=true. [2022-05-19 20:42:09,341][INFO][JRaft-RPC-Processor-8][com.alipay.sofa.jraft.core.NodeImpl]: Node <TEST2/192.168.5.98:8084> start vote and grant vote self, term=392. [2022-05-19 20:42:09,341][INFO][JRaft-RPC-Processor-8][com.alipay.sofa.jraft.core.NodeImpl]: Node <TEST2/192.168.5.98:8085> start vote and grant vote self, term=392. [2022-05-19 20:42:10,343][ERROR][JRaft-RPC-Processor-8][com.alipay.sofa.jraft.rpc.impl.AbstractClientService]: Fail to connect 192.168.5.98:8086, remoting exception: com.alipay.remoting.exception.RemotingException: Create connection failed. The address is 192.168.5.98:8086. [2022-05-19 20:42:10,343][ERROR][JRaft-RPC-Processor-8][com.alipay.sofa.jraft.rpc.impl.AbstractClientService]: Fail to connect 192.168.5.98:8086, remoting exception: com.alipay.remoting.exception.RemotingException: Create connection failed. The address is 192.168.5.98:8086. [2022-05-19 20:42:10,343][WARN][JRaft-RPC-Processor-8][com.alipay.sofa.jraft.core.NodeImpl]: Node <TEST2/192.168.5.98:8084> channel init failed, address=192.168.5.98:8086. [2022-05-19 20:42:10,343][WARN][JRaft-RPC-Processor-8][com.alipay.sofa.jraft.core.NodeImpl]: Node <TEST2/192.168.5.98:8085> channel init failed, address=192.168.5.98:8086. [2022-05-19 20:42:10,346][WARN][JRaft-RPC-Processor-8][com.alipay.sofa.jraft.util.Utils]: Unable to fsync directory \database\jraft\backup\data\raft_meta on windows. [2022-05-19 20:42:10,346][WARN][JRaft-RPC-Processor-8][com.alipay.sofa.jraft.util.Utils]: Unable to fsync directory \database\jraft\master\data\raft_meta on windows. [2022-05-19 20:42:10,346][INFO][JRaft-RPC-Processor-8][com.alipay.sofa.jraft.storage.impl.LocalRaftMetaStorage]: Save raft meta, path=/database/jraft/backup/data\raft_meta, term=393, votedFor=192.168.5.98:8085, cost time=3 ms [2022-05-19 20:42:10,346][INFO][JRaft-RPC-Processor-8][com.alipay.sofa.jraft.storage.impl.LocalRaftMetaStorage]: Save raft meta, path=/database/jraft/master/data\raft_meta, term=393, votedFor=192.168.5.98:8084, cost time=3 ms [2022-05-19 20:42:10,346][INFO][Bolt-default-executor-6-thread-2][com.alipay.sofa.jraft.core.NodeImpl]: Node <TEST2/192.168.5.98:8085> received RequestVoteRequest from 192.168.5.98:8084, term=393, currTerm=393. [2022-05-19 20:42:10,346][INFO][Bolt-default-executor-6-thread-16][com.alipay.sofa.jraft.core.NodeImpl]: Node <TEST2/192.168.5.98:8084> received RequestVoteRequest from 192.168.5.98:8085, term=393, currTerm=393. [2022-05-19 20:42:10,915][WARN][JRaft-VoteTimer-<TEST2/192.168.5.98:8085>0][com.alipay.sofa.jraft.core.NodeImpl]: Candidate node <TEST2/192.168.5.98:8085> term 393 steps down when election reaching vote timeout: fail to get quorum vote-granted. [2022-05-19 20:42:10,915][INFO][JRaft-VoteTimer-<TEST2/192.168.5.98:8085>0][com.alipay.sofa.jraft.core.NodeImpl]: Node <TEST2/192.168.5.98:8085> term 393 start preVote. [2022-05-19 20:42:11,187][WARN][JRaft-VoteTimer-<TEST2/192.168.5.98:8084>0][com.alipay.sofa.jraft.core.NodeImpl]: Candidate node <TEST2/192.168.5.98:8084> term 393 steps down when election reaching vote timeout: fail to get quorum vote-granted. [2022-05-19 20:42:11,187][INFO][JRaft-VoteTimer-<TEST2/192.168.5.98:8084>0][com.alipay.sofa.jraft.core.NodeImpl]: Node <TEST2/192.168.5.98:8084> term 393 start preVote. [2022-05-19 20:42:11,348][ERROR][scheduling-1][com.alipay.sofa.jraft.rpc.impl.AbstractClientService]: Fail to connect 192.168.5.98:8086, remoting exception: com.alipay.remoting.exception.RemotingException: Create connection failed. The address is 192.168.5.98:8086. [2022-05-19 20:42:11,348][ERROR][scheduling-1][com.alipay.sofa.jraft.core.CliServiceImpl]: Fail to connect peer 192.168.5.98:8086 to get leader for group TEST2. 2、这个是等待一些时间后，B机器自己连接成功，A机器并没有进行操作开启服务 [2022-05-19 20:43:32,798][ERROR][JRaft-RPC-Processor-18][com.alipay.sofa.jraft.rpc.impl.AbstractClientService]: Fail to connect 192.168.5.98:8086, remoting exception: com.alipay.remoting.exception.RemotingException: Create connection failed. The address is 192.168.5.98:8086. [2022-05-19 20:43:32,798][WARN][JRaft-RPC-Processor-18][com.alipay.sofa.jraft.core.NodeImpl]: Node <TEST2/192.168.5.98:8084> channel init failed, address=192.168.5.98:8086. [2022-05-19 20:43:32,801][WARN][JRaft-RPC-Processor-18][com.alipay.sofa.jraft.util.Utils]: Unable to fsync directory \database\jraft\master\data\raft_meta on windows. [2022-05-19 20:43:32,801][INFO][JRaft-RPC-Processor-18][com.alipay.sofa.jraft.storage.impl.LocalRaftMetaStorage]: Save raft meta, path=/database/jraft/master/data\raft_meta, term=420, votedFor=192.168.5.98:8084, cost time=3 ms [2022-05-19 20:43:32,876][WARN][JRaft-VoteTimer-<TEST2/192.168.5.98:8084>0][com.alipay.sofa.jraft.core.NodeImpl]: Candidate node <TEST2/192.168.5.98:8084> term 420 steps down when election reaching vote timeout: fail to get quorum vote-granted. [2022-05-19 20:43:32,876][INFO][JRaft-VoteTimer-<TEST2/192.168.5.98:8084>0][com.alipay.sofa.jraft.core.NodeImpl]: Node <TEST2/192.168.5.98:8084> term 420 start preVote. [2022-05-19 20:43:33,172][ERROR][JRaft-VoteTimer-<TEST2/192.168.5.98:8085>0][com.alipay.sofa.jraft.rpc.impl.AbstractClientService]: Fail to connect 192.168.5.98:8086, remoting exception: com.alipay.remoting.exception.RemotingException: Create connection failed. The address is 192.168.5.98:8086. [2022-05-19 20:43:33,172][WARN][JRaft-VoteTimer-<TEST2/192.168.5.98:8085>0][com.alipay.sofa.jraft.core.NodeImpl]: Node <TEST2/192.168.5.98:8085> channel init failed, address=192.168.5.98:8086. [2022-05-19 20:43:33,172][INFO][Bolt-default-executor-6-thread-8][com.alipay.sofa.jraft.core.NodeImpl]: Node <TEST2/192.168.5.98:8085> received RequestVoteRequest from 192.168.5.98:8084, term=420, currTerm=419. [2022-05-19 20:43:33,175][WARN][Bolt-default-executor-6-thread-8][com.alipay.sofa.jraft.util.Utils]: Unable to fsync directory \database\jraft\backup\data\raft_meta on windows. [2022-05-19 20:43:33,175][INFO][Bolt-default-executor-6-thread-8][com.alipay.sofa.jraft.storage.impl.LocalRaftMetaStorage]: Save raft meta, path=/database/jraft/backup/data\raft_meta, term=420, votedFor=0.0.0.0:0, cost time=3 ms [2022-05-19 20:43:33,176][WARN][Bolt-default-executor-6-thread-8][com.alipay.sofa.jraft.util.Utils]: Unable to fsync directory \database\jraft\backup\data\raft_meta on windows. [2022-05-19 20:43:33,176][INFO][Bolt-default-executor-6-thread-8][com.alipay.sofa.jraft.storage.impl.LocalRaftMetaStorage]: Save raft meta, path=/database/jraft/backup/data\raft_meta, term=420, votedFor=192.168.5.98:8084, cost time=1 ms [2022-05-19 20:43:33,890][ERROR][JRaft-VoteTimer-<TEST2/192.168.5.98:8084>0][com.alipay.sofa.jraft.rpc.impl.AbstractClientService]: Fail to connect 192.168.5.98:8086, remoting exception: com.alipay.remoting.exception.RemotingException: Create connection failed. The address is 192.168.5.98:8086. [2022-05-19 20:43:33,890][WARN][JRaft-VoteTimer-<TEST2/192.168.5.98:8084>0][com.alipay.sofa.jraft.core.NodeImpl]: Node <TEST2/192.168.5.98:8084> channel init failed, address=192.168.5.98:8086. [2022-05-19 20:43:33,891][INFO][Bolt-default-executor-6-thread-2][com.alipay.sofa.jraft.core.NodeImpl]: Node <TEST2/192.168.5.98:8084> received PreVoteRequest from 192.168.5.98:8085, term=420, currTerm=420, granted=true, requestLastLogId=LogId [index=109, term=389], lastLogId=LogId [index=109, term=389]. [2022-05-19 20:43:33,891][WARN][JRaft-RPC-Processor-19][com.alipay.sofa.jraft.core.NodeImpl]: Node <TEST2/192.168.5.98:8084> received invalid RequestVoteResponse from 192.168.5.98:8085, state not in STATE_CANDIDATE but STATE_FOLLOWER. [2022-05-19 20:43:33,891][INFO][Bolt-default-executor-6-thread-3][com.alipay.sofa.jraft.core.NodeImpl]: Node <TEST2/192.168.5.98:8085> received PreVoteRequest from 192.168.5.98:8084, term=421, currTerm=420, granted=true, requestLastLogId=LogId [index=109, term=389], lastLogId=LogId [index=109, term=389]. [2022-05-19 20:43:33,891][WARN][JRaft-RPC-Processor-20][com.alipay.sofa.jraft.core.NodeImpl]: Node <TEST2/192.168.5.98:8085> received invalid PreVoteResponse from 192.168.5.98:8084, term=419, currTerm=420. [2022-05-19 20:43:33,892][INFO][JRaft-RPC-Processor-20][com.alipay.sofa.jraft.core.NodeImpl]: Node <TEST2/192.168.5.98:8084> received PreVoteResponse from 192.168.5.98:8085, term=420, granted=true. [2022-05-19 20:43:33,892][INFO][JRaft-RPC-Processor-20][com.alipay.sofa.jraft.core.NodeImpl]: Node <TEST2/192.168.5.98:8084> start vote and grant vote self, term=420. [2022-05-19 20:43:34,185][ERROR][scheduling-1][com.alipay.sofa.jraft.rpc.impl.AbstractClientService]: Fail to connect 192.168.5.98:8086, remoting exception: com.alipay.remoting.exception.RemotingException: Create connection failed. The address is 192.168.5.98:8086. [2022-05-19 20:43:34,185][ERROR][scheduling-1][com.alipay.sofa.jraft.core.CliServiceImpl]: Fail to connect peer 192.168.5.98:8086 to get leader for group TEST2. [2022-05-19 20:43:34,201][INFO][JRaft-ElectionTimer-<TEST2/192.168.5.98:8085>0][com.alipay.sofa.jraft.core.NodeImpl]: Node <TEST2/192.168.5.98:8085> term 420 start preVote. [2022-05-19 20:43:34,903][ERROR][JRaft-RPC-Processor-20][com.alipay.sofa.jraft.rpc.impl.AbstractClientService]: Fail to connect 192.168.5.98:8086, remoting exception: com.alipay.remoting.exception.RemotingException: Create connection failed. The address is 192.168.5.98:8086. [2022-05-19 20:43:34,903][WARN][JRaft-RPC-Processor-20][com.alipay.sofa.jraft.core.NodeImpl]: Node <TEST2/192.168.5.98:8084> channel init failed, address=192.168.5.98:8086. [2022-05-19 20:43:34,905][WARN][JRaft-RPC-Processor-20][com.alipay.sofa.jraft.util.Utils]: Unable to fsync directory \database\jraft\master\data\raft_meta on windows. [2022-05-19 20:43:34,905][INFO][JRaft-RPC-Processor-20][com.alipay.sofa.jraft.storage.impl.LocalRaftMetaStorage]: Save raft meta, path=/database/jraft/master/data\raft_meta, term=421, votedFor=192.168.5.98:8084, cost time=2 ms [2022-05-19 20:43:35,202][ERROR][JRaft-ElectionTimer-<TEST2/192.168.5.98:8085>0][com.alipay.sofa.jraft.rpc.impl.AbstractClientService]: Fail to connect 192.168.5.98:8086, remoting exception: com.alipay.remoting.exception.RemotingException: Create connection failed. The address is 192.168.5.98:8086. [2022-05-19 20:43:35,202][WARN][JRaft-ElectionTimer-<TEST2/192.168.5.98:8085>0][com.alipay.sofa.jraft.core.NodeImpl]: Node <TEST2/192.168.5.98:8085> channel init failed, address=192.168.5.98:8086. [2022-05-19 20:43:35,202][INFO][Bolt-default-executor-6-thread-10][com.alipay.sofa.jraft.core.NodeImpl]: Node <TEST2/192.168.5.98:8085> received RequestVoteRequest from 192.168.5.98:8084, term=421, currTerm=420. [2022-05-19 20:43:35,203][INFO][Bolt-default-executor-6-thread-1][com.alipay.sofa.jraft.core.NodeImpl]: Node <TEST2/192.168.5.98:8084> received PreVoteRequest from 192.168.5.98:8085, term=421, currTerm=421, granted=true, requestLastLogId=LogId [index=109, term=389], lastLogId=LogId [index=109, term=389]. [2022-05-19 20:43:35,209][WARN][Bolt-default-executor-6-thread-10][com.alipay.sofa.jraft.util.Utils]: Unable to fsync directory \database\jraft\backup\data\raft_meta on windows. [2022-05-19 20:43:35,209][INFO][Bolt-default-executor-6-thread-10][com.alipay.sofa.jraft.storage.impl.LocalRaftMetaStorage]: Save raft meta, path=/database/jraft/backup/data\raft_meta, term=421, votedFor=0.0.0.0:0, cost time=7 ms [2022-05-19 20:43:35,211][WARN][Bolt-default-executor-6-thread-10][com.alipay.sofa.jraft.util.Utils]: Unable to fsync directory \database\jraft\backup\data\raft_meta on windows. [2022-05-19 20:43:35,211][INFO][Bolt-default-executor-6-thread-10][com.alipay.sofa.jraft.storage.impl.LocalRaftMetaStorage]: Save raft meta, path=/database/jraft/backup/data\raftmeta, term=421, votedFor=192.168.5.98:8084, cost time=2 ms [2022-05-19 20:43:35,212][WARN][JRaft-RPC-Processor-21][com.alipay.sofa.jraft.core.NodeImpl]: Node <TEST2/192.168.5.98:8085> received invalid PreVoteResponse from 192.168.5.98:8084, term=420, currTerm=421. [2022-05-19 20:43:35,212][INFO][JRaft-RPC-Processor-21][com.alipay.sofa.jraft.core.NodeImpl]: Node <TEST2/192.168.5.98:8084> become leader of group, term=421, conf=192.168.5.98:8084,192.168.5.98:8085,192.168.5.98:8086, oldConf=. [2022-05-19 20:43:36,217][ERROR][scheduling-1][com.alipay.sofa.jraft.rpc.impl.AbstractClientService]: Fail to connect 192.168.5.98:8086, remoting exception: com.alipay.remoting.exception.RemotingException: Create connection failed. The address is 192.168.5.98:8086. [2022-05-19 20:43:36,217][ERROR][scheduling-1][com.alipay.sofa.jraft.core.CliServiceImpl]: Fail to connect peer 192.168.5.98:8086 to get leader for group TEST2. [2022-05-19 20:43:36,217][ERROR][JRaft-RPC-Processor-21][com.alipay.sofa.jraft.rpc.impl.AbstractClientService]: Fail to connect 192.168.5.98:8086, remoting exception: com.alipay.remoting.exception.RemotingException: Create connection failed. The address is 192.168.5.98:8086. [2022-05-19 20:43:36,217][ERROR][JRaft-RPC-Processor-21][com.alipay.sofa.jraft.core.Replicator]: Fail to init sending channel to 192.168.5.98:8086. [2022-05-19 20:43:36,217][ERROR][JRaft-RPC-Processor-21][com.alipay.sofa.jraft.core.ReplicatorGroupImpl]: Fail to start replicator to peer=192.168.5.98:8086, replicatorType=Follower. [2022-05-19 20:43:36,217][ERROR][JRaft-RPC-Processor-21][com.alipay.sofa.jraft.core.NodeImpl]: Fail to add a replicator, peer=192.168.5.98:8086. [2022-05-19 20:43:36,218][INFO][JRaft-RPC-Processor-21][com.alipay.sofa.jraft.core.Replicator]: Replicator=Replicator [state=Created, statInfo=, peerId=192.168.5.98:8085, type=Follower]@192.168.5.98:8085 is started [2022-05-19 20:43:36,222][WARN][Rpc-netty-server-worker-1-thread-2][com.alipay.sofa.jraft.rpc.impl.BoltRaftRpcFactory]: JRaft SET bolt.rpc.dispatch-msg-list-in-default-executor to be false for replicator pipeline optimistic. [2022-05-19 20:43:36,225][INFO][JRaft-RPC-Processor-21][com.alipay.sofa.jraft.util.Recyclers]: -Djraft.recyclers.maxCapacityPerThread: 4096. [2022-05-19 20:43:36,227][INFO][JRaft-FSMCaller-Disruptor-0][com.alipay.sofa.jraft.core.StateMachineAdapter]: onStartFollowing: LeaderChangeContext [leaderId=192.168.5.98:8084, term=421, status=Status[ENEWLEADER<10011>: Follower receives message from new leader with the same term.]]. Hibernate: select sysraftclu0.c_id as c_id1_100, sysraftclu0_.c_balance_mode as c_balanc2_100, sysraftclu0_.c_cluster_mode as c_cluste3_100, sysraftclu0_.c_cluster_status as c_cluste4_100, sysraftclu0_.c_endpoints as c_endpoi5_100, sysraftclu0_.c_endpoints_vip as c_endpoi6_100, sysraftclu0_.c_group_id as c_group_7_100, sysraftclu0_.c_ip as c_ip8_100, sysraftclu0_.c_joined as c_joined9_100, sysraftclu0_.c_license_port as c_licen10_100, sysraftclu0_.c_master as c_maste11_100, sysraftclu0_.c_node as c_node12_100, sysraftclu0_.c_node2 as c_node13_100, sysraftclu0_.c_other_endpoints_vip as c_other14_100, sysraftclu0_.c_port as c_port15_100, sysraftclu0_.c_status as c_statu16_100, sysraftclu0_.c_sync_status as c_sync_17_100, sysraftclu0_.c_vip_iface as c_vip_i18_100 from sys_raftcluster sysraftclu0 where sysraftclu0_.c_id=?

[2022-05-19 20:43:36,240][INFO][JRaft-FSMCaller-Disruptor-0][com.alipay.sofa.jraft.core.StateMachineAdapter]: onLeaderStart: term=421.

[2022-05-19 20:43:36,244][INFO][Bolt-conn-event-executor-5-thread-1][com.alipay.sofa.jraft.rpc.impl.core.ClientServiceConnectionEventProcessor]: Peer 192.168.5.98:8084 is connected [2022-05-19 20:43:36,319][INFO][JRaft-FSMCaller-Disruptor-0][com.visec.platform.webstarter.cluster.core.jraft.RaftStateMachine]: Added value, logIndex=111

[2022-05-19 20:43:36,328][INFO][JRaft-FSMCaller-Disruptor-0][com.visec.platform.webstarter.cluster.core.jraft.RaftStateMachine]: Added value, logIndex=111

[2022-05-19 20:43:36,329][INFO][scheduling-1][com.visec.platform.webstarter.cluster.core.tasks.ScheduledRaftTasks]: 版本信息, leader[1.1.1], 本地[1.1.1] [2022-05-19 20:43:36,329][INFO][scheduling-1][com.visec.platform.webstarter.cluster.core.tasks.ScheduledRaftTasks]: 模块信息, leader[], 本地[]

Hibernate: update sys_raft_cluster set c_balance_mode=?, c_cluster_mode=?, c_cluster_status=?, c_endpoints=?, c_endpoints_vip=?, c_group_id=?, c_ip=?, c_joined=?, c_license_port=?, c_master=?, c_node=?, c_node2=?, c_other_endpoints_vip=?, c_port=?, c_status=?, c_sync_status=?, c_vip_iface=? where c_id=? [2022-05-19 20:43:36,388][INFO][scheduling-1][com.visec.example.cluster.SubscriberImpl]: 业务系统收到cluster发来的消息:[{"newValue":true,"endpoint":"192.168.5.98:8084","groupId":"TEST2","self":true,"noticeType":"LEADER","oldValue":false}, {"newValue":true,"endpoint":"192.168.5.98:8084","groupId":"TEST2","self":true,"noticeType":"HEALTH","oldValue":false}, {"newValue":true,"endpoint":"192.168.5.98:8085","groupId":"TEST2","self":false,"noticeType":"HEALTH","oldValue":false}] [2022-05-19 20:43:36,730][ERROR][JRaft-StepDownTimer-<TEST2/192.168.5.98:8084>0][com.alipay.sofa.jraft.core.ReplicatorGroupImpl]: Fail to check replicator connection to peer=192.168.5.98:8086, replicatorType=Follower. [2022-05-19 20:43:37,243][ERROR][JRaft-StepDownTimer-<TEST2/192.168.5.98:8084>0][com.alipay.sofa.jraft.core.ReplicatorGroupImpl]: Fail to check replicator connection to peer=192.168.5.98:8086, replicatorType=Follower. [2022-05-19 20:43:37,758][ERROR][JRaft-StepDownTimer-<TEST2/192.168.5.98:8084>0][com.alipay.sofa.jraft.core.ReplicatorGroupImpl]: Fail to check replicator connection to peer=192.168.5.98:8086, replicatorType=Follower. [2022-05-19 20:43:38,272][ERROR][JRaft-StepDownTimer-<TEST2/192.168.5.98:8084>0][com.alipay.sofa.jraft.core.ReplicatorGroupImpl]: Fail to check replicator connection to peer=192.168.5.98:8086, replicatorType=Follower. [2022-05-19 20:43:38,784][ERROR][JRaft-StepDownTimer-<TEST2/192.168.5.98:8084>0][com.alipay.sofa.jraft.core.ReplicatorGroupImpl]: Fail to check replicator connection to peer=192.168.5.98:8086, replicatorType=Follower. [2022-05-19 20:43:39,297][ERROR][JRaft-StepDownTimer-<TEST2/192.168.5.98:8084>0][com.alipay.sofa.jraft.core.ReplicatorGroupImpl]: Fail to check replicator connection to peer=192.168.5.98:8086, replicatorType=Follower. [2022-05-19 20:43:39,811][ERROR][JRaft-StepDownTimer-<TEST2/192.168.5.98:8084>0][com.alipay.sofa.jraft.core.ReplicatorGroupImpl]: Fail to check replicator connection to peer=192.168.5.98:8086, replicatorType=Follower. [2022-05-19 20:43:40,324][ERROR][JRaft-StepDownTimer-<TEST2/192.168.5.98:8084>0][com.alipay.sofa.jraft.core.ReplicatorGroupImpl]: Fail to check replicator connection to peer=192.168.5.98:8086, replicatorType=Follower. [2022-05-19 20:43:40,835][ERROR][JRaft-StepDownTimer-<TEST2/192.168.5.98:8084>0][com.alipay.sofa.jraft.core.ReplicatorGroupImpl]: Fail to check replicator connection to peer=192.168.5.98:8086, replicatorType=Follower. [2022-05-19 20:43:41,349][ERROR][JRaft-StepDownTimer-<TEST2/192.168.5.98:8084>0][com.alipay.sofa.jraft.core.ReplicatorGroupImpl]: Fail to check replicator connection to peer=192.168.5.98:8086, replicatorType=Follower. [2022-05-19 20:43:41,862][ERROR][JRaft-StepDownTimer-<TEST2/192.168.5.98:8084>0][com.alipay.sofa.jraft.core.ReplicatorGroupImpl]: Fail to check replicator connection to peer=192.168.5.98:8086, replicatorType=Follower. [2022-05-19 20:43:42,376][ERROR][JRaft-StepDownTimer-<TEST2/192.168.5.98:8084>0][com.alipay.sofa.jraft.core.ReplicatorGroupImpl]: Fail to check replicator connection to peer=192.168.5.98:8086, replicatorType=Follower. [2022-05-19 20:43:42,890][ERROR][JRaft-StepDownTimer-<TEST2/192.168.5.98:8084>0][com.alipay.sofa.jraft.core.ReplicatorGroupImpl]: Fail to check replicator connection to peer=192.168.5.98:8086, replicatorType=Follower. [2022-05-19 20:43:43,406][ERROR][JRaft-StepDownTimer-<TEST2/192.168.5.98:8084>0][com.alipay.sofa.jraft.core.ReplicatorGroupImpl]: Fail to check replicator connection to peer=192.168.5.98:8086, replicatorType=Follower. [2022-05-19 20:43:43,920][ERROR][JRaft-StepDownTimer-<TEST2/192.168.5.98:8084>0][com.alipay.sofa.jraft.core.ReplicatorGroupImpl]: Fail to check replicator connection to peer=192.168.5.98:8086, replicatorType=Follower. [2022-05-19 20:43:44,430][ERROR][JRaft-StepDownTimer-<TEST2/192.168.5.98:8084>0][com.alipay.sofa.jraft.core.ReplicatorGroupImpl]: Fail to check replicator connection to peer=192.168.5.98:8086, replicatorType=Follower. [2022-05-19 20:43:44,944][ERROR][JRaft-StepDownTimer-<TEST2/192.168.5.98:8084>0][com.alipay.sofa.jraft.core.ReplicatorGroupImpl]: Fail to check replicator connection to peer=192.168.5.98:8086, replicatorType=Follower. [2022-05-19 20:43:45,458][ERROR][JRaft-StepDownTimer-<TEST2/192.168.5.98:8084>0][com.alipay.sofa.jraft.core.ReplicatorGroupImpl]: Fail to check replicator connection to peer=192.168.5.98:8086, replicatorType=Follower. [2022-05-19 20:43:45,961][ERROR][JRaft-StepDownTimer-<TEST2/192.168.5.98:8084>0][com.alipay.sofa.jraft.core.ReplicatorGroupImpl]: Fail to check replicator connection to peer=192.168.5.98:8086, replicatorType=Follower. [2022-05-19 20:43:46,464][ERROR][JRaft-StepDownTimer-<TEST2/192.168.5.98:8084>0][com.alipay.sofa.jraft.core.ReplicatorGroupImpl]: Fail to check replicator connection to peer=192.168.5.98:8086, replicatorType=Follower. [2022-05-19 20:43:46,980][ERROR][JRaft-StepDownTimer-<TEST2/192.168.5.98:8084>0][com.alipay.sofa.jraft.core.ReplicatorGroupImpl]: Fail to check replicator connection to peer=192.168.5.98:8086, replicatorType=Follower. [2022-05-19 20:43:47,483][ERROR][JRaft-StepDownTimer-<TEST2/192.168.5.98:8084>0][com.alipay.sofa.jraft.core.ReplicatorGroupImpl]: Fail to check replicator connection to peer=192.168.5.98:8086, replicatorType=Follower. [2022-05-19 20:43:47,997][ERROR][JRaft-StepDownTimer-<TEST2/192.168.5.98:8084>0][com.alipay.sofa.jraft.core.ReplicatorGroupImpl]: Fail to check replicator connection to peer=192.168.5.98:8086, replicatorType=Follower. [2022-05-19 20:43:48,500][ERROR][JRaft-StepDownTimer-<TEST2/192.168.5.98:8084>0][com.alipay.sofa.jraft.core.ReplicatorGroupImpl]: Fail to check replicator connection to peer=192.168.5.98:8086, replicatorType=Follower. [2022-05-19 20:43:49,001][ERROR][JRaft-StepDownTimer-<TEST2/192.168.5.98:8084>0][com.alipay.sofa.jraft.core.ReplicatorGroupImpl]: Fail to check replicator connection to peer=192.168.5.98:8086, replicatorType=Follower. [2022-05-19 20:43:49,504][ERROR][JRaft-StepDownTimer-<TEST2/192.168.5.98:8084>0][com.alipay.sofa.jraft.core.ReplicatorGroupImpl]: Fail to check replicator connection to peer=192.168.5.98:8086, replicatorType=Follower. [2022-05-19 20:43:50,007][ERROR][JRaft-StepDownTimer-<TEST2/192.168.5.98:8084>0][com.alipay.sofa.jraft.core.ReplicatorGroupImpl]: Fail to check replicator connection to peer=192.168.5.98:8086, replicatorType=Follower.

[2022-05-19 20:43:50,153][WARN][JRaft-Closure-Executor-10][com.alipay.sofa.jraft.util.Utils]: Unable to fsync directory \database\jraft\master\data\snapshot\temp on windows. [2022-05-19 20:43:50,153][INFO][JRaft-Closure-Executor-10][com.alipay.sofa.jraft.storage.snapshot.local.LocalSnapshotStorage]: Deleting snapshot /database/jraft/master/data\snapshot\snapshot_111. [2022-05-19 20:43:50,153][INFO][JRaft-Closure-Executor-10][com.alipay.sofa.jraft.storage.snapshot.local.LocalSnapshotStorage]: Renaming /database/jraft/master/data\snapshot\temp to /database/jraft/master/data\snapshot\snapshot_111. [2022-05-19 20:43:50,154][WARN][JRaft-Closure-Executor-10][com.alipay.sofa.jraft.util.Utils]: Unable to fsync directory \database\jraft\master\data\snapshot on windows. [2022-05-19 20:43:50,154][INFO][JRaft-Closure-Executor-10][com.alipay.sofa.jraft.storage.snapshot.local.LocalSnapshotStorage]: Deleting snapshot /database/jraft/master/data\snapshot\snapshot_109. [2022-05-19 20:43:50,158][INFO][JRaft-Closure-Executor-10][com.alipay.sofa.jraft.storage.impl.RocksDBLogStorage]: Truncated prefix logs in data path: /database/jraft/master/data\log from log index 110 to 110, cost 0 ms. [2022-05-19 20:43:50,508][ERROR][JRaft-StepDownTimer-<TEST2/192.168.5.98:8084>0][com.alipay.sofa.jraft.core.ReplicatorGroupImpl]: Fail to check replicator connection to peer=192.168.5.98:8086, replicatorType=Follower. [2022-05-19 20:43:51,009][ERROR][JRaft-StepDownTimer-<TEST2/192.168.5.98:8084>0][com.alipay.sofa.jraft.core.ReplicatorGroupImpl]: Fail to check replicator connection to peer=192.168.5.98:8086, replicatorType=Follower. [2022-05-19 20:43:51,511][ERROR][JRaft-StepDownTimer-<TEST2/192.168.5.98:8084>0][com.alipay.sofa.jraft.core.ReplicatorGroupImpl]: Fail to check replicator connection to peer=192.168.5.98:8086, replicatorType=Follower. [2022-05-19 20:43:52,013][ERROR][JRaft-StepDownTimer-<TEST2/192.168.5.98:8084>0][com.alipay.sofa.jraft.core.ReplicatorGroupImpl]: Fail to check replicator connection to peer=192.168.5.98:8086, replicatorType=Follower. [2022-05-19 20:43:52,514][ERROR][JRaft-StepDownTimer-<TEST2/192.168.5.98:8084>0][com.alipay.sofa.jraft.core.ReplicatorGroupImpl]: Fail to check replicator connection to peer=192.168.5.98:8086, replicatorType=Follower. [2022-05-19 20:43:53,017][ERROR][JRaft-StepDownTimer-<TEST2/192.168.5.98:8084>0][com.alipay.sofa.jraft.core.ReplicatorGroupImpl]: Fail to check replicator connection to peer=192.168.5.98:8086, replicatorType=Follower. [2022-05-19 20:43:53,518][ERROR][JRaft-StepDownTimer-<TEST2/192.168.5.98:8084>0][com.alipay.sofa.jraft.core.ReplicatorGroupImpl]: Fail to check replicator connection to peer=192.168.5.98:8086, replicatorType=Follower. [2022-05-19 20:43:54,019][ERROR][JRaft-StepDownTimer-<TEST2/192.168.5.98:8084>0][com.alipay.sofa.jraft.core.ReplicatorGroupImpl]: Fail to check replicator connection to peer=192.168.5.98:8086, replicatorType=Follower. [2022-05-19 20:43:54,521][ERROR][JRaft-StepDownTimer-<TEST2/192.168.5.98:8084>0][com.alipay.sofa.jraft.core.ReplicatorGroupImpl]: Fail to check replicator connection to peer=192.168.5.98:8086, replicatorType=Follower. [2022-05-19 20:43:55,022][ERROR][JRaft-StepDownTimer-<TEST2/192.168.5.98:8084>0][com.alipay.sofa.jraft.core.ReplicatorGroupImpl]: Fail to check replicator connection to peer=192.168.5.98:8086, replicatorType=Follower.

[2022-05-19 20:43:55,379][WARN][JRaft-Closure-Executor-11][com.alipay.sofa.jraft.util.Utils]: Unable to fsync directory \database\jraft\backup\data\snapshot\temp on windows. [2022-05-19 20:43:55,379][INFO][JRaft-Closure-Executor-11][com.alipay.sofa.jraft.storage.snapshot.local.LocalSnapshotStorage]: Deleting snapshot /database/jraft/backup/data\snapshot\snapshot_111. [2022-05-19 20:43:55,379][INFO][JRaft-Closure-Executor-11][com.alipay.sofa.jraft.storage.snapshot.local.LocalSnapshotStorage]: Renaming /database/jraft/backup/data\snapshot\temp to /database/jraft/backup/data\snapshot\snapshot_111. [2022-05-19 20:43:55,379][WARN][JRaft-Closure-Executor-11][com.alipay.sofa.jraft.util.Utils]: Unable to fsync directory \database\jraft\backup\data\snapshot on windows. [2022-05-19 20:43:55,379][INFO][JRaft-Closure-Executor-11][com.alipay.sofa.jraft.storage.snapshot.local.LocalSnapshotStorage]: Deleting snapshot /database/jraft/backup/data\snapshot\snapshot_109. [2022-05-19 20:43:55,381][INFO][JRaft-Closure-Executor-11][com.alipay.sofa.jraft.storage.impl.RocksDBLogStorage]: Truncated prefix logs in data path: /database/jraft/backup/data\log from log index 110 to 110, cost 0 ms. [2022-05-19 20:43:55,523][ERROR][JRaft-StepDownTimer-<TEST2/192.168.5.98:8084>0][com.alipay.sofa.jraft.core.ReplicatorGroupImpl]: Fail to check replicator connection to peer=192.168.5.98:8086, replicatorType=Follower.

killme2008 commented 2 years ago

初步看了下日志，似乎是两个 election 定时器完全同步，导致选举冲突了，在同一进程启动多个 raft 节点情况下很可能出现。

我们的调整建议：

  1. 同一个进程内的多个 raft 节点，建议启动时间通过增加一些随机时间（sleep) 错开启动
  2. 后续我们会改进下，针对这种情况，内部增加随机因子
SpringLin97 commented 2 years ago

初步看了下日志，似乎是两个 election 定时器完全同步，导致选举冲突了，在同一进程启动多个 raft 节点情况下很可能出现。

我们的调整建议：

  1. 同一个进程内的多个 raft 节点，建议启动时间通过增加一些随机时间（sleep) 错开启动
  2. 后续我们会改进下，针对这种情况，内部增加随机因子

加了sleep几秒效果也是一样，不过部署在linux就很少出现这种情况，window出现概率比较大

funky-eyes commented 1 year ago

我也遇到了类似的问题，同台机器启动2个节点时，第三个不启动时选举很慢，如果启动第三个节点选举马上就成功了，这个时候干掉leader重选举也很快成功了