Closed lilinghai closed 4 months ago
the region has peer on 29(tiflash-2) and 31(tiflash-0).
store 31 is which is tiflash-0, send a remote read request to store 29 which is tiflash-2 at 2023/07/30 23:27:05.167 +08:00
, and got an error remote reader meets error: Region epoch not match after retries: Region {319305,31,167} not in region cache.
We can see logs on tiflash-2
[2023/07/30 23:26:20.643 +08:00] [INFO] [FlashService.cpp:169] ["Handling coprocessor request, is_remote_read: true, start ts: 443214871671341074, region info: 319305, region epoch: conf_ver: 31\nversion: 167\n"] [source=FlashService] [thread_id=2280]
at that time, the region does not exist in store 6, it actually is removed since 2023/07/30 21:18:43.436 +08:00
.
[2023/07/30 21:18:43.434 +08:00] [INFO] [apply.rs:2465] ["conf change successfully"] ["current region"="id: 319305 start_key: 7480000000000000FFFE5F728000000000FF0100D20000000000FA end_key: 7480000000000000FFFE5F728000000000FF01678C0000000000FA region_epoch { conf_ver: 28 version: 167 } peers { id: 319306 store_id: 4 } peers { id: 319307 store_id: 6 role: DemotingVoter } peers { id: 319308 store_id: 1 } peers { id: 319309 store_id: 31 role: Learner } peers { id: 319310 store_id: 29 role: Learner } peers { id: 347199 store_id: 5 role: IncomingVoter }"] ["original region"="id: 319305 start_key: 7480000000000000FFFE5F728000000000FF0100D20000000000FA end_key: 7480000000000000FFFE5F728000000000FF01678C0000000000FA region_epoch { conf_ver: 26 version: 167 } peers { id: 319306 store_id: 4 } peers { id: 319307 store_id: 6 } peers { id: 319308 store_id: 1 } peers { id: 319309 store_id: 31 role: Learner } peers { id: 319310 store_id: 29 role: Learner } peers { id: 347199 store_id: 5 role: Learner }"] [changes="[peer { id: 347199 store_id: 5 }, change_type: AddLearnerNode peer { id: 319307 store_id: 6 role: Learner }]"] [peer_id=319309] [region_id=319305]
[2023/07/30 21:18:43.436 +08:00] [INFO] [apply.rs:2465] ["conf change successfully"] ["current region"="id: 319305 start_key: 7480000000000000FFFE5F728000000000FF0100D20000000000FA end_key: 7480000000000000FFFE5F728000000000FF01678C0000000000FA region_epoch { conf_ver: 31 version: 167 } peers { id: 319306 store_id: 4 } peers { id: 319308 store_id: 1 } peers { id: 319309 store_id: 31 role: Learner } peers { id: 319310 store_id: 29 role: Learner } peers { id: 347199 store_id: 5 }"] ["original region"="id: 319305 start_key: 7480000000000000FFFE5F728000000000FF0100D20000000000FA end_key: 7480000000000000FFFE5F728000000000FF01678C0000000000FA region_epoch { conf_ver: 30 version: 167 } peers { id: 319306 store_id: 4 } peers { id: 319307 store_id: 6 role: Learner } peers { id: 319308 store_id: 1 } peers { id: 319309 store_id: 31 role: Learner } peers { id: 319310 store_id: 29 role: Learner } peers { id: 347199 store_id: 5 }"] [changes="[change_type: RemoveNode peer { id: 319307 store_id: 6 role: Learner }]"] [peer_id=319309] [region_id=319305]
The local peer on store 31, which is 319309, creates at 2023/07/30 21:07:20.873 +08:00
[2023/07/30 21:07:20.873 +08:00] [INFO] [raft.rs:1127] ["became follower at term 6"] [term=6] [raft_id=319309] [region_id=319305]
[2023/07/30 21:07:20.873 +08:00] [INFO] [region.rs:103] [" 31:319305 319309, peer created"] [is_replicated=false] [role=Follower] [leader_id=319306] [region_id=319305]
The region epoch on store 31 is advanced to { conf_ver: 31 version: 167 }
[2023/07/30 21:18:43.436 +08:00] [INFO] [apply.rs:2465] ["conf change successfully"] ["current region"="id: 319305 start_key: 7480000000000000FFFE5F728000000000FF0100D20000000000FA end_key: 7480000000000000FFFE5F728000000000FF01678C0000000000FA region_epoch { conf_ver: 31 version: 167 } peers { id: 319306 store_id: 4 } peers { id: 319308 store_id: 1 } peers { id: 319309 store_id: 31 role: Learner } peers { id: 319310 store_id: 29 role: Learner } peers { id: 347199 store_id: 5 }"] ["original region"="id: 319305 start_key: 7480000000000000FFFE5F728000000000FF0100D20000000000FA end_key: 7480000000000000FFFE5F728000000000FF01678C0000000000FA region_epoch { conf_ver: 30 version: 167 } peers { id: 319306 store_id: 4 } peers { id: 319307 store_id: 6 role: Learner } peers { id: 319308 store_id: 1 } peers { id: 319309 store_id: 31 role: Learner } peers { id: 319310 store_id: 29 role: Learner } peers { id: 347199 store_id: 5 }"] [changes="[change_type: RemoveNode peer { id: 319307 store_id: 6 role: Learner }]"] [peer_id=319309] [region_id=319305]
[2023/07/30 21:21:13.668 +08:00] [INFO] [apply.rs:1690] ["execute admin command"] [command="cmd_type: UpdateGcPeer update_gc_peers { peer_id: 319307 }"] [index=12577] [term=7] [peer_id=319309] [region_id=319305]
[2023/07/30 22:12:10.945 +08:00] [INFO] [command.rs:169] ["can't flush data, filter CompactLog"] [compact_term=8] [compact_index=12875] [term=8] [index=12877] [region_epoch="conf_ver: 31 version: 167"] [region_id=319305]
The region epoch on store 29 is advanced to { conf_ver: 31 version: 167 }
[2023/07/30 21:18:38.465 +08:00] [INFO] [apply.rs:2284] ["exec ConfChangeV2"] [epoch="conf_ver: 30 version: 167"] [kind=Simple] [peer_id=319310] [region_id=319305]
[2023/07/30 21:18:38.465 +08:00] [INFO] [apply.rs:2465] ["conf change successfully"] ["current region"="id: 319305 start_key: 7480000000000000FFFE5F728000000000FF0100D20000000000FA end_key: 7480000000000000FFFE5F728000000000FF01678C0000000000FA region_epoch { conf_ver: 31 version: 167 } peers { id: 319306 store_id: 4 } peers { id: 319308 store_id: 1 } peers { id: 319309 store_id: 31 role: Learner } peers { id: 319310 store_id: 29 role: Learner } peers { id: 347199 store_id: 5 }"] ["original region"="id: 319305 start_key: 7480000000000000FFFE5F728000000000FF0100D20000000000FA end_key: 7480000000000000FFFE5F728000000000FF01678C0000000000FA region_epoch { conf_ver: 30 version: 167 } peers { id: 319306 store_id: 4 } peers { id: 319307 store_id: 6 role: Learner } peers { id: 319308 store_id: 1 } peers { id: 319309 store_id: 31 role: Learner } peers { id: 319310 store_id: 29 role: Learner } peers { id: 347199 store_id: 5 }"] [changes="[change_type: RemoveNode peer { id: 319307 store_id: 6 role: Learner }]"] [peer_id=319310] [region_id=319305]
[2023/07/30 22:12:12.446 +08:00] [INFO] [command.rs:169] ["can't flush data, filter CompactLog"] [compact_term=8] [compact_index=12875] [term=8] [index=12877] [region_epoch="conf_ver: 31 version: 167"] [region_id=319305]
However
[2023/07/30 23:27:04.667 +08:00] [INFO] [FlashService.cpp:169] ["Handling coprocessor request, is_remote_read: true, start ts: 443214871671341074, region info: 319305, region epoch: conf_ver: 31\nversion: 167\n"] [source=FlashService] [thread_id=2249]
[2023/07/30 23:27:04.667 +08:00] [INFO] [FlashService.cpp:221] ["Begin process cop request after wait 0 ms, start ts: 443214871671341074, region info: 319305, region epoch: conf_ver: 31\nversion: 167\n"] [source=FlashService] [thread_id=322]
[2023/07/30 23:27:04.667 +08:00] [DEBUG] [CoprocessorHandler.cpp:91] ["Handling DAG request: time_zone_offset: 28800\noutput_offsets: 0\nencode_type: TypeCHBlock\ntime_zone_name: \"Asia/Shanghai\"\ncollect_execution_summaries: false\nroot_executor {\n tp: TypeTableScan\n tbl_scan {\n table_id: 254\n columns {\n column_id: -1\n tp: 8\n collation: -63\n columnLen: 20\n decimal: 0\n flag: 3\n pk_handle: true\n }\n desc: false\n next_read_engine: Local\n keep_order: false\n is_fast_scan: false\n max_wait_time_ms: 0\n }\n executor_id: \"TableFullScan_16\"\n}\nforce_encode_type: true\n"] [source=CoprocessorHandler] [thread_id=322]
[2023/07/30 23:27:04.667 +08:00] [DEBUG] [executeQuery.cpp:386] ["(from 10.233.105.28:56294, query_id: c1352e2a-a220-493d-90e1-601003bcc055) time_zone_offset: 28800 output_offsets: 0 encode_type: TypeCHBlock time_zone_name: \"Asia/Shanghai\" collect_execution_summaries: false root_executor { tp: TypeTableScan tbl_scan { table_id: 254 columns { column_id: -1 tp: 8 collation: -63 columnLen: 20 decimal: 0 flag: 3 pk_handle: true } desc: false next_read_engine: Local keep_order: false is_fast_scan: false max_wait_time_ms: 0 } executor_id: \"TableFullScan_16\" } force_encode_type: true "] [source=CoprocessorHandler] [thread_id=322]
[2023/07/30 23:27:04.667 +08:00] [DEBUG] [DeltaValueSpace.cpp:338] ["Update index start, delta=<delta_id=165>"] [source="keyspace=4294967295 table_id=261 segment_id=164 epoch=2"] [thread_id=369]
[2023/07/30 23:27:04.667 +08:00] [DEBUG] [PhysicalPlan.cpp:80] ["tidb table scan has runtime filter size:0"] [source=CoprocessorHandler] [thread_id=322]
[2023/07/30 23:27:04.667 +08:00] [DEBUG] [DeltaValueSpace.cpp:340] ["Update index done, delta=<delta_id=165>"] [source="keyspace=4294967295 table_id=261 segment_id=164 epoch=2"] [thread_id=369]
[2023/07/30 23:27:04.667 +08:00] [DEBUG] [PhysicalPlan.cpp:261] ["build unoptimized physical plan: \n<Projection, TableFullScan_16> | is_tidb_operator: false, schema: <TableFullScan_16__tidb_rowid, Int64>\n <TableScan, TableFullScan_16> | is_tidb_operator: true, schema: <_tidb_rowid, Int64>\n"] [source=CoprocessorHandler] [thread_id=322]
[2023/07/30 23:27:04.667 +08:00] [DEBUG] [PhysicalPlan.cpp:267] ["build optimized physical plan: \n<Projection, TableFullScan_16> | is_tidb_operator: false, schema: <TableFullScan_16__tidb_rowid, Int64>\n <TableScan, TableFullScan_16> | is_tidb_operator: true, schema: <_tidb_rowid, Int64>\n"] [source=CoprocessorHandler] [thread_id=322]
[2023/07/30 23:27:04.667 +08:00] [DEBUG] [LearnerRead.cpp:287] ["Batch read index, original size 1, send & get 1 message, cost 0ms"] [source=CoprocessorHandler] [thread_id=322]
[2023/07/30 23:27:04.667 +08:00] [DEBUG] [LearnerRead.cpp:409] ["Finish wait index | resolve locks, wait_cost=0ms n_regions=1 n_unavailable=1"] [source=CoprocessorHandler] [thread_id=322]
[2023/07/30 23:27:04.667 +08:00] [DEBUG] [ColumnFilePersistedSet.cpp:256] ["ColumnFilePersistedSet [165]: 24 column files, 97137 rows, 27781182 bytes, 0 deletes., after append 1 column files, persisted column files: [T_4863,T_4137,T_4137,T_4726,T_4863,T_4137,T_4137,T_4863,T_4137,T_4863,T_4137,T_4863,T_4137,T_4863,T_4137,T_4863,T_3137,T_4863,T_3000,T_137,T_1000,T_4863,T_4137,T_4137]"] [source="keyspace=4294967295 table_id=261 segment_id=164 epoch=2"] [thread_id=369]
[2023/07/30 23:27:04.667 +08:00] [WARN] [CoprocessorHandler.cpp:152] ["RegionException: region_id=319305, message: Region error NOT_FOUND"] [source=CoprocessorHandler] [thread_id=322]
[2023/07/30 23:27:04.667 +08:00] [DEBUG] [DeltaValueSpace.cpp:373] ["Flush end, flush_tasks=1 flush_rows=4137 flush_deletes=0 delta=MemTableSet: 0 column files, 0 rows, 0 bytes, 0 deletes. ColumnFilePersistedSet [165]: 24 column files, 97137 rows, 27781182 bytes, 0 deletes."] [source="keyspace=4294967295 table_id=261 segment_id=164 epoch=2"] [thread_id=369]
[2023/07/30 23:27:04.667 +08:00] [INFO] [FlashService.cpp:238] ["Handle coprocessor request done: 0, "] [source=FlashService] [thread_id=2249]
[2023/07/30 23:27:04.667 +08:00] [DEBUG] [DeltaValueSpace.h:210] ["Cannot get update lock because DeltaValueSpace is updating. Current update operation will be discarded, delta=<delta_id=165>"] [source="keyspace=4294967295 table_id=261 segment_id=164 epoch=2"] [thread_id=369]
The reason is tikv-0
[2023/07/30 23:23:55.635 +08:00] [WARN] [endpoint.rs:836] [error-response] [err="Region error (will back off and retry) message: \"peer is not leader for region 319305, leader may None\" not_leader { region_id: 319305 }"]
[2023/07/30 23:23:55.657 +08:00] [WARN] [endpoint.rs:836] [error-response] [err="Region error (will back off and retry) message: \"peer is not leader for region 319305, leader may None\" not_leader { region_id: 319305 }"]
[2023/07/30 23:23:55.735 +08:00] [WARN] [endpoint.rs:836] [error-response] [err="Region error (will back off and retry) message: \"peer is not leader for region 319305, leader may None\" not_leader { region_id: 319305 }"]
[2023/07/30 23:23:55.806 +08:00] [WARN] [endpoint.rs:836] [error-response] [err="Region error (will back off and retry) message: \"peer is not leader for region 319305, leader may None\" not_leader { region_id: 319305 }"]
...
[2023/07/30 23:28:24.053 +08:00] [WARN] [endpoint.rs:836] [error-response] [err="Region error (will back off and retry) message: \"peer is not leader for region 319305, leader may None\" not_leader { region_id: 319305 }"]
[2023/07/30 23:28:25.072 +08:00] [WARN] [endpoint.rs:836] [error-response] [err="Region error (will back off and retry) message: \"peer is not leader for region 319305, leader may None\" not_leader { region_id: 319305 }"]
[2023/07/30 23:28:26.075 +08:00] [WARN] [endpoint.rs:836] [error-response] [err="Region error (will back off and retry) message: \"peer is not leader for region 319305, leader may None\" not_leader { region_id: 319305 }"]
[2023/07/30 23:28:27.079 +08:00] [WARN] [endpoint.rs:836] [error-response] [err="Region error (will back off and retry) message: \"peer is not leader for region 319305, leader may None\" not_leader { region_id: 319305 }"]
[2023/07/30 23:28:28.092 +08:00] [WARN] [endpoint.rs:836] [error-response] [err="Region error (will back off and retry) message: \"peer is not leader for region 319305, leader may None\" not_leader { region_id: 319305 }"]
tikv-1
[2023/07/30 21:37:43.976 +08:00] [WARN] [endpoint.rs:836] [error-response] [err="Region error (will back off and retry) message: \"peer is not leader for region 319305, leader may Some(id: 319306 store_id: 4)\" not_leader { region_id: 319305 leader { id: 319306 store_id: 4 } }"]
tikv-1 thinks tikv-0 is leader, however, tikv-0 does not think so.
The transfer leader log
[2023/07/30 21:19:09.607 +08:00] [INFO] [raft.rs:1235] ["became leader at term 7"] [term=7] [raft_id=347199] [peer_id=347199] [region_id=319305]
[2023/07/30 21:19:09.607 +08:00] [INFO] [txn_ext.rs:122] ["require updating max ts"] [initial_status=30064771406] [peer_id=347199] [region_id=319305]
[2023/07/30 21:19:09.608 +08:00] [INFO] [misc.rs:92] ["succeed to update max timestamp"] [region_id=319305]
[2023/07/30 21:19:09.608 +08:00] [INFO] [endpoint.rs:342] ["register observe region"] [region="id: 319305 start_key: 7480000000000000FFFE5F728000000000FF0100D20000000000FA end_key: 7480000000000000FFFE5F728000000000FF01678C0000000000FA region_epoch { conf_ver: 31 version: 167 } peers { id: 319306 store_id: 4 } peers { id: 319308 store_id: 1 } peers { id: 319309 store_id: 31 role: Learner } peers { id: 319310 store_id: 29 role: Learner } peers { id: 347199 store_id: 5 }"]
[2023/07/30 21:19:09.608 +08:00] [INFO] [capture.rs:146] ["capture update observe level"] [level=LockRelated] [peer_id=347199] [region_id=319305]
[2023/07/30 21:19:09.608 +08:00] [INFO] [endpoint.rs:258] ["Resolver initialized"] [pending_data_index=0] [snapshot_index=12576] [observe_id=ObserveId(3740)] [region=319305]
[2023/07/30 21:19:09.615 +08:00] [ERROR] [mod.rs:439] ["raft step error"] [err=StepLocalMsg] [peer_id=347199] [region_id=319305]
[2023/07/30 21:21:09.609 +08:00] [INFO] [conf_change.rs:589] ["update gc peer"] [merged_peers="[]"] [gc_peers="[id: 319307 store_id: 6 role: Learner]"] [updates="[319307]"] [index=12577] [peer_id=347199] [region_id=319305]
[2023/07/30 21:21:09.609 +08:00] [ERROR] [mod.rs:439] ["raft step error"] [err=StepLocalMsg] [peer_id=347199] [region_id=319305]
[2023/07/30 21:21:09.625 +08:00] [ERROR] [mod.rs:439] ["raft step error"] [err=StepLocalMsg] [peer_id=347199] [region_id=319305]
[2023/07/30 21:21:11.416 +08:00] [ERROR] [mod.rs:439] ["raft step error"] [err=StepLocalMsg] [peer_id=347199] [region_id=319305]
[2023/07/30 21:21:27.654 +08:00] [ERROR] [mod.rs:439] ["raft step error"] [err=StepLocalMsg] [peer_id=347199] [region_id=319305]
[2023/07/30 21:21:29.429 +08:00] [ERROR] [mod.rs:439] ["raft step error"] [err=StepLocalMsg] [peer_id=347199] [region_id=319305]
[2023/07/30 21:21:31.432 +08:00] [ERROR] [mod.rs:439] ["raft step error"] [err=StepLocalMsg] [peer_id=347199] [region_id=319305]
[2023/07/30 21:37:39.086 +08:00] [INFO] [region.rs:258] ["try to transfer leader"] [to_peers="[]"] [to_peer="id: 319306 store_id: 4"] [from_peer="id: 347199 store_id: 5"] [region_id=319305]
[2023/07/30 21:37:39.086 +08:00] [INFO] [transfer_leader.rs:211] ["transfer leader"] [peer="id: 319306 store_id: 4"] [peer_id=347199] [region_id=319305]
[2023/07/30 21:37:39.086 +08:00] [INFO] [raft.rs:1911] ["[term 7] starts to transfer leadership to 319306"] [lead_transferee=319306] [term=7] [raft_id=347199] [peer_id=347199] [region_id=319305]
[2023/07/30 21:37:39.086 +08:00] [INFO] [raft.rs:1924] ["sends MsgTimeoutNow to 319306 immediately as 319306 already has up-to-date log"] [lead_transferee=319306] [raft_id=347199] [peer_id=347199] [region_id=319305]
[2023/07/30 21:37:39.087 +08:00] [INFO] [raft.rs:1371] ["received a message with higher term from 319306"] ["msg type"=MsgRequestVote] [message_term=8] [term=7] [from=319306] [raft_id=347199] [peer_id=347199] [region_id=319305]
[2023/07/30 21:37:39.087 +08:00] [INFO] [raft.rs:1127] ["became follower at term 8"] [term=8] [raft_id=347199] [peer_id=347199] [region_id=319305]
[2023/07/30 21:37:39.087 +08:00] [INFO] [raft.rs:1572] ["[logterm: 7, index: 12578, vote: 0] cast vote for 319306 [logterm: 7, index: 12578] at term 8"] ["msg type"=MsgRequestVote] [term=8] [msg_index=12578] [msg_term=7] [from=319306] [vote=0] [log_index=12578] [log_term=7] [raft_id=347199] [peer_id=347199] [region_id=319305]
[2023/07/30 21:37:39.087 +08:00] [INFO] [endpoint.rs:419] ["deregister observe region"] [observe_id=ObserveId(3740)] [region_id=319305] [store_id=Some(5)]
[2023/07/30 21:37:43.976 +08:00] [WARN] [endpoint.rs:836] [error-response] [err="Region error (will back off and retry) message: \"peer is not leader for region 319305, leader may Some(id: 319306 store_id: 4)\" not_leader { region_id: 319305 leader { id: 319306 store_id: 4 } }"]
sh-5.1# ./pd-ctl store
{
"count": 7,
"stores": [
{
"store": {
"id": 6,
"address": "tc-tikv-3.tc-tikv-peer.endless-htap-consistency2-tps-1862284-1-819.svc:20160",
"version": "7.3.0-alpha",
"peer_address": "tc-tikv-3.tc-tikv-peer.endless-htap-consistency2-tps-1862284-1-819.svc:20160",
"status_address": "0.0.0.0:20180",
"git_hash": "42882f8bcf4ce52ea93f93d325c000147ac2e137",
"start_timestamp": 1690740539,
"deploy_path": "/",
"last_heartbeat": 1690876258188093155,
"state_name": "Up"
},
"status": {
"capacity": "3.581TiB",
"available": "2.919TiB",
"used_size": "437.7GiB",
"leader_count": 3831,
"leader_weight": 1,
"leader_score": 3831,
"leader_size": 267106,
"region_count": 12382,
"region_weight": 1,
"region_score": 839580.0612361203,
"region_size": 771584,
"slow_trend": {
"cause_value": 50746.10833333333,
"cause_rate": 0,
"result_value": 4,
"result_rate": 0
},
"start_ts": "2023-07-31T02:08:59+08:00",
"last_heartbeat_ts": "2023-08-01T15:50:58.188093155+08:00",
"uptime": "37h41m59.188093155s"
}
},
{
"store": {
"id": 29,
"address": "tc-tiflash-2.tc-tiflash-peer.endless-htap-consistency2-tps-1862284-1-819.svc:3930",
"labels": [
{
"key": "engine",
"value": "tiflash"
}
],
"version": "v7.3.0-alpha",
"peer_address": "tc-tiflash-2.tc-tiflash-peer.endless-htap-consistency2-tps-1862284-1-819.svc:20170",
"status_address": "tc-tiflash-2.tc-tiflash-peer.endless-htap-consistency2-tps-1862284-1-819.svc:20292",
"git_hash": "4ac78e669c33854cd50b334e4e633e9aac8b2ea2",
"start_timestamp": 1690709418,
"deploy_path": "/tiflash",
"last_heartbeat": 1690876259489487542,
"state_name": "Up"
},
"status": {
"capacity": "3.581TiB",
"available": "3.081TiB",
"used_size": "274.6GiB",
"leader_count": 0,
"leader_weight": 1,
"leader_score": 0,
"leader_size": 0,
"region_count": 8174,
"region_weight": 1,
"region_score": 500237.2872790887,
"region_size": 460934,
"learner_count": 8174,
"slow_score": 1,
"slow_trend": {
"cause_value": 250004.60833333334,
"cause_rate": 0,
"result_value": 0,
"result_rate": 0
},
"start_ts": "2023-07-30T17:30:18+08:00",
"last_heartbeat_ts": "2023-08-01T15:50:59.489487542+08:00",
"uptime": "46h20m41.489487542s"
}
},
{
"store": {
"id": 30,
"address": "tc-tiflash-1.tc-tiflash-peer.endless-htap-consistency2-tps-1862284-1-819.svc:3930",
"labels": [
{
"key": "engine",
"value": "tiflash"
}
],
"version": "v7.3.0-alpha",
"peer_address": "tc-tiflash-1.tc-tiflash-peer.endless-htap-consistency2-tps-1862284-1-819.svc:20170",
"status_address": "tc-tiflash-1.tc-tiflash-peer.endless-htap-consistency2-tps-1862284-1-819.svc:20292",
"git_hash": "4ac78e669c33854cd50b334e4e633e9aac8b2ea2",
"start_timestamp": 1690709419,
"deploy_path": "/tiflash",
"last_heartbeat": 1690876256148261847,
"state_name": "Up"
},
"status": {
"capacity": "3.581TiB",
"available": "3.059TiB",
"used_size": "281.2GiB",
"leader_count": 0,
"leader_weight": 1,
"leader_score": 0,
"leader_size": 0,
"region_count": 8083,
"region_weight": 1,
"region_score": 496216.9478029116,
"region_size": 457071,
"learner_count": 8083,
"slow_score": 1,
"slow_trend": {
"cause_value": 250005.425,
"cause_rate": 0,
"result_value": 0,
"result_rate": 0
},
"start_ts": "2023-07-30T17:30:19+08:00",
"last_heartbeat_ts": "2023-08-01T15:50:56.148261847+08:00",
"uptime": "46h20m37.148261847s"
}
},
{
"store": {
"id": 31,
"address": "tc-tiflash-0.tc-tiflash-peer.endless-htap-consistency2-tps-1862284-1-819.svc:3930",
"labels": [
{
"key": "engine",
"value": "tiflash"
}
],
"version": "v7.3.0-alpha",
"peer_address": "tc-tiflash-0.tc-tiflash-peer.endless-htap-consistency2-tps-1862284-1-819.svc:20170",
"status_address": "tc-tiflash-0.tc-tiflash-peer.endless-htap-consistency2-tps-1862284-1-819.svc:20292",
"git_hash": "4ac78e669c33854cd50b334e4e633e9aac8b2ea2",
"start_timestamp": 1690709420,
"deploy_path": "/tiflash",
"last_heartbeat": 1690876260794801687,
"state_name": "Up"
},
"status": {
"capacity": "3.581TiB",
"available": "3.072TiB",
"used_size": "281GiB",
"leader_count": 0,
"leader_weight": 1,
"leader_score": 0,
"leader_size": 0,
"region_count": 8443,
"region_weight": 1,
"region_score": 503622.5732538102,
"region_size": 463987,
"learner_count": 8443,
"slow_score": 1,
"slow_trend": {
"cause_value": 250004.90833333333,
"cause_rate": 0,
"result_value": 0,
"result_rate": 0
},
"start_ts": "2023-07-30T17:30:20+08:00",
"last_heartbeat_ts": "2023-08-01T15:51:00.794801687+08:00",
"uptime": "46h20m40.794801687s"
}
},
{
"store": {
"id": 1,
"address": "tc-tikv-2.tc-tikv-peer.endless-htap-consistency2-tps-1862284-1-819.svc:20160",
"version": "7.3.0-alpha",
"peer_address": "tc-tikv-2.tc-tikv-peer.endless-htap-consistency2-tps-1862284-1-819.svc:20160",
"status_address": "0.0.0.0:20180",
"git_hash": "42882f8bcf4ce52ea93f93d325c000147ac2e137",
"start_timestamp": 1690740427,
"deploy_path": "/",
"last_heartbeat": 1690876257655957712,
"state_name": "Up"
},
"status": {
"capacity": "3.581TiB",
"available": "2.926TiB",
"used_size": "436.7GiB",
"leader_count": 3831,
"leader_weight": 1,
"leader_score": 3831,
"leader_size": 270725,
"region_count": 11668,
"region_weight": 1,
"region_score": 840395.6502692404,
"region_size": 772434,
"slow_trend": {
"cause_value": 50019.88333333333,
"cause_rate": 0,
"result_value": 1,
"result_rate": 0
},
"start_ts": "2023-07-31T02:07:07+08:00",
"last_heartbeat_ts": "2023-08-01T15:50:57.655957712+08:00",
"uptime": "37h43m50.655957712s"
}
},
{
"store": {
"id": 4,
"address": "tc-tikv-0.tc-tikv-peer.endless-htap-consistency2-tps-1862284-1-819.svc:20160",
"version": "7.3.0-alpha",
"peer_address": "tc-tikv-0.tc-tikv-peer.endless-htap-consistency2-tps-1862284-1-819.svc:20160",
"status_address": "0.0.0.0:20180",
"git_hash": "42882f8bcf4ce52ea93f93d325c000147ac2e137",
"start_timestamp": 1690740410,
"deploy_path": "/",
"last_heartbeat": 1690876257291037749,
"state_name": "Up"
},
"status": {
"capacity": "1.433TiB",
"available": "967.9GiB",
"used_size": "380.1GiB",
"leader_count": 3827,
"leader_weight": 1,
"leader_score": 3827,
"leader_size": 230300,
"region_count": 10262,
"region_weight": 1,
"region_score": 835416.3450990797,
"region_size": 667887,
"slow_trend": {
"cause_value": 50022.833333333336,
"cause_rate": 0,
"result_value": 0.5,
"result_rate": 0
},
"start_ts": "2023-07-31T02:06:50+08:00",
"last_heartbeat_ts": "2023-08-01T15:50:57.291037749+08:00",
"uptime": "37h44m7.291037749s"
}
},
{
"store": {
"id": 5,
"address": "tc-tikv-1.tc-tikv-peer.endless-htap-consistency2-tps-1862284-1-819.svc:20160",
"version": "7.3.0-alpha",
"peer_address": "tc-tikv-1.tc-tikv-peer.endless-htap-consistency2-tps-1862284-1-819.svc:20160",
"status_address": "0.0.0.0:20180",
"git_hash": "42882f8bcf4ce52ea93f93d325c000147ac2e137",
"start_timestamp": 1690739965,
"deploy_path": "/",
"last_heartbeat": 1690876260869411436,
"state_name": "Up"
},
"status": {
"capacity": "3.581TiB",
"available": "2.928TiB",
"used_size": "437.4GiB",
"leader_count": 3834,
"leader_weight": 1,
"leader_score": 3834,
"leader_size": 220793,
"region_count": 11657,
"region_weight": 1,
"region_score": 821258.4574588499,
"region_size": 754867,
"slow_trend": {
"cause_value": 52325.666666666664,
"cause_rate": 0,
"result_value": 21.5,
"result_rate": 0
},
"start_ts": "2023-07-31T01:59:25+08:00",
"last_heartbeat_ts": "2023-08-01T15:51:00.869411436+08:00",
"uptime": "37h51m35.869411436s"
}
}
]
}
sh-5.1#
log of 319306 of tikv-0
[2023/07/30 21:36:55.586 +08:00] [INFO] [raft.rs:2663] ["switched to configuration"] [config="Configuration { voters: Configuration { incoming: Configuration { voters: {319308, 319306, 319307} }, outgoing: Configuration { voters: {} } }, learners: {319309, 319310}, learners_next: {}, auto_leave: false }"] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 21:36:55.586 +08:00] [INFO] [raft.rs:1127] ["became follower at term 7"] [term=7] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 21:36:55.586 +08:00] [INFO] [raft.rs:388] [newRaft] [peers="Configuration { incoming: Configuration { voters: {319308, 319306, 319307} }, outgoing: Configuration { voters: {} } }"] ["last term"=7] ["last index"=12578] [applied=12567] [commit=12578] [term=7] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 21:36:55.586 +08:00] [INFO] [raw_node.rs:315] ["RawNode created with id 319306."] [id=319306] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 21:36:55.593 +08:00] [INFO] [peer.rs:50] ["create peer"] [region_state="region { id: 319305 start_key: 7480000000000000FFFE5F728000000000FF0100D20000000000FA end_key: 7480000000000000FFFE5F728000000000FF01678C0000000000FA region_epoch { conf_ver: 25 version: 167 } peers { id: 319306 store_id: 4 } peers { id: 319307 store_id: 6 } peers { id: 319308 store_id: 1 } peers { id: 319309 store_id: 31 role: Learner } peers { id: 319310 store_id: 29 role: Learner } } tablet_index: 12567"] [apply_state="applied_index: 12567 truncated_state { index: 10959 term: 6 }"] [raft_state="hard_state { term: 7 vote: 347199 commit: 12578 } last_index: 12578"] [peer_id=319306] [region_id=319305]
[2023/07/30 21:37:20.088 +08:00] [INFO] [bucket.rs:278] ["refreshed region bucket info"] [first_refresh=true] [estimated_region_size=52428800] [buckets_count=1] [bucket_version=30064771072] [peer_id=319306] [region_id=319305]
[2023/07/30 21:37:21.019 +08:00] [INFO] [conf_change.rs:264] ["exec ConfChangeV2"] [index=12571] [epoch="conf_ver: 25 version: 167"] [legacy=false] [kind=Simple] [peer_id=319306] [region_id=319305]
[2023/07/30 21:37:21.019 +08:00] [INFO] [conf_change.rs:294] ["conf change successfully"] ["current region"="id: 319305 start_key: 7480000000000000FFFE5F728000000000FF0100D20000000000FA end_key: 7480000000000000FFFE5F728000000000FF01678C0000000000FA region_epoch { conf_ver: 26 version: 167 } peers { id: 319306 store_id: 4 } peers { id: 319307 store_id: 6 } peers { id: 319308 store_id: 1 } peers { id: 319309 store_id: 31 role: Learner } peers { id: 319310 store_id: 29 role: Learner } peers { id: 347199 store_id: 5 role: Learner }"] ["original region"="id: 319305 start_key: 7480000000000000FFFE5F728000000000FF0100D20000000000FA end_key: 7480000000000000FFFE5F728000000000FF01678C0000000000FA region_epoch { conf_ver: 25 version: 167 } peers { id: 319306 store_id: 4 } peers { id: 319307 store_id: 6 } peers { id: 319308 store_id: 1 } peers { id: 319309 store_id: 31 role: Learner } peers { id: 319310 store_id: 29 role: Learner }"] [legacy=false] [changes="[change_type: AddLearnerNode peer { id: 347199 store_id: 5 role: Learner }]"] [peer_id=319306] [region_id=319305]
[2023/07/30 21:37:21.019 +08:00] [INFO] [conf_change.rs:264] ["exec ConfChangeV2"] [index=12572] [epoch="conf_ver: 26 version: 167"] [legacy=false] [kind=EnterJoint] [peer_id=319306] [region_id=319305]
[2023/07/30 21:37:21.019 +08:00] [INFO] [conf_change.rs:294] ["conf change successfully"] ["current region"="id: 319305 start_key: 7480000000000000FFFE5F728000000000FF0100D20000000000FA end_key: 7480000000000000FFFE5F728000000000FF01678C0000000000FA region_epoch { conf_ver: 28 version: 167 } peers { id: 319306 store_id: 4 } peers { id: 319307 store_id: 6 role: DemotingVoter } peers { id: 319308 store_id: 1 } peers { id: 319309 store_id: 31 role: Learner } peers { id: 319310 store_id: 29 role: Learner } peers { id: 347199 store_id: 5 role: IncomingVoter }"] ["original region"="id: 319305 start_key: 7480000000000000FFFE5F728000000000FF0100D20000000000FA end_key: 7480000000000000FFFE5F728000000000FF01678C0000000000FA region_epoch { conf_ver: 26 version: 167 } peers { id: 319306 store_id: 4 } peers { id: 319307 store_id: 6 } peers { id: 319308 store_id: 1 } peers { id: 319309 store_id: 31 role: Learner } peers { id: 319310 store_id: 29 role: Learner } peers { id: 347199 store_id: 5 role: Learner }"] [legacy=false] [changes="[peer { id: 347199 store_id: 5 }, change_type: AddLearnerNode peer { id: 319307 store_id: 6 role: Learner }]"] [peer_id=319306] [region_id=319305]
[2023/07/30 21:37:21.019 +08:00] [INFO] [conf_change.rs:264] ["exec ConfChangeV2"] [index=12573] [epoch="conf_ver: 28 version: 167"] [legacy=false] [kind=LeaveJoint] [peer_id=319306] [region_id=319305]
[2023/07/30 21:37:21.019 +08:00] [INFO] [conf_change.rs:374] ["leave joint state successfully"] [region="id: 319305 start_key: 7480000000000000FFFE5F728000000000FF0100D20000000000FA end_key: 7480000000000000FFFE5F728000000000FF01678C0000000000FA region_epoch { conf_ver: 30 version: 167 } peers { id: 319306 store_id: 4 } peers { id: 319307 store_id: 6 role: Learner } peers { id: 319308 store_id: 1 } peers { id: 319309 store_id: 31 role: Learner } peers { id: 319310 store_id: 29 role: Learner } peers { id: 347199 store_id: 5 }"] [peer_id=319306] [region_id=319305]
[2023/07/30 21:37:21.019 +08:00] [INFO] [conf_change.rs:294] ["conf change successfully"] ["current region"="id: 319305 start_key: 7480000000000000FFFE5F728000000000FF0100D20000000000FA end_key: 7480000000000000FFFE5F728000000000FF01678C0000000000FA region_epoch { conf_ver: 30 version: 167 } peers { id: 319306 store_id: 4 } peers { id: 319307 store_id: 6 role: Learner } peers { id: 319308 store_id: 1 } peers { id: 319309 store_id: 31 role: Learner } peers { id: 319310 store_id: 29 role: Learner } peers { id: 347199 store_id: 5 }"] ["original region"="id: 319305 start_key: 7480000000000000FFFE5F728000000000FF0100D20000000000FA end_key: 7480000000000000FFFE5F728000000000FF01678C0000000000FA region_epoch { conf_ver: 28 version: 167 } peers { id: 319306 store_id: 4 } peers { id: 319307 store_id: 6 role: DemotingVoter } peers { id: 319308 store_id: 1 } peers { id: 319309 store_id: 31 role: Learner } peers { id: 319310 store_id: 29 role: Learner } peers { id: 347199 store_id: 5 role: IncomingVoter }"] [legacy=false] [changes="[]"] [peer_id=319306] [region_id=319305]
[2023/07/30 21:37:21.019 +08:00] [INFO] [conf_change.rs:264] ["exec ConfChangeV2"] [index=12574] [epoch="conf_ver: 30 version: 167"] [legacy=false] [kind=Simple] [peer_id=319306] [region_id=319305]
[2023/07/30 21:37:21.019 +08:00] [INFO] [conf_change.rs:294] ["conf change successfully"] ["current region"="id: 319305 start_key: 7480000000000000FFFE5F728000000000FF0100D20000000000FA end_key: 7480000000000000FFFE5F728000000000FF01678C0000000000FA region_epoch { conf_ver: 31 version: 167 } peers { id: 319306 store_id: 4 } peers { id: 319308 store_id: 1 } peers { id: 319309 store_id: 31 role: Learner } peers { id: 319310 store_id: 29 role: Learner } peers { id: 347199 store_id: 5 }"] ["original region"="id: 319305 start_key: 7480000000000000FFFE5F728000000000FF0100D20000000000FA end_key: 7480000000000000FFFE5F728000000000FF01678C0000000000FA region_epoch { conf_ver: 30 version: 167 } peers { id: 319306 store_id: 4 } peers { id: 319307 store_id: 6 role: Learner } peers { id: 319308 store_id: 1 } peers { id: 319309 store_id: 31 role: Learner } peers { id: 319310 store_id: 29 role: Learner } peers { id: 347199 store_id: 5 }"] [legacy=false] [changes="[change_type: RemoveNode peer { id: 319307 store_id: 6 role: Learner }]"] [peer_id=319306] [region_id=319305]
[2023/07/30 21:37:21.019 +08:00] [INFO] [conf_change.rs:589] ["update gc peer"] [merged_peers="[]"] [gc_peers="[id: 319307 store_id: 6 role: Learner]"] [updates="[319307]"] [index=12577] [peer_id=319306] [region_id=319305]
[2023/07/30 21:37:25.060 +08:00] [INFO] [compact_log.rs:588] ["compact log"] [truncated="applied_index: 12567 truncated_state { index: 11084 term: 6 }"] [apply_trace="ApplyTrace { data_cfs: [Progress { flushed: 12566, last_modified: 12566 }, Progress { flushed: 12566, last_modified: 12566 }, Progress { flushed: 12566, last_modified: 12566 }], admin: Progress { flushed: 12567, last_modified: 12567 }, persisted_applied: 12567, last_flush_trigger: 12567, try_persist: false }"] [index=11085] [peer_id=319306] [region_id=319305]
[2023/07/30 21:37:25.060 +08:00] [INFO] [compact_log.rs:588] ["compact log"] [truncated="applied_index: 12567 truncated_state { index: 11275 term: 6 }"] [apply_trace="ApplyTrace { data_cfs: [Progress { flushed: 12566, last_modified: 12566 }, Progress { flushed: 12566, last_modified: 12566 }, Progress { flushed: 12566, last_modified: 12566 }], admin: Progress { flushed: 12567, last_modified: 12567 }, persisted_applied: 12567, last_flush_trigger: 12567, try_persist: false }"] [index=11276] [peer_id=319306] [region_id=319305]
[2023/07/30 21:37:25.060 +08:00] [INFO] [compact_log.rs:588] ["compact log"] [truncated="applied_index: 12567 truncated_state { index: 12568 term: 6 }"] [apply_trace="ApplyTrace { data_cfs: [Progress { flushed: 12566, last_modified: 12566 }, Progress { flushed: 12566, last_modified: 12566 }, Progress { flushed: 12566, last_modified: 12566 }], admin: Progress { flushed: 12567, last_modified: 12567 }, persisted_applied: 12567, last_flush_trigger: 12567, try_persist: false }"] [index=12567] [peer_id=319306] [region_id=319305]
[2023/07/30 21:37:25.060 +08:00] [INFO] [raft.rs:2663] ["switched to configuration"] [config="Configuration { voters: Configuration { incoming: Configuration { voters: {319308, 319306, 319307} }, outgoing: Configuration { voters: {} } }, learners: {319309, 319310, 347199}, learners_next: {}, auto_leave: false }"] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 21:37:25.060 +08:00] [INFO] [raft.rs:2663] ["switched to configuration"] [config="Configuration { voters: Configuration { incoming: Configuration { voters: {319306, 347199, 319308} }, outgoing: Configuration { voters: {319308, 319306, 319307} } }, learners: {319309, 319310}, learners_next: {319307}, auto_leave: false }"] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 21:37:25.060 +08:00] [INFO] [raft.rs:2663] ["switched to configuration"] [config="Configuration { voters: Configuration { incoming: Configuration { voters: {319306, 347199, 319308} }, outgoing: Configuration { voters: {} } }, learners: {319309, 319310, 319307}, learners_next: {}, auto_leave: false }"] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 21:37:25.060 +08:00] [INFO] [raft.rs:2663] ["switched to configuration"] [config="Configuration { voters: Configuration { incoming: Configuration { voters: {319306, 347199, 319308} }, outgoing: Configuration { voters: {} } }, learners: {319309, 319310}, learners_next: {}, auto_leave: false }"] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 21:37:39.103 +08:00] [INFO] [raft.rs:2348] ["[term 7] received MsgTimeoutNow from 347199 and starts an election to get leadership."] [from=347199] [term=7] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 21:37:39.103 +08:00] [INFO] [raft.rs:1557] ["starting a new election"] [term=7] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 21:37:39.103 +08:00] [INFO] [raft.rs:1151] ["became candidate at term 8"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 21:37:39.103 +08:00] [INFO] [raft.rs:1306] ["broadcasting vote request"] [to="[347199, 319308]"] [log_index=12578] [log_term=7] [term=8] [type=MsgRequestVote] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 21:37:39.103 +08:00] [INFO] [raft.rs:2219] ["received votes response"] [term=8] [type=MsgRequestVoteResponse] [approvals=2] [rejections=0] [from=347199] [vote=true] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 21:37:39.103 +08:00] [INFO] [raft.rs:1235] ["became leader at term 8"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 21:37:39.103 +08:00] [INFO] [txn_ext.rs:122] ["require updating max ts"] [initial_status=34359738702] [peer_id=319306] [region_id=319305]
[2023/07/30 21:37:45.654 +08:00] [INFO] [endpoint.rs:342] ["register observe region"] [region="id: 319305 start_key: 7480000000000000FFFE5F728000000000FF0100D20000000000FA end_key: 7480000000000000FFFE5F728000000000FF01678C0000000000FA region_epoch { conf_ver: 31 version: 167 } peers { id: 319306 store_id: 4 } peers { id: 319308 store_id: 1 } peers { id: 319309 store_id: 31 role: Learner } peers { id: 319310 store_id: 29 role: Learner } peers { id: 347199 store_id: 5 }"]
[2023/07/30 21:37:45.656 +08:00] [INFO] [capture.rs:146] ["capture update observe level"] [level=LockRelated] [peer_id=319306] [region_id=319305]
[2023/07/30 22:11:48.920 +08:00] [INFO] [apply_trace.rs:543] ["persisting admin flushed"] [flushed=12769] [tablet_index=12567] [peer_id=319306] [region_id=319305]
[2023/07/30 22:11:48.920 +08:00] [INFO] [compact_log.rs:588] ["compact log"] [truncated="applied_index: 12773 truncated_state { index: 12771 term: 8 }"] [apply_trace="ApplyTrace { data_cfs: [Progress { flushed: 12566, last_modified: 12566 }, Progress { flushed: 12769, last_modified: 12772 }, Progress { flushed: 12772, last_modified: 12772 }], admin: Progress { flushed: 12769, last_modified: 12567 }, persisted_applied: 12769, last_flush_trigger: 12567, try_persist: false }"] [index=12769] [peer_id=319306] [region_id=319305]
[2023/07/30 22:17:37.298 +08:00] [INFO] [apply_trace.rs:543] ["persisting admin flushed"] [flushed=12879] [tablet_index=12567] [peer_id=319306] [region_id=319305]
[2023/07/30 22:17:37.298 +08:00] [INFO] [compact_log.rs:588] ["compact log"] [truncated="applied_index: 12879 truncated_state { index: 12877 term: 8 }"] [apply_trace="ApplyTrace { data_cfs: [Progress { flushed: 12566, last_modified: 12566 }, Progress { flushed: 12876, last_modified: 12876 }, Progress { flushed: 12876, last_modified: 12876 }], admin: Progress { flushed: 12879, last_modified: 12567 }, persisted_applied: 12879, last_flush_trigger: 12567, try_persist: false }"] [index=12878] [peer_id=319306] [region_id=319305]
[2023/07/30 23:00:00.622 +08:00] [INFO] [region.rs:258] ["try to transfer leader"] [to_peers="[]"] [to_peer="id: 319308 store_id: 1"] [from_peer="id: 319306 store_id: 4"] [region_id=319305]
[2023/07/30 23:00:03.093 +08:00] [INFO] [region.rs:258] ["try to transfer leader"] [to_peers="[]"] [to_peer="id: 319308 store_id: 1"] [from_peer="id: 319306 store_id: 4"] [region_id=319305]
[2023/07/30 23:00:05.593 +08:00] [INFO] [region.rs:258] ["try to transfer leader"] [to_peers="[]"] [to_peer="id: 319308 store_id: 1"] [from_peer="id: 319306 store_id: 4"] [region_id=319305]
[2023/07/30 23:00:07.593 +08:00] [INFO] [region.rs:258] ["try to transfer leader"] [to_peers="[]"] [to_peer="id: 319308 store_id: 1"] [from_peer="id: 319306 store_id: 4"] [region_id=319305]
[2023/07/30 23:00:09.593 +08:00] [INFO] [region.rs:258] ["try to transfer leader"] [to_peers="[]"] [to_peer="id: 319308 store_id: 1"] [from_peer="id: 319306 store_id: 4"] [region_id=319305]
[2023/07/30 23:00:12.093 +08:00] [INFO] [region.rs:258] ["try to transfer leader"] [to_peers="[]"] [to_peer="id: 319308 store_id: 1"] [from_peer="id: 319306 store_id: 4"] [region_id=319305]
[2023/07/30 23:00:14.592 +08:00] [INFO] [region.rs:258] ["try to transfer leader"] [to_peers="[]"] [to_peer="id: 319308 store_id: 1"] [from_peer="id: 319306 store_id: 4"] [region_id=319305]
[2023/07/30 23:00:16.593 +08:00] [INFO] [region.rs:258] ["try to transfer leader"] [to_peers="[]"] [to_peer="id: 319308 store_id: 1"] [from_peer="id: 319306 store_id: 4"] [region_id=319305]
[2023/07/30 23:00:19.093 +08:00] [INFO] [region.rs:258] ["try to transfer leader"] [to_peers="[]"] [to_peer="id: 319308 store_id: 1"] [from_peer="id: 319306 store_id: 4"] [region_id=319305]
[2023/07/30 23:00:20.124 +08:00] [INFO] [region.rs:258] ["try to transfer leader"] [to_peers="[]"] [to_peer="id: 319308 store_id: 1"] [from_peer="id: 319306 store_id: 4"] [region_id=319305]
[2023/07/30 23:00:21.093 +08:00] [INFO] [region.rs:258] ["try to transfer leader"] [to_peers="[]"] [to_peer="id: 319308 store_id: 1"] [from_peer="id: 319306 store_id: 4"] [region_id=319305]
[2023/07/30 23:00:23.593 +08:00] [INFO] [region.rs:258] ["try to transfer leader"] [to_peers="[]"] [to_peer="id: 319308 store_id: 1"] [from_peer="id: 319306 store_id: 4"] [region_id=319305]
[2023/07/30 23:00:25.593 +08:00] [INFO] [region.rs:258] ["try to transfer leader"] [to_peers="[]"] [to_peer="id: 319308 store_id: 1"] [from_peer="id: 319306 store_id: 4"] [region_id=319305]
[2023/07/30 23:00:28.093 +08:00] [INFO] [region.rs:258] ["try to transfer leader"] [to_peers="[]"] [to_peer="id: 319308 store_id: 1"] [from_peer="id: 319306 store_id: 4"] [region_id=319305]
[2023/07/30 23:00:30.093 +08:00] [INFO] [region.rs:258] ["try to transfer leader"] [to_peers="[]"] [to_peer="id: 319308 store_id: 1"] [from_peer="id: 319306 store_id: 4"] [region_id=319305]
[2023/07/30 23:00:32.592 +08:00] [INFO] [region.rs:258] ["try to transfer leader"] [to_peers="[]"] [to_peer="id: 319308 store_id: 1"] [from_peer="id: 319306 store_id: 4"] [region_id=319305]
[2023/07/30 23:00:34.593 +08:00] [INFO] [region.rs:258] ["try to transfer leader"] [to_peers="[]"] [to_peer="id: 319308 store_id: 1"] [from_peer="id: 319306 store_id: 4"] [region_id=319305]
[2023/07/30 23:00:37.092 +08:00] [INFO] [region.rs:258] ["try to transfer leader"] [to_peers="[]"] [to_peer="id: 319308 store_id: 1"] [from_peer="id: 319306 store_id: 4"] [region_id=319305]
[2023/07/30 23:00:39.093 +08:00] [INFO] [region.rs:258] ["try to transfer leader"] [to_peers="[]"] [to_peer="id: 319308 store_id: 1"] [from_peer="id: 319306 store_id: 4"] [region_id=319305]
[2023/07/30 23:00:41.093 +08:00] [INFO] [region.rs:258] ["try to transfer leader"] [to_peers="[]"] [to_peer="id: 319308 store_id: 1"] [from_peer="id: 319306 store_id: 4"] [region_id=319305]
[2023/07/30 23:00:43.093 +08:00] [INFO] [region.rs:258] ["try to transfer leader"] [to_peers="[]"] [to_peer="id: 319308 store_id: 1"] [from_peer="id: 319306 store_id: 4"] [region_id=319305]
[2023/07/30 23:00:45.593 +08:00] [INFO] [region.rs:258] ["try to transfer leader"] [to_peers="[]"] [to_peer="id: 319308 store_id: 1"] [from_peer="id: 319306 store_id: 4"] [region_id=319305]
[2023/07/30 23:00:48.093 +08:00] [INFO] [region.rs:258] ["try to transfer leader"] [to_peers="[]"] [to_peer="id: 319308 store_id: 1"] [from_peer="id: 319306 store_id: 4"] [region_id=319305]
[2023/07/30 23:00:50.593 +08:00] [INFO] [region.rs:258] ["try to transfer leader"] [to_peers="[]"] [to_peer="id: 319308 store_id: 1"] [from_peer="id: 319306 store_id: 4"] [region_id=319305]
[2023/07/30 23:00:53.092 +08:00] [INFO] [region.rs:258] ["try to transfer leader"] [to_peers="[]"] [to_peer="id: 319308 store_id: 1"] [from_peer="id: 319306 store_id: 4"] [region_id=319305]
[2023/07/30 23:00:55.093 +08:00] [INFO] [region.rs:258] ["try to transfer leader"] [to_peers="[]"] [to_peer="id: 319308 store_id: 1"] [from_peer="id: 319306 store_id: 4"] [region_id=319305]
[2023/07/30 23:00:57.093 +08:00] [INFO] [region.rs:258] ["try to transfer leader"] [to_peers="[]"] [to_peer="id: 319308 store_id: 1"] [from_peer="id: 319306 store_id: 4"] [region_id=319305]
[2023/07/30 23:00:59.593 +08:00] [INFO] [region.rs:258] ["try to transfer leader"] [to_peers="[]"] [to_peer="id: 319308 store_id: 1"] [from_peer="id: 319306 store_id: 4"] [region_id=319305]
[2023/07/30 23:22:20.524 +08:00] [WARN] [raft.rs:2008] ["stepped down to follower since quorum is not active"] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:22:20.524 +08:00] [INFO] [raft.rs:1127] ["became follower at term 8"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:22:35.547 +08:00] [INFO] [raft.rs:1557] ["starting a new election"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:22:35.548 +08:00] [INFO] [raft.rs:1177] ["became pre-candidate at term 8"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:22:35.548 +08:00] [INFO] [raft.rs:1306] ["broadcasting vote request"] [to="[347199, 319308]"] [log_index=12879] [log_term=8] [term=8] [type=MsgRequestPreVote] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:22:50.565 +08:00] [INFO] [raft.rs:1557] ["starting a new election"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:22:50.565 +08:00] [INFO] [raft.rs:1177] ["became pre-candidate at term 8"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:22:50.565 +08:00] [INFO] [raft.rs:1306] ["broadcasting vote request"] [to="[347199, 319308]"] [log_index=12879] [log_term=8] [term=8] [type=MsgRequestPreVote] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:23:05.584 +08:00] [INFO] [raft.rs:1557] ["starting a new election"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:23:05.584 +08:00] [INFO] [raft.rs:1177] ["became pre-candidate at term 8"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:23:05.584 +08:00] [INFO] [raft.rs:1306] ["broadcasting vote request"] [to="[347199, 319308]"] [log_index=12879] [log_term=8] [term=8] [type=MsgRequestPreVote] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:23:20.601 +08:00] [INFO] [raft.rs:1557] ["starting a new election"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:23:20.602 +08:00] [INFO] [raft.rs:1177] ["became pre-candidate at term 8"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:23:20.602 +08:00] [INFO] [raft.rs:1306] ["broadcasting vote request"] [to="[347199, 319308]"] [log_index=12879] [log_term=8] [term=8] [type=MsgRequestPreVote] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:23:35.618 +08:00] [INFO] [raft.rs:1557] ["starting a new election"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:23:35.618 +08:00] [INFO] [raft.rs:1177] ["became pre-candidate at term 8"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:23:35.618 +08:00] [INFO] [raft.rs:1306] ["broadcasting vote request"] [to="[347199, 319308]"] [log_index=12879] [log_term=8] [term=8] [type=MsgRequestPreVote] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:23:50.645 +08:00] [INFO] [raft.rs:1557] ["starting a new election"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:23:50.645 +08:00] [INFO] [raft.rs:1177] ["became pre-candidate at term 8"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:23:50.645 +08:00] [INFO] [raft.rs:1306] ["broadcasting vote request"] [to="[347199, 319308]"] [log_index=12879] [log_term=8] [term=8] [type=MsgRequestPreVote] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:24:05.663 +08:00] [INFO] [raft.rs:1557] ["starting a new election"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:24:05.663 +08:00] [INFO] [raft.rs:1177] ["became pre-candidate at term 8"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:24:05.663 +08:00] [INFO] [raft.rs:1306] ["broadcasting vote request"] [to="[347199, 319308]"] [log_index=12879] [log_term=8] [term=8] [type=MsgRequestPreVote] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:24:20.678 +08:00] [INFO] [raft.rs:1557] ["starting a new election"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:24:20.678 +08:00] [INFO] [raft.rs:1177] ["became pre-candidate at term 8"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:24:20.678 +08:00] [INFO] [raft.rs:1306] ["broadcasting vote request"] [to="[347199, 319308]"] [log_index=12879] [log_term=8] [term=8] [type=MsgRequestPreVote] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:24:35.693 +08:00] [INFO] [raft.rs:1557] ["starting a new election"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:24:35.693 +08:00] [INFO] [raft.rs:1177] ["became pre-candidate at term 8"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:24:35.693 +08:00] [INFO] [raft.rs:1306] ["broadcasting vote request"] [to="[347199, 319308]"] [log_index=12879] [log_term=8] [term=8] [type=MsgRequestPreVote] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:24:50.711 +08:00] [INFO] [raft.rs:1557] ["starting a new election"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:24:50.711 +08:00] [INFO] [raft.rs:1177] ["became pre-candidate at term 8"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:24:50.711 +08:00] [INFO] [raft.rs:1306] ["broadcasting vote request"] [to="[347199, 319308]"] [log_index=12879] [log_term=8] [term=8] [type=MsgRequestPreVote] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:25:05.727 +08:00] [INFO] [raft.rs:1557] ["starting a new election"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:25:05.727 +08:00] [INFO] [raft.rs:1177] ["became pre-candidate at term 8"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:25:05.727 +08:00] [INFO] [raft.rs:1306] ["broadcasting vote request"] [to="[347199, 319308]"] [log_index=12879] [log_term=8] [term=8] [type=MsgRequestPreVote] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:25:20.757 +08:00] [INFO] [raft.rs:1557] ["starting a new election"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:25:20.757 +08:00] [INFO] [raft.rs:1177] ["became pre-candidate at term 8"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:25:20.757 +08:00] [INFO] [raft.rs:1306] ["broadcasting vote request"] [to="[347199, 319308]"] [log_index=12879] [log_term=8] [term=8] [type=MsgRequestPreVote] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:25:37.418 +08:00] [INFO] [raft.rs:1557] ["starting a new election"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:25:37.418 +08:00] [INFO] [raft.rs:1177] ["became pre-candidate at term 8"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:25:37.418 +08:00] [INFO] [raft.rs:1306] ["broadcasting vote request"] [to="[347199, 319308]"] [log_index=12879] [log_term=8] [term=8] [type=MsgRequestPreVote] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:25:52.747 +08:00] [INFO] [raft.rs:1557] ["starting a new election"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:25:52.747 +08:00] [INFO] [raft.rs:1177] ["became pre-candidate at term 8"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:25:52.747 +08:00] [INFO] [raft.rs:1306] ["broadcasting vote request"] [to="[347199, 319308]"] [log_index=12879] [log_term=8] [term=8] [type=MsgRequestPreVote] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:26:07.807 +08:00] [INFO] [raft.rs:1557] ["starting a new election"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:26:07.807 +08:00] [INFO] [raft.rs:1177] ["became pre-candidate at term 8"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:26:07.807 +08:00] [INFO] [raft.rs:1306] ["broadcasting vote request"] [to="[347199, 319308]"] [log_index=12879] [log_term=8] [term=8] [type=MsgRequestPreVote] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:26:22.861 +08:00] [INFO] [raft.rs:1557] ["starting a new election"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:26:22.861 +08:00] [INFO] [raft.rs:1177] ["became pre-candidate at term 8"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:26:22.861 +08:00] [INFO] [raft.rs:1306] ["broadcasting vote request"] [to="[347199, 319308]"] [log_index=12879] [log_term=8] [term=8] [type=MsgRequestPreVote] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:26:37.902 +08:00] [INFO] [raft.rs:1557] ["starting a new election"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:26:37.902 +08:00] [INFO] [raft.rs:1177] ["became pre-candidate at term 8"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:26:37.902 +08:00] [INFO] [raft.rs:1306] ["broadcasting vote request"] [to="[347199, 319308]"] [log_index=12879] [log_term=8] [term=8] [type=MsgRequestPreVote] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:26:52.918 +08:00] [INFO] [raft.rs:1557] ["starting a new election"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:26:52.918 +08:00] [INFO] [raft.rs:1177] ["became pre-candidate at term 8"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:26:52.918 +08:00] [INFO] [raft.rs:1306] ["broadcasting vote request"] [to="[347199, 319308]"] [log_index=12879] [log_term=8] [term=8] [type=MsgRequestPreVote] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:27:07.935 +08:00] [INFO] [raft.rs:1557] ["starting a new election"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:27:07.935 +08:00] [INFO] [raft.rs:1177] ["became pre-candidate at term 8"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:27:07.935 +08:00] [INFO] [raft.rs:1306] ["broadcasting vote request"] [to="[347199, 319308]"] [log_index=12879] [log_term=8] [term=8] [type=MsgRequestPreVote] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:27:22.959 +08:00] [INFO] [raft.rs:1557] ["starting a new election"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:27:22.959 +08:00] [INFO] [raft.rs:1177] ["became pre-candidate at term 8"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:27:22.959 +08:00] [INFO] [raft.rs:1306] ["broadcasting vote request"] [to="[347199, 319308]"] [log_index=12879] [log_term=8] [term=8] [type=MsgRequestPreVote] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:27:37.981 +08:00] [INFO] [raft.rs:1557] ["starting a new election"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:27:37.981 +08:00] [INFO] [raft.rs:1177] ["became pre-candidate at term 8"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:27:37.981 +08:00] [INFO] [raft.rs:1306] ["broadcasting vote request"] [to="[347199, 319308]"] [log_index=12879] [log_term=8] [term=8] [type=MsgRequestPreVote] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:27:53.001 +08:00] [INFO] [raft.rs:1557] ["starting a new election"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:27:53.001 +08:00] [INFO] [raft.rs:1177] ["became pre-candidate at term 8"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:27:53.001 +08:00] [INFO] [raft.rs:1306] ["broadcasting vote request"] [to="[347199, 319308]"] [log_index=12879] [log_term=8] [term=8] [type=MsgRequestPreVote] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:28:08.323 +08:00] [INFO] [raft.rs:1557] ["starting a new election"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:28:08.323 +08:00] [INFO] [raft.rs:1177] ["became pre-candidate at term 8"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:28:08.323 +08:00] [INFO] [raft.rs:1306] ["broadcasting vote request"] [to="[347199, 319308]"] [log_index=12879] [log_term=8] [term=8] [type=MsgRequestPreVote] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:28:23.361 +08:00] [INFO] [raft.rs:1557] ["starting a new election"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:28:23.361 +08:00] [INFO] [raft.rs:1177] ["became pre-candidate at term 8"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:28:23.361 +08:00] [INFO] [raft.rs:1306] ["broadcasting vote request"] [to="[347199, 319308]"] [log_index=12879] [log_term=8] [term=8] [type=MsgRequestPreVote] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:28:38.420 +08:00] [INFO] [raft.rs:1557] ["starting a new election"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:28:38.420 +08:00] [INFO] [raft.rs:1177] ["became pre-candidate at term 8"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:28:38.420 +08:00] [INFO] [raft.rs:1306] ["broadcasting vote request"] [to="[347199, 319308]"] [log_index=12879] [log_term=8] [term=8] [type=MsgRequestPreVote] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:28:53.458 +08:00] [INFO] [raft.rs:1557] ["starting a new election"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:28:53.459 +08:00] [INFO] [raft.rs:1177] ["became pre-candidate at term 8"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:28:53.459 +08:00] [INFO] [raft.rs:1306] ["broadcasting vote request"] [to="[347199, 319308]"] [log_index=12879] [log_term=8] [term=8] [type=MsgRequestPreVote] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:29:08.498 +08:00] [INFO] [raft.rs:1557] ["starting a new election"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:29:08.502 +08:00] [INFO] [raft.rs:1177] ["became pre-candidate at term 8"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:29:08.502 +08:00] [INFO] [raft.rs:1306] ["broadcasting vote request"] [to="[347199, 319308]"] [log_index=12879] [log_term=8] [term=8] [type=MsgRequestPreVote] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:29:23.534 +08:00] [INFO] [raft.rs:1557] ["starting a new election"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:29:23.534 +08:00] [INFO] [raft.rs:1177] ["became pre-candidate at term 8"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:29:23.534 +08:00] [INFO] [raft.rs:1306] ["broadcasting vote request"] [to="[347199, 319308]"] [log_index=12879] [log_term=8] [term=8] [type=MsgRequestPreVote] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:29:38.558 +08:00] [INFO] [raft.rs:1557] ["starting a new election"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:29:38.558 +08:00] [INFO] [raft.rs:1177] ["became pre-candidate at term 8"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:29:38.558 +08:00] [INFO] [raft.rs:1306] ["broadcasting vote request"] [to="[347199, 319308]"] [log_index=12879] [log_term=8] [term=8] [type=MsgRequestPreVote] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:29:53.597 +08:00] [INFO] [raft.rs:1557] ["starting a new election"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:29:53.597 +08:00] [INFO] [raft.rs:1177] ["became pre-candidate at term 8"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:29:53.597 +08:00] [INFO] [raft.rs:1306] ["broadcasting vote request"] [to="[347199, 319308]"] [log_index=12879] [log_term=8] [term=8] [type=MsgRequestPreVote] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:30:08.647 +08:00] [INFO] [raft.rs:1557] ["starting a new election"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:30:08.647 +08:00] [INFO] [raft.rs:1177] ["became pre-candidate at term 8"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:30:08.647 +08:00] [INFO] [raft.rs:1306] ["broadcasting vote request"] [to="[347199, 319308]"] [log_index=12879] [log_term=8] [term=8] [type=MsgRequestPreVote] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:30:23.685 +08:00] [INFO] [raft.rs:1557] ["starting a new election"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:30:23.685 +08:00] [INFO] [raft.rs:1177] ["became pre-candidate at term 8"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:30:23.685 +08:00] [INFO] [raft.rs:1306] ["broadcasting vote request"] [to="[347199, 319308]"] [log_index=12879] [log_term=8] [term=8] [type=MsgRequestPreVote] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:30:38.720 +08:00] [INFO] [raft.rs:1557] ["starting a new election"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:30:38.720 +08:00] [INFO] [raft.rs:1177] ["became pre-candidate at term 8"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:30:38.720 +08:00] [INFO] [raft.rs:1306] ["broadcasting vote request"] [to="[347199, 319308]"] [log_index=12879] [log_term=8] [term=8] [type=MsgRequestPreVote] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:30:53.750 +08:00] [INFO] [raft.rs:1557] ["starting a new election"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:30:53.750 +08:00] [INFO] [raft.rs:1177] ["became pre-candidate at term 8"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:30:53.750 +08:00] [INFO] [raft.rs:1306] ["broadcasting vote request"] [to="[347199, 319308]"] [log_index=12879] [log_term=8] [term=8] [type=MsgRequestPreVote] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:31:08.773 +08:00] [INFO] [raft.rs:1557] ["starting a new election"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:31:08.774 +08:00] [INFO] [raft.rs:1177] ["became pre-candidate at term 8"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:31:08.774 +08:00] [INFO] [raft.rs:1306] ["broadcasting vote request"] [to="[347199, 319308]"] [log_index=12879] [log_term=8] [term=8] [type=MsgRequestPreVote] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:31:23.807 +08:00] [INFO] [raft.rs:1557] ["starting a new election"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:31:23.807 +08:00] [INFO] [raft.rs:1177] ["became pre-candidate at term 8"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:31:23.807 +08:00] [INFO] [raft.rs:1306] ["broadcasting vote request"] [to="[347199, 319308]"] [log_index=12879] [log_term=8] [term=8] [type=MsgRequestPreVote] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:31:38.824 +08:00] [INFO] [raft.rs:1557] ["starting a new election"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:31:38.824 +08:00] [INFO] [raft.rs:1177] ["became pre-candidate at term 8"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:31:38.824 +08:00] [INFO] [raft.rs:1306] ["broadcasting vote request"] [to="[347199, 319308]"] [log_index=12879] [log_term=8] [term=8] [type=MsgRequestPreVote] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:31:53.845 +08:00] [INFO] [raft.rs:1557] ["starting a new election"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:31:53.845 +08:00] [INFO] [raft.rs:1177] ["became pre-candidate at term 8"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:31:53.845 +08:00] [INFO] [raft.rs:1306] ["broadcasting vote request"] [to="[347199, 319308]"] [log_index=12879] [log_term=8] [term=8] [type=MsgRequestPreVote] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:32:08.870 +08:00] [INFO] [raft.rs:1557] ["starting a new election"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:32:08.870 +08:00] [INFO] [raft.rs:1177] ["became pre-candidate at term 8"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:32:08.870 +08:00] [INFO] [raft.rs:1306] ["broadcasting vote request"] [to="[347199, 319308]"] [log_index=12879] [log_term=8] [term=8] [type=MsgRequestPreVote] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:32:23.905 +08:00] [INFO] [raft.rs:1557] ["starting a new election"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:32:23.905 +08:00] [INFO] [raft.rs:1177] ["became pre-candidate at term 8"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:32:23.905 +08:00] [INFO] [raft.rs:1306] ["broadcasting vote request"] [to="[347199, 319308]"] [log_index=12879] [log_term=8] [term=8] [type=MsgRequestPreVote] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:32:38.926 +08:00] [INFO] [raft.rs:1557] ["starting a new election"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:32:38.926 +08:00] [INFO] [raft.rs:1177] ["became pre-candidate at term 8"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:32:38.926 +08:00] [INFO] [raft.rs:1306] ["broadcasting vote request"] [to="[347199, 319308]"] [log_index=12879] [log_term=8] [term=8] [type=MsgRequestPreVote] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:32:53.941 +08:00] [INFO] [raft.rs:1557] ["starting a new election"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:32:53.941 +08:00] [INFO] [raft.rs:1177] ["became pre-candidate at term 8"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:32:53.941 +08:00] [INFO] [raft.rs:1306] ["broadcasting vote request"] [to="[347199, 319308]"] [log_index=12879] [log_term=8] [term=8] [type=MsgRequestPreVote] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:33:08.957 +08:00] [INFO] [raft.rs:1557] ["starting a new election"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:33:08.957 +08:00] [INFO] [raft.rs:1177] ["became pre-candidate at term 8"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:33:08.957 +08:00] [INFO] [raft.rs:1306] ["broadcasting vote request"] [to="[347199, 319308]"] [log_index=12879] [log_term=8] [term=8] [type=MsgRequestPreVote] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:33:23.972 +08:00] [INFO] [raft.rs:1557] ["starting a new election"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:33:23.972 +08:00] [INFO] [raft.rs:1177] ["became pre-candidate at term 8"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:33:23.972 +08:00] [INFO] [raft.rs:1306] ["broadcasting vote request"] [to="[347199, 319308]"] [log_index=12879] [log_term=8] [term=8] [type=MsgRequestPreVote] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:33:38.986 +08:00] [INFO] [raft.rs:1557] ["starting a new election"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:33:38.986 +08:00] [INFO] [raft.rs:1177] ["became pre-candidate at term 8"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:33:38.986 +08:00] [INFO] [raft.rs:1306] ["broadcasting vote request"] [to="[347199, 319308]"] [log_index=12879] [log_term=8] [term=8] [type=MsgRequestPreVote] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:33:54.002 +08:00] [INFO] [raft.rs:1557] ["starting a new election"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:33:54.003 +08:00] [INFO] [raft.rs:1177] ["became pre-candidate at term 8"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:33:54.003 +08:00] [INFO] [raft.rs:1306] ["broadcasting vote request"] [to="[347199, 319308]"] [log_index=12879] [log_term=8] [term=8] [type=MsgRequestPreVote] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:34:09.017 +08:00] [INFO] [raft.rs:1557] ["starting a new election"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:34:09.017 +08:00] [INFO] [raft.rs:1177] ["became pre-candidate at term 8"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:34:09.017 +08:00] [INFO] [raft.rs:1306] ["broadcasting vote request"] [to="[347199, 319308]"] [log_index=12879] [log_term=8] [term=8] [type=MsgRequestPreVote] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:34:24.068 +08:00] [INFO] [raft.rs:1557] ["starting a new election"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:34:24.068 +08:00] [INFO] [raft.rs:1177] ["became pre-candidate at term 8"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:34:24.068 +08:00] [INFO] [raft.rs:1306] ["broadcasting vote request"] [to="[347199, 319308]"] [log_index=12879] [log_term=8] [term=8] [type=MsgRequestPreVote] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:34:39.109 +08:00] [INFO] [raft.rs:1557] ["starting a new election"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:34:39.109 +08:00] [INFO] [raft.rs:1177] ["became pre-candidate at term 8"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:34:39.109 +08:00] [INFO] [raft.rs:1306] ["broadcasting vote request"] [to="[347199, 319308]"] [log_index=12879] [log_term=8] [term=8] [type=MsgRequestPreVote] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:34:54.126 +08:00] [INFO] [raft.rs:1557] ["starting a new election"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:34:54.126 +08:00] [INFO] [raft.rs:1177] ["became pre-candidate at term 8"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:34:54.126 +08:00] [INFO] [raft.rs:1306] ["broadcasting vote request"] [to="[347199, 319308]"] [log_index=12879] [log_term=8] [term=8] [type=MsgRequestPreVote] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:35:09.141 +08:00] [INFO] [raft.rs:1557] ["starting a new election"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:35:09.141 +08:00] [INFO] [raft.rs:1177] ["became pre-candidate at term 8"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:35:09.141 +08:00] [INFO] [raft.rs:1306] ["broadcasting vote request"] [to="[347199, 319308]"] [log_index=12879] [log_term=8] [term=8] [type=MsgRequestPreVote] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:35:24.158 +08:00] [INFO] [raft.rs:1557] ["starting a new election"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:35:24.158 +08:00] [INFO] [raft.rs:1177] ["became pre-candidate at term 8"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:35:24.158 +08:00] [INFO] [raft.rs:1306] ["broadcasting vote request"] [to="[347199, 319308]"] [log_index=12879] [log_term=8] [term=8] [type=MsgRequestPreVote] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:35:39.696 +08:00] [INFO] [raft.rs:1557] ["starting a new election"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:35:39.696 +08:00] [INFO] [raft.rs:1177] ["became pre-candidate at term 8"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:35:39.696 +08:00] [INFO] [raft.rs:1306] ["broadcasting vote request"] [to="[347199, 319308]"] [log_index=12879] [log_term=8] [term=8] [type=MsgRequestPreVote] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:35:54.721 +08:00] [INFO] [raft.rs:1557] ["starting a new election"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:35:54.721 +08:00] [INFO] [raft.rs:1177] ["became pre-candidate at term 8"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:35:54.721 +08:00] [INFO] [raft.rs:1306] ["broadcasting vote request"] [to="[347199, 319308]"] [log_index=12879] [log_term=8] [term=8] [type=MsgRequestPreVote] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:36:09.747 +08:00] [INFO] [raft.rs:1557] ["starting a new election"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:36:09.747 +08:00] [INFO] [raft.rs:1177] ["became pre-candidate at term 8"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:36:09.747 +08:00] [INFO] [raft.rs:1306] ["broadcasting vote request"] [to="[347199, 319308]"] [log_index=12879] [log_term=8] [term=8] [type=MsgRequestPreVote] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:36:24.762 +08:00] [INFO] [raft.rs:1557] ["starting a new election"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:36:24.762 +08:00] [INFO] [raft.rs:1177] ["became pre-candidate at term 8"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:36:24.762 +08:00] [INFO] [raft.rs:1306] ["broadcasting vote request"] [to="[347199, 319308]"] [log_index=12879] [log_term=8] [term=8] [type=MsgRequestPreVote] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:36:39.777 +08:00] [INFO] [raft.rs:1557] ["starting a new election"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:36:39.777 +08:00] [INFO] [raft.rs:1177] ["became pre-candidate at term 8"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:36:39.777 +08:00] [INFO] [raft.rs:1306] ["broadcasting vote request"] [to="[347199, 319308]"] [log_index=12879] [log_term=8] [term=8] [type=MsgRequestPreVote] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:36:54.807 +08:00] [INFO] [raft.rs:1557] ["starting a new election"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:36:54.807 +08:00] [INFO] [raft.rs:1177] ["became pre-candidate at term 8"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:36:54.807 +08:00] [INFO] [raft.rs:1306] ["broadcasting vote request"] [to="[347199, 319308]"] [log_index=12879] [log_term=8] [term=8] [type=MsgRequestPreVote] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:37:09.829 +08:00] [INFO] [raft.rs:1557] ["starting a new election"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:37:09.830 +08:00] [INFO] [raft.rs:1177] ["became pre-candidate at term 8"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:37:09.830 +08:00] [INFO] [raft.rs:1306] ["broadcasting vote request"] [to="[347199, 319308]"] [log_index=12879] [log_term=8] [term=8] [type=MsgRequestPreVote] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:37:24.845 +08:00] [INFO] [raft.rs:1557] ["starting a new election"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:37:24.845 +08:00] [INFO] [raft.rs:1177] ["became pre-candidate at term 8"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:37:24.845 +08:00] [INFO] [raft.rs:1306] ["broadcasting vote request"] [to="[347199, 319308]"] [log_index=12879] [log_term=8] [term=8] [type=MsgRequestPreVote] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:37:39.863 +08:00] [INFO] [raft.rs:1557] ["starting a new election"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:37:39.863 +08:00] [INFO] [raft.rs:1177] ["became pre-candidate at term 8"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:37:39.863 +08:00] [INFO] [raft.rs:1306] ["broadcasting vote request"] [to="[347199, 319308]"] [log_index=12879] [log_term=8] [term=8] [type=MsgRequestPreVote] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:37:54.881 +08:00] [INFO] [raft.rs:1557] ["starting a new election"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:37:54.881 +08:00] [INFO] [raft.rs:1177] ["became pre-candidate at term 8"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:37:54.881 +08:00] [INFO] [raft.rs:1306] ["broadcasting vote request"] [to="[347199, 319308]"] [log_index=12879] [log_term=8] [term=8] [type=MsgRequestPreVote] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:38:09.897 +08:00] [INFO] [raft.rs:1557] ["starting a new election"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:38:09.897 +08:00] [INFO] [raft.rs:1177] ["became pre-candidate at term 8"] [term=8] [raft_id=319306] [peer_id=319306] [region_id=319305]
[2023/07/30 23:38:09.897 +08:00] [INFO] [raft.rs:1306] ["broadcasting vote request"] [to="[347199, 319308]"] [log_index=12879] [log_term=8] [term=8] [type=MsgRequestPreVote] [raft_id=319306] [peer_id=319306] [region_id=319305]
We can't find any vote messages of this region since the Leader step down. Maybe there are some network error.
cat /var/lib/tikv/log/tikv*.log | grep -E "(RequestPreVote|RequestVote).*region_id=319305\]"
[2023/07/30 21:07:18.444 +08:00] [INFO] [raft.rs:1572] ["[logterm: 0, index: 0, vote: 0] cast vote for 319306 [logterm: 5, index: 5] at term 0"] ["msg type"=MsgRequestPreVote] [term=0] [msg_index=5] [msg_term=5] [from=319306] [vote=0] [log_index=0] [log_term=0] [raft_id=319307] [peer_id=319307] [region_id=319305]
[2023/07/30 21:07:18.791 +08:00] [INFO] [raft.rs:1371] ["received a message with higher term from 319306"] ["msg type"=MsgRequestVote] [message_term=6] [term=5] [from=319306] [raft_id=319307] [peer_id=319307] [region_id=319305]
[2023/07/30 21:07:18.791 +08:00] [INFO] [raft.rs:1572] ["[logterm: 5, index: 5, vote: 0] cast vote for 319306 [logterm: 5, index: 5] at term 6"] ["msg type"=MsgRequestVote] [term=6] [msg_index=5] [msg_term=5] [from=319306] [vote=0] [log_index=5] [log_term=5] [raft_id=319307] [peer_id=319307] [region_id=319305]
In another test case, We found that a region only have one replica in the TiKV cluster.
sh-5.1# ./tikv-ctl --host [tc-tikv-3.tc](http://tc-tikv-3.tc/)-tikv-peer.endless-htap-consistency2-tps-2100560-1-148.svc:20160 raft region -r 12462
DebugClient::region_info: RpcFailure: 5-NOT_FOUND info for region 12462
sh-5.1# ./tikv-ctl --host [tc-tikv-2.tc](http://tc-tikv-2.tc/)-tikv-peer.endless-htap-consistency2-tps-2100560-1-148.svc:20160 raft region -r 12462
DebugClient::region_info: RpcFailure: 5-NOT_FOUND info for region 12462
sh-5.1# ./tikv-ctl --host [tc-tikv-1.tc](http://tc-tikv-1.tc/)-tikv-peer.endless-htap-consistency2-tps-2100560-1-148.svc:20160 raft region -r 12462
DebugClient::region_info: RpcFailure: 5-NOT_FOUND info for region 12462
sh-5.1# ./tikv-ctl --host [tc-tikv-0.tc](http://tc-tikv-0.tc/)-tikv-peer.endless-htap-consistency2-tps-2100560-1-148.svc:20160 raft region -r 12462
{
"region_infos": {
"12462": {
"region_id": 12462,
"region_local_state": {
"region": {
"id": 12462,
"start_key": "7480000000000001FF5B5F728000000000FF7856650000000000FA",
"end_key": "7480000000000001FF5B5F728000000000FF87BD360000000000FA",
"region_epoch": {
"conf_ver": 37,
"version": 180
},
"peers": [
{
"id": 12463,
"store_id": 1,
"role": "Voter"
},
{
"id": 12464,
"store_id": 26,
"role": "Voter"
},
{
"id": 12465,
"store_id": 4,
"role": "Voter"
},
{
"id": 12466,
"store_id": 19,
"role": "Learner"
},
{
"id": 12467,
"store_id": 27,
"role": "Learner"
}
]
}
},
"raft_local_state": {
"hard_state": {
"term": 9,
"vote": 12465,
"commit": 11326
},
"last_index": 11326
},
"raft_apply_state": {
"applied_index": 11325,
"commit_index": 0,
"commit_term": 0,
"truncated_state": {
"index": 11323,
"term": 8
}
}
}
}
}
In a deeper investigate of the previous cases, we found no log for region_id=12462 in tikv-1 and tikv-2. However, there are logs for peer_id=12462 in tikv-1.
The error is raised after the whole workload is finished. Which is to say, there are two phases in this test:
The case failed in phase 2.
Close as the "Dynamic Region" aka "Partitioned Raft KV" has been stopped for a long time
Bug Report
Please answer these questions before submitting your issue. Thanks!
1. Minimal reproduce step (Required)
consistency test
2. What did you expect to see? (Required)
3. What did you see instead (Required)
4. What is your TiFlash version? (Required)
master