Open vzong opened 5 months ago
The IP address 10.25.248.131:2380 (VMS584328), which previously belonged to the tikv-oversea cluster, underwent a scale-in operation on 2024/04/08 at 10:19:26. The tiup cluster display command for tikv-oversea has shown that 10.25.248.131:2380 has been removed, and subsequently, the server VMS584328 was taken offline. However, the pd.log indicates that the tikv-oversea cluster is still attempting to connect to 10.25.248.131:2380 and is reporting connection errors, which persisted until 2024/04/10.
On 2024/04/10, a new server VMS602679 was brought online, and its IP address, 10.25.248.131, was reused. At 13:47 on the same day, 10.25.248.131:2380 (VMS602679) was scaled out to the tikv-dal-test cluster, which transitioned the tikv-dal-test cluster into a 3+1 configuration. During this time, the six nodes of the tikv-oversea cluster also reconnected to 10.25.248.131:2380, forming a 6+1 configuration. Following this, the combination of 3+1+6, all ten pd nodes were connected, creating a ten-node pd cluster. At this point, data confusion occurred.
tikv-oversea 10.109.220.10:2379 10.109.220.9:2379 10.25.248.208:2379 10.25.248.246:2379 10.58.228.76:2379 10.58.228.86:2379
tikv-dal-test 10.58.228.37 10.109.216.124 10.25.248.212
tikv-oversea pd log:
[2024/04/07 18:37:25.977 +08:00] [INFO] [etcdutil.go:309] ["update endpoints"] [num-change=7->8] [last-endpoints="[http://10.58.228.76:2379,http://10.58.228.86:2379,http://10.109.220.9:2379,http://10.109.220.10:2379,http://10.25.248.246:2379,http://10.25.248.131:2379,http://10.25.249.164:2379]"] [endpoints="[http://10.58.228.76:2379,http://10.58.228.86:2379,http://10.109.220.10:2379,http://10.25.248.246:2379,http://10.109.220.9:2379,http://10.25.248.131:2379,http://10.25.249.164:2379,http://10.25.248.208:2379]"]
[2024/04/08 10:19:26.254 +08:00] [INFO] [cluster.go:422] ["removed member"] [cluster-id=468758231b5b0393] [local-member-id=edff54aa33575887] [removed-remote-peer-id=f67c161a4e9b9cb8] [removed-remote-peer-urls="[http://10.25.248.131:2380]"]
[2024/04/08 10:19:27.958 +08:00] [WARN] [grpclog.go:60] ["grpc: addrConn.createTransport failed to connect to {http://10.25.248.131:2379
Subsequently, I attempted to find a way to cancel the reconnection of the removed pd node. I discovered that changing the pd leader did not work; only by reload the pd could the error be eliminated.
reload pd:
[2024/04/11 10:55:45.584 +08:00] [WARN] [grpclog.go:60] ["grpc: addrConn.createTransport failed to connect to {http://10.25.248.208:2379
Why do you need such a number of PDs?
I reproduced this. What's the solution so far?
reproduced steps:
pd-ctl member delete name xxx
to delete two members(pd2, pd3), and now it's 1-1-1As I understand it, these two nodes have already been scaled-in, and there should be no attempt to reconnect the scaled-in node
Is the pd2 and pd3 still in the output of pd-ctl member
?
Is the pd2 and pd3 still in the output of
pd-ctl member
?
Neither pd2 or pd3 are shown in pd-ctl member
.
Which tool do you use to manage the cluster, tiup or tidb-operator?
Does your start script still contain those deleted member?
Which tool do you use to manage the cluster, tiup or tidb-operator?
Instead of using tiup to deploy, I just deployed manually using docker.
Maybe it has something to do with this etcd upgrade, which was reproduced with 7.1.5 but not 7.1.4
We have a tikv cluster, architecture 2:2:2. PD gourp loss leader, here are the relevant logs for the PD leader, the probable cause appears to be that meta data can’t be written to etcd. Is there any other way to troubleshoot the issue?
12:04:40, The network experienced a temporary outage.![image](https://github.com/tikv/pd/assets/7782693/e85dde66-43f7-4bfc-9c8c-9c3c5a3f3587)
13:48:55, write meta data failed.![image](https://github.com/tikv/pd/assets/7782693/28908fe5-de66-4b1b-a13c-5fd9bb5b1a1b)
TSO gap![image](https://github.com/tikv/pd/assets/7782693/c63730c4-23b1-4595-a3f1-a9d5d128c4ad)
Except for the leader, other nodes have only a small amount of logs: region sync with leader meet error.