openark / orchestrator

MySQL replication topology management and HA
Apache License 2.0
5.64k stars 933 forks source link

Should set old master's read_only to false under DeadCoMaster failure #1480

Open ZhangJiaQiao opened 1 year ago

ZhangJiaQiao commented 1 year ago

There is an attempt of setting old master's read_only to false while recovering DeadMaster failure, but there is no such operation for DeadCoMaster failure, which is not reasonable. See the code below:

DeadMaster https://github.com/openark/orchestrator/blob/1a6c3cd6634ce72bb068de81b6af73691e0ce32c/go/logic/topology_recovery.go#L920-L928

DeadCoMaster https://github.com/openark/orchestrator/blob/1a6c3cd6634ce72bb068de81b6af73691e0ce32c/go/logic/topology_recovery.go#L1440-L1443

Old master is supposed to be dead and should not accept new write requests. If we don't try to set old master's read_only to false and the master is not dead completely, there might be living connections still accepting new write requests. When there is only one master is allowed to accept write requests for some purposes, such as generating auto-increment id for new records, the living connections of the old master might still accept new write requests and its data will be conflict with the new master's.