search

openark / orchestrator

MySQL replication topology management and HA
Apache License 2.0
5.62k stars 928 forks source link

recovery from master/slave failure results in co-masters #525

Open lokanadhamm opened 6 years ago

lokanadhamm commented 6 years ago

I have 3 mysql servers mysql-A and mysql-B and mysql-C in topology as follows,

mysql-A -> mysql B and mysql-A -> mysql-C

Now i made mysql-A and mysql-C down. After the recovery mysql-B is becoming master as expected. After some time i made mysql-A up. Now mysql-A is becoming slave to mysql-B but showing revovery symbol. And topoly is showing mysql-C as slave to mysql-A( but this is okay as orchestrator cannot ping mysql-C as it is still down). After this the topology is as follows,

mysql-B -> mysql-A(with recovery label) -> mysql-C(still down)

Now i made mysql-B down and immediately mysql-A became master.After some time i made mysql-B up. But now mysql-B is becoming co-master to mysql-A instead of becoming slave to mysql-A.

After this i am seeing 2 masters(mysql-A and mysql-B) in topology which are replicating from one another.

Is there any config i am missing here or this is an issue?

shlomi-noach commented 6 years ago

Did you set "ApplyMySQLPromotionAfterMasterFailover": true?

See https://github.com/github/orchestrator/blob/master/docs/configuration-recovery.md

This is by far the largest confusion people have with the configuration and I think I should change the default of that flag to true.

shlomi-noach commented 6 years ago

related: https://github.com/github/orchestrator/pull/528, where ApplyMySQLPromotionAfterMasterFailover defaults true

shlomi-noach commented 6 years ago

Thoughts? Will close if not responded.