signal18 / replication-manager

Signal 18 repman - Replication Manager for MySQL / MariaDB / Percona Server
https://signal18.io/products/srm
GNU General Public License v3.0
655 stars 167 forks source link

When the slave sets maintenance, there is a problem with the status change of proxysql. #752

Open peterpan104 opened 2 months ago

peterpan104 commented 2 months ago

This is the state when repman parameters, repman and proxysql are first start:

## repman parameter
proxy-servers-read-on-master = false
proxy-servers-read-on-master-no-slave = true

image

image

I set 172.24.44.182 as maintenance, the status of repman and proxysql is as follows:

image

image

Then set 172.24.47.188 as maintenance, the status of repman and proxysql is as follows:

image

image

Now set 172.24.47.188 as online, the status of repman and proxysql is as follows:

image

image

then set 172.24.43.182 as online, the status of repman and proxysql is as follows:

image

image

proxy-servers-read-on-master-no-slave seems to have no effect, but proxysql status is normal

Now I trigger the switchover, the status of repman and proxysql is as follows:

image

image

I set 172.24.43.182 as maintenance, the status of repman and proxysql is as follows:

image

image

Then set 172.24.47.188 maintenance, the status of repman and proxysql is as follows:

image

image

You can see that 172.24.43.182 is online in the proxysql status!

Now I restore the status to after switchover:

image

image

I set 172.24.47.188 as maintenance, the status of repman and proxysql is as follows:

image

image

Then set 172.24.43.183 as maintenance, the status of repman and proxysql is as follows:

image

image

At this time, proxysql status is still normal.

Now set 172.24.47.188 as online, the status of repman and proxysql is as follows:

image

image

You can see that 172.24.43.182 is online in the proxysql status!

I tried setting proxysql-bootstrap=true, It just allows proxy-servers-read-on-master-no-slave to work, However, the status of proxysql will still be the same as the above test. Can this problem be fixed?

svaroqui commented 2 months ago

Hi peterpan104 We have to think on it a lot, the solution was never made to allow to put master in maintenance it is just made to avoid that situation to do that and to to preserve the writer traffic . A maintenance would be done on slave one by one , once all slaves have finish maintenance one is promote as master and the old master now a slave or a broken slave will receive the maintenance and get reintroduce as the master

svaroqui commented 2 months ago

Sorry please correct I set 172.24.43.182 as maintenance on the first lines as it lead to miss understand of the issue , we need to reproduce as what you are trying to do make sens, still i would never stop all my slaves to put all read traffic on a single node

svaroqui commented 2 months ago

Is this scripting on your side or is it reproductible manually by using the same steps from the gui ?

peterpan104 commented 2 months ago

@svaroqui Sorry, I described it wrong. The above maintenance is all for slave actions.

Under normal conditions, all slaves will not be maintenance. I am simulating the processing after all slaves are failed. The above tests don't use any monitor scripts. The operation can be reproduced through the gui. I will find that it is because my script has set maintenance, But I found that proxysql's changes were not what I expected.

thanks for your reply.

caffeinated92 commented 2 months ago

Do you enable proxysql bootstrap?

peterpan104 commented 2 months ago

HI @caffeinated92 I didn't enable proxysql bootstrap for the above test, but I also enabled this parameter test and the result was the same. The only difference is that when there is no readable machine, master will be used as readable.

caffeinated92 commented 2 months ago

Added bug fix in v2.3.44 but it will need bootstrap to ON. Since we don't want the script to add master in reader list except with acknowledgement

peterpan104 commented 9 hours ago

Hi @caffeinated92 I'm using v2.3.44 and setting bootstrap to ON but the problem is not solved. When I set maintenance on both slaves, one will always show online in proxysql. Can this issue be opened and fixed?