search

sysown / proxysql

High-performance MySQL proxy with a GPL license.
http://www.proxysql.com
GNU General Public License v3.0
5.99k stars 976 forks source link

Question : when a node from reader hostgroup and lost his READ_ONLY #4515

Closed Esysteme closed 5 months ago

Esysteme commented 6 months ago

Maybe it's useless, but I would to try the scheduler ! In the cool stuff, I don't like the configuration at all, that's why imagined that.

First there our situation initial situation :

1 proxysql 1 master with 2 slaves

Capture d’écran du 2024-04-20 14-11-00

[by the way, I don't put 10.68.68.18 in slave hostgroup, I guess when my hg=2 was empty, but when the back ProxySQL should remove him from hg=2 ?]

Now we switch off a slave :

root@vol-master-slave-3:~# service mysql stop

Capture d’écran du 2024-04-20 14-18-09

Node got shunned from ProxySQL, all good there !

Now we restart the node 3 

root@vol-master-slave-3:~# service mysql start

Of course the server, don't have READ_ONLY setting properly, and here it's the disaster ! So as specified in documentation ProxySQL set him in WRITER hostgroup

Capture d’écran du 2024-04-20 14-29-29

and of course what should happen, happened :

Capture d’écran du 2024-04-20 14-36-41

Here a surprise, my application detected more faster the broken link, than ProxySQL, I get information from all servers to each 10 seconds, 1 second for ProxySQL. (I need to investigate to be sure otherwise it's mean ProxySQL, send around and avg ~5 seconds of query to the wrong hostgroup).

after course after :

Capture d’écran du 2024-04-20 14-46-44

Here a miss on my side => need to add on explanation a broken replication of course. I forgot to show you the message we got on replication : Capture d’écran du 2024-04-20 14-48-29

Let's fix the replication : Capture d’écran du 2024-04-20 15-06-22

And there again the same, but at this point I prefer that the proxy take his time to be sure to put back the server online :

Capture d’écran du 2024-04-20 14-57-17 Capture d’écran du 2024-04-20 14-57-22

Well, I got a disater with this point 2 years, ago where the broken slave keep continue to receive write, even if it's stay maybe 10 sec, it's cost more than one week to try to fix everything.

By the way my script (to install complete M/S and ProxySQL) is used still there : https://github.com/PmaControl/Toolkit/blob/master/install-topology-master-slave.sh (I probably made a mistake there)

For me, lost READ_ONLY flag with a reboot, is a mistake. If someone got a use case please let me comment there !

A simple task to prevent this case it's just to control UPTIME with READ_ONLY. Yesterday evening instead of watch a movie, i developed a short script to fix (Or to try ) it !

So if we got this :

READ_ONLY = ON 
UPTIME = 4567

and after we got this :

READ_ONLY = OFF
UPTIME = 3

if

UPTIME_NOW < UPTIME_PREVIOUS

We need to execute : 

mysql> SET GLOBAL READ_ONLY=ON;

To be more simple to monitor I created a new table in main :

MySQL [(none)]> show create table pmacontrol_keep_read_only\G
*************************** 1. row ***************************
       table: pmacontrol_keep_read_only
Create Table: CREATE TABLE pmacontrol_keep_read_only (

    hostname VARCHAR NOT NULL,
    port INT CHECK (port >= 0 AND port <= 65535) NOT NULL DEFAULT 3306, 
    read_only INT CHECK (read_only >= 0 AND read_only <= 1) NOT NULL, 
    uptime_now INT NOT NULL DEFAULT -1,
    uptime_previous INT NOT NULL DEFAULT -1,
    date_now datetime DEFAULT '0000-00-00 00:00:00',
    date_previous datetime DEFAULT '0000-00-00 00:00:00',
    PRIMARY KEY (hostname, port)
    )
1 row in set (0,000 sec)

MySQL [(none)]> select * from pmacontrol_keep_read_only;
+-------------+------+-----------+------------+-----------------+---------------------+---------------------+
| hostname    | port | read_only | uptime_now | uptime_previous | date_now            | date_previous       |
+-------------+------+-----------+------------+-----------------+---------------------+---------------------+
| 10.68.68.19 | 3306 | 1         | 45027      | 45026           | 2024-04-20 14:22:06 | 2024-04-20 14:22:05 |
| 10.68.68.20 | 3306 | 1         | 36516      | 36516           | 2024-04-20 14:17:29 | 2024-04-20 14:17:29 |
+-------------+------+-----------+------------+-----------------+---------------------+---------------------+
2 rows in set (0,000 sec)

Now let do the script who will make it for us : filename : pmacontrol_proxysql_keep_read_only.sh (to move in /var/lib/proxysql)

#!/bin/bash
## inspired by https://dasini.net/blog/2017/01/11/configurer-proxysql-pour-mysql-group-replication/
# Author: Aurélien LEQUOY
# version: 0.1
# 2024-04-05

# CHANGE THOSE
PROXYSQL_USERNAME="pmacontrol"
PROXYSQL_PASSWORD="pmacontrol"
PROXYSQL_HOSTNAME="127.0.0.1"
PROXYSQL_PORT="6032"
#

base_name=$(basename "$0" .sh)
ERR_FILE="${base_name}.log"

function usage()
{
  echo "Usage: $0 --init"
  exit 0
}

if [ "$1" = '-h' -o "$1" = '--help'  -o -z "$1" ]
then
  usage
fi

if [ $# -lt 1 ]
then
  echo "Invalid number of arguments"
  usage
fi

NUMBER_WRITERS="${9:-0}"

#Timeout exists for instances where mysqld may be hung
TIMEOUT=1

#--socket /tmp/proxysql.sock
PROXYSQL_CMDLINE="mysql -u$PROXYSQL_USERNAME -p$PROXYSQL_PASSWORD -h $PROXYSQL_HOSTNAME -P $PROXYSQL_PORT -Nse"
MYSQL_CREDENTIALS=$($PROXYSQL_CMDLINE "SELECT variable_value FROM global_variables WHERE variable_name IN ('mysql-monitor_username','mysql-monitor_password') ORDER BY variable_name")

# Lire les deux lignes séparément
read -r MYSQL_USERNAME <<< $(echo "$MYSQL_CREDENTIALS" | sed -n '1p')
read -r MYSQL_PASSWORD <<< $(echo "$MYSQL_CREDENTIALS" | sed -n '2p')

# Afficher pour vérification
#echo "Username: $MYSQL_USERNAME"
#echo "Password: $MYSQL_PASSWORD"

read HOSTGROUP_WRITER_ID HOSTGROUP_READER_ID CHECK_TYPE <<<  $($PROXYSQL_CMDLINE "SELECT writer_hostgroup, reader_hostgroup, check_type FROM mysql_replication_hostgroups")
#echo "Hostgroup writers $HOSTGROUP_WRITER_ID"
#echo "Hostgroup readers $HOSTGROUP_READER_ID"
#echo "check_type $CHECK_TYPE"
#echo "Number of writers $NUMBER_WRITERS"
#echo "Writers are readers $WRITER_IS_READER"
#echo "log file $ERR_FILE"

if [ "$CHECK_TYPE" != "read_only" ]
then
  echo "Script incompatible, CHECK_TYPE != read_only, or ProxySQL not set with mysql_replication_hostgroups"
  exit 1
fi

# creation de notre table 

if [ ! -f pmacontrol_keep_read_only.log ] 
then
  date=$(date)
  echo "[${date}] Create table pmacontrol_keep_read_only" > pmacontrol_keep_read_only.log

  $PROXYSQL_CMDLINE "CREATE TABLE IF NOT EXISTS pmacontrol_keep_read_only (
    hostname VARCHAR NOT NULL,
    port INT CHECK (port >= 0 AND port <= 65535) NOT NULL DEFAULT 3306, 
    read_only INT CHECK (read_only >= 0 AND read_only <= 1) NOT NULL, 
    uptime_now INT NOT NULL DEFAULT -1,
    uptime_previous INT NOT NULL DEFAULT -1,
    date_now datetime DEFAULT '0000-00-00 00:00:00',
    date_previous datetime DEFAULT '0000-00-00 00:00:00',
    PRIMARY KEY (hostname, port)
    );"
fi

$PROXYSQL_CMDLINE "UPDATE pmacontrol_keep_read_only SET uptime_previous=uptime_now, date_previous=date_now;"

PROXYSQL_HOSTGROUPS=$($PROXYSQL_CMDLINE "SELECT variable_value FROM global_variables 
  WHERE variable_name IN ('mysql-monitor_username','mysql-monitor_password') ORDER BY variable_name")

MYSQL_CMDLINE="timeout $TIMEOUT mysql -nNE -u$MYSQL_USERNAME -p$MYSQL_PASSWORD "

DATE=$(date "+%Y-%m-%d %H:%M:%S")

sql="SELECT a.hostname, a.port
FROM runtime_mysql_servers a
LEFT JOIN runtime_mysql_servers b ON a.hostname = b.hostname AND a.port = b.port AND b.hostgroup_id in ($HOSTGROUP_WRITER_ID)
WHERE a.hostgroup_id in ($HOSTGROUP_READER_ID) AND b.hostname IS NULL;"

$PROXYSQL_CMDLINE "$sql" | while read BACKEND_HOSTNAME PORT 
do
  sql3="SELECT count(1) FROM pmacontrol_keep_read_only WHERE hostname='$BACKEND_HOSTNAME' and port='$PORT'"
  cpt=$($PROXYSQL_CMDLINE "$sql3")

  if [ $cpt == "0" ]; then
    read READ_ONLY <<< $($MYSQL_CMDLINE -h $BACKEND_HOSTNAME -P $PORT -e "show global variables like 'read_only';" 2>>/dev/null | tail -1 2>>${ERR_FILE})
    read UPTIME <<< $($MYSQL_CMDLINE -h $BACKEND_HOSTNAME -P $PORT -e "show global status like 'uptime';" 2>>/dev/null | tail -1 2>>${ERR_FILE})

    if [ "$READ_ONLY" == "ON" ]; then 
      READ_ONLY=1
    else 
      READ_ONLY=0 
    fi;
    sql2="INSERT INTO pmacontrol_keep_read_only (hostname,port,read_only,uptime_now,date_now) VALUES ('$BACKEND_HOSTNAME','$PORT','$READ_ONLY', '$UPTIME','$DATE' );"
    echo "$DATE [INFO] [KEEP_READ_ONLY_AFTER_REBOOT] Add new server $BACKEND_HOSTNAME:$PORT to READ_ONLY tracking"
    $PROXYSQL_CMDLINE "$sql2"
  fi
done

#on prend tout le monde en cas de détection par le proxy même si cela a peu de chance d'arrivé (ca évite de modifier la table mysql_servers)
sql6="SELECT hostname, port FROM pmacontrol_keep_read_only;"

$PROXYSQL_CMDLINE "$sql6" | while read BACKEND_HOSTNAME PORT 
do
  #echo " hostname $BACKEND_HOSTNAME port $PORT "

  read READ_ONLY <<< $($MYSQL_CMDLINE -h $BACKEND_HOSTNAME -P $PORT -e "show global variables like 'read_only';" 2>>/dev/null | tail -1 2>>${ERR_FILE})
  read UPTIME <<< $($MYSQL_CMDLINE -h $BACKEND_HOSTNAME -P $PORT -e "show global status like 'uptime';" 2>>/dev/null | tail -1 2>>${ERR_FILE})

  if [ "$READ_ONLY" == "ON" ]; then 
    READ_ONLY=1
  else 
    READ_ONLY=0 
  fi;

    re='^[0-9]+$'
    if [[ $UPTIME =~ $re ]] ; then     
        sql2="UPDATE pmacontrol_keep_read_only SET read_only='$READ_ONLY', uptime_now='$UPTIME', date_now='$DATE' WHERE hostname ='$BACKEND_HOSTNAME' and port='$PORT';"
        $PROXYSQL_CMDLINE "$sql2"
    fi
done

sql4="SELECT hostname, port FROM pmacontrol_keep_read_only WHERE uptime_now < uptime_previous AND read_only=0 AND uptime_now < 60"

$PROXYSQL_CMDLINE "$sql4" | while read BACKEND_HOSTNAME PORT 
do
  $MYSQL_CMDLINE -h $BACKEND_HOSTNAME -P $PORT -e "SET GLOBAL READ_ONLY=1"

  echo "$DATE [NOTICE] [KEEP_READ_ONLY_AFTER_REBOOT] Detected a reboot of mysql server '$BACKEND_HOSTNAME:$PORT' => Executed : 'SET GLOBAL READ_ONLY=1'"

  #need update proxy ? if fact no, he will update himself
  #$PROXYSQL_CMDLINE "SAVE MYSQL SERVERS TO RUNTIME;" 2>> ${ERR_FILE}
  #sql5="DELETE FROM ....." 
  #$PROXYSQL_CMDLINE "LOAD MYSQL SERVERS TO RUNTIME;" 2>> ${ERR_FILE}
done

# si on élit un nouveau master, il faut l'exclure, donc on exclu tout le monde les slaves seront reload avec le bon read_only
# (et si on garde un master en mémoire et s'il reboot c'et le drame)

$PROXYSQL_CMDLINE "DELETE FROM pmacontrol_keep_read_only WHERE read_only=0;"

What we go in this script :

We got credential to connect on backend server

SELECT variable_value FROM global_variables WHERE variable_name IN ('mysql-monitor_username','mysql-monitor_password') ORDER BY variable_name

We go config for mysql_replication_hostgroups only (not interested by other kind of server

SELECT writer_hostgroup, reader_hostgroup, check_type FROM mysql_replication_hostgroups

We move uptime_now of last run to uptime_previous (I would to day we don't use date, i should remove but I was lazy)

UPDATE pmacontrol_keep_read_only SET uptime_previous=uptime_now, date_previous=date_now;

Important thing, in all request to mysql we need a really shot timeout (here 1 second, but lower will be better)

We need to get all backend we have to follow, here is simple we need to get all backend who are in HOSTGROUP_READER but not in HOSTGROUP_READER

SELECT a.hostname, a.port
FROM runtime_mysql_servers a
LEFT JOIN runtime_mysql_servers b ON a.hostname = b.hostname AND a.port = b.port AND b.hostgroup_id in ($HOSTGROUP_WRITER_ID)
WHERE a.hostgroup_id in ($HOSTGROUP_READER_ID) AND b.hostname IS NULL;

We import these lines in our table (in fact there we could make it at end to get a micro optimization, we could save an extra query for all run each time we add a new server there :D)

then for each line in our table we get new value for READ_ONLY and UPTIME, and we update our table with there value.

UPDATE pmacontrol_keep_read_only SET read_only='$READ_ONLY', uptime_now='$UPTIME', date_now='$DATE' WHERE hostname ='$BACKEND_HOSTNAME' and port='$PORT';

Now we are able to detect all server who was restarted with their READ_ONLY been updated

SELECT hostname, port FROM pmacontrol_keep_read_only WHERE uptime_now < uptime_previous AND read_only=0

For all these lines we update the READ_ONLY flag : 

SET GLOBAL READ_ONLY=1;

To finish we need to remove, all read_only who was updated without a restart of MySQL.

DELETE FROM pmacontrol_keep_read_only WHERE read_only=0;

After this script can be improved with store date in file, and save 4 queries + (number of node to follow) * 2, there 8 queries. And other case will be to add the condition to make it only where uptime not more than 60 secondes. (we never know if ProxySQL was stopped and someone decided to start it after a long time, after a reboot of one MySQL server) => I just added in script.

Now let's try it !

well it's work as expected :

In 80% we got this : (even not detected by ProxySQL)

2024-04-20 03:10:13 [INFO] Scheduler starting id: 1 , filename: /var/lib/proxysql/pmacontrol_proxysql_keep_read_only.sh
2024-04-20 03:10:13 [NOTICE] [KEEP_READ_ONLY_AFTER_REBOOT] Detected a reboot of mysql server '10.68.68.20:3306' => Executed : 'SET GLOBAL READ_ONLY=1'
2024-04-20 03:10:14 [INFO] Scheduler starting id: 1 , filename: /var/lib/proxysql/pmacontrol_proxysql_keep_read_only.sh
2024-04-20 03:10:14 [INFO] [KEEP_READ_ONLY_AFTER_REBOOT] Add new server 10.68.68.20:3306 to READ_ONLY tracking
2024-04-20 03:10:15 [INFO] Scheduler starting id: 1 , filename: /var/lib/proxysql/pmacontrol_proxysql_keep_read_only.sh

But in some case when I guess the connexion got stun with timeout, ProxySQL detect the switch of READ_ONLY before us (I guess ProxySQL made it each 3 seconds)

Look about tag : [KEEP_READ_ONLY_AFTER_REBOOT]

2024-04-20 03:12:04 MySQL_Monitor.cpp:7101:monitor_ping_process_ready_tasks(): [ERROR] Error after 0ms on server 10.68.68.20:3306 : Lost connection to server during query
2024-04-20 03:12:05 [INFO] Scheduler starting id: 1 , filename: /var/lib/proxysql/pmacontrol_proxysql_keep_read_only.sh
2024-04-20 03:12:05 [INFO] Server '10.68.68.20:3306' found with 'read_only=0', but not found as writer
2024-04-20 03:12:05 [INFO] Creating new server in HG 1 : 10.68.68.20:3306 , gtid_port=0, weight=1, status=0
2024-04-20 03:12:05 [INFO] Regenerating table 'mysql_servers' due to actions on server '10.68.68.20:3306'
2024-04-20 03:12:05 [INFO] Dumping current MySQL Servers structures for hostgroup ALL
HID: 1 , address: 10.68.68.18 , port: 3306 , gtid_port: 0 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 10 , use_ssl: 0 , max_latency_ms: 0 , comment: 
HID: 1 , address: 10.68.68.20 , port: 3306 , gtid_port: 0 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 10 , use_ssl: 0 , max_latency_ms: 0 , comment: 
HID: 2 , address: 10.68.68.19 , port: 3306 , gtid_port: 0 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 10 , use_ssl: 0 , max_latency_ms: 0 , comment: 
HID: 2 , address: 10.68.68.20 , port: 3306 , gtid_port: 0 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 10 , use_ssl: 0 , max_latency_ms: 0 , comment: 
HID: 2 , address: 10.68.68.18 , port: 3306 , gtid_port: 0 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment: 
HID: 3 , address: 10.68.68.202 , port: 3306 , gtid_port: 0 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment: 
2024-04-20 03:12:05 [INFO] Dumping mysql_servers: ALL
+-----+--------------+------+------+--------+--------+-----+-----------+---------+-----+---------+---------+-----------------+
| hid | hostname     | port | gtid | weight | status | cmp | max_conns | max_lag | ssl | max_lat | comment | mem_pointer     |
+-----+--------------+------+------+--------+--------+-----+-----------+---------+-----+---------+---------+-----------------+
| 1   | 10.68.68.18  | 3306 | 0    | 1      | 0      | 0   | 1000      | 10      | 0   | 0       |         | 130678809716736 |
| 3   | 10.68.68.202 | 3306 | 0    | 1      | 0      | 0   | 1000      | 0       | 0   | 0       |         | 130678809717312 |
| 2   | 10.68.68.18  | 3306 | 0    | 1      | 0      | 0   | 1000      | 0       | 0   | 0       |         | 130678578666688 |
| 2   | 10.68.68.20  | 3306 | 0    | 1      | 0      | 0   | 1000      | 10      | 0   | 0       |         | 130678809717120 |
| 2   | 10.68.68.19  | 3306 | 0    | 1      | 0      | 0   | 1000      | 10      | 0   | 0       |         | 130678809716928 |
| 1   | 10.68.68.20  | 3306 | 0    | 1      | 0      | 0   | 1000      | 10      | 0   | 0       |         | 130678610075840 |
+-----+--------------+------+------+--------+--------+-----+-----------+---------+-----+---------+---------+-----------------+
2024-04-20 03:12:05 [INFO] Checksum for table mysql_servers is 0xAA35A8FBD6459F74
2024-04-20 03:12:05 [NOTICE] [KEEP_READ_ONLY_AFTER_REBOOT] Detected a reboot of mysql server '10.68.68.20:3306' => Executed : 'SET GLOBAL READ_ONLY=1'
2024-04-20 03:12:06 [INFO] Scheduler starting id: 1 , filename: /var/lib/proxysql/pmacontrol_proxysql_keep_read_only.sh
2024-04-20 03:12:07 [INFO] Scheduler starting id: 1 , filename: /var/lib/proxysql/pmacontrol_proxysql_keep_read_only.sh
2024-04-20 03:12:07 [INFO] Server '10.68.68.20:3306' found with 'read_only=1', but not found as reader
2024-04-20 03:12:07 MySQL_HostGroups_Manager.cpp:8565:remove_HGM(): [WARNING] Removed server at address 0x76d9fc2220c0, hostgroup 1, address 10.68.68.20 port 3306. Setting status OFFLINE HARD and immediately dropping all free connections. Used connections will be dropped when trying to use them
2024-04-20 03:12:07 [INFO] Regenerating table 'mysql_servers' due to actions on server '10.68.68.20:3306'
2024-04-20 03:12:07 [INFO] Dumping current MySQL Servers structures for hostgroup ALL
HID: 1 , address: 10.68.68.18 , port: 3306 , gtid_port: 0 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 10 , use_ssl: 0 , max_latency_ms: 0 , comment: 
HID: 2 , address: 10.68.68.19 , port: 3306 , gtid_port: 0 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 10 , use_ssl: 0 , max_latency_ms: 0 , comment: 
HID: 2 , address: 10.68.68.20 , port: 3306 , gtid_port: 0 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 10 , use_ssl: 0 , max_latency_ms: 0 , comment: 
HID: 2 , address: 10.68.68.18 , port: 3306 , gtid_port: 0 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment: 
HID: 3 , address: 10.68.68.202 , port: 3306 , gtid_port: 0 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment: 
2024-04-20 03:12:07 [INFO] Dumping mysql_servers: ALL
+-----+--------------+------+------+--------+--------+-----+-----------+---------+-----+---------+---------+-----------------+
| hid | hostname     | port | gtid | weight | status | cmp | max_conns | max_lag | ssl | max_lat | comment | mem_pointer     |
+-----+--------------+------+------+--------+--------+-----+-----------+---------+-----+---------+---------+-----------------+
| 1   | 10.68.68.18  | 3306 | 0    | 1      | 0      | 0   | 1000      | 10      | 0   | 0       |         | 130678809716736 |
| 3   | 10.68.68.202 | 3306 | 0    | 1      | 0      | 0   | 1000      | 0       | 0   | 0       |         | 130678809717312 |
| 2   | 10.68.68.18  | 3306 | 0    | 1      | 0      | 0   | 1000      | 0       | 0   | 0       |         | 130678578666688 |
| 2   | 10.68.68.20  | 3306 | 0    | 1      | 0      | 0   | 1000      | 10      | 0   | 0       |         | 130678809717120 |
| 2   | 10.68.68.19  | 3306 | 0    | 1      | 0      | 0   | 1000      | 10      | 0   | 0       |         | 130678809716928 |
+-----+--------------+------+------+--------+--------+-----+-----------+---------+-----+---------+---------+-----------------+
2024-04-20 03:12:07 [INFO] Checksum for table mysql_servers is 0x346EE26CC78E1121
2024-04-20 03:12:07 [INFO] [KEEP_READ_ONLY_AFTER_REBOOT] Add new server 10.68.68.20:3306 to READ_ONLY tracking
2024-04-20 03:12:08 [INFO] Scheduler starting id: 1 , filename: /var/lib/proxysql/pmacontrol_proxysql_keep_read_only.sh

It's not often, but of course it's a problem we miss 2 seconds, it still exist a small windows where the problem can happen, let's try to make a test, maybe ProxySQL have some internal delay (I should open source code, I will check it tomorow)

Here my test : open 10 simultaneous connexions and generate an insert avg each second (0.5~1.5) on same table.

CREATE TABLE a (
    id INT AUTO_INCREMENT NOT NULL,
    PRIMARY KEY (id)
);
#!/bin/bash
for i in {1..10}
do
  (
    while true
    do
      random_sleep=$(echo "scale=2; 0.5 + $RANDOM % 1000 / 1000" | bc)
      sleep $random_sleep
      mysql -h 10.68.68.73 -u stnduser -pstnduser -P6033 test -e "insert into a VALUES (NULL);"
    done
  ) &
done
wait

Now let's produce random start and stop of MySQL on slave, let it's for all night we will see on morning if we arrive to broke the replication.

we have 2 cases :

  1. service mysql restart => it's never happened
  2. service mysql stop && sleep rdm time && service mysql start => it's crashed there !

Capture d’écran du 2024-04-21 01-40-50

So for the moment this solution is a fail :(, in high concurrent database ! (we could execute this script more often but i am not sure it's a good solution at all). The best should be to add it directly inside of ProxySQL as default, and propose an option to deactivate it ?

Maybe one of you got an idea ?

Esysteme commented 6 months ago

I made more time to write this post than to make the script lol :D

MichalisDBA commented 2 months ago

Hey @Esysteme, can you tell me what software you use to make these diagrams?

Esysteme commented 2 months ago

@MichalisDBA it's sub part of PmaControl (my own software) this view will be available when my version 4.0 will be ready.

send me your email in PM