Collecting grastate.dat loop

bartoszx commented 7 years ago

Hello

I am experiencing following loop problem after updating my XML and redeploy stack docker stack deploy --with-registry-auth --compose-file=mystack-stack.yml mystack

mystack_mysql.0.t7xla5nq2k4a@VmystackDB3    | mysqld.sh: Found view from gvwstate.dat with (4) members: 3 11404823-ea44-11e6-84b1-ea05a2b4e40c 9
mystack_mysql.0.t7xla5nq2k4a@VmystackDB3    | mysqld.sh: Collecting grastate.dat and gvwstate.dat info from other nodes...
mystack_mysql.0.lzsskm1zj97u@VmystackDB2    | 2017-02-03 20:54:15 140011364018112 [Note] WSREP: Recovered position: 158fdb18-ea44-11e6-b446-4b7b438f0daa:238
mystack_mysql.0.lzsskm1zj97u@VmystackDB2    | mysqld.sh: --------------------------------------------------
mystack_mysql.0.lzsskm1zj97u@VmystackDB2    | mysqld.sh: Found view from gvwstate.dat with (4) members: 3 11404823-ea44-11e6-84b1-ea05a2b4e40c 9
mystack_mysql.0.lzsskm1zj97u@VmystackDB2    | mysqld.sh: Collecting grastate.dat and gvwstate.dat info from other nodes...
mystack_mysql.0.1zbprj7lcyxa@VmystackDB1    | 2017-02-03 20:54:15 140042962524096 [Note] WSREP: Recovered position: 158fdb18-ea44-11e6-b446-4b7b438f0daa:238
mystack_mysql.0.1zbprj7lcyxa@VmystackDB1    | mysqld.sh: --------------------------------------------------
mystack_mysql.0.1zbprj7lcyxa@VmystackDB1    | mysqld.sh: Found view from gvwstate.dat with (4) members: 3 11404823-ea44-11e6-84b1-ea05a2b4e40c 9
mystack_mysql.0.1zbprj7lcyxa@VmystackDB1    | mysqld.sh: Collecting grastate.dat and gvwstate.dat info from other nodes...
mystack_mysql.0.ymaf8jzf0hh3@VmystackAPP2    | 2017-02-03 20:54:18 140462857877440 [Note] WSREP: Recovered position: 158fdb18-ea44-11e6-b446-4b7b438f0daa:238
mystack_mysql.0.ymaf8jzf0hh3@VmystackAPP2    | mysqld.sh: --------------------------------------------------
mystack_mysql.0.ymaf8jzf0hh3@VmystackAPP2    | mysqld.sh: Collecting grastate.dat and gvwstate.dat info from other nodes...
mystack_mysql.0.uscp9uug9rq3@VmystackAPP1    | ---===--- MariaDB Galera Start Script ---===---
mystack_mysql.0.uscp9uug9rq3@VmystackAPP1    | Got NODE_ADDRESS=10.0.1.7
mystack_mysql.0.uscp9uug9rq3@VmystackAPP1    | Starting node, connecting to gcomm://
mystack_mysql.0.uscp9uug9rq3@VmystackAPP1    | mysqld.sh: uuid is known but seqno is not...
mystack_mysql.0.uscp9uug9rq3@VmystackAPP1    | mysqld.sh: --------------------------------------------------
mystack_mysql.0.uscp9uug9rq3@VmystackAPP1    | mysqld.sh: Attempting to recover GTID positon...
mystack_mysql.0.uscp9uug9rq3@VmystackAPP1    | 2017-02-03 20:54:45 140385227855808 [Note] WSREP: Recovered position: 158fdb18-ea44-11e6-b446-4b7b438f0daa:238
mystack_mysql.0.uscp9uug9rq3@VmystackAPP1    | mysqld.sh: --------------------------------------------------
mystack_mysql.0.uscp9uug9rq3@VmystackAPP1    | mysqld.sh: Found view from gvwstate.dat with (4) members: 3 11404823-ea44-11e6-84b1-ea05a2b4e40c 9
mystack_mysql.0.uscp9uug9rq3@VmystackAPP1    | mysqld.sh: Collecting grastate.dat and gvwstate.dat info from other nodes...
mystack_mysql.0.t7xla5nq2k4a@VmystackDB3    | Received TERM|INT signal. Shutting down...
mystack_mysql.0.t7xla5nq2k4a@VmystackDB3    | ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111 "Connection refused")
mystack_mysql.0.t7xla5nq2k4a@VmystackDB3    | start.sh: Trapped error on line 201
mystack_mysql.0.lzsskm1zj97u@VmystackDB2    | Received TERM|INT signal. Shutting down...
mystack_mysql.0.lzsskm1zj97u@VmystackDB2    | ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111 "Connection refused")
mystack_mysql.0.lzsskm1zj97u@VmystackDB2    | start.sh: Trapped error on line 201
mystack_mysql.0.1zbprj7lcyxa@VmystackDB1    | Received TERM|INT signal. Shutting down...
mystack_mysql.0.1zbprj7lcyxa@VmystackDB1    | ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111 "Connection refused")
mystack_mysql.0.1zbprj7lcyxa@VmystackDB1    | start.sh: Trapped error on line 201
mystack_mysql.0.ymaf8jzf0hh3@VmystackAPP2    | Received TERM|INT signal. Shutting down...
mystack_mysql.0.ymaf8jzf0hh3@VmystackAPP2    | ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111 "Connection refused")
mystack_mysql.0.ymaf8jzf0hh3@VmystackAPP2    | start.sh: Trapped error on line 201
mystack_mysql.0.sv8q5ueu718t@VmystackDB3    | ---===--- MariaDB Galera Start Script ---===---
mystack_mysql.0.sv8q5ueu718t@VmystackDB3    | Got NODE_ADDRESS=10.0.1.4
mystack_mysql.0.sv8q5ueu718t@VmystackDB3    | Starting node, connecting to gcomm://
mystack_mysql.0.sv8q5ueu718t@VmystackDB3    | mysqld.sh: uuid is known but seqno is not...
mystack_mysql.0.sv8q5ueu718t@VmystackDB3    | mysqld.sh: --------------------------------------------------
mystack_mysql.0.sv8q5ueu718t@VmystackDB3    | mysqld.sh: Attempting to recover GTID positon...
mystack_mysql.0.sjv8sz5a7eao@VmystackDB2    | ---===--- MariaDB Galera Start Script ---===---
mystack_mysql.0.sjv8sz5a7eao@VmystackDB2    | Got NODE_ADDRESS=10.0.1.5
mystack_mysql.0.sjv8sz5a7eao@VmystackDB2    | Starting node, connecting to gcomm://
mystack_mysql.0.sjv8sz5a7eao@VmystackDB2    | mysqld.sh: uuid is known but seqno is not...
mystack_mysql.0.sjv8sz5a7eao@VmystackDB2    | mysqld.sh: --------------------------------------------------
mystack_mysql.0.sjv8sz5a7eao@VmystackDB2    | mysqld.sh: Attempting to recover GTID positon...
mystack_mysql.0.sjbr5cnt46um@VmystackDB1    | ---===--- MariaDB Galera Start Script ---===---
mystack_mysql.0.sjbr5cnt46um@VmystackDB1    | Got NODE_ADDRESS=10.0.1.6
mystack_mysql.0.sjbr5cnt46um@VmystackDB1    | Starting node, connecting to gcomm://
mystack_mysql.0.sjbr5cnt46um@VmystackDB1    | mysqld.sh: uuid is known but seqno is not...
mystack_mysql.0.sjbr5cnt46um@VmystackDB1    | mysqld.sh: --------------------------------------------------
mystack_mysql.0.sjbr5cnt46um@VmystackDB1    | mysqld.sh: Attempting to recover GTID positon...
mystack_mysql.0.sv8q5ueu718t@VmystackDB3    | 2017-02-03 20:55:51 140703859263424 [Note] WSREP: Recovered position: 158fdb18-ea44-11e6-b446-4b7b438f0daa:238
mystack_mysql.0.sv8q5ueu718t@VmystackDB3    | mysqld.sh: --------------------------------------------------
mystack_mysql.0.sv8q5ueu718t@VmystackDB3    | mysqld.sh: Found view from gvwstate.dat with (4) members: 3 11404823-ea44-11e6-84b1-ea05a2b4e40c 9
mystack_mysql.0.sv8q5ueu718t@VmystackDB3    | mysqld.sh: Collecting grastate.dat and gvwstate.dat info from other nodes...
mystack_mysql.0.sjv8sz5a7eao@VmystackDB2    | 2017-02-03 20:55:51 140317358938048 [Note] WSREP: Recovered position: 158fdb18-ea44-11e6-b446-4b7b438f0daa:238
mystack_mysql.0.sjv8sz5a7eao@VmystackDB2    | mysqld.sh: --------------------------------------------------
mystack_mysql.0.sjv8sz5a7eao@VmystackDB2    | mysqld.sh: Found view from gvwstate.dat with (4) members: 3 11404823-ea44-11e6-84b1-ea05a2b4e40c 9
mystack_mysql.0.sjv8sz5a7eao@VmystackDB2    | mysqld.sh: Collecting grastate.dat and gvwstate.dat info from other nodes...
mystack_mysql.0.sjbr5cnt46um@VmystackDB1    | 2017-02-03 20:55:51 140419770664896 [Note] WSREP: Recovered position: 158fdb18-ea44-11e6-b446-4b7b438f0daa:238
mystack_mysql.0.sjbr5cnt46um@VmystackDB1    | mysqld.sh: --------------------------------------------------
mystack_mysql.0.sjbr5cnt46um@VmystackDB1    | mysqld.sh: Found view from gvwstate.dat with (4) members: 3 11404823-ea44-11e6-84b1-ea05a2b4e40c 9
mystack_mysql.0.sjbr5cnt46um@VmystackDB1    | mysqld.sh: Collecting grastate.dat and gvwstate.dat info from other nodes...
mystack_mysql.0.8e0k8y2tnau3@VmystackAPP2    | ---===--- MariaDB Galera Start Script ---===---
mystack_mysql.0.8e0k8y2tnau3@VmystackAPP2    | Got NODE_ADDRESS=10.0.1.9
mystack_mysql.0.8e0k8y2tnau3@VmystackAPP2    | Starting node, connecting to gcomm://
mystack_mysql.0.8e0k8y2tnau3@VmystackAPP2    | mysqld.sh: uuid is known but seqno is not...
mystack_mysql.0.8e0k8y2tnau3@VmystackAPP2    | mysqld.sh: --------------------------------------------------
mystack_mysql.0.8e0k8y2tnau3@VmystackAPP2    | mysqld.sh: Attempting to recover GTID positon...
mystack_mysql.0.8e0k8y2tnau3@VmystackAPP2    | 2017-02-03 20:55:56 140042340394944 [Note] WSREP: Recovered position: 158fdb18-ea44-11e6-b446-4b7b438f0daa:238
mystack_mysql.0.8e0k8y2tnau3@VmystackAPP2    | mysqld.sh: --------------------------------------------------
mystack_mysql.0.8e0k8y2tnau3@VmystackAPP2    | mysqld.sh: Collecting grastate.dat and gvwstate.dat info from other nodes...

colinmollenhour commented 7 years ago

The problem I see is:

Starting node, connecting to gcomm://

There should be some IP addresses for the other nodes after gcomm://. These are also used for inter-node communication which it tries for up to 3 minutes but it looks like Docker is killing it before it completes (it isn't doing anything though since there are no nodes to communicate with). If the addresses were there it may not be an issue. If you want to avoid Docker killing them during this phase probably another file used as a flag would be the best way. E.g. touch /var/lib/mysql/pre-boot at mysqld.sh:50 and then rm /var/lib/mysql/pre-boot at line 279 and then add the test -f /var/lib/mysql/pre-boot to the healthcheck.

bartoszx commented 7 years ago

Hi. Thanks for reply. I've tried your hint but this resolves only problem with loop (service is healthy but it is not) However gcom:// is still empty after redeploy

mystack_mysql.0.ip90ouvqewx7@VmystackAPP1 | Got NODE_ADDRESS=10.0.1.4 mystack_mysql.0.ip90ouvqewx7@VmystackAPP1 | Starting node, connecting to gcomm:// mystack_mysql.0.ip90ouvqewx7@VmystackAPP1 | mysqld.sh: uuid is known but seqno is not... mystack_mysql.0.ip90ouvqewx7@VmystackAPP1 | mysqld.sh: -------------------------------------------------- mystack_mysql.0.ip90ouvqewx7@VmystackAPP1 | mysqld.sh: Attempting to recover GTID positon... mystack_mysql.0.ip90ouvqewx7@VmystackAPP1 | 2017-02-04 8:22:47 140378916956096 [Note] WSREP: Recovered position: 0652cdaf-ea61-11e6-844e-cef75cf1b9b7:238 mystack_mysql.0.ip90ouvqewx7@VmystackAPP1 | mysqld.sh: -------------------------------------------------- mystack_mysql.0.ip90ouvqewx7@VmystackAPP1 | mysqld.sh: Found view from gvwstate.dat with (6) members: 3 061cf80d-ea61-11e6-84f9-fb7f234cfcc9 12 mystack_mysql.0.ip90ouvqewx7@VmystackAPP1 | mysqld.sh: Collecting grastate.dat and gvwstate.dat info from other nodes...

colinmollenhour commented 7 years ago

I would start by confirming that within the container the DNS resolves as expected. E.g., whatever your second command argument is for "node" containers should resolve to all of the IPs of the other nodes. The "seed" node does not have this second parameter, but there should only be one "seed" instance.

bartoszx commented 7 years ago

Hi. This is what I did to simplify my tests.

I disabled healtcheck to stop containers exit
I check getent in each of them

root@23d10569cef6:/# getent hosts tasks.mysql 10.0.0.7 tasks.mysql 10.0.0.10 tasks.mysql 10.0.0.8 tasks.mysql 10.0.0.9 tasks.mysql 10.0.0.6 tasks.mysql

my args contains only tasks.mysql

"Id": "23d10569cef67e30900d7d24b2531d3f6f8e1597a776b677cd1ab727bcf2673d", "Created": "2017-02-05T17:28:50.289581995Z", "Path": "start.sh", "Args": [ "node", "tasks.mystack_mysql" ],

Results:

mystack_mysql.0.afnzj9a7kpi2@VmystackAPP1 | ---===--- MariaDB Galera Start Script ---===--- mystack_mysql.0.afnzj9a7kpi2@VmystackAPP1 | Got NODE_ADDRESS=10.0.0.7 mystack_mysql.0.ws15i9ixpqk7@VmystackAPP2 | ---===--- MariaDB Galera Start Script ---===--- mystack_mysql.0.ws15i9ixpqk7@VmystackAPP2 | Got NODE_ADDRESS=10.0.0.8 mystack_mysql.0.ws15i9ixpqk7@VmystackAPP2 | Starting node, connecting to gcomm:// mystack_mysql.0.y41djxuwu3ft@VmystackDB2 | ---===--- MariaDB Galera Start Script ---===--- mystack_mysql.0.y41djxuwu3ft@VmystackDB2 | Got NODE_ADDRESS=10.0.0.10 mystack_mysql.0.y41djxuwu3ft@VmystackDB2 | Starting node, connecting to gcomm:// mystack_mysql.0.nshetb2mzy9i@VmystackDB1 | ---===--- MariaDB Galera Start Script ---===--- mystack_mysql.0.nshetb2mzy9i@VmystackDB1 | Got NODE_ADDRESS=10.0.0.9 mystack_mysql.0.nshetb2mzy9i@VmystackDB1 | Starting node, connecting to gcomm:// mystack_mysql.0.nshetb2mzy9i@VmystackDB1 | mysqld.sh: uuid is known but seqno is not... mystack_mysql.0.nshetb2mzy9i@VmystackDB1 | mysqld.sh: -------------------------------------------------- mystack_mysql.0.nshetb2mzy9i@VmystackDB1 | mysqld.sh: Attempting to recover GTID positon... mystack_mysql.0.nshetb2mzy9i@VmystackDB1 | 2017-02-05 18:28:51 140484534290368 [Note] WSREP: Recovered position: 0652cdaf-ea61-11e6-844e-cef75cf1b9b7:238 mystack_mysql.0.nshetb2mzy9i@VmystackDB1 | mysqld.sh: -------------------------------------------------- mystack_mysql.0.nshetb2mzy9i@VmystackDB1 | mysqld.sh: Found view from gvwstate.dat with (5) members: 3 061cf80d-ea61-11e6-84f9-fb7f234cfcc9 21 mystack_mysql.0.nshetb2mzy9i@VmystackDB1 | mysqld.sh: Collecting grastate.dat and gvwstate.dat info from other nodes... mystack_mysql.0.ws15i9ixpqk7@VmystackAPP2 | mysqld.sh: uuid is known but seqno is not... mystack_mysql.0.ws15i9ixpqk7@VmystackAPP2 | mysqld.sh: -------------------------------------------------- mystack_mysql.0.ws15i9ixpqk7@VmystackAPP2 | mysqld.sh: Attempting to recover GTID positon... mystack_mysql.0.ws15i9ixpqk7@VmystackAPP2 | 2017-02-05 18:28:52 140615837607872 [Note] WSREP: Recovered position: 0652cdaf-ea61-11e6-844e-cef75cf1b9b7:238 mystack_mysql.0.ws15i9ixpqk7@VmystackAPP2 | mysqld.sh: -------------------------------------------------- mystack_mysql.0.ws15i9ixpqk7@VmystackAPP2 | mysqld.sh: Found view from gvwstate.dat with (5) members: 3 061cf80d-ea61-11e6-84f9-fb7f234cfcc9 21 mystack_mysql.0.ws15i9ixpqk7@VmystackAPP2 | mysqld.sh: Collecting grastate.dat and gvwstate.dat info from other nodes... mystack_mysql.0.rojbzwm77s6i@VmystackDB3 | ---===--- MariaDB Galera Start Script ---===--- mystack_mysql.0.rojbzwm77s6i@VmystackDB3 | Got NODE_ADDRESS=10.0.0.6 mystack_mysql.0.rojbzwm77s6i@VmystackDB3 | Starting node, connecting to gcomm:// mystack_mysql.0.rojbzwm77s6i@VmystackDB3 | mysqld.sh: Recovered position from grastate.dat: 0652cdaf-ea61-11e6-844e-cef75cf1b9b7:238 mystack_mysql.0.rojbzwm77s6i@VmystackDB3 | mysqld.sh: Collecting grastate.dat and gvwstate.dat info from other nodes... mystack_mysql.0.y41djxuwu3ft@VmystackDB2 | mysqld.sh: Recovered position from grastate.dat: 0652cdaf-ea61-11e6-844e-cef75cf1b9b7:238 mystack_mysql.0.y41djxuwu3ft@VmystackDB2 | mysqld.sh: Collecting grastate.dat and gvwstate.dat info from other nodes... mystack_mysql.0.afnzj9a7kpi2@VmystackAPP1 | Starting node, connecting to gcomm:// mystack_mysql.0.afnzj9a7kpi2@VmystackAPP1 | mysqld.sh: uuid is known but seqno is not... mystack_mysql.0.afnzj9a7kpi2@VmystackAPP1 | mysqld.sh: -------------------------------------------------- mystack_mysql.0.afnzj9a7kpi2@VmystackAPP1 | mysqld.sh: Attempting to recover GTID positon... mystack_mysql.0.afnzj9a7kpi2@VmystackAPP1 | 2017-02-05 18:28:51 140339704223680 [Note] WSREP: Recovered position: 0652cdaf-ea61-11e6-844e-cef75cf1b9b7:238 mystack_mysql.0.afnzj9a7kpi2@VmystackAPP1 | mysqld.sh: -------------------------------------------------- mystack_mysql.0.afnzj9a7kpi2@VmystackAPP1 | mysqld.sh: Found view from gvwstate.dat with (5) members: 3 061cf80d-ea61-11e6-84f9-fb7f234cfcc9 21 mystack_mysql.0.afnzj9a7kpi2@VmystackAPP1 | mysqld.sh: Collecting grastate.dat and gvwstate.dat info from other nodes...

Everytime I run

docker exec -it $(docker ps | grep mysql | awk -F ' ' '{print $1}') touch /var/lib/mysql/wsrep-new-cluster && docker service update --env-add up=1 mystack_mysql

cluster starts as it should

colinmollenhour commented 7 years ago

When you checked DNS with getent you checked "tasks.mysql", but your command argument is for "tasks.mystack_mysql".. Seems your command should be changed to "node tasks.mysql".

bartoszx commented 7 years ago

You are right. Anyway both working correctly

root@1ba9e7f188fc:/# getent hosts tasks.mystack_mysql
10.0.0.9        tasks.mystack_mysql
10.0.0.6        tasks.mystack_mysql
10.0.0.7        tasks.mystack_mysql
10.0.0.8        tasks.mystack_mysql
10.0.0.5        tasks.mystack_mysql
root@1ba9e7f188fc:/# getent hosts tasks.mysql
10.0.0.7        tasks.mysql
10.0.0.5        tasks.mysql
10.0.0.8        tasks.mysql
10.0.0.9        tasks.mysql
10.0.0.6        tasks.mysql

other test

root@1ba9e7f188fc:/# getent hosts tasks.mystack_proxy
10.0.0.3        tasks.mystack_proxy
root@1ba9e7f188fc:/# getent hosts tasks.proxy
10.0.0.3        tasks.proxy
root@1ba9e7f188fc:/# getent hosts tasks.mystack_wrongone

colinmollenhour commented 7 years ago

I think you should enable tracing (set env variable TRACE=y) because something weird is happening.. If there are no hosts resolved you should be seeing the line "Waiting for at least $GCOMM_MINIMUM IP addresses to resolve..." but that isn't in your logs.. It is like it does resolve but the value is still empty..

bartoszx commented 7 years ago

Full log

mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | + '[' -z eth2 ']'
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | + '[' -z eth2 ']'
mystack_mysql.0.s4c1nd937wzv@MYSTACKAPP2    | sleep|no-galera|bash|seed|node <othernode>,...
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | + '[' -z eth2 ']'
mystack_mysql.0.1blgnbzrjn3a@MYSTACKAPP2    | sleep|no-galera|bash|seed|node <othernode>,...
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | + [[ eth2 =~ [a-zA-Z][a-zA-Z0-9:]+ ]]
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | + [[ eth2 =~ [a-zA-Z][a-zA-Z0-9:]+ ]]
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | ++ ip addr
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | ++ ip addr
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | ++ awk '/inet/ && / eth2$/{sub(/\/.*$/,"",$2); print $2}'
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | ++ awk '/inet/ && / eth2$/{sub(/\/.*$/,"",$2); print $2}'
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | ++ head -n 1
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | ++ head -n 1
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | ---===--- MariaDB Galera Start Script ---===---
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | ---===--- MariaDB Galera Start Script ---===---
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | Got NODE_ADDRESS=10.0.0.10
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | Got NODE_ADDRESS=10.0.0.5
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | + NODE_ADDRESS=10.0.0.5
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | + NODE_ADDRESS=10.0.0.10
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | + [[ 10.0.0.10 =~ ^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+$ ]]
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | + [[ 10.0.0.5 =~ ^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+$ ]]
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | + echo '---===--- MariaDB Galera Start Script ---===---'
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | + echo '---===--- MariaDB Galera Start Script ---===---'
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | + echo 'Got NODE_ADDRESS=10.0.0.10'
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | + echo 'Got NODE_ADDRESS=10.0.0.5'
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | + '[' -z **************** ']'
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | + '[' -z **************** ']'
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | + '[' -z ']'
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | + '[' -z ']'
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | + '[' -f ']'
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | + '[' -f ']'
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | ++ cat
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | ++ cat
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | + SYSTEM_PASSWORD=
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | + SYSTEM_PASSWORD=
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | + '[' -z *************** ']'
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | + '[' -z *************** ']'
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | + '[' -z ']'
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | + '[' -z ']'
mystack_mysql.0.os1sqw2ldrk9@MYSTACKDB1    | + '[' -z *************** ']'
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | + '[' -f ']'
mystack_mysql.0.os1sqw2ldrk9@MYSTACKDB1    | + '[' -z ']'
mystack_mysql.0.os1sqw2ldrk9@MYSTACKDB1    | + '[' -f ']'
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | ++ cat
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | + MYSQL_PASSWORD=
mystack_mysql.0.os1sqw2ldrk9@MYSTACKDB1    | ++ cat
mystack_mysql.0.os1sqw2ldrk9@MYSTACKDB1    | + MYSQL_PASSWORD=
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | + test -n ****************
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | ++ echo ****************
mystack_mysql.0.os1sqw2ldrk9@MYSTACKDB1    | + test -n ****************
mystack_mysql.0.os1sqw2ldrk9@MYSTACKDB1    | ++ echo ****************
mystack_mysql.0.62bu8zhio1b1@MYSTACKDB2    | sleep|no-galera|bash|seed|node <othernode>,...
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | + '[' -z eth2 ']'
mystack_mysql.0.tsmk14equni1@MYSTACKDB2    | sleep|no-galera|bash|seed|node <othernode>,...
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | + [[ eth2 =~ [a-zA-Z][a-zA-Z0-9:]+ ]]
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | ++ ip addr
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | ++ awk '/inet/ && / eth2$/{sub(/\/.*$/,"",$2); print $2}'
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | ++ head -n 1
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | + NODE_ADDRESS=10.0.0.8
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | + [[ 10.0.0.8 =~ ^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+$ ]]
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | + echo '---===--- MariaDB Galera Start Script ---===---'
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | + echo 'Got NODE_ADDRESS=10.0.0.8'
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | + '[' -z **************** ']'
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | + '[' -z ']'
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | + '[' -f ']'
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | ---===--- MariaDB Galera Start Script ---===---
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | Got NODE_ADDRESS=10.0.0.8
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | Got NODE_ADDRESS=10.0.0.9
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | + echo '---===--- MariaDB Galera Start Script ---===---'
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | + echo 'Got NODE_ADDRESS=10.0.0.9'
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | + '[' -z **************** ']'
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | + '[' -z ']'
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | + '[' -f ']'
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | ++ cat
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | + SYSTEM_PASSWORD=
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | + '[' -z *************** ']'
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | + '[' -z ']'
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | + '[' -f ']'
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | ++ cat
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | + MYSQL_PASSWORD=
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | + test -n ****************
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | ++ echo ****************
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | ++ sha256sum
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | ++ awk '{print $1;}'
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | + SYSTEM_PASSWORD=****************************************************************
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | + case "$1" in
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | + '[' -z **************** ']'
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | + '[' -z **************************************************************** ']'
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | + CLUSTER_NAME=xcluster
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | + GCOMM_MINIMUM=2
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | + GCOMM=
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | + MYSQL_MODE_ARGS=
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | + '[' -f /usr/local/lib/startup.sh ']'
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | + '[' node = node ']'
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | + '[' -f /var/lib/mysql/force-cluster-bootstrapping ']'
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | + '[' node = seed ']'
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | + case "$1" in
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | + MYSQL_MODE_ARGS+=' --wsrep-on=ON'
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | + '[' -z tasks.mystack_mysql ']'
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | + ADDRS=tasks.mystack_mysql
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | + RESOLVE=0
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | + SLEEPS=0
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | + true
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | + SEP=
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | + GCOMM=
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | + for ADDR in '${ADDRS//,/ }'
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | + [[ tasks.mystack_mysql =~ ^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+$ ]]
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | + RESOLVE=1
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | ++ getent hosts tasks.mystack_mysql
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | ++ awk '{ print $1 }'
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | ++ paste -sd ,
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | + GCOMM+=
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | + '[' -n '' ']'
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | + GCOMM=
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | ++ echo ''
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | ++ sed 's/,\+/,/g'
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | + GCOMM=
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | ++ echo ''
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | ++ tr , '\n'
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | ++ sort -u
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | ++ grep -v -e '^10.0.0.9$'
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | ++ wc -l
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | + COUNT=1
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | + '[' 1 -eq 1 ']'
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | Starting node, connecting to gcomm://
mystack_mysql.0.s9k3b4oxqi5w@MYSTACKAPP1    | ---===--- MariaDB Galera Start Script ---===---
mystack_mysql.0.6xpzfvwpo1bd@MYSTACKAPP1    | ---===--- MariaDB Galera Start Script ---===---
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | ++ cat
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | + SYSTEM_PASSWORD=
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | + '[' -z *************** ']'
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | + '[' -z ']'
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | + '[' -f ']'
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | ++ cat
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | + MYSQL_PASSWORD=
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | + test -n ****************
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | ++ echo ****************
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | ++ sha256sum
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | ++ awk '{print $1;}'
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | + SYSTEM_PASSWORD=****************************************************************
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | + case "$1" in
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | + '[' -z **************** ']'
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | + '[' -z **************************************************************** ']'
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | + CLUSTER_NAME=xcluster
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | + GCOMM_MINIMUM=2
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | + GCOMM=
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | + MYSQL_MODE_ARGS=
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | + '[' -f /usr/local/lib/startup.sh ']'
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | + '[' node = node ']'
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | + '[' -f /var/lib/mysql/force-cluster-bootstrapping ']'
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | + '[' node = seed ']'
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | + case "$1" in
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | + MYSQL_MODE_ARGS+=' --wsrep-on=ON'
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | + '[' -z tasks.mystack_mysql ']'
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | + ADDRS=tasks.mystack_mysql
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | + RESOLVE=0
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | + SLEEPS=0
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | + true
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | + SEP=
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | + GCOMM=
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | + for ADDR in '${ADDRS//,/ }'
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | + [[ tasks.mystack_mysql =~ ^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+$ ]]
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | + RESOLVE=1
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | ++ getent hosts tasks.mystack_mysql
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | ++ awk '{ print $1 }'
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | ++ paste -sd ,
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | + GCOMM+=
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | + '[' -n '' ']'
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | + GCOMM=
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | ++ echo ''
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | ++ sed 's/,\+/,/g'
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | + GCOMM=
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | ++ echo ''
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | ++ tr , '\n'
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | ++ sort -u
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | ++ grep -v -e '^10.0.0.8$'
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | ++ wc -l
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | + COUNT=1
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | + '[' 1 -eq 1 ']'
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | + '[' 1 -lt 1 ']'
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | + break
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | + shift 2
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | + echo 'Starting node, connecting to gcomm://'
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | + set +e -m
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | Starting node, connecting to gcomm://
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | + trap shutdown TERM INT
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | + galera-healthcheck -user=system -password=**************************************************************** -port=8080 -availWhenDonor=false -availWhenReadOnly=false -pidfile=/var/run/galera-healthcheck-1.pid
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | + galera-healthcheck -user=system -password=**************************************************************** -port=8081 -availWhenDonor=true -availWhenReadOnly=true -pidfile=/var/run/galera-healthcheck-2.pid
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | + wait 35
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | + gosu mysql mysqld.sh --console --wsrep-on=ON --wsrep_cluster_name=xcluster --wsrep_cluster_address=gcomm:// --wsrep_node_address=10.0.0.8:4567 --wsrep_sst_auth=xtrabackup:**************** --default-time-zone=+00:00
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | + [[ --console --wsrep-on=ON --wsrep_cluster_name=xcluster --wsrep_cluster_address=gcomm:// --wsrep_node_address=10.0.0.8:4567 --wsrep_sst_auth=xtrabackup:**************** --default-time-zone=+00:00 =~ /--wsrep-new-cluster/ ]]
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | + test -f /var/lib/mysql/ibdata1
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | + POSITION=
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | + test -f /var/lib/mysql/grastate.dat
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | + grep -q seqno: /var/lib/mysql/grastate.dat
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | + grep -q 00000000-0000-0000-0000-000000000000 /var/lib/mysql/grastate.dat
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | ++ awk '/^uuid:/{print $2}' /var/lib/mysql/grastate.dat
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | + uuid=0652cdaf-ea61-11e6-844e-cef75cf1b9b7
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | ++ awk '/^seqno:/{print $2}' /var/lib/mysql/grastate.dat
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | + seqno=238
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | + '[' 238 = -1 ']'
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | + '[' -n 0652cdaf-ea61-11e6-844e-cef75cf1b9b7 ']'
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | + '[' -n 238 ']'
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | + POSITION=0652cdaf-ea61-11e6-844e-cef75cf1b9b7:238
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | + echo 'mysqld.sh: Recovered position from grastate.dat: 0652cdaf-ea61-11e6-844e-cef75cf1b9b7:238'
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | mysqld.sh: Recovered position from grastate.dat: 0652cdaf-ea61-11e6-844e-cef75cf1b9b7:238
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | + [[ -z 0652cdaf-ea61-11e6-844e-cef75cf1b9b7:238 ]]
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | ++ sed -E 's#.*--wsrep_node_address=([0-9\.]+):4567.*#\1#'
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | + NODE_ADDRESS=10.0.0.8
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | ++ sed -E 's#.*gcomm://([0-9\.,]+)\s+.*#\1#'
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | + GCOMM='--console --wsrep-on=ON --wsrep_cluster_name=xcluster --wsrep_cluster_address=gcomm:// --wsrep_node_address=10.0.0.8:4567 --wsrep_sst_auth=xtrabackup:**************** --default-time-zone=+00:00'
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | + [[ -f /var/lib/mysql/wsrep-new-cluster ]]
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | + [[ -z 0652cdaf-ea61-11e6-844e-cef75cf1b9b7:238 ]]
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | + check_nodes --console --wsrep-on=ON --wsrep_cluster_name=xcluster --wsrep_cluster_address=gcomm:// --wsrep_node_address=10.0.0.8:4567 --wsrep_sst_auth=xtrabackup:**************** --default-time-zone=+00:00 10.0.0.8
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | + for node in '${1//,/ }'
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | + '[' --console = --wsrep-on=ON ']'
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | + curl -f -s -o - http://--console:8081
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | + return 1
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | + LISTEN_PORT=3309
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | + EXPECT_NODES=3
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | + [[ -f /var/lib/mysql/gvwstate.dat ]]
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | ++ wc -w
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | + [[ 7 -gt 3 ]]
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | ++ wc -w
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | + EXPECT_NODES=7
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | + EXPECT_NODES=6
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | + echo 'mysqld.sh: Collecting grastate.dat and gvwstate.dat info from other nodes...'
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | mysqld.sh: Collecting grastate.dat and gvwstate.dat info from other nodes...
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | + set -m
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | ++ mktemp -t socat.XXXX
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | + tmpfile=/tmp/socat.tCXU
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | + PID_SERVER=59
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | + SENT_NODES=
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | + for i in '{36..0}'
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | ++ sha256sum
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | ++ awk '{print $1;}'
mystack_mysql.0.os1sqw2ldrk9@MYSTACKDB1    | ++ sha256sum
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | + SYSTEM_PASSWORD=****************************************************************
mystack_mysql.0.os1sqw2ldrk9@MYSTACKDB1    | ++ awk '{print $1;}'
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | + case "$1" in
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | + '[' -z **************** ']'
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | + '[' -z **************************************************************** ']'
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | + CLUSTER_NAME=xcluster
mystack_mysql.0.os1sqw2ldrk9@MYSTACKDB1    | + SYSTEM_PASSWORD=****************************************************************
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | + GCOMM_MINIMUM=2
mystack_mysql.0.os1sqw2ldrk9@MYSTACKDB1    | + case "$1" in
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | + GCOMM=
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | + MYSQL_MODE_ARGS=
mystack_mysql.0.os1sqw2ldrk9@MYSTACKDB1    | + '[' -z **************** ']'
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | + '[' -f /usr/local/lib/startup.sh ']'
mystack_mysql.0.os1sqw2ldrk9@MYSTACKDB1    | + '[' -z **************************************************************** ']'
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | + '[' node = node ']'
mystack_mysql.0.os1sqw2ldrk9@MYSTACKDB1    | + CLUSTER_NAME=xcluster
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | + '[' -f /var/lib/mysql/force-cluster-bootstrapping ']'
mystack_mysql.0.os1sqw2ldrk9@MYSTACKDB1    | + GCOMM_MINIMUM=2
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | + '[' node = seed ']'
mystack_mysql.0.os1sqw2ldrk9@MYSTACKDB1    | + GCOMM=
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | + case "$1" in
mystack_mysql.0.os1sqw2ldrk9@MYSTACKDB1    | + MYSQL_MODE_ARGS=
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | + MYSQL_MODE_ARGS+=' --wsrep-on=ON'
mystack_mysql.0.os1sqw2ldrk9@MYSTACKDB1    | + '[' -f /usr/local/lib/startup.sh ']'
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | + '[' -z tasks.mystack_mysql ']'
mystack_mysql.0.os1sqw2ldrk9@MYSTACKDB1    | + '[' node = node ']'
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | + ADDRS=tasks.mystack_mysql
mystack_mysql.0.os1sqw2ldrk9@MYSTACKDB1    | + '[' -f /var/lib/mysql/force-cluster-bootstrapping ']'
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | + RESOLVE=0
mystack_mysql.0.os1sqw2ldrk9@MYSTACKDB1    | + '[' node = seed ']'
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | + SLEEPS=0
mystack_mysql.0.os1sqw2ldrk9@MYSTACKDB1    | + case "$1" in
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | + true
mystack_mysql.0.os1sqw2ldrk9@MYSTACKDB1    | + MYSQL_MODE_ARGS+=' --wsrep-on=ON'
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | + SEP=
mystack_mysql.0.os1sqw2ldrk9@MYSTACKDB1    | + '[' -z tasks.mystack_mysql ']'
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | + GCOMM=
mystack_mysql.0.os1sqw2ldrk9@MYSTACKDB1    | + ADDRS=tasks.mystack_mysql
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | + for ADDR in '${ADDRS//,/ }'
mystack_mysql.0.os1sqw2ldrk9@MYSTACKDB1    | + RESOLVE=0
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | + [[ tasks.mystack_mysql =~ ^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+$ ]]
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | + RESOLVE=1
mystack_mysql.0.os1sqw2ldrk9@MYSTACKDB1    | + SLEEPS=0
mystack_mysql.0.os1sqw2ldrk9@MYSTACKDB1    | + true
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | ++ getent hosts tasks.mystack_mysql
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | ++ awk '{ print $1 }'
mystack_mysql.0.os1sqw2ldrk9@MYSTACKDB1    | + SEP=
mystack_mysql.0.os1sqw2ldrk9@MYSTACKDB1    | + GCOMM=
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | ++ paste -sd ,
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | + GCOMM+=
mystack_mysql.0.os1sqw2ldrk9@MYSTACKDB1    | + for ADDR in '${ADDRS//,/ }'
mystack_mysql.0.os1sqw2ldrk9@MYSTACKDB1    | + [[ tasks.mystack_mysql =~ ^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+$ ]]
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | + '[' -n '' ']'
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | + GCOMM=
mystack_mysql.0.os1sqw2ldrk9@MYSTACKDB1    | + RESOLVE=1
mystack_mysql.0.os1sqw2ldrk9@MYSTACKDB1    | ++ getent hosts tasks.mystack_mysql
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | ++ echo ''
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | ++ sed 's/,\+/,/g'
mystack_mysql.0.os1sqw2ldrk9@MYSTACKDB1    | ++ awk '{ print $1 }'
mystack_mysql.0.os1sqw2ldrk9@MYSTACKDB1    | ++ paste -sd ,
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | + GCOMM=
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | ++ echo ''
mystack_mysql.0.os1sqw2ldrk9@MYSTACKDB1    | + GCOMM+=
mystack_mysql.0.os1sqw2ldrk9@MYSTACKDB1    | + '[' -n '' ']'
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | ++ tr , '\n'
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | ++ sort -u
mystack_mysql.0.os1sqw2ldrk9@MYSTACKDB1    | + GCOMM=
mystack_mysql.0.os1sqw2ldrk9@MYSTACKDB1    | ++ echo ''
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | ++ wc -l
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | ++ grep -v -e '^10.0.0.12$'
mystack_mysql.0.os1sqw2ldrk9@MYSTACKDB1    | ++ sed 's/,\+/,/g'
mystack_mysql.0.os1sqw2ldrk9@MYSTACKDB1    | + GCOMM=
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | + COUNT=1
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | + '[' 1 -eq 1 ']'
mystack_mysql.0.os1sqw2ldrk9@MYSTACKDB1    | ++ echo ''
mystack_mysql.0.os1sqw2ldrk9@MYSTACKDB1    | ++ tr , '\n'
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | + '[' 1 -lt 1 ']'
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | + break
mystack_mysql.0.os1sqw2ldrk9@MYSTACKDB1    | ++ grep -v -e '^10.0.0.7$'
mystack_mysql.0.os1sqw2ldrk9@MYSTACKDB1    | ++ wc -l
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | + shift 2
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | + echo 'Starting node, connecting to gcomm://'
mystack_mysql.0.os1sqw2ldrk9@MYSTACKDB1    | ++ sort -u
mystack_mysql.0.os1sqw2ldrk9@MYSTACKDB1    | Starting node, connecting to gcomm://
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | + set +e -m
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | + trap shutdown TERM INT
mystack_mysql.0.os1sqw2ldrk9@MYSTACKDB1    | + COUNT=1
mystack_mysql.0.os1sqw2ldrk9@MYSTACKDB1    | + '[' 1 -eq 1 ']'
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | Starting node, connecting to gcomm://
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | + galera-healthcheck -user=system -password=**************************************************************** -port=8080 -availWhenDonor=false -availWhenReadOnly=false -pidfile=/var/run/galera-healthcheck-1.pid
mystack_mysql.0.os1sqw2ldrk9@MYSTACKDB1    | + '[' 1 -lt 1 ']'
mystack_mysql.0.os1sqw2ldrk9@MYSTACKDB1    | + break
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | + galera-healthcheck -user=system -password=**************************************************************** -port=8081 -availWhenDonor=true -availWhenReadOnly=true -pidfile=/var/run/galera-healthcheck-2.pid
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | + wait 34
mystack_mysql.0.os1sqw2ldrk9@MYSTACKDB1    | + shift 2
mystack_mysql.0.os1sqw2ldrk9@MYSTACKDB1    | + echo 'Starting node, connecting to gcomm://'
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | + gosu mysql mysqld.sh --console --wsrep-on=ON --wsrep_cluster_name=xcluster --wsrep_cluster_address=gcomm:// --wsrep_node_address=10.0.0.12:4567 --wsrep_sst_auth=xtrabackup:**************** --default-time-zone=+00:00
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | + [[ --console --wsrep-on=ON --wsrep_cluster_name=xcluster --wsrep_cluster_address=gcomm:// --wsrep_node_address=10.0.0.12:4567 --wsrep_sst_auth=xtrabackup:**************** --default-time-zone=+00:00 =~ /--wsrep-new-cluster/ ]]
mystack_mysql.0.os1sqw2ldrk9@MYSTACKDB1    | + set +e -m
mystack_mysql.0.os1sqw2ldrk9@MYSTACKDB1    | + trap shutdown TERM INT
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | + test -f /var/lib/mysql/ibdata1
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | + POSITION=
mystack_mysql.0.os1sqw2ldrk9@MYSTACKDB1    | + galera-healthcheck -user=system -password=**************************************************************** -port=8080 -availWhenDonor=false -availWhenReadOnly=false -pidfile=/var/run/galera-healthcheck-1.pid
mystack_mysql.0.os1sqw2ldrk9@MYSTACKDB1    | + galera-healthcheck -user=system -password=**************************************************************** -port=8081 -availWhenDonor=true -availWhenReadOnly=true -pidfile=/var/run/galera-healthcheck-2.pid
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | + test -f /var/lib/mysql/grastate.dat
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | + grep -q seqno: /var/lib/mysql/grastate.dat
mystack_mysql.0.os1sqw2ldrk9@MYSTACKDB1    | + wait 35
mystack_mysql.0.os1sqw2ldrk9@MYSTACKDB1    | + gosu mysql mysqld.sh --console --wsrep-on=ON --wsrep_cluster_name=xcluster --wsrep_cluster_address=gcomm:// --wsrep_node_address=10.0.0.7:4567 --wsrep_sst_auth=xtrabackup:**************** --default-time-zone=+00:00
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | + grep -q 00000000-0000-0000-0000-000000000000 /var/lib/mysql/grastate.dat
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | ++ awk '/^uuid:/{print $2}' /var/lib/mysql/grastate.dat
mystack_mysql.0.os1sqw2ldrk9@MYSTACKDB1    | + [[ --console --wsrep-on=ON --wsrep_cluster_name=xcluster --wsrep_cluster_address=gcomm:// --wsrep_node_address=10.0.0.7:4567 --wsrep_sst_auth=xtrabackup:**************** --default-time-zone=+00:00 =~ /--wsrep-new-cluster/ ]]
mystack_mysql.0.os1sqw2ldrk9@MYSTACKDB1    | + test -f /var/lib/mysql/ibdata1
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | + uuid=0652cdaf-ea61-11e6-844e-cef75cf1b9b7
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | ++ awk '/^seqno:/{print $2}' /var/lib/mysql/grastate.dat
mystack_mysql.0.os1sqw2ldrk9@MYSTACKDB1    | + POSITION=
mystack_mysql.0.os1sqw2ldrk9@MYSTACKDB1    | + test -f /var/lib/mysql/grastate.dat
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | + seqno=238
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | + '[' 238 = -1 ']'
mystack_mysql.0.os1sqw2ldrk9@MYSTACKDB1    | + grep -q seqno: /var/lib/mysql/grastate.dat
mystack_mysql.0.os1sqw2ldrk9@MYSTACKDB1    | + grep -q 00000000-0000-0000-0000-000000000000 /var/lib/mysql/grastate.dat
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | + '[' -n 0652cdaf-ea61-11e6-844e-cef75cf1b9b7 ']'
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | + '[' -n 238 ']'
mystack_mysql.0.os1sqw2ldrk9@MYSTACKDB1    | ++ awk '/^uuid:/{print $2}' /var/lib/mysql/grastate.dat
mystack_mysql.0.os1sqw2ldrk9@MYSTACKDB1    | + uuid=0652cdaf-ea61-11e6-844e-cef75cf1b9b7
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | + POSITION=0652cdaf-ea61-11e6-844e-cef75cf1b9b7:238
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | + echo 'mysqld.sh: Recovered position from grastate.dat: 0652cdaf-ea61-11e6-844e-cef75cf1b9b7:238'
mystack_mysql.0.os1sqw2ldrk9@MYSTACKDB1    | ++ awk '/^seqno:/{print $2}' /var/lib/mysql/grastate.dat
mystack_mysql.0.os1sqw2ldrk9@MYSTACKDB1    | + seqno=238
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | mysqld.sh: Recovered position from grastate.dat: 0652cdaf-ea61-11e6-844e-cef75cf1b9b7:238
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | + [[ -z 0652cdaf-ea61-11e6-844e-cef75cf1b9b7:238 ]]
mystack_mysql.0.os1sqw2ldrk9@MYSTACKDB1    | + '[' 238 = -1 ']'
mystack_mysql.0.os1sqw2ldrk9@MYSTACKDB1    | + '[' -n 0652cdaf-ea61-11e6-844e-cef75cf1b9b7 ']'
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | ++ sed -E 's#.*--wsrep_node_address=([0-9\.]+):4567.*#\1#'
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | + NODE_ADDRESS=10.0.0.12
mystack_mysql.0.os1sqw2ldrk9@MYSTACKDB1    | + '[' -n 238 ']'
mystack_mysql.0.os1sqw2ldrk9@MYSTACKDB1    | + POSITION=0652cdaf-ea61-11e6-844e-cef75cf1b9b7:238
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | ++ sed -E 's#.*gcomm://([0-9\.,]+)\s+.*#\1#'
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | + GCOMM='--console --wsrep-on=ON --wsrep_cluster_name=xcluster --wsrep_cluster_address=gcomm:// --wsrep_node_address=10.0.0.12:4567 --wsrep_sst_auth=xtrabackup:**************** --default-time-zone=+00:00'
mystack_mysql.0.os1sqw2ldrk9@MYSTACKDB1    | + echo 'mysqld.sh: Recovered position from grastate.dat: 0652cdaf-ea61-11e6-844e-cef75cf1b9b7:238'
mystack_mysql.0.os1sqw2ldrk9@MYSTACKDB1    | mysqld.sh: Recovered position from grastate.dat: 0652cdaf-ea61-11e6-844e-cef75cf1b9b7:238
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | + [[ -f /var/lib/mysql/wsrep-new-cluster ]]
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | + [[ -z 0652cdaf-ea61-11e6-844e-cef75cf1b9b7:238 ]]
mystack_mysql.0.os1sqw2ldrk9@MYSTACKDB1    | + [[ -z 0652cdaf-ea61-11e6-844e-cef75cf1b9b7:238 ]]
mystack_mysql.0.os1sqw2ldrk9@MYSTACKDB1    | ++ sed -E 's#.*--wsrep_node_address=([0-9\.]+):4567.*#\1#'
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | + check_nodes --console --wsrep-on=ON --wsrep_cluster_name=xcluster --wsrep_cluster_address=gcomm:// --wsrep_node_address=10.0.0.12:4567 --wsrep_sst_auth=xtrabackup:**************** --default-time-zone=+00:00 10.0.0.12
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | + for node in '${1//,/ }'
mystack_mysql.0.os1sqw2ldrk9@MYSTACKDB1    | + NODE_ADDRESS=10.0.0.7
mystack_mysql.0.os1sqw2ldrk9@MYSTACKDB1    | ++ sed -E 's#.*gcomm://([0-9\.,]+)\s+.*#\1#'
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | + '[' --console = --wsrep-on=ON ']'
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | + curl -f -s -o - http://--console:8081
mystack_mysql.0.os1sqw2ldrk9@MYSTACKDB1    | + GCOMM='--console --wsrep-on=ON --wsrep_cluster_name=xcluster --wsrep_cluster_address=gcomm:// --wsrep_node_address=10.0.0.7:4567 --wsrep_sst_auth=xtrabackup:**************** --default-time-zone=+00:00'
mystack_mysql.0.os1sqw2ldrk9@MYSTACKDB1    | + [[ -f /var/lib/mysql/wsrep-new-cluster ]]
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | + return 1
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | + LISTEN_PORT=3309
mystack_mysql.0.os1sqw2ldrk9@MYSTACKDB1    | + [[ -z 0652cdaf-ea61-11e6-844e-cef75cf1b9b7:238 ]]
mystack_mysql.0.os1sqw2ldrk9@MYSTACKDB1    | + check_nodes --console --wsrep-on=ON --wsrep_cluster_name=xcluster --wsrep_cluster_address=gcomm:// --wsrep_node_address=10.0.0.7:4567 --wsrep_sst_auth=xtrabackup:**************** --default-time-zone=+00:00 10.0.0.7
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | + EXPECT_NODES=3
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | + [[ -f /var/lib/mysql/gvwstate.dat ]]
mystack_mysql.0.os1sqw2ldrk9@MYSTACKDB1    | + for node in '${1//,/ }'
mystack_mysql.0.os1sqw2ldrk9@MYSTACKDB1    | + '[' --console = --wsrep-on=ON ']'
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | ++ wc -w
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | + [[ 7 -gt 3 ]]
mystack_mysql.0.os1sqw2ldrk9@MYSTACKDB1    | + curl -f -s -o - http://--console:8081
mystack_mysql.0.os1sqw2ldrk9@MYSTACKDB1    | + return 1
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | ++ wc -w
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | + EXPECT_NODES=7
mystack_mysql.0.os1sqw2ldrk9@MYSTACKDB1    | + LISTEN_PORT=3309
mystack_mysql.0.os1sqw2ldrk9@MYSTACKDB1    | + EXPECT_NODES=3
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | + EXPECT_NODES=6
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | + echo 'mysqld.sh: Collecting grastate.dat and gvwstate.dat info from other nodes...'
mystack_mysql.0.os1sqw2ldrk9@MYSTACKDB1    | + [[ -f /var/lib/mysql/gvwstate.dat ]]
mystack_mysql.0.os1sqw2ldrk9@MYSTACKDB1    | ++ wc -w
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | mysqld.sh: Collecting grastate.dat and gvwstate.dat info from other nodes...
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | + set -m
mystack_mysql.0.os1sqw2ldrk9@MYSTACKDB1    | + [[ 7 -gt 3 ]]
mystack_mysql.0.os1sqw2ldrk9@MYSTACKDB1    | ++ wc -w
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | ++ mktemp -t socat.XXXX
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | + tmpfile=/tmp/socat.VzbR
mystack_mysql.0.os1sqw2ldrk9@MYSTACKDB1    | + EXPECT_NODES=7
mystack_mysql.0.os1sqw2ldrk9@MYSTACKDB1    | + EXPECT_NODES=6
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | + PID_SERVER=56
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | + SENT_NODES=
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | + '[' -z ']'
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | + '[' -f ']'
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | + '[' -f ']'
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | ++ cat
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | ++ cat
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | + MYSQL_PASSWORD=
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | + MYSQL_PASSWORD=
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | + test -n ****************
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | + test -n ****************
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | ++ echo ****************
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | ++ echo ****************
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | ++ sha256sum
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | ++ sha256sum
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | ++ awk '{print $1;}'
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | ++ awk '{print $1;}'
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | + SYSTEM_PASSWORD=****************************************************************
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | + SYSTEM_PASSWORD=****************************************************************
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | + case "$1" in
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | + '[' 1 -lt 1 ']'
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | + break
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | + shift 2
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | + echo 'Starting node, connecting to gcomm://'
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | + set +e -m
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | + trap shutdown TERM INT
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | + galera-healthcheck -user=system -password=**************************************************************** -port=8080 -availWhenDonor=false -availWhenReadOnly=false -pidfile=/var/run/galera-healthcheck-1.pid
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | + galera-healthcheck -user=system -password=**************************************************************** -port=8081 -availWhenDonor=true -availWhenReadOnly=true -pidfile=/var/run/galera-healthcheck-2.pid
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | + wait 35
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | + gosu mysql mysqld.sh --console --wsrep-on=ON --wsrep_cluster_name=xcluster --wsrep_cluster_address=gcomm:// --wsrep_node_address=10.0.0.9:4567 --wsrep_sst_auth=xtrabackup:**************** --default-time-zone=+00:00
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | + [[ --console --wsrep-on=ON --wsrep_cluster_name=xcluster --wsrep_cluster_address=gcomm:// --wsrep_node_address=10.0.0.9:4567 --wsrep_sst_auth=xtrabackup:**************** --default-time-zone=+00:00 =~ /--wsrep-new-cluster/ ]]
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | + test -f /var/lib/mysql/ibdata1
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | + POSITION=
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | + test -f /var/lib/mysql/grastate.dat
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | + grep -q seqno: /var/lib/mysql/grastate.dat
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | + grep -q 00000000-0000-0000-0000-000000000000 /var/lib/mysql/grastate.dat
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | ++ awk '/^uuid:/{print $2}' /var/lib/mysql/grastate.dat
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | + uuid=0652cdaf-ea61-11e6-844e-cef75cf1b9b7
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | ++ awk '/^seqno:/{print $2}' /var/lib/mysql/grastate.dat
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | + seqno=238
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | + '[' 238 = -1 ']'
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | + '[' -n 0652cdaf-ea61-11e6-844e-cef75cf1b9b7 ']'
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | + '[' -n 238 ']'
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | + POSITION=0652cdaf-ea61-11e6-844e-cef75cf1b9b7:238
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | + echo 'mysqld.sh: Recovered position from grastate.dat: 0652cdaf-ea61-11e6-844e-cef75cf1b9b7:238'
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | mysqld.sh: Recovered position from grastate.dat: 0652cdaf-ea61-11e6-844e-cef75cf1b9b7:238
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | + [[ -z 0652cdaf-ea61-11e6-844e-cef75cf1b9b7:238 ]]
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | ++ sed -E 's#.*--wsrep_node_address=([0-9\.]+):4567.*#\1#'
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | + NODE_ADDRESS=10.0.0.9
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | ++ sed -E 's#.*gcomm://([0-9\.,]+)\s+.*#\1#'
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | + GCOMM='--console --wsrep-on=ON --wsrep_cluster_name=xcluster --wsrep_cluster_address=gcomm:// --wsrep_node_address=10.0.0.9:4567 --wsrep_sst_auth=xtrabackup:**************** --default-time-zone=+00:00'
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | + [[ -f /var/lib/mysql/wsrep-new-cluster ]]
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | + [[ -z 0652cdaf-ea61-11e6-844e-cef75cf1b9b7:238 ]]
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | + check_nodes --console --wsrep-on=ON --wsrep_cluster_name=xcluster --wsrep_cluster_address=gcomm:// --wsrep_node_address=10.0.0.9:4567 --wsrep_sst_auth=xtrabackup:**************** --default-time-zone=+00:00 10.0.0.9
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | + for node in '${1//,/ }'
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | + '[' --console = --wsrep-on=ON ']'
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | + curl -f -s -o - http://--console:8081
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | + return 1
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | + LISTEN_PORT=3309
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | + EXPECT_NODES=3
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | + [[ -f /var/lib/mysql/gvwstate.dat ]]
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | ++ awk '/^view_id:/{print $2 " " $3 " " $4}'
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | + VIEW_ID='3 104d5fd4-ebcc-11e6-a17f-337f5a7c90f7 30'
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | ++ grep '^member:'
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | ++ wc -l
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | + GVW_MEMBERS=5
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | + echo 'mysqld.sh: Found view from gvwstate.dat with (5) members: 3 104d5fd4-ebcc-11e6-a17f-337f5a7c90f7 30'
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | mysqld.sh: Found view from gvwstate.dat with (5) members: 3 104d5fd4-ebcc-11e6-a17f-337f5a7c90f7 30
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | + [[ 5 -gt 3 ]]
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | + EXPECT_NODES=5
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | ++ wc -w
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | + [[ 7 -gt 5 ]]
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | ++ wc -w
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | + EXPECT_NODES=7
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | + EXPECT_NODES=6
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | + echo 'mysqld.sh: Collecting grastate.dat and gvwstate.dat info from other nodes...'
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | mysqld.sh: Collecting grastate.dat and gvwstate.dat info from other nodes...
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | + set -m
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | ++ mktemp -t socat.XXXX
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | + tmpfile=/tmp/socat.Kbp3
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | + PID_SERVER=64
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | + SENT_NODES=
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | + for i in '{36..0}'
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | + socat -u TCP-LISTEN:3309,bind=10.0.0.9,fork OPEN:/tmp/socat.Kbp3,append
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | + for node in '${GCOMM//,/ }'
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | + [[ --console = 10.0.0.9 ]]
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | + socat - TCP:--console:3309
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | + case "$1" in
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | + '[' -z **************** ']'
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | + '[' -z **************** ']'
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | + '[' -z **************************************************************** ']'
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | + '[' -z **************************************************************** ']'
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | + CLUSTER_NAME=xcluster
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | + CLUSTER_NAME=xcluster
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | + GCOMM_MINIMUM=2
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | + GCOMM_MINIMUM=2
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | + GCOMM=
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | + GCOMM=
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | + MYSQL_MODE_ARGS=
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | + MYSQL_MODE_ARGS=
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | + '[' -f /usr/local/lib/startup.sh ']'
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | + '[' -f /usr/local/lib/startup.sh ']'
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | + '[' node = node ']'
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | + '[' node = node ']'
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | + '[' -f /var/lib/mysql/force-cluster-bootstrapping ']'
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | + '[' -f /var/lib/mysql/force-cluster-bootstrapping ']'
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | + '[' node = seed ']'
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | + '[' node = seed ']'
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | + case "$1" in
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | + case "$1" in
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | + MYSQL_MODE_ARGS+=' --wsrep-on=ON'
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | + MYSQL_MODE_ARGS+=' --wsrep-on=ON'
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | + '[' -z tasks.mystack_mysql ']'
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | + '[' -z tasks.mystack_mysql ']'
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | + ADDRS=tasks.mystack_mysql
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | + ADDRS=tasks.mystack_mysql
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | + RESOLVE=0
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | + RESOLVE=0
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | + SLEEPS=0
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | + SLEEPS=0
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | + true
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | + true
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | + SEP=
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | + SEP=
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | + GCOMM=
mystack_mysql.0.os1sqw2ldrk9@MYSTACKDB1    | + echo 'mysqld.sh: Collecting grastate.dat and gvwstate.dat info from other nodes...'
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | + for node in '${GCOMM//,/ }'
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | + socat -u TCP-LISTEN:3309,bind=10.0.0.8,fork OPEN:/tmp/socat.tCXU,append
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | + [[ --console = 10.0.0.8 ]]
mystack_mysql.0.x5x0lj1qxjeg@MYSTACKDB2    | + socat - TCP:--console:3309
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | + GCOMM=
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | + for ADDR in '${ADDRS//,/ }'
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | + for ADDR in '${ADDRS//,/ }'
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | + [[ tasks.mystack_mysql =~ ^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+$ ]]
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | + [[ tasks.mystack_mysql =~ ^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+$ ]]
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | + RESOLVE=1
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | + RESOLVE=1
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | ++ getent hosts tasks.mystack_mysql
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | ++ getent hosts tasks.mystack_mysql
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | ++ paste -sd ,
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | ++ awk '{ print $1 }'
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | ++ paste -sd ,
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | ++ awk '{ print $1 }'
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | + GCOMM+=
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | + GCOMM+=
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | + '[' -n '' ']'
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | + '[' -n '' ']'
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | + GCOMM=
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | + GCOMM=
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | ++ echo ''
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | ++ echo ''
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | ++ sed 's/,\+/,/g'
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | ++ sed 's/,\+/,/g'
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | + GCOMM=
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | + GCOMM=
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | ++ echo ''
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | ++ echo ''
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | ++ sort -u
mystack_mysql.0.os1sqw2ldrk9@MYSTACKDB1    | mysqld.sh: Collecting grastate.dat and gvwstate.dat info from other nodes...
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | + for i in '{36..0}'
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | + for node in '${GCOMM//,/ }'
mystack_mysql.0.os1sqw2ldrk9@MYSTACKDB1    | + set -m
mystack_mysql.0.os1sqw2ldrk9@MYSTACKDB1    | ++ mktemp -t socat.XXXX
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | + [[ --console = 10.0.0.12 ]]
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | + socat - TCP:--console:3309
mystack_mysql.0.os1sqw2ldrk9@MYSTACKDB1    | + tmpfile=/tmp/socat.fbCD
mystack_mysql.0.os1sqw2ldrk9@MYSTACKDB1    | + PID_SERVER=57
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | + socat -u TCP-LISTEN:3309,bind=10.0.0.12,fork OPEN:/tmp/socat.VzbR,append
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | ++ shutdown
mystack_mysql.0.os1sqw2ldrk9@MYSTACKDB1    | + SENT_NODES=
mystack_mysql.0.os1sqw2ldrk9@MYSTACKDB1    | + for i in '{36..0}'
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | ++ echo 'Received TERM|INT signal. Shutting down...'
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | Received TERM|INT signal. Shutting down...
mystack_mysql.0.os1sqw2ldrk9@MYSTACKDB1    | + socat -u TCP-LISTEN:3309,bind=10.0.0.7,fork OPEN:/tmp/socat.fbCD,append
mystack_mysql.0.os1sqw2ldrk9@MYSTACKDB1    | + for node in '${GCOMM//,/ }'
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | ++ mysql -u system -h 127.0.0.1 -p**************************************************************** -e SHUTDOWN
mystack_mysql.0.os1sqw2ldrk9@MYSTACKDB1    | + [[ --console = 10.0.0.7 ]]
mystack_mysql.0.os1sqw2ldrk9@MYSTACKDB1    | + socat - TCP:--console:3309
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111 "Connection refused")
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | start.sh: Trapped error on line 217
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | +++ err_report 217
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | +++ echo 'start.sh: Trapped error on line 217'
mystack_mysql.0.qq59sjw5xgx7@MYSTACKDB1    | +++ exit
mystack_mysql.0.6xpzfvwpo1bd@MYSTACKAPP1    | Got NODE_ADDRESS=10.0.0.7
mystack_mysql.0.s9k3b4oxqi5w@MYSTACKAPP1    | Got NODE_ADDRESS=10.0.0.7
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | ++ tr , '\n'
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | ++ wc -l
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | ++ tr , '\n'
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | ++ grep -v -e '^10.0.0.5$'
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | ++ grep -v -e '^10.0.0.10$'
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | ++ sort -u
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | ++ wc -l
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | + COUNT=1
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | + COUNT=1
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | + '[' 1 -eq 1 ']'
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | + '[' 1 -eq 1 ']'
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | + '[' 1 -lt 1 ']'
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | + '[' 1 -lt 1 ']'
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | + break
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | + break
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | + shift 2
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | + shift 2
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | + echo 'Starting node, connecting to gcomm://'
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | + echo 'Starting node, connecting to gcomm://'
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | + set +e -m
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | + set +e -m
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | + trap shutdown TERM INT
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | + trap shutdown TERM INT
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | + galera-healthcheck -user=system -password=**************************************************************** -port=8080 -availWhenDonor=false -availWhenReadOnly=false -pidfile=/var/run/galera-healthcheck-1.pid
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | + galera-healthcheck -user=system -password=**************************************************************** -port=8080 -availWhenDonor=false -availWhenReadOnly=false -pidfile=/var/run/galera-healthcheck-1.pid
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | + wait 33
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | + wait 33
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | + galera-healthcheck -user=system -password=**************************************************************** -port=8081 -availWhenDonor=true -availWhenReadOnly=true -pidfile=/var/run/galera-healthcheck-2.pid
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | + gosu mysql mysqld.sh --console --wsrep-on=ON --wsrep_cluster_name=xcluster --wsrep_cluster_address=gcomm:// --wsrep_node_address=10.0.0.10:4567 --wsrep_sst_auth=xtrabackup:**************** --default-time-zone=+00:00
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | + galera-healthcheck -user=system -password=**************************************************************** -port=8081 -availWhenDonor=true -availWhenReadOnly=true -pidfile=/var/run/galera-healthcheck-2.pid
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | + gosu mysql mysqld.sh --console --wsrep-on=ON --wsrep_cluster_name=xcluster --wsrep_cluster_address=gcomm:// --wsrep_node_address=10.0.0.5:4567 --wsrep_sst_auth=xtrabackup:**************** --default-time-zone=+00:00
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | Starting node, connecting to gcomm://
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | Starting node, connecting to gcomm://
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | + [[ --console --wsrep-on=ON --wsrep_cluster_name=xcluster --wsrep_cluster_address=gcomm:// --wsrep_node_address=10.0.0.10:4567 --wsrep_sst_auth=xtrabackup:**************** --default-time-zone=+00:00 =~ /--wsrep-new-cluster/ ]]
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | + [[ --console --wsrep-on=ON --wsrep_cluster_name=xcluster --wsrep_cluster_address=gcomm:// --wsrep_node_address=10.0.0.5:4567 --wsrep_sst_auth=xtrabackup:**************** --default-time-zone=+00:00 =~ /--wsrep-new-cluster/ ]]
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | + test -f /var/lib/mysql/ibdata1
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | + test -f /var/lib/mysql/ibdata1
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | + POSITION=
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | + POSITION=
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | + test -f /var/lib/mysql/grastate.dat
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | + test -f /var/lib/mysql/grastate.dat
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | + grep -q seqno: /var/lib/mysql/grastate.dat
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | + grep -q seqno: /var/lib/mysql/grastate.dat
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | + grep -q 00000000-0000-0000-0000-000000000000 /var/lib/mysql/grastate.dat
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | + grep -q 00000000-0000-0000-0000-000000000000 /var/lib/mysql/grastate.dat
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | ++ awk '/^uuid:/{print $2}' /var/lib/mysql/grastate.dat
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | ++ awk '/^uuid:/{print $2}' /var/lib/mysql/grastate.dat
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | + uuid=0652cdaf-ea61-11e6-844e-cef75cf1b9b7
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | + uuid=0652cdaf-ea61-11e6-844e-cef75cf1b9b7
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | ++ awk '/^seqno:/{print $2}' /var/lib/mysql/grastate.dat
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | ++ awk '/^seqno:/{print $2}' /var/lib/mysql/grastate.dat
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | + seqno=238
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | + seqno=238
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | + '[' 238 = -1 ']'
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | + '[' 238 = -1 ']'
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | + '[' -n 0652cdaf-ea61-11e6-844e-cef75cf1b9b7 ']'
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | + '[' -n 0652cdaf-ea61-11e6-844e-cef75cf1b9b7 ']'
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | + '[' -n 238 ']'
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | + '[' -n 238 ']'
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | + POSITION=0652cdaf-ea61-11e6-844e-cef75cf1b9b7:238
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | + POSITION=0652cdaf-ea61-11e6-844e-cef75cf1b9b7:238
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | + echo 'mysqld.sh: Recovered position from grastate.dat: 0652cdaf-ea61-11e6-844e-cef75cf1b9b7:238'
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | mysqld.sh: Recovered position from grastate.dat: 0652cdaf-ea61-11e6-844e-cef75cf1b9b7:238
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | + echo 'mysqld.sh: Recovered position from grastate.dat: 0652cdaf-ea61-11e6-844e-cef75cf1b9b7:238'
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | mysqld.sh: Recovered position from grastate.dat: 0652cdaf-ea61-11e6-844e-cef75cf1b9b7:238
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | + [[ -z 0652cdaf-ea61-11e6-844e-cef75cf1b9b7:238 ]]
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | ++ sed -E 's#.*--wsrep_node_address=([0-9\.]+):4567.*#\1#'
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | + [[ -z 0652cdaf-ea61-11e6-844e-cef75cf1b9b7:238 ]]
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | ++ sed -E 's#.*--wsrep_node_address=([0-9\.]+):4567.*#\1#'
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | + NODE_ADDRESS=10.0.0.5
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | ++ sed -E 's#.*gcomm://([0-9\.,]+)\s+.*#\1#'
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | + NODE_ADDRESS=10.0.0.10
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | ++ sed -E 's#.*gcomm://([0-9\.,]+)\s+.*#\1#'
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | + GCOMM='--console --wsrep-on=ON --wsrep_cluster_name=xcluster --wsrep_cluster_address=gcomm:// --wsrep_node_address=10.0.0.5:4567 --wsrep_sst_auth=xtrabackup:**************** --default-time-zone=+00:00'
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | + [[ -f /var/lib/mysql/wsrep-new-cluster ]]
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | + GCOMM='--console --wsrep-on=ON --wsrep_cluster_name=xcluster --wsrep_cluster_address=gcomm:// --wsrep_node_address=10.0.0.10:4567 --wsrep_sst_auth=xtrabackup:**************** --default-time-zone=+00:00'
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | + [[ -f /var/lib/mysql/wsrep-new-cluster ]]
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | + [[ -z 0652cdaf-ea61-11e6-844e-cef75cf1b9b7:238 ]]
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | + check_nodes --console --wsrep-on=ON --wsrep_cluster_name=xcluster --wsrep_cluster_address=gcomm:// --wsrep_node_address=10.0.0.5:4567 --wsrep_sst_auth=xtrabackup:**************** --default-time-zone=+00:00 10.0.0.5
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | + [[ -z 0652cdaf-ea61-11e6-844e-cef75cf1b9b7:238 ]]
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | + check_nodes --console --wsrep-on=ON --wsrep_cluster_name=xcluster --wsrep_cluster_address=gcomm:// --wsrep_node_address=10.0.0.10:4567 --wsrep_sst_auth=xtrabackup:**************** --default-time-zone=+00:00 10.0.0.10
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | + for node in '${1//,/ }'
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | + '[' --console = --wsrep-on=ON ']'
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | + for node in '${1//,/ }'
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | + '[' --console = --wsrep-on=ON ']'
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | + curl -f -s -o - http://--console:8081
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | + return 1
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | + curl -f -s -o - http://--console:8081
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | + return 1
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | + LISTEN_PORT=3309
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | + EXPECT_NODES=3
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | + LISTEN_PORT=3309
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | + EXPECT_NODES=3
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | + [[ -f /var/lib/mysql/gvwstate.dat ]]
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | ++ wc -w
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | + [[ -f /var/lib/mysql/gvwstate.dat ]]
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | ++ wc -w
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | + [[ 7 -gt 3 ]]
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | ++ wc -w
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | + [[ 7 -gt 3 ]]
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | ++ wc -w
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | + EXPECT_NODES=7
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | + EXPECT_NODES=6
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | + EXPECT_NODES=7
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | + EXPECT_NODES=6
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | + echo 'mysqld.sh: Collecting grastate.dat and gvwstate.dat info from other nodes...'
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | mysqld.sh: Collecting grastate.dat and gvwstate.dat info from other nodes...
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | + echo 'mysqld.sh: Collecting grastate.dat and gvwstate.dat info from other nodes...'
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | mysqld.sh: Collecting grastate.dat and gvwstate.dat info from other nodes...
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | + set -m
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | ++ mktemp -t socat.XXXX
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | + set -m
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | ++ mktemp -t socat.XXXX
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | + tmpfile=/tmp/socat.3rv5
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | + PID_SERVER=54
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | + tmpfile=/tmp/socat.y0dD
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | + PID_SERVER=55
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | + SENT_NODES=
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | + for i in '{36..0}'
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | + SENT_NODES=
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | + for i in '{36..0}'
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | + for node in '${GCOMM//,/ }'
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | + [[ --console = 10.0.0.5 ]]
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | + for node in '${GCOMM//,/ }'
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | + socat -u TCP-LISTEN:3309,bind=10.0.0.10,fork OPEN:/tmp/socat.y0dD,append
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | + socat - TCP:--console:3309
mystack_mysql.0.s1v0xuod6ip3@MYSTACKAPP2    | + socat -u TCP-LISTEN:3309,bind=10.0.0.5,fork OPEN:/tmp/socat.3rv5,append
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | + [[ --console = 10.0.0.10 ]]
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | + socat - TCP:--console:3309
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | Received TERM|INT signal. Shutting down...
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | ++ shutdown
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | ++ echo 'Received TERM|INT signal. Shutting down...'
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | ++ mysql -u system -h 127.0.0.1 -p**************************************************************** -e SHUTDOWN
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111 "Connection refused")
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | +++ err_report 217
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | +++ echo 'start.sh: Trapped error on line 217'
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | +++ exit
mystack_mysql.0.q0gv942ytsh1@MYSTACKAPP2    | start.sh: Trapped error on line 217
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | + chown mysql /var/lib/mysql
mystack_mysql.0.6xpzfvwpo1bd@MYSTACKAPP1    | sleep|no-galera|bash|seed|node <othernode>,...
mystack_mysql.0.s9k3b4oxqi5w@MYSTACKAPP1    | sleep|no-galera|bash|seed|node <othernode>,...
mystack_mysql.0.zhk460d4fk71@MYSTACKAPP1    | ---===--- MariaDB Galera Start Script ---===---
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | + chown mysql /var/lib/mysql
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | + '[' -z eth2 ']'
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | + '[' -z eth2 ']'
mystack_mysql.0.zhk460d4fk71@MYSTACKAPP1    | Got NODE_ADDRESS=10.0.0.7
mystack_mysql.0.zhk460d4fk71@MYSTACKAPP1    | sleep|no-galera|bash|seed|node <othernode>,...
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | + '[' -z eth2 ']'
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | + '[' -z eth2 ']'
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | + [[ eth2 =~ [a-zA-Z][a-zA-Z0-9:]+ ]]
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | + [[ eth2 =~ [a-zA-Z][a-zA-Z0-9:]+ ]]
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | ++ awk '/inet/ && / eth2$/{sub(/\/.*$/,"",$2); print $2}'
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | ++ ip addr
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | ++ awk '/inet/ && / eth2$/{sub(/\/.*$/,"",$2); print $2}'
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | ++ ip addr
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | ++ head -n 1
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | ++ head -n 1
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | + NODE_ADDRESS=10.0.0.11
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | + NODE_ADDRESS=10.0.0.6
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | + [[ 10.0.0.6 =~ ^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+$ ]]
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | + [[ 10.0.0.11 =~ ^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+$ ]]
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | + echo '---===--- MariaDB Galera Start Script ---===---'
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | + echo '---===--- MariaDB Galera Start Script ---===---'
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | + echo 'Got NODE_ADDRESS=10.0.0.6'
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | + echo 'Got NODE_ADDRESS=10.0.0.11'
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | + '[' -z **************** ']'
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | + '[' -z **************** ']'
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | + '[' -z ']'
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | + '[' -z ']'
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | + '[' -f ']'
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | + '[' -f ']'
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | ++ cat
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | ---===--- MariaDB Galera Start Script ---===---
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | Got NODE_ADDRESS=10.0.0.11
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | ---===--- MariaDB Galera Start Script ---===---
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | Got NODE_ADDRESS=10.0.0.6
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | ++ cat
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | + SYSTEM_PASSWORD=
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | + SYSTEM_PASSWORD=
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | + '[' -z *************** ']'
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | + '[' -z *************** ']'
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | + '[' -z ']'
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | + '[' -z ']'
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | + '[' -f ']'
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | + '[' -f ']'
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | ++ cat
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | ++ cat
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | + MYSQL_PASSWORD=
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | + MYSQL_PASSWORD=
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | + test -n ****************
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | + test -n ****************
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | ++ echo ****************
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | ++ echo ****************
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | ++ sha256sum
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | ++ sha256sum
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | ++ awk '{print $1;}'
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | ++ awk '{print $1;}'
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | + SYSTEM_PASSWORD=****************************************************************
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | + SYSTEM_PASSWORD=****************************************************************
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | + case "$1" in
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | + case "$1" in
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | + '[' -z **************** ']'
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | + '[' -z **************** ']'
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | + '[' -z **************************************************************** ']'
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | + '[' -z **************************************************************** ']'
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | + CLUSTER_NAME=xcluster
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | + CLUSTER_NAME=xcluster
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | + GCOMM_MINIMUM=2
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | + GCOMM_MINIMUM=2
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | + GCOMM=
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | + GCOMM=
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | + MYSQL_MODE_ARGS=
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | + MYSQL_MODE_ARGS=
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | + '[' -f /usr/local/lib/startup.sh ']'
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | + '[' -f /usr/local/lib/startup.sh ']'
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | + '[' node = node ']'
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | + '[' node = node ']'
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | + '[' -f /var/lib/mysql/force-cluster-bootstrapping ']'
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | + '[' -f /var/lib/mysql/force-cluster-bootstrapping ']'
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | + '[' node = seed ']'
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | + '[' node = seed ']'
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | + case "$1" in
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | + case "$1" in
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | + MYSQL_MODE_ARGS+=' --wsrep-on=ON'
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | + MYSQL_MODE_ARGS+=' --wsrep-on=ON'
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | + '[' -z tasks.mystack_mysql ']'
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | + '[' -z tasks.mystack_mysql ']'
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | + ADDRS=tasks.mystack_mysql
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | + ADDRS=tasks.mystack_mysql
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | + RESOLVE=0
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | + RESOLVE=0
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | + SLEEPS=0
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | + SLEEPS=0
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | + true
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | + true
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | + SEP=
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | + SEP=
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | + GCOMM=
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | + GCOMM=
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | + for ADDR in '${ADDRS//,/ }'
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | + for ADDR in '${ADDRS//,/ }'
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | + [[ tasks.mystack_mysql =~ ^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+$ ]]
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | + [[ tasks.mystack_mysql =~ ^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+$ ]]
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | + RESOLVE=1
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | + RESOLVE=1
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | ++ getent hosts tasks.mystack_mysql
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | ++ getent hosts tasks.mystack_mysql
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | ++ awk '{ print $1 }'
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | ++ awk '{ print $1 }'
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | ++ paste -sd ,
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | ++ paste -sd ,
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | + GCOMM+=
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | + GCOMM+=
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | + '[' -n '' ']'
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | + '[' -n '' ']'
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | + GCOMM=
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | + GCOMM=
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | ++ sed 's/,\+/,/g'
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | ++ echo ''
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | ++ sed 's/,\+/,/g'
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | ++ echo ''
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | + GCOMM=
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | + GCOMM=
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | ++ echo ''
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | ++ echo ''
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | ++ tr , '\n'
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | ++ grep -v -e '^10.0.0.6$'
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | ++ tr , '\n'
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | ++ grep -v -e '^10.0.0.11$'
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | ++ sort -u
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | ++ wc -l
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | ++ sort -u
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | ++ wc -l
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | + COUNT=1
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | + COUNT=1
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | + '[' 1 -eq 1 ']'
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | + '[' 1 -eq 1 ']'
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | + '[' 1 -lt 1 ']'
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | + '[' 1 -lt 1 ']'
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | + break
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | + break
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | + shift 2
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | + shift 2
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | + echo 'Starting node, connecting to gcomm://'
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | + echo 'Starting node, connecting to gcomm://'
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | + set +e -m
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | + set +e -m
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | + trap shutdown TERM INT
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | + trap shutdown TERM INT
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | Starting node, connecting to gcomm://
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | Starting node, connecting to gcomm://
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | + galera-healthcheck -user=system -password=**************************************************************** -port=8080 -availWhenDonor=false -availWhenReadOnly=false -pidfile=/var/run/galera-healthcheck-1.pid
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | + galera-healthcheck -user=system -password=**************************************************************** -port=8080 -availWhenDonor=false -availWhenReadOnly=false -pidfile=/var/run/galera-healthcheck-1.pid
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | + wait 33
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | + wait 33
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | + gosu mysql mysqld.sh --console --wsrep-on=ON --wsrep_cluster_name=xcluster --wsrep_cluster_address=gcomm:// --wsrep_node_address=10.0.0.6:4567 --wsrep_sst_auth=xtrabackup:**************** --default-time-zone=+00:00
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | + gosu mysql mysqld.sh --console --wsrep-on=ON --wsrep_cluster_name=xcluster --wsrep_cluster_address=gcomm:// --wsrep_node_address=10.0.0.11:4567 --wsrep_sst_auth=xtrabackup:**************** --default-time-zone=+00:00
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | + galera-healthcheck -user=system -password=**************************************************************** -port=8081 -availWhenDonor=true -availWhenReadOnly=true -pidfile=/var/run/galera-healthcheck-2.pid
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | + galera-healthcheck -user=system -password=**************************************************************** -port=8081 -availWhenDonor=true -availWhenReadOnly=true -pidfile=/var/run/galera-healthcheck-2.pid
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | + [[ --console --wsrep-on=ON --wsrep_cluster_name=xcluster --wsrep_cluster_address=gcomm:// --wsrep_node_address=10.0.0.6:4567 --wsrep_sst_auth=xtrabackup:**************** --default-time-zone=+00:00 =~ /--wsrep-new-cluster/ ]]
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | + [[ --console --wsrep-on=ON --wsrep_cluster_name=xcluster --wsrep_cluster_address=gcomm:// --wsrep_node_address=10.0.0.11:4567 --wsrep_sst_auth=xtrabackup:**************** --default-time-zone=+00:00 =~ /--wsrep-new-cluster/ ]]
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | + test -f /var/lib/mysql/ibdata1
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | + test -f /var/lib/mysql/ibdata1
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | + POSITION=
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | + POSITION=
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | + test -f /var/lib/mysql/grastate.dat
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | + test -f /var/lib/mysql/grastate.dat
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | + grep -q seqno: /var/lib/mysql/grastate.dat
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | + grep -q seqno: /var/lib/mysql/grastate.dat
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | + grep -q 00000000-0000-0000-0000-000000000000 /var/lib/mysql/grastate.dat
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | + grep -q 00000000-0000-0000-0000-000000000000 /var/lib/mysql/grastate.dat
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | ++ awk '/^uuid:/{print $2}' /var/lib/mysql/grastate.dat
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | ++ awk '/^uuid:/{print $2}' /var/lib/mysql/grastate.dat
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | + uuid=0652cdaf-ea61-11e6-844e-cef75cf1b9b7
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | + uuid=0652cdaf-ea61-11e6-844e-cef75cf1b9b7
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | ++ awk '/^seqno:/{print $2}' /var/lib/mysql/grastate.dat
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | ++ awk '/^seqno:/{print $2}' /var/lib/mysql/grastate.dat
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | + seqno=238
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | + seqno=238
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | + '[' 238 = -1 ']'
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | + '[' 238 = -1 ']'
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | + '[' -n 0652cdaf-ea61-11e6-844e-cef75cf1b9b7 ']'
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | + '[' -n 0652cdaf-ea61-11e6-844e-cef75cf1b9b7 ']'
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | + '[' -n 238 ']'
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | + '[' -n 238 ']'
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | + POSITION=0652cdaf-ea61-11e6-844e-cef75cf1b9b7:238
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | + POSITION=0652cdaf-ea61-11e6-844e-cef75cf1b9b7:238
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | + echo 'mysqld.sh: Recovered position from grastate.dat: 0652cdaf-ea61-11e6-844e-cef75cf1b9b7:238'
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | + echo 'mysqld.sh: Recovered position from grastate.dat: 0652cdaf-ea61-11e6-844e-cef75cf1b9b7:238'
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | mysqld.sh: Recovered position from grastate.dat: 0652cdaf-ea61-11e6-844e-cef75cf1b9b7:238
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | mysqld.sh: Recovered position from grastate.dat: 0652cdaf-ea61-11e6-844e-cef75cf1b9b7:238
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | + [[ -z 0652cdaf-ea61-11e6-844e-cef75cf1b9b7:238 ]]
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | + [[ -z 0652cdaf-ea61-11e6-844e-cef75cf1b9b7:238 ]]
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | ++ sed -E 's#.*--wsrep_node_address=([0-9\.]+):4567.*#\1#'
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | ++ sed -E 's#.*--wsrep_node_address=([0-9\.]+):4567.*#\1#'
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | + NODE_ADDRESS=10.0.0.6
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | + NODE_ADDRESS=10.0.0.11
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | ++ sed -E 's#.*gcomm://([0-9\.,]+)\s+.*#\1#'
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | ++ sed -E 's#.*gcomm://([0-9\.,]+)\s+.*#\1#'
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | + GCOMM='--console --wsrep-on=ON --wsrep_cluster_name=xcluster --wsrep_cluster_address=gcomm:// --wsrep_node_address=10.0.0.6:4567 --wsrep_sst_auth=xtrabackup:**************** --default-time-zone=+00:00'
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | + GCOMM='--console --wsrep-on=ON --wsrep_cluster_name=xcluster --wsrep_cluster_address=gcomm:// --wsrep_node_address=10.0.0.11:4567 --wsrep_sst_auth=xtrabackup:**************** --default-time-zone=+00:00'
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | + [[ -f /var/lib/mysql/wsrep-new-cluster ]]
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | + [[ -f /var/lib/mysql/wsrep-new-cluster ]]
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | + [[ -z 0652cdaf-ea61-11e6-844e-cef75cf1b9b7:238 ]]
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | + [[ -z 0652cdaf-ea61-11e6-844e-cef75cf1b9b7:238 ]]
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | + check_nodes --console --wsrep-on=ON --wsrep_cluster_name=xcluster --wsrep_cluster_address=gcomm:// --wsrep_node_address=10.0.0.11:4567 --wsrep_sst_auth=xtrabackup:**************** --default-time-zone=+00:00 10.0.0.11
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | + check_nodes --console --wsrep-on=ON --wsrep_cluster_name=xcluster --wsrep_cluster_address=gcomm:// --wsrep_node_address=10.0.0.6:4567 --wsrep_sst_auth=xtrabackup:**************** --default-time-zone=+00:00 10.0.0.6
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | + for node in '${1//,/ }'
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | + for node in '${1//,/ }'
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | + '[' --console = --wsrep-on=ON ']'
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | + '[' --console = --wsrep-on=ON ']'
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | + curl -f -s -o - http://--console:8081
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | + curl -f -s -o - http://--console:8081
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | + return 1
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | + return 1
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | + LISTEN_PORT=3309
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | + LISTEN_PORT=3309
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | + EXPECT_NODES=3
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | + EXPECT_NODES=3
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | + [[ -f /var/lib/mysql/gvwstate.dat ]]
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | + [[ -f /var/lib/mysql/gvwstate.dat ]]
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | ++ awk '/^view_id:/{print $2 " " $3 " " $4}'
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | ++ awk '/^view_id:/{print $2 " " $3 " " $4}'
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | + VIEW_ID='3 104d5fd4-ebcc-11e6-a17f-337f5a7c90f7 30'
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | + VIEW_ID='3 104d5fd4-ebcc-11e6-a17f-337f5a7c90f7 30'
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | ++ grep '^member:'
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | ++ grep '^member:'
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | ++ wc -l
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | ++ wc -l
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | + GVW_MEMBERS=5
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | + GVW_MEMBERS=5
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | + echo 'mysqld.sh: Found view from gvwstate.dat with (5) members: 3 104d5fd4-ebcc-11e6-a17f-337f5a7c90f7 30'
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | + echo 'mysqld.sh: Found view from gvwstate.dat with (5) members: 3 104d5fd4-ebcc-11e6-a17f-337f5a7c90f7 30'
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | mysqld.sh: Found view from gvwstate.dat with (5) members: 3 104d5fd4-ebcc-11e6-a17f-337f5a7c90f7 30
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | mysqld.sh: Found view from gvwstate.dat with (5) members: 3 104d5fd4-ebcc-11e6-a17f-337f5a7c90f7 30
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | + [[ 5 -gt 3 ]]
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | + [[ 5 -gt 3 ]]
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | + EXPECT_NODES=5
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | + EXPECT_NODES=5
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | ++ wc -w
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | ++ wc -w
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | + [[ 7 -gt 5 ]]
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | + [[ 7 -gt 5 ]]
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | ++ wc -w
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | ++ wc -w
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | + EXPECT_NODES=7
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | + EXPECT_NODES=7
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | + EXPECT_NODES=6
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | + EXPECT_NODES=6
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | + echo 'mysqld.sh: Collecting grastate.dat and gvwstate.dat info from other nodes...'
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | + echo 'mysqld.sh: Collecting grastate.dat and gvwstate.dat info from other nodes...'
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | mysqld.sh: Collecting grastate.dat and gvwstate.dat info from other nodes...
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | mysqld.sh: Collecting grastate.dat and gvwstate.dat info from other nodes...
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | + set -m
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | + set -m
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | ++ mktemp -t socat.XXXX
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | ++ mktemp -t socat.XXXX
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | + tmpfile=/tmp/socat.wZGA
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | + tmpfile=/tmp/socat.4NJH
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | + PID_SERVER=60
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | + PID_SERVER=60
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | + SENT_NODES=
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | + SENT_NODES=
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | + for i in '{36..0}'
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | + for i in '{36..0}'
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | + for node in '${GCOMM//,/ }'
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | + socat -u TCP-LISTEN:3309,bind=10.0.0.6,fork OPEN:/tmp/socat.4NJH,append
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | + for node in '${GCOMM//,/ }'
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | + socat -u TCP-LISTEN:3309,bind=10.0.0.11,fork OPEN:/tmp/socat.wZGA,append
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | + [[ --console = 10.0.0.11 ]]
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | + [[ --console = 10.0.0.6 ]]
mystack_mysql.0.uam9gxzh6c16@MYSTACKAPP1    | + socat - TCP:--console:3309
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | + socat - TCP:--console:3309
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | ++ shutdown
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | ++ echo 'Received TERM|INT signal. Shutting down...'
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | ++ mysql -u system -h 127.0.0.1 -p**************************************************************** -e SHUTDOWN
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | Received TERM|INT signal. Shutting down...
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | start.sh: Trapped error on line 217
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111 "Connection refused")
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | +++ err_report 217
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | +++ echo 'start.sh: Trapped error on line 217'
mystack_mysql.0.off2o33ujy67@MYSTACKAPP1    | +++ exit

colinmollenhour commented 7 years ago

Thanks for the logs. Please note to do blocks of code you use three ticks (```) on a separate line before and after (I've been editing your comments to make them readable). Also just a tip, sorting by container helps too:

$ awk '{print $1}' log.txt | sort -u | while read index; do grep -F "$index" log.txt; done

Now here is proof that your DNS is not resolving while the script runs:

mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | + for ADDR in '${ADDRS//,/ }'
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | + [[ tasks.mystack_mysql =~ ^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+$ ]]
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | + RESOLVE=1
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | ++ getent hosts tasks.mystack_mysql
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | ++ awk '{ print $1 }'
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | ++ paste -sd ,
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | + GCOMM+=
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | + '[' -n '' ']'
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | + GCOMM=
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | ++ echo ''
mystack_mysql.0.al5sowpt2dxv@MYSTACKDB3    | ++ sed 's/,\+/,/g'

However, the part after exposed a bug because COUNT was assigned 1 instead of 0, so I just fixed this by excluding blank lines from the count. You can give it a try with that fix but I think there is probably still another problem.

The nodes need to all be starting up simultaneously.. If there is some sort of "slow start" parameter in Docker Swarm it needs to be disabled or worked around. Unlike other containers that are stateless, the Galera nodes depend on communicating with each other during startup.

bartoszx commented 7 years ago

Hi

I updated code and built new image. Here you can find more logs https://dl.dropboxusercontent.com/u/7783158/log.txt

ystack_mysql.0.ndfzy8caasj0@MYSTACKDB3    | + '[' -n 0652cdaf-ea61-11e6-844e-cef75cf1b9b7 ']'
mystack_mysql.0.4m07zyvh6ebn@MYSTACKDB3    | + GCOMM='--console --wsrep-on=ON --wsrep_CLUSTER_NAME=xcluster --wsrep_cluster_address=gcomm:// --wsrep_node_address=10.0.0.9:4567 --wsrep_sst_auth=xtrabackup:*************** --default-time-zone=+00:00'
mystack_mysql.0.g17v6oe8uglb@MYSTACKDB3    | ++ awk '/^uuid:/{print $2}' /var/lib/mysql/grastate.dat
mystack_mysql.0.r2cxqjg4k02l@MYSTACKDB3    | + POSITION=0652cdaf-ea61-11e6-844e-cef75cf1b9b7:238
mystack_mysql.0.ndfzy8caasj0@MYSTACKDB3    | + '[' -n 238 ']'
mystack_mysql.0.g17v6oe8uglb@MYSTACKDB3    | + uuid=0652cdaf-ea61-11e6-844e-cef75cf1b9b7
mystack_mysql.0.sree7cwscci4@MYSTACKDB1    | ++ echo ''
mystack_mysql.0.sree7cwscci4@MYSTACKDB1    | ++ tr , '\n'
mystack_mysql.0.59ryujs2lkai@MYSTACKDB1    | + '[' 0 -lt 1 ']'
mystack_mysql.0.ifsf2la1fuuo@MYSTACKDB1    | + SLEEPS=15
mystack_mysql.0.4m07zyvh6ebn@MYSTACKDB3    | + [[ -f /var/lib/mysql/wsrep-new-cluster ]]
mystack_mysql.0.r2cxqjg4k02l@MYSTACKDB3    | + echo 'mysqld.sh: Recovered position from grastate.dat: 0652cdaf-ea61-11e6-844e-cef75cf1b9b7:238'
mystack_mysql.0.ndfzy8caasj0@MYSTACKDB3    | + POSITION=0652cdaf-ea61-11e6-844e-cef75cf1b9b7:238
mystack_mysql.0.ndfzy8caasj0@MYSTACKDB3    | + echo 'mysqld.sh: Recovered position from grastate.dat: 0652cdaf-ea61-11e6-844e-cef75cf1b9b7:238'
mystack_mysql.0.g17v6oe8uglb@MYSTACKDB3    | ++ awk '/^seqno:/{print $2}' /var/lib/mysql/grastate.dat
mystack_mysql.0.ifsf2la1fuuo@MYSTACKDB1    | + sleep 3
mystack_mysql.0.sree7cwscci4@MYSTACKDB1    | ++ sort -u
mystack_mysql.0.59ryujs2lkai@MYSTACKDB1    | + echo 'Waiting for at least 2 IP addresses to resolve...'
mystack_mysql.0.4m07zyvh6ebn@MYSTACKDB3    | + [[ -z 0652cdaf-ea61-11e6-844e-cef75cf1b9b7:238 ]]
mystack_mysql.0.r2cxqjg4k02l@MYSTACKDB3    | mysqld.sh: Recovered position from grastate.dat: 0652cdaf-ea61-11e6-844e-cef75cf1b9b7:238
mystack_mysql.0.4m07zyvh6ebn@MYSTACKDB3    | + check_nodes --console --wsrep-on=ON --wsrep_CLUSTER_NAME=xcluster --wsrep_cluster_address=gcomm:// --wsrep_node_address=10.0.0.9:4567 --wsrep_sst_auth=xtrabackup:*************** --default-time-zone=+00:00 10.0.0.9
mystack_mysql.0.g17v6oe8uglb@MYSTACKDB3    | + seqno=238
mystack_mysql.0.ndfzy8caasj0@MYSTACKDB3    | mysqld.sh: Recovered position from grastate.dat: 0652cdaf-ea61-11e6-844e-cef75cf1b9b7:238
mystack_mysql.0.bd9h8hrmovc0@MYSTACKDB1    | + [[ tasks.mystack_mysql =~ ^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+$ ]]
mystack_mysql.0.59ryujs2lkai@MYSTACKDB1    | + SLEEPS=15
mystack_mysql.0.ifsf2la1fuuo@MYSTACKDB1    | + '[' 15 -ge 30 ']'
mystack_mysql.0.sree7cwscci4@MYSTACKDB1    | ++ wc -l
mystack_mysql.0.sree7cwscci4@MYSTACKDB1    | ++ grep -v -e '^10.0.0.6$' -e '^$'
mystack_mysql.0.bd9h8hrmovc0@MYSTACKDB1    | + RESOLVE=1
mystack_mysql.0.ifsf2la1fuuo@MYSTACKDB1    | + true
mystack_mysql.0.sree7cwscci4@MYSTACKDB1    | Waiting for at least 2 IP addresses to resolve...
mystack_mysql.0.59ryujs2lkai@MYSTACKDB1    | + sleep 3
mystack_mysql.0.ifsf2la1fuuo@MYSTACKDB1    | + SEP=
mystack_mysql.0.bd9h8hrmovc0@MYSTACKDB1    | ++ getent hosts tasks.mystack_mysql
mystack_mysql.0.59ryujs2lkai@MYSTACKDB1    | Waiting for at least 2 IP addresses to resolve...
mystack_mysql.0.r2cxqjg4k02l@MYSTACKDB3    | + [[ -z 0652cdaf-ea61-11e6-844e-cef75cf1b9b7:238 ]]
mystack_mysql.0.4m07zyvh6ebn@MYSTACKDB3    | + for node in '${1//,/ }'
mystack_mysql.0.g17v6oe8uglb@MYSTACKDB3    | + '[' 238 = -1 ']'
mystack_mysql.0.ndfzy8caasj0@MYSTACKDB3    | + [[ -z 0652cdaf-ea61-11e6-844e-cef75cf1b9b7:238 ]]
mystack_mysql.0.r2cxqjg4k02l@MYSTACKDB3    | ++ sed -E 's#.*--wsrep_node_address=([0-9\.]+):4567.*#\1#'
mystack_mysql.0.r2cxqjg4k02l@MYSTACKDB3    | + NODE_ADDRESS=10.0.0.9
mystack_mysql.0.g17v6oe8uglb@MYSTACKDB3    | + '[' -n 0652cdaf-ea61-11e6-844e-cef75cf1b9b7 ']'
mystack_mysql.0.ndfzy8caasj0@MYSTACKDB3    | ++ sed -E 's#.*--wsrep_node_address=([0-9\.]+):4567.*#\1#'
mystack_mysql.0.4m07zyvh6ebn@MYSTACKDB3    | + '[' --console = --wsrep-on=ON ']'
mystack_mysql.0.4m07zyvh6ebn@MYSTACKDB3    | + curl -f -s -o - http://--console:8081
mystack_mysql.0.ifsf2la1fuuo@MYSTACKDB1    | + GCOMM=
mystack_mysql.0.sree7cwscci4@MYSTACKDB1    | + COUNT=0
mystack_mysql.0.ifsf2la1fuuo@MYSTACKDB1    | + for ADDR in '${ADDRS//,/ }'
mystack_mysql.0.bd9h8hrmovc0@MYSTACKDB1    | ++ awk '{ print $1 }'
mystack_mysql.0.59ryujs2lkai@MYSTACKDB1    | + '[' 15 -ge 30 ']'
mystack_mysql.0.sree7cwscci4@MYSTACKDB1    | + '[' 1 -eq 1 ']'
mystack_mysql.0.59ryujs2lkai@MYSTACKDB1    | + true
mystack_mysql.0.bd9h8hrmovc0@MYSTACKDB1    | ++ paste -sd ,
mystack_mysql.0.59ryujs2lkai@MYSTACKDB1    | + SEP=
mystack_mysql.0.sree7cwscci4@MYSTACKDB1    | + '[' 0 -lt 1 ']'
mystack_mysql.0.bd9h8hrmovc0@MYSTACKDB1    | + GCOMM+=
mystack_mysql.0.sree7cwscci4@MYSTACKDB1    | + echo 'Waiting for at least 2 IP addresses to resolve...'
mystack_mysql.0.bd9h8hrmovc0@MYSTACKDB1    | + '[' -n '' ']'
mystack_mysql.0.59ryujs2lkai@MYSTACKDB1    | + GCOMM=
mystack_mysql.0.sree7cwscci4@MYSTACKDB1    | + SLEEPS=15
mystack_mysql.0.bd9h8hrmovc0@MYSTACKDB1    | + GCOMM=
mystack_mysql.0.sree7cwscci4@MYSTACKDB1    | + sleep 3
mystack_mysql.0.g17v6oe8uglb@MYSTACKDB3    | + '[' -n 238 ']'
mystack_mysql.0.ndfzy8caasj0@MYSTACKDB3    | + NODE_ADDRESS=10.0.0.9
mystack_mysql.0.r2cxqjg4k02l@MYSTACKDB3    | ++ sed -E 's#.*gcomm://([0-9\.,]+)\s+.*#\1#'
mystack_mysql.0.ifsf2la1fuuo@MYSTACKDB1    | + [[ tasks.mystack_mysql =~ ^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+$ ]]
mystack_mysql.0.4m07zyvh6ebn@MYSTACKDB3    | + return 1
mystack_mysql.0.ifsf2la1fuuo@MYSTACKDB1    | + RESOLVE=1
mystack_mysql.0.59ryujs2lkai@MYSTACKDB1    | + for ADDR in '${ADDRS//,/ }'
mystack_mysql.0.ndfzy8caasj0@MYSTACKDB3    | ++ sed -E 's#.*gcomm://([0-9\.,]+)\s+.*#\1#'
mystack_mysql.0.g17v6oe8uglb@MYSTACKDB3    | + POSITION=0652cdaf-ea61-11e6-844e-cef75cf1b9b7:238
mystack_mysql.0.4m07zyvh6ebn@MYSTACKDB3    | + LISTEN_PORT=3309
mystack_mysql.0.r2cxqjg4k02l@MYSTACKDB3    | + GCOMM='--console --wsrep-on=ON --wsrep_CLUSTER_NAME=xcluster --wsrep_cluster_address=gcomm:// --wsrep_node_address=10.0.0.9:4567 --wsrep_sst_auth=xtrabackup:*************** --default-time-zone=+00:00'
mystack_mysql.0.ndfzy8caasj0@MYSTACKDB3    | + GCOMM='--console --wsrep-on=ON --wsrep_CLUSTER_NAME=xcluster --wsrep_cluster_address=gcomm:// --wsrep_node_address=10.0.0.9:4567 --wsrep_sst_auth=xtrabackup:*************** --default-time-zone=+00:00'
mystack_mysql.0.g17v6oe8uglb@MYSTACKDB3    | + echo 'mysqld.sh: Recovered position from grastate.dat: 0652cdaf-ea61-11e6-844e-cef75cf1b9b7:238'
mystack_mysql.0.4m07zyvh6ebn@MYSTACKDB3    | + EXPECT_NODES=3
mystack_mysql.0.g17v6oe8uglb@MYSTACKDB3    | mysqld.sh: Recovered position from grastate.dat: 0652cdaf-ea61-11e6-844e-cef75cf1b9b7:238
mystack_mysql.0.r2cxqjg4k02l@MYSTACKDB3    | + [[ -f /var/lib/mysql/wsrep-new-cluster ]]
mystack_mysql.0.ndfzy8caasj0@MYSTACKDB3    | + [[ -f /var/lib/mysql/wsrep-new-cluster ]]
mystack_mysql.0.g17v6oe8uglb@MYSTACKDB3    | + [[ -z 0652cdaf-ea61-11e6-844e-cef75cf1b9b7:238 ]]
mystack_mysql.0.ndfzy8caasj0@MYSTACKDB3    | + [[ -z 0652cdaf-ea61-11e6-844e-cef75cf1b9b7:238 ]]
mystack_mysql.0.r2cxqjg4k02l@MYSTACKDB3    | + [[ -z 0652cdaf-ea61-11e6-844e-cef75cf1b9b7:238 ]]
mystack_mysql.0.4m07zyvh6ebn@MYSTACKDB3    | + [[ -f /var/lib/mysql/gvwstate.dat ]]
mystack_mysql.0.4m07zyvh6ebn@MYSTACKDB3    | ++ awk '/^view_id:/{print $2 " " $3 " " $4}'
mystack_mysql.0.ndfzy8caasj0@MYSTACKDB3    | + check_nodes --console --wsrep-on=ON --wsrep_CLUSTER_NAME=xcluster --wsrep_cluster_address=gcomm:// --wsrep_node_address=10.0.0.9:4567 --wsrep_sst_auth=xtrabackup:*************** --default-time-zone=+00:00 10.0.0.9
mystack_mysql.0.g17v6oe8uglb@MYSTACKDB3    | ++ sed -E 's#.*--wsrep_node_address=([0-9\.]+):4567.*#\1#'
mystack_mysql.0.r2cxqjg4k02l@MYSTACKDB3    | + check_nodes --console --wsrep-on=ON --wsrep_CLUSTER_NAME=xcluster --wsrep_cluster_address=gcomm:// --wsrep_node_address=10.0.0.9:4567 --wsrep_sst_auth=xtrabackup:*************** --default-time-zone=+00:00 10.0.0.9
mystack_mysql.0.r2cxqjg4k02l@MYSTACKDB3    | + for node in '${1//,/ }'
mystack_mysql.0.ndfzy8caasj0@MYSTACKDB3    | + for node in '${1//,/ }'
mystack_mysql.0.g17v6oe8uglb@MYSTACKDB3    | + NODE_ADDRESS=10.0.0.9
mystack_mysql.0.4m07zyvh6ebn@MYSTACKDB3    | + VIEW_ID='3 104d5fd4-ebcc-11e6-a17f-337f5a7c90f7 30'
mystack_mysql.0.bd9h8hrmovc0@MYSTACKDB1    | ++ echo ''
mystack_mysql.0.ndfzy8caasj0@MYSTACKDB3    | + '[' --console = --wsrep-on=ON ']'
mystack_mysql.0.r2cxqjg4k02l@MYSTACKDB3    | + '[' --console = --wsrep-on=ON ']'
mystack_mysql.0.sree7cwscci4@MYSTACKDB1    | + '[' 15 -ge 30 ']'
mystack_mysql.0.59ryujs2lkai@MYSTACKDB1    | + [[ tasks.mystack_mysql =~ ^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+$ ]]
mystack_mysql.0.sree7cwscci4@MYSTACKDB1    | + true
mystack_mysql.0.ifsf2la1fuuo@MYSTACKDB1    | ++ getent hosts tasks.mystack_mysql
mystack_mysql.0.bd9h8hrmovc0@MYSTACKDB1    | ++ sed 's/,\+/,/g'
mystack_mysql.0.sree7cwscci4@MYSTACKDB1    | + SEP=
mystack_mysql.0.59ryujs2lkai@MYSTACKDB1    | + RESOLVE=1
mystack_mysql.0.sree7cwscci4@MYSTACKDB1    | + GCOMM=
mystack_mysql.0.ifsf2la1fuuo@MYSTACKDB1    | ++ awk '{ print $1 }'
mystack_mysql.0.bd9h8hrmovc0@MYSTACKDB1    | + GCOMM=
mystack_mysql.0.59ryujs2lkai@MYSTACKDB1    | ++ getent hosts tasks.mystack_mysql
mystack_mysql.0.sree7cwscci4@MYSTACKDB1    | + for ADDR in '${ADDRS//,/ }'
mystack_mysql.0.59ryujs2lkai@MYSTACKDB1    | ++ awk '{ print $1 }'
mystack_mysql.0.bd9h8hrmovc0@MYSTACKDB1    | ++ echo ''
mystack_mysql.0.ifsf2la1fuuo@MYSTACKDB1    | ++ paste -sd ,
mystack_mysql.0.ifsf2la1fuuo@MYSTACKDB1    | + GCOMM+=
mystack_mysql.0.g17v6oe8uglb@MYSTACKDB3    | ++ sed -E 's#.*gcomm://([0-9\.,]+)\s+.*#\1#'
mystack_mysql.0.59ryujs2lkai@MYSTACKDB1    | ++ paste -sd ,
mystack_mysql.0.4m07zyvh6ebn@MYSTACKDB3    | ++ grep '^member:'
mystack_mysql.0.g17v6oe8uglb@MYSTACKDB3    | + GCOMM='--console --wsrep-on=ON --wsrep_CLUSTER_NAME=xcluster --wsrep_cluster_address=gcomm:// --wsrep_node_address=10.0.0.9:4567 --wsrep_sst_auth=xtrabackup:*************** --default-time-zone=+00:00'
mystack_mysql.0.ndfzy8caasj0@MYSTACKDB3    | + curl -f -s -o - http://--console:8081
mystack_mysql.0.r2cxqjg4k02l@MYSTACKDB3    | + curl -f -s -o - http://--console:8081
mystack_mysql.0.r2cxqjg4k02l@MYSTACKDB3    | + return 1
mystack_mysql.0.g17v6oe8uglb@MYSTACKDB3    | + [[ -f /var/lib/mysql/wsrep-new-cluster ]]
mystack_mysql.0.4m07zyvh6ebn@MYSTACKDB3    | ++ wc -l
mystack_mysql.0.ndfzy8caasj0@MYSTACKDB3    | + return 1
mystack_mysql.0.4m07zyvh6ebn@MYSTACKDB3    | + GVW_MEMBERS=5
mystack_mysql.0.r2cxqjg4k02l@MYSTACKDB3    | + LISTEN_PORT=3309
mystack_mysql.0.g17v6oe8uglb@MYSTACKDB3    | + [[ -z 0652cdaf-ea61-11e6-844e-cef75cf1b9b7:238 ]]
mystack_mysql.0.4m07zyvh6ebn@MYSTACKDB3    | + echo 'mysqld.sh: Found view from gvwstate.dat with (5) members: 3 104d5fd4-ebcc-11e6-a17f-337f5a7c90f7 30'
mystack_mysql.0.r2cxqjg4k02l@MYSTACKDB3    | + EXPECT_NODES=3
mystack_mysql.0.ndfzy8caasj0@MYSTACKDB3    | + LISTEN_PORT=3309
mystack_mysql.0.g17v6oe8uglb@MYSTACKDB3    | + check_nodes --console --wsrep-on=ON --wsrep_CLUSTER_NAME=xcluster --wsrep_cluster_address=gcomm:// --wsrep_node_address=10.0.0.9:4567 --wsrep_sst_auth=xtrabackup:*************** --default-time-zone=+00:00 10.0.0.9
mystack_mysql.0.4m07zyvh6ebn@MYSTACKDB3    | mysqld.sh: Found view from gvwstate.dat with (5) members: 3 104d5fd4-ebcc-11e6-a17f-337f5a7c90f7 30
mystack_mysql.0.r2cxqjg4k02l@MYSTACKDB3    | + [[ -f /var/lib/mysql/gvwstate.dat ]]
mystack_mysql.0.g17v6oe8uglb@MYSTACKDB3    | + for node in '${1//,/ }'
mystack_mysql.0.ndfzy8caasj0@MYSTACKDB3    | + EXPECT_NODES=3
mystack_mysql.0.r2cxqjg4k02l@MYSTACKDB3    | ++ awk '/^view_id:/{print $2 " " $3 " " $4}'
mystack_mysql.0.4m07zyvh6ebn@MYSTACKDB3    | + [[ 5 -gt 3 ]]
mystack_mysql.0.4m07zyvh6ebn@MYSTACKDB3    | + EXPECT_NODES=5
mystack_mysql.0.g17v6oe8uglb@MYSTACKDB3    | + '[' --console = --wsrep-on=ON ']'
mystack_mysql.0.ndfzy8caasj0@MYSTACKDB3    | + [[ -f /var/lib/mysql/gvwstate.dat ]]
mystack_mysql.0.s8ckbomwfdq6@MYSTACKAPP1    | + SEP=
mystack_mysql.0.bd9h8hrmovc0@MYSTACKDB1    | ++ tr , '\n'
mystack_mysql.0.ifsf2la1fuuo@MYSTACKDB1    | + '[' -n '' ']'
mystack_mysql.0.bd9h8hrmovc0@MYSTACKDB1    | ++ sort -u
mystack_mysql.0.59ryujs2lkai@MYSTACKDB1    | + GCOMM+=
mystack_mysql.0.sree7cwscci4@MYSTACKDB1    | + [[ tasks.mystack_mysql =~ ^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+$ ]]
mystack_mysql.0.sree7cwscci4@MYSTACKDB1    | + RESOLVE=1
mystack_mysql.0.59ryujs2lkai@MYSTACKDB1    | + '[' -n '' ']'
mystack_mysql.0.bd9h8hrmovc0@MYSTACKDB1    | ++ grep -v -e '^10.0.0.6$' -e '^$'
mystack_mysql.0.bd9h8hrmovc0@MYSTACKDB1    | ++ wc -l
mystack_mysql.0.sree7cwscci4@MYSTACKDB1    | ++ getent hosts tasks.mystack_mysql
mystack_mysql.0.ifsf2la1fuuo@MYSTACKDB1    | + GCOMM=
mystack_mysql.0.59ryujs2lkai@MYSTACKDB1    | + GCOMM=
mystack_mysql.0.r2cxqjg4k02l@MYSTACKDB3    | + VIEW_ID='3 104d5fd4-ebcc-11e6-a17f-337f5a7c90f7 30'
mystack_mysql.0.g17v6oe8uglb@MYSTACKDB3    | + curl -f -s -o - http://--console:8081
mystack_mysql.0.4m07zyvh6ebn@MYSTACKDB3    | ++ wc -w
mystack_mysql.0.ndfzy8caasj0@MYSTACKDB3    | ++ awk '/^view_id:/{print $2 " " $3 " " $4}'
mystack_mysql.0.r2cxqjg4k02l@MYSTACKDB3    | ++ grep '^member:'
mystack_mysql.0.g17v6oe8uglb@MYSTACKDB3    | + return 1
mystack_mysql.0.ndfzy8caasj0@MYSTACKDB3    | + VIEW_ID='3 104d5fd4-ebcc-11e6-a17f-337f5a7c90f7 30'
mystack_mysql.0.4m07zyvh6ebn@MYSTACKDB3    | + [[ 7 -gt 5 ]]
mystack_mysql.0.r2cxqjg4k02l@MYSTACKDB3    | ++ wc -l
mystack_mysql.0.r2cxqjg4k02l@MYSTACKDB3    | + GVW_MEMBERS=5
mystack_mysql.0.ndfzy8caasj0@MYSTACKDB3    | ++ grep '^member:'
mystack_mysql.0.4m07zyvh6ebn@MYSTACKDB3    | ++ wc -w
mystack_mysql.0.g17v6oe8uglb@MYSTACKDB3    | + LISTEN_PORT=3309
mystack_mysql.0.bd9h8hrmovc0@MYSTACKDB1    | + COUNT=0
mystack_mysql.0.ndfzy8caasj0@MYSTACKDB3    | ++ wc -l
mystack_mysql.0.r2cxqjg4k02l@MYSTACKDB3    | + echo 'mysqld.sh: Found view from gvwstate.dat with (5) members: 3 104d5fd4-ebcc-11e6-a17f-337f5a7c90f7 30'
mystack_mysql.0.4m07zyvh6ebn@MYSTACKDB3    | + EXPECT_NODES=7
mystack_mysql.0.g17v6oe8uglb@MYSTACKDB3    | + EXPECT_NODES=3
mystack_mysql.0.4m07zyvh6ebn@MYSTACKDB3    | + EXPECT_NODES=6
mystack_mysql.0.ndfzy8caasj0@MYSTACKDB3    | + GVW_MEMBERS=5
mystack_mysql.0.r2cxqjg4k02l@MYSTACKDB3    | mysqld.sh: Found view from gvwstate.dat with (5) members: 3 104d5fd4-ebcc-11e6-a17f-337f5a7c90f7 30
mystack_mysql.0.ndfzy8caasj0@MYSTACKDB3    | + echo 'mysqld.sh: Found view from gvwstate.dat with (5) members: 3 104d5fd4-ebcc-11e6-a17f-337f5a7c90f7 30'
mystack_mysql.0.yb5t5inrebqy@MYSTACKAPP1    | + for ADDR in '${ADDRS//,/ }'
mystack_mysql.0.sy3ym0960h63@MYSTACKAPP1    | + for ADDR in '${ADDRS//,/ }'
mystack_mysql.0.uh6n2wxvxe4q@MYSTACKAPP1    | + SEP=
mystack_mysql.0.jiivbdxib3u6@MYSTACKAPP1    | + SLEEPS=2
mystack_mysql.0.owul6utnat2v@MYSTACKAPP2    | + true
mystack_mysql.0.sy3l55nera9m@MYSTACKAPP2    | + true
mystack_mysql.0.0a9mdtpmnxtr@MYSTACKAPP2    | + for ADDR in '${ADDRS//,/ }'
mystack_mysql.0.sy3l55nera9m@MYSTACKAPP2    | + SEP=
mystack_mysql.0.sqbgoyxbonr4@MYSTACKAPP2    | + true
mystack_mysql.0.owul6utnat2v@MYSTACKAPP2    | + SEP=
mystack_mysql.0.pchn4p8c31ya@MYSTACKAPP2    | + true
mystack_mysql.0.pchn4p8c31ya@MYSTACKAPP2    | + SEP=
mystack_mysql.0.0a9mdtpmnxtr@MYSTACKAPP2    | + [[ tasks.mystack_mysql =~ ^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+$ ]]
mystack_mysql.0.sqbgoyxbonr4@MYSTACKAPP2    | + SEP=
mystack_mysql.0.sy3l55nera9m@MYSTACKAPP2    | + GCOMM=
mystack_mysql.0.owul6utnat2v@MYSTACKAPP2    | + GCOMM=
mystack_mysql.0.owul6utnat2v@MYSTACKAPP2    | + for ADDR in '${ADDRS//,/ }'
mystack_mysql.0.sqbgoyxbonr4@MYSTACKAPP2    | + GCOMM=
mystack_mysql.0.59ryujs2lkai@MYSTACKDB1    | ++ echo ''
mystack_mysql.0.4m07zyvh6ebn@MYSTACKDB3    | + echo 'mysqld.sh: Collecting grastate.dat and gvwstate.dat info from other nodes...'
mystack_mysql.0.g17v6oe8uglb@MYSTACKDB3    | + [[ -f /var/lib/mysql/gvwstate.dat ]]
mystack_mysql.0.r2cxqjg4k02l@MYSTACKDB3    | + [[ 5 -gt 3 ]]
mystack_mysql.0.ndfzy8caasj0@MYSTACKDB3    | mysqld.sh: Found view from gvwstate.dat with (5) members: 3 104d5fd4-ebcc-11e6-a17f-337f5a7c90f7 30
mystack_mysql.0.4m07zyvh6ebn@MYSTACKDB3    | mysqld.sh: Collecting grastate.dat and gvwstate.dat info from other nodes...
mystack_mysql.0.g17v6oe8uglb@MYSTACKDB3    | ++ awk '/^view_id:/{print $2 " " $3 " " $4}'
mystack_mysql.0.ndfzy8caasj0@MYSTACKDB3    | + [[ 5 -gt 3 ]]
mystack_mysql.0.r2cxqjg4k02l@MYSTACKDB3    | + EXPECT_NODES=5
mystack_mysql.0.4m07zyvh6ebn@MYSTACKDB3    | + set -m
mystack_mysql.0.g17v6oe8uglb@MYSTACKDB3    | + VIEW_ID='3 104d5fd4-ebcc-11e6-a17f-337f5a7c90f7 30'
mystack_mysql.0.pchn4p8c31ya@MYSTACKAPP2    | + GCOMM=
mystack_mysql.0.0a9mdtpmnxtr@MYSTACKAPP2    | + RESOLVE=1
mystack_mysql.0.sy3l55nera9m@MYSTACKAPP2    | + for ADDR in '${ADDRS//,/ }'
mystack_mysql.0.0a9mdtpmnxtr@MYSTACKAPP2    | ++ getent hosts tasks.mystack_mysql
mystack_mysql.0.owul6utnat2v@MYSTACKAPP2    | + [[ tasks.mystack_mysql =~ ^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+$ ]]
mystack_mysql.0.sqbgoyxbonr4@MYSTACKAPP2    | + for ADDR in '${ADDRS//,/ }'
mystack_mysql.0.pchn4p8c31ya@MYSTACKAPP2    | + for ADDR in '${ADDRS//,/ }'
mystack_mysql.0.sree7cwscci4@MYSTACKDB1    | ++ awk '{ print $1 }'
mystack_mysql.0.ifsf2la1fuuo@MYSTACKDB1    | ++ echo ''
mystack_mysql.0.59ryujs2lkai@MYSTACKDB1    | ++ sed 's/,\+/,/g'
mystack_mysql.0.bd9h8hrmovc0@MYSTACKDB1    | + '[' 1 -eq 1 ']'
mystack_mysql.0.bd9h8hrmovc0@MYSTACKDB1    | + '[' 0 -lt 1 ']'
mystack_mysql.0.sree7cwscci4@MYSTACKDB1    | ++ paste -sd ,
mystack_mysql.0.ndfzy8caasj0@MYSTACKDB3    | + EXPECT_NODES=5
mystack_mysql.0.4m07zyvh6ebn@MYSTACKDB3    | ++ mktemp -t socat.XXXX
mystack_mysql.0.r2cxqjg4k02l@MYSTACKDB3    | ++ wc -w
mystack_mysql.0.g17v6oe8uglb@MYSTACKDB3    | ++ grep '^member:'
mystack_mysql.0.ifsf2la1fuuo@MYSTACKDB1    | ++ sed 's/,\+/,/g'
mystack_mysql.0.4m07zyvh6ebn@MYSTACKDB3    | + tmpfile=/tmp/socat.i77R
mystack_mysql.0.59ryujs2lkai@MYSTACKDB1    | + GCOMM=
mystack_mysql.0.bd9h8hrmovc0@MYSTACKDB1    | + echo 'Waiting for at least 2 IP addresses to resolve...'
mystack_mysql.0.sree7cwscci4@MYSTACKDB1    | + GCOMM+=
mystack_mysql.0.59ryujs2lkai@MYSTACKDB1    | ++ echo ''
mystack_mysql.0.ifsf2la1fuuo@MYSTACKDB1    | + GCOMM=
mystack_mysql.0.59ryujs2lkai@MYSTACKDB1    | ++ tr , '\n'
mystack_mysql.0.bd9h8hrmovc0@MYSTACKDB1    | + SLEEPS=15
mystack_mysql.0.sree7cwscci4@MYSTACKDB1    | + '[' -n '' ']'
mystack_mysql.0.bd9h8hrmovc0@MYSTACKDB1    | + sleep 3
mystack_mysql.0.59ryujs2lkai@MYSTACKDB1    | ++ sort -u
mystack_mysql.0.ifsf2la1fuuo@MYSTACKDB1    | ++ echo ''
mystack_mysql.0.sree7cwscci4@MYSTACKDB1    | + GCOMM=
mystack_mysql.0.ifsf2la1fuuo@MYSTACKDB1    | ++ tr , '\n'
mystack_mysql.0.bd9h8hrmovc0@MYSTACKDB1    | Waiting for at least 2 IP addresses to resolve...
mystack_mysql.0.59ryujs2lkai@MYSTACKDB1    | ++ grep -v -e '^10.0.0.6$' -e '^$'
mystack_mysql.0.ifsf2la1fuuo@MYSTACKDB1    | ++ grep -v -e '^10.0.0.6$' -e '^$'
mystack_mysql.0.ndfzy8caasj0@MYSTACKDB3    | ++ wc -w
mystack_mysql.0.sree7cwscci4@MYSTACKDB1    | ++ echo ''
mystack_mysql.0.bd9h8hrmovc0@MYSTACKDB1    | + '[' 15 -ge 30 ']'
mystack_mysql.0.59ryujs2lkai@MYSTACKDB1    | ++ wc -l
mystack_mysql.0.bd9h8hrmovc0@MYSTACKDB1    | + true
mystack_mysql.0.ifsf2la1fuuo@MYSTACKDB1    | ++ wc -l
mystack_mysql.0.sree7cwscci4@MYSTACKDB1    | ++ sed 's/,\+/,/g'
mystack_mysql.0.59ryujs2lkai@MYSTACKDB1    | + COUNT=0
mystack_mysql.0.bd9h8hrmovc0@MYSTACKDB1    | + SEP=
mystack_mysql.0.sree7cwscci4@MYSTACKDB1    | + GCOMM=
mystack_mysql.0.ifsf2la1fuuo@MYSTACKDB1    | ++ sort -u
mystack_mysql.0.59ryujs2lkai@MYSTACKDB1    | + '[' 1 -eq 1 ']'
mystack_mysql.0.ndfzy8caasj0@MYSTACKDB3    | + [[ 7 -gt 5 ]]
mystack_mysql.0.g17v6oe8uglb@MYSTACKDB3    | ++ wc -l
mystack_mysql.0.r2cxqjg4k02l@MYSTACKDB3    | + [[ 7 -gt 5 ]]
mystack_mysql.0.59ryujs2lkai@MYSTACKDB1    | + '[' 0 -lt 1 ']'
mystack_mysql.0.4m07zyvh6ebn@MYSTACKDB3    | + PID_SERVER=505
mystack_mysql.0.4m07zyvh6ebn@MYSTACKDB3    | + SENT_NODES=
mystack_mysql.0.ndfzy8caasj0@MYSTACKDB3    | ++ wc -w
mystack_mysql.0.g17v6oe8uglb@MYSTACKDB3    | + GVW_MEMBERS=5
mystack_mysql.0.r2cxqjg4k02l@MYSTACKDB3    | ++ wc -w
mystack_mysql.0.r2cxqjg4k02l@MYSTACKDB3    | + EXPECT_NODES=7
mystack_mysql.0.4m07zyvh6ebn@MYSTACKDB3    | + for i in '{36..0}'
mystack_mysql.0.ndfzy8caasj0@MYSTACKDB3    | + EXPECT_NODES=7
mystack_mysql.0.g17v6oe8uglb@MYSTACKDB3    | + echo 'mysqld.sh: Found view from gvwstate.dat with (5) members: 3 104d5fd4-ebcc-11e6-a17f-337f5a7c90f7 30'
mystack_mysql.0.4m07zyvh6ebn@MYSTACKDB3    | + for node in '${GCOMM//,/ }'
mystack_mysql.0.r2cxqjg4k02l@MYSTACKDB3    | + EXPECT_NODES=6
mystack_mysql.0.ndfzy8caasj0@MYSTACKDB3    | + EXPECT_NODES=6
mystack_mysql.0.g17v6oe8uglb@MYSTACKDB3    | mysqld.sh: Found view from gvwstate.dat with (5) members: 3 104d5fd4-ebcc-11e6-a17f-337f5a7c90f7 30
mystack_mysql.0.4m07zyvh6ebn@MYSTACKDB3    | + [[ --console = 10.0.0.9 ]]
mystack_mysql.0.r2cxqjg4k02l@MYSTACKDB3    | + echo 'mysqld.sh: Collecting grastate.dat and gvwstate.dat info from other nodes...'
mystack_mysql.0.ndfzy8caasj0@MYSTACKDB3    | + echo 'mysqld.sh: Collecting grastate.dat and gvwstate.dat info from other nodes...'
mystack_mysql.0.g17v6oe8uglb@MYSTACKDB3    | + [[ 5 -gt 3 ]]
mystack_mysql.0.r2cxqjg4k02l@MYSTACKDB3    | mysqld.sh: Collecting grastate.dat and gvwstate.dat info from other nodes...
mystack_mysql.0.4m07zyvh6ebn@MYSTACKDB3    | + socat - TCP:--console:3309
mystack_mysql.0.ndfzy8caasj0@MYSTACKDB3    | mysqld.sh: Collecting grastate.dat and gvwstate.dat info from other nodes...
mystack_mysql.0.g17v6oe8uglb@MYSTACKDB3    | + EXPECT_NODES=5
mystack_mysql.0.ndfzy8caasj0@MYSTACKDB3    | + set -m
mystack_mysql.0.4m07zyvh6ebn@MYSTACKDB3    | + socat -u TCP-LISTEN:3309,bind=10.0.0.9,fork OPEN:/tmp/socat.i77R,append
mystack_mysql.0.r2cxqjg4k02l@MYSTACKDB3    | + set -m
mystack_mysql.0.ifsf2la1fuuo@MYSTACKDB1    | Waiting for at least 2 IP addresses to resolve...
mystack_mysql.0.bd9h8hrmovc0@MYSTACKDB1    | + GCOMM=
mystack_mysql.0.sree7cwscci4@MYSTACKDB1    | ++ echo ''
mystack_mysql.0.ifsf2la1fuuo@MYSTACKDB1    | + COUNT=0
mystack_mysql.0.59ryujs2lkai@MYSTACKDB1    | + echo 'Waiting for at least 2 IP addresses to resolve...'
mystack_mysql.0.sree7cwscci4@MYSTACKDB1    | ++ tr , '\n'
mystack_mysql.0.59ryujs2lkai@MYSTACKDB1    | + SLEEPS=16
mystack_mysql.0.ndfzy8caasj0@MYSTACKDB3    | ++ mktemp -t socat.XXXX
mystack_mysql.0.bd9h8hrmovc0@MYSTACKDB1    | + for ADDR in '${ADDRS//,/ }'
mystack_mysql.0.ifsf2la1fuuo@MYSTACKDB1    | + '[' 1 -eq 1 ']'

colinmollenhour commented 7 years ago

So the count of GCOMM addresses is fixed because now I see "Waiting for at least 2 IP addresses to resolve..." but I have no idea why they are not resolving. 'getent hosts' is definitely not resolving the IPs as evidenced by 'GCOMM+=' so I don't know what else to say.. Gotta get the DNS resolution working within the script before anything else will work.

bartoszx commented 7 years ago

One Hundred Thanks for your help and great software. Try to figure out the rest by myself

bartoszx commented 7 years ago

I did one trick. I disabled healtcheck

    healthcheck:
      #test: ["CMD-SHELL", "test -f /var/lib/mysql/sst_in_progress || curl -f -o - http://localhost:8080/ || exit 1"]
      test: "exit 0"

Guess what? :) Now DNS lookup works correctly. Any idea why? This seems to be logical but it breaks your approach :)

Find part of my logs

mystack_mysql.0.yvon5y141yfn@MYSTACKDB1    | + '[' 10.0.1.7 = 10.0.1.3 ']'
mystack_mysql.0.yvon5y141yfn@MYSTACKDB1    | + curl -f -s -o - http://10.0.1.7:8081
mystack_mysql.0.yvon5y141yfn@MYSTACKDB1    | + return 1
mystack_mysql.0.yvon5y141yfn@MYSTACKDB1    | + [[ -s /tmp/socat.06if ]]
mystack_mysql.0.yvon5y141yfn@MYSTACKDB1    | ++ awk -F: '/^seqno:/{print $2}'
mystack_mysql.0.yvon5y141yfn@MYSTACKDB1    | ++ sort -u
mystack_mysql.0.yvon5y141yfn@MYSTACKDB1    | ++ paste -sd ,
mystack_mysql.0.yvon5y141yfn@MYSTACKDB1    | + _GCOMM=10.0.1.6,10.0.1.7,10.0.1.4,10.0.1.5,10.0.1.6,10.0.1.7
mystack_mysql.0.yvon5y141yfn@MYSTACKDB1    | ++ sed 's/,\+/,/g'
mystack_mysql.0.yvon5y141yfn@MYSTACKDB1    | ++ tr , '\n'
mystack_mysql.0.yvon5y141yfn@MYSTACKDB1    | ++ sort -u
mystack_mysql.0.yvon5y141yfn@MYSTACKDB1    | ++ paste -sd ,
mystack_mysql.0.yvon5y141yfn@MYSTACKDB1    | + GCOMM=10.0.1.6,10.0.1.7
mystack_mysql.0.yvon5y141yfn@MYSTACKDB1    | ++ sed -E 's#gcomm://[0-9\.,]+#gcomm://10.0.1.6,10.0.1.7#'
mystack_mysql.0.yvon5y141yfn@MYSTACKDB1    | + OPT='--console --wsrep-on=ON --wsrep_CLUSTER_NAME=xcluster --wsrep_cluster_address=gcomm://10.0.1.6,10.0.1.7 --wsrep_node_address=10.0.1.3:4567 --wsrep_sst_auth=xtrabackup:************** --default-time-zone=+00:00'
mystack_mysql.0.yvon5y141yfn@MYSTACKDB1    | + [[ 21 -eq 24 ]]
mystack_mysql.0.yvon5y141yfn@MYSTACKDB1    | + [[ 21 -eq 0 ]]
mystack_mysql.0.yvon5y141yfn@MYSTACKDB1    | + sleep 5
mystack_mysql.0.p3e27amfluk9@MYSTACKDB3    | + for i in '{36..0}'
mystack_mysql.0.p3e27amfluk9@MYSTACKDB3    | + for node in '${GCOMM//,/ }'
mystack_mysql.0.p3e27amfluk9@MYSTACKDB3    | + [[ 10.0.1.3 = 10.0.1.6 ]]
mystack_mysql.0.p3e27amfluk9@MYSTACKDB3    | + socat - TCP:10.0.1.3:3309
mystack_mysql.0.p3e27amfluk9@MYSTACKDB3    | + SENT_NODES=,10.0.1.4,10.0.1.3,10.0.1.5,10.0.1.4,10.0.1.3,10.0.1.5,10.0.1.3,10.0.1.4,10.0.1.5,10.0.1.3,10.0.1.4,10.0.1.5,10.0.1.3,10.0.1.4,10.0.1.5,10.0.1.3,10.0.1.4,10.0.1.5,10.0.1.3,10.0.1.4,10.0.1.5,10.0.1.3,10.0.1.4,10.0.1.5,10.0.1.3,10.0.1.4,10.0.1.5,10.0.1.3,10.0.1.4,10.0.1.5,10.0.1.3
mystack_mysql.0.p3e27amfluk9@MYSTACKDB3    | + [[ -n '' ]]
mystack_mysql.0.p3e27amfluk9@MYSTACKDB3    | + for node in '${GCOMM//,/ }'
mystack_mysql.0.p3e27amfluk9@MYSTACKDB3    | + [[ 10.0.1.4 = 10.0.1.6 ]]
mystack_mysql.0.p3e27amfluk9@MYSTACKDB3    | + socat - TCP:10.0.1.4:3309
mystack_mysql.0.p3e27amfluk9@MYSTACKDB3    | + SENT_NODES=,10.0.1.4,10.0.1.3,10.0.1.5,10.0.1.4,10.0.1.3,10.0.1.5,10.0.1.3,10.0.1.4,10.0.1.5,10.0.1.3,10.0.1.4,10.0.1.5,10.0.1.3,10.0.1.4,10.0.1.5,10.0.1.3,10.0.1.4,10.0.1.5,10.0.1.3,10.0.1.4,10.0.1.5,10.0.1.3,10.0.1.4,10.0.1.5,10.0.1.3,10.0.1.4,10.0.1.5,10.0.1.3,10.0.1.4,10.0.1.5,10.0.1.3,10.0.1.4
mystack_mysql.0.p3e27amfluk9@MYSTACKDB3    | + [[ -n '' ]]
mystack_mysql.0.p3e27amfluk9@MYSTACKDB3    | + for node in '${GCOMM//,/ }'
mystack_mysql.0.p3e27amfluk9@MYSTACKDB3    | + [[ 10.0.1.5 = 10.0.1.6 ]]
mystack_mysql.0.p3e27amfluk9@MYSTACKDB3    | + socat - TCP:10.0.1.5:3309
mystack_mysql.0.p3e27amfluk9@MYSTACKDB3    | + SENT_NODES=,10.0.1.4,10.0.1.3,10.0.1.5,10.0.1.4,10.0.1.3,10.0.1.5,10.0.1.3,10.0.1.4,10.0.1.5,10.0.1.3,10.0.1.4,10.0.1.5,10.0.1.3,10.0.1.4,10.0.1.5,10.0.1.3,10.0.1.4,10.0.1.5,10.0.1.3,10.0.1.4,10.0.1.5,10.0.1.3,10.0.1.4,10.0.1.5,10.0.1.3,10.0.1.4,10.0.1.5,10.0.1.3,10.0.1.4,10.0.1.5,10.0.1.3,10.0.1.4,10.0.1.5
mystack_mysql.0.p3e27amfluk9@MYSTACKDB3    | + [[ -n '' ]]
mystack_mysql.0.p3e27amfluk9@MYSTACKDB3    | ++ awk -F: '/^seqno:/{print $2}'
mystack_mysql.0.p3e27amfluk9@MYSTACKDB3    | ++ sort -u
mystack_mysql.0.p3e27amfluk9@MYSTACKDB3    | ++ wc -w
mystack_mysql.0.p3e27amfluk9@MYSTACKDB3    | + [[ 2 -eq 3 ]]
mystack_mysql.0.p3e27amfluk9@MYSTACKDB3    | + check_nodes 10.0.1.3,10.0.1.4,10.0.1.5 10.0.1.6
mystack_mysql.0.p3e27amfluk9@MYSTACKDB3    | + for node in '${1//,/ }'
mystack_mysql.0.p3e27amfluk9@MYSTACKDB3    | + '[' 10.0.1.3 = 10.0.1.6 ']'
mystack_mysql.0.p3e27amfluk9@MYSTACKDB3    | + curl -f -s -o - http://10.0.1.3:8081
mystack_mysql.0.p3e27amfluk9@MYSTACKDB3    | + for node in '${1//,/ }'
mystack_mysql.0.p3e27amfluk9@MYSTACKDB3    | + '[' 10.0.1.4 = 10.0.1.6 ']'
mystack_mysql.0.p3e27amfluk9@MYSTACKDB3    | + curl -f -s -o - http://10.0.1.4:8081
mystack_mysql.0.p3e27amfluk9@MYSTACKDB3    | + for node in '${1//,/ }'
mystack_mysql.0.p3e27amfluk9@MYSTACKDB3    | + '[' 10.0.1.5 = 10.0.1.6 ']'
mystack_mysql.0.p3e27amfluk9@MYSTACKDB3    | + curl -f -s -o - http://10.0.1.5:8081
mystack_mysql.0.p3e27amfluk9@MYSTACKDB3    | + return 1
mystack_mysql.0.p3e27amfluk9@MYSTACKDB3    | + [[ -s /tmp/socat.KuCU ]]
mystack_mysql.0.p3e27amfluk9@MYSTACKDB3    | ++ awk -F: '/^seqno:/{print $2}'
mystack_mysql.0.p3e27amfluk9@MYSTACKDB3    | ++ sort -u
mystack_mysql.0.p3e27amfluk9@MYSTACKDB3    | ++ paste -sd ,
mystack_mysql.0.p3e27amfluk9@MYSTACKDB3    | + _GCOMM=10.0.1.3,10.0.1.4,10.0.1.5,10.0.1.3,10.0.1.4,10.0.1.5
mystack_mysql.0.p3e27amfluk9@MYSTACKDB3    | ++ sed 's/,\+/,/g'
mystack_mysql.0.p3e27amfluk9@MYSTACKDB3    | ++ tr , '\n'
mystack_mysql.0.p3e27amfluk9@MYSTACKDB3    | ++ sort -u
mystack_mysql.0.p3e27amfluk9@MYSTACKDB3    | ++ paste -sd ,
mystack_mysql.0.p3e27amfluk9@MYSTACKDB3    | + GCOMM=10.0.1.3,10.0.1.4,10.0.1.5
mystack_mysql.0.p3e27amfluk9@MYSTACKDB3    | ++ sed -E 's#gcomm://[0-9\.,]+#gcomm://10.0.1.3,10.0.1.4,10.0.1.5#'
mystack_mysql.0.p3e27amfluk9@MYSTACKDB3    | + OPT='--console --wsrep-on=ON --wsrep_CLUSTER_NAME=xcluster --wsrep_cluster_address=gcomm://10.0.1.3,10.0.1.4,10.0.1.5 --wsrep_node_address=10.0.1.6:4567 --wsrep_sst_auth=xtrabackup:************** --default-time-zone=+00:00'
mystack_mysql.0.p3e27amfluk9@MYSTACKDB3    | + [[ 26 -eq 24 ]]
mystack_mysql.0.p3e27amfluk9@MYSTACKDB3    | + [[ 26 -eq 0 ]]
mystack_mysql.0.p3e27amfluk9@MYSTACKDB3    | + sleep 5

colinmollenhour commented 7 years ago

I thought you said you previously disabled healthcheck? Anyway, I assume the healthcheck was coming back unhealthy and docker swarm was doing a slow startup so waiting for a node to be healthy before starting the next. I don't see a flag to disable this behavior so the healthcheck needs to be mocked to "healthy" during the entire initial startup. This reduces the value of healthcheck but is the only option for now.

As I stated previously you could try this:

If you want to avoid Docker killing them during this phase probably another file used as a flag would be the best way. E.g. touch /var/lib/mysql/pre-boot at mysqld.sh:50 and then rm /var/lib/mysql/pre-boot at line 279 and then add the test -f /var/lib/mysql/pre-boot to the healthcheck.

colinmollenhour commented 7 years ago

I fixed a handful of issues, some of which were caused by the new Galera 3.19 safe_to_bootstrap feature. Now this feature is fully supported however so it should improve cluster recovery from a clean shutdown.

I also added two new flags so that the HEALTHCHECK reports healthy while the node is in the shell scripts. Your safest bet is to just keep it disabled but if you want to try it again please let me know how it goes.

bartoszx commented 7 years ago

You are the man! Thanks for your effort. Works perfeclty.

colinmollenhour commented 7 years ago

Cool, I hope it serves you well!

colinmollenhour / mariadb-galera-swarm

Collecting grastate.dat loop #12