search

colinmollenhour / mariadb-galera-swarm

MariaDb Galera Cluster container based on official mariadb image which can auto-bootstrap and recover cluster state.
https://hub.docker.com/r/colinmollenhour/mariadb-galera-swarm
Apache License 2.0
219 stars 103 forks source link

swarm example works with SST_METHOD=rsync but no with default #70

Closed gentunian closed 5 years ago

gentunian commented 5 years ago

The example docker-compose.yml for swarm stack is not working with default SST_METHOD.

I needed to set to rsync.

This are related lines to the error:

SREP_SST: [INFO] Waiting for SST streaming to complete! (20190418 16:32:18.888)
2019-04-18 16:32:20 140445766117120 [Note] WSREP: (878a5918, 'tcp://0.0.0.0:4567') turning message relay requesting off
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
2019-04-18 16:32:28 140445757724416 [Warning] WSREP: 0.0 (3eb692141971): State transfer to 1.0 (370ccf625f86) failed: -22 (Invalid argument)
2019-04-18 16:32:28 140445757724416 [ERROR] WSREP: gcs/src/gcs_group.cpp:gcs_group_handle_join_msg():737: Will never receive state. Need to abort.
2019-04-18 16:32:28 140445757724416 [Note] WSREP: gcomm: terminating thread
2019-04-18 16:32:28 140445757724416 [Note] WSREP: gcomm: joining thread
2019-04-18 16:32:28 140445757724416 [Note] WSREP: gcomm: closing backend
WSREP_SST: [ERROR] xtrabackup_checkpoints missing, failed innobackupex/SST on donor (20190418 16:32:28.978)
WSREP_SST: [ERROR] Cleanup after exit with status:2 (20190418 16:32:28.979)
WSREP_SST: [INFO] Cleaning up fifo file /tmp/mysql-console/fifo (20190418 16:32:28.980)
rm: cannot remove '/tmp/mysql-console/fifo': Permission denied
2019-04-18 16:32:28 140445749331712 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'joiner' --address '10.0.11.5' --datadir '/var/lib/mysql/'   --parent '44'  ''  '': 1 (Operation not permitted)
2019-04-18 16:32:28 140445749331712 [ERROR] WSREP: Failed to read uuid:seqno and wsrep_gtid_domain_id from joiner script.
2019-04-18 16:32:28 140448150955968 [ERROR] WSREP: SST failed: 1 (Operation not permitted)
2019-04-18 16:32:28 140448150955968 [ERROR] Aborting

Steps to reproduce

  1. git clone https://github.com/colinmollenhour/mariadb-galera-swarm.git
  2. cd mariadb-galera/swarm/examples/swarm
  3. Follow this steps: https://github.com/colinmollenhour/mariadb-galera-swarm/tree/master/examples/swarm
colinmollenhour commented 5 years ago

I think the fifo has issue on some filesystems... See https://github.com/colinmollenhour/mariadb-galera-swarm/issues/33 as well.

gentunian commented 5 years ago

hey @colinmollenhour ! I've read that post before, I didn't get to manage the nodes working at all with default SST_METHOD. And that post point me to use the image mentioned in this comment but the image didn't work for me. I made a reply to that post that I decided to remove it in order to avoid noise in the post.

Would you recommend rsync method in production?

colinmollenhour commented 5 years ago

The docs go over the differences between the two in a fair amount of detail and there is no reason rsync cannot be used in production and may be the recommended method anyway:

http://galeracluster.com/documentation-webpages/sst.html#rsync

It seems the xtrabackup method is quite fragile so I may make it the default anyway...

colinmollenhour commented 5 years ago

The default is now mariabackup as it seems MariaDb isn't interested in maintaining xtrabackup-v2 even though it is still bundled..