search

colinmollenhour / mariadb-galera-swarm

MariaDb Galera Cluster container based on official mariadb image which can auto-bootstrap and recover cluster state.
https://hub.docker.com/r/colinmollenhour/mariadb-galera-swarm
Apache License 2.0
219 stars 103 forks source link
docker docker-swarm-mode galera kontena mariadb mysql

readme

MariaDb Galera Cluster

This Docker container is based on the official Docker mariadb image and is designed to be compatible with auto-scheduling systems, specifically Kubernetes, Docker Swarm Mode and Kontena Classic. It could also work with manual scheduling (docker run) by specifying the correct environment variables or possibly other scheduling systems that use similar conventions.

Tags

It is recommended to test all upgrades carefully! The latest tag should not be used for production!

Several version are supported and rebuilt and tagged occasionally with both a version number and date built so just use the latest build for your major version that suits you or build your own version. Be sure to test replication functionality when building new versions using the test.sh script!

How It Works

This is not a simple config update, much effort has gone into making this container automate the initialization and recovery of a cluster. The entrypoint script orchestrates full recovery on simultaneous reset by having nodes communicate with each other before starting the mysqld process to ensure that the cluster is recovered correctly. It does this by examining Galera's state files, recovering the GTID position on all nodes and then communicating this between nodes to find the most up-to-date one to form a new cluster if needed. It also provides multiple healthcheck endpoints for varying degress of healthiness to aid with integration of load balancers and scheduling systems.

Examples

Please submit more examples for Mesos, etc. and also improvements for existing examples!

By using DNS resolution to discover other nodes, they don't have to be specified explicitly so this container should work with any system with DNS-based service discovery such as Kubernetes, Docker Swarm, Consul, etc.

Commands

The entrypoint takes as a command one of the following startup "modes". Additional arguments will be passed to the mysqld command line. For example, you will typically want to add --log-bin=mysqld-bin to enable binary logging.

seed

Used only to initialize a new cluster and after initialization and other nodes are joined the "seed" container should be stopped and replaced with a "node" container using the same volume.

node

Join an existing cluster. Takes as a second argument a comma-separated list of IPs or hostnames to resolve which are used to build the --wsrep_cluster_address option for joining a cluster.

A "node" can actually also be used to bootstrap a cluster as the "seed" does described above by placing a flag file in the data volume before boot name /var/lib/mysql/new-cluster.

no-galera

Start server with Galera disabled. Useful for maintenance tasks like performing mysql_upgrade and resetting root credentials.

Reset root password

For example, to reset the root user password, with the Galera container stopped you can run a new temporary container like so:

shell1 $ docker run --rm -v {volume-name}:/var/lib/mysql --name no-galera-temp {image} no-galera --skip-grant-tables
shell2 $ docker exec -it no-galera-temp mysql -u root mysql
MariaDB> update user set password=password("YOUR_NEW_PASSWORD") where user='root' and host='127.0.0.1';
MariaDB> flush privileges;
MariaDB> quit
shell2 $ exit
shell1 $ <CTRL+C>

And now start your Galera container back up with the new root password.

sleep

Start the container but not the server. Runs "sleep infinity". Useful just to get volumes initialized or if you want to docker exec without the server running.

bash

Open a bash shell instead of starting the container. For example bash -c 'touch /var/lib/mysql/new-cluster' to touch the flag file that indicates to start a new cluster on the next startup.

Environment Variables

Additional variables for "seed":

Additional variables for "node":

Providing secrets through files

It's also possible to configure the sensitive variables using files, a method used by Docker Swarm, Rancher and perhaps others. The paths to the secret files defaults to /run/secrets/{lower_case_variable_name} but can be specified explicitly as well using the following environment variables:

Flag Files

In order to accomodate controlling the bootstrapping phase without having to change the CMD which is sometimes hard to do with automated schedulers you can touch the following files to change the bootstrapping behavior before starting the container. All files are expected to be in the /var/lib/mysql directory which you should be mounting as a container volume.

Health Checks

By default there are two HTTP-based healthcheck servers running in the background.

The default HEALTHCHECK command also returns healthy status if /var/lib/mysql/sst_in_progress is present to avoid a node being killed during an SST. Otherwise it uses the second health check (port 8081) to return healthy only if it is 'synced' to prevent the node from being killed if it is a donor for a long period of time. How you want the healthcheck command to behave will vary on your uses for the healthcheck so you may need to override it depending on the behavior you desire. Regardless, both healthcheck servers will be started and will use negligible resources unless they are actually being pinged.

Additionally, if a LISTEN_WHEN_HEALTHY port number is specified then the container will start a loop checking it's own port 8080 health check described above until it reports healthy at which point it will open a new socket on this port which just forwards to port 8080. This can be used with Kontena's wait_for_port feature to accomodate the rolling update mechanism.

Upgrading

In general, since MariaDb is not a particularly stable server, often introducing regressions or backwards-compatibility breaks with minor version numbers it is always advised to use specific tags in production and test upgrades on a staging environment first. Also it is advised to read the commit log to see what changes have been made. No warranty is provided, use at your own risk!

Upgrading from 10.1 to 10.2

Before upgrading you need to grant the PROCESS privilege to the xtrabackup user:

mysql> GRANT PROCESS ON *.* TO 'xtrabackup'@'localhost';

Running Tests

The test.sh script is a simple script that starts the seed and node containers using docker run. It starts the seed in the background, outputs the seed logs to the console, then starts a node in the foreground. Success of the test should be determined by inspecting the logs and additional actions like starting additional nodes and killing one or more nodes should be performed to verify that recovery works as expected.

Example

make test-10.4

Or build and run your own image for 10.4 and start a single seed and node:

$ docker build . -f Dockerfile-10.4 -t galera-test-10.4
$ ./test.sh galera-test-10.4

In another console, verify the cluster status and size:

$ docker exec -it cm-galera-test-node mysql -be "show status like 'wsrep_cluster_s%';"
+----------------------------+--------------------------------------+
| Variable_name              | Value                                |
+----------------------------+--------------------------------------+
| wsrep_cluster_size         | 2                                    |
| wsrep_cluster_state_uuid   | e54090dc-f892-11ec-ad6f-f2704a681293 |
| wsrep_cluster_status       | Primary                              |
+----------------------------+--------------------------------------+

More Info

Credit