mysqld process won't start when using a data volume

[anweiss]

When mounting a data volume to /var/lib/mysql, I noticed that mysqld seems to hang at:

[Note] mysqld (mysqld 5.6.34-79.1-56) starting as process 38 ...

My docker-compose.yml file is as follows:

version: '3.1'

services:
  etcd:
    deploy:
      replicas: 1
    image: elcolio/etcd:latest
    command: ["-name etcd", "-discovery <my_discovery_endpoint>"]
    volumes:
      - ./etcd_data:/data
    networks:
      - galera_net

  galera:
    deploy:
      replicas: 1
      restart_policy:
        condition: on-failure
        delay: 30s
        max_attempts: 3
        window: 60s
      update_config:
        parallelism: 1
        delay: 10s
        max_failure_ratio: 0.3
    image: severalnines/pxc56
    volumes:
      - galera_data:/var/lib/mysql
    environment:
      MYSQL_ROOT_PASSWORD: "mypassword"
      CLUSTER_NAME: "my_wsrep_cluster"
      XTRABACKUP_PASSWORD: "mypassword"
      DISCOVERY_SERVICE: 'etcd:2379'
      MYSQL_DATABASE: 'mydatabase'
    networks:
      - galera_net

volumes:
  galera_data:

networks:
  galera_net:
    driver: overlay

[sdelrio]

Seems that is already started, I already got the samebehaviour and is working. In my case:

$ docker stack deploy -c docker-compose.yml c1
Creating network c1_galera_net
Creating service c1_galera
Creating service c1_etcd

$ docker stack ps c1
ID            NAME                                 IMAGE                       NODE     DESIRED STATE  CURRENT STATE          ERROR  PORTS
kvblrj6iq96o  c1_galera.so19u08ewrf7y9b2fcrm2i4l1  severalnines/pxc56:latest   swarm-6  Running        Running 5 minutes ago
rka20sm7w6m1  c1_galera.u7qyk2uapmsft30yuts1d24eq  severalnines/pxc56:latest   swarm-5  Running        Running 5 minutes ago
9x5v7sizoj5q  c1_galera.mc55mgme3aknnfrlalemzvgtb  severalnines/pxc56:latest   swarm-4  Running        Running 5 minutes ago
zjrsqgrmzsqa  c1_etcd.1                            elcolio/etcd:latest         swarm-1  Running        Running 6 minutes ago

$ eval $(docker-machine env swarm-6); docker exec -ti $(docker ps |grep galera|head -n 1|cut -f 1 -d' ') mysql -u root --password=mypassword
Warning: Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 36
Server version: 5.6.34-79.1-56 Percona XtraDB Cluster (GPL), Release rel79.1, Revision 7c38350, WSREP version 26.19, wsrep_26.19

Copyright (c) 2009-2016 Percona LLC and/or its affiliates
Copyright (c) 2000, 2016, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> show global status like 'wsrep_cluster%';
+--------------------------+--------------------------------------+
| Variable_name            | Value                                |
+--------------------------+--------------------------------------+
| wsrep_cluster_conf_id    | 3                                    |
| wsrep_cluster_size       | 3                                    |
| wsrep_cluster_state_uuid | 6bed439f-0731-11e7-89eb-b2991b43069e |
| wsrep_cluster_status     | Primary                              |
+--------------------------+--------------------------------------+
4 rows in set (0.00 sec)
mysql> show global status like 'wsrep_local_stat%';
+---------------------------+--------------------------------------+
| Variable_name             | Value                                |
+---------------------------+--------------------------------------+
| wsrep_local_state_uuid    | 6bed439f-0731-11e7-89eb-b2991b43069e |
| wsrep_local_state         | 4                                    |
| wsrep_local_state_comment | Synced                               |
+---------------------------+--------------------------------------+
3 rows in set (0.00 sec)

[sdelrio]

I've tested again mounting data volume. And after a working cluster you stop all machines and start again you can get the behaviour you told, the las log line said is started but there is no service up.

This is written on the Known Limitations parts on the blog post:

https://severalnines.com/blog/mysql-docker-deploy-homogeneous-galera-cluster-etcd

If you enter on the cotainer with the problem at your volume (/var/lib/docker/volumes/<yourvolumename>/_data) and examine the last part of error.logyou could see something like this:

2017-03-14 18:25:48 415 [ERROR] WSREP: It may not be safe to bootstrap the cluster from this node. It was not the last one to leave the cluster and may not contain all the updates. To force cluster bootstrap with this node, edit the grastate.dat file manually and set safe_to_bootstrap to 1 .
2017-03-14 18:25:48 415 [ERROR] WSREP: wsrep::connect(gcomm://) failed: 7
2017-03-14 18:25:48 415 [ERROR] Aborting

Then you have to edit the grastate.dat and modify

safe_to_bootstrap: 0

To

safe_to_bootstrap: 1

And after a while the bootstrap begins and all start in sync again.

[anweiss]

Awesome, thanks