DB failing on reprovision

banderon commented 4 years ago

The DB breaks when trying to reprovision with MariaDB.

My system:

OSX Catalina 10.15.4
Vagrant 2.2.7
VirtualBox 6.1.4 r136177 (Qt5.6.3)
No other current vagrants

Steps to reproduce:

$ git clone --recursive https://github.com/Chassis/Chassis test
$ cd test
$ mkdir -p content/{themes,plugins}

Add the MariaDB extension to the vanilla config.yaml:

extensions:
  - chassis/mariadb

Create the vagrant:

$ vagrant up

The vagrant is created and works without issue. However, when I attempt to reprovision, the DB setup breaks:

$ vagrant up --provision
Bringing machine 'default' up with 'virtualbox' provider...
==> default: Running action triggers before up ...
==> default: Running trigger...
==> default: 
==> default: Checking if box 'chassis/chassis' version '3.0.3' is up to date...
==> default: Running provisioner: shell...
    default: Running: /var/folders/hl/knq98cv16_j4w86ylb5y2zxc0000gn/T/vagrant-shell20200412-97023-1jnujnz.sh
==> default: Running provisioner: shell...
    default: Running: inline script
    default: Notice: Compiled catalog for vagrant.local in environment production in 0.93 seconds
    default: Notice: /Stage[main]/Main/Package[git-core]/ensure: created
    default: Notice: /Stage[main]/Mysql::Server::Install/Package[mysql-server]/ensure: created
    default: Error: Systemd start for mysql failed!
    default: journalctl log for mysql:
    default: -- Logs begin at Sun 2020-04-12 19:03:22 UTC, end at Sun 2020-04-12 19:39:48 UTC. --
    default: Apr 12 19:39:47 vagrant systemd[1]: Starting MySQL Community Server...
    default: Apr 12 19:39:48 vagrant mysqld[20813]: Initialization of mysqld failed: 0
    default: Apr 12 19:39:48 vagrant systemd[1]: mysql.service: Control process exited, code=exited status=1
    default: Apr 12 19:39:48 vagrant systemd[1]: mysql.service: Failed with result 'exit-code'.
    default: Apr 12 19:39:48 vagrant systemd[1]: Failed to start MySQL Community Server.
    default: 
    default: Error: /Stage[main]/Mysql::Server::Service/Service[mysqld]/ensure: change from 'stopped' to 'running' failed: Systemd start for mysql failed!
    default: journalctl log for mysql:
    default: -- Logs begin at Sun 2020-04-12 19:03:22 UTC, end at Sun 2020-04-12 19:39:48 UTC. --
    default: Apr 12 19:39:47 vagrant systemd[1]: Starting MySQL Community Server...
    default: Apr 12 19:39:48 vagrant mysqld[20813]: Initialization of mysqld failed: 0
    default: Apr 12 19:39:48 vagrant systemd[1]: mysql.service: Control process exited, code=exited status=1
    default: Apr 12 19:39:48 vagrant systemd[1]: mysql.service: Failed with result 'exit-code'.
    default: Apr 12 19:39:48 vagrant systemd[1]: Failed to start MySQL Community Server.
    default: 
    default: Notice: /Stage[main]/Mysql::Server::Service/Exec[wait_for_mysql_socket_to_open]: Dependency Service[mysqld] has failures: true
    default: Warning: /Stage[main]/Mysql::Server::Service/Exec[wait_for_mysql_socket_to_open]: Skipping because of failed dependencies
    default: Notice: /Stage[main]/Mysql::Server::Root_password/Exec[remove install pass]: Dependency Service[mysqld] has failures: true
    default: Warning: /Stage[main]/Mysql::Server::Root_password/Exec[remove install pass]: Skipping because of failed dependencies
    default: Error: Failed to apply catalog: Execution of '/usr/bin/mysql --defaults-extra-file=/root/.my.cnf -NBe SELECT CONCAT(User, '@',Host) AS User FROM mysql.user' returned 1: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)
The SSH command responded with a non-zero exit status. Vagrant
assumes that this means the command failed. The output for this command
should be in the log above. Please read the output to determine what
went wrong.

The DB is now inaccessible and doesn't work:

$ vagrant ssh
-- SSH into vagrant ---
vagrant@vagrant:~$ mysql -uwordpress -pvagrantpassword wordpress
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)

However, I can get the DB working again by manually reinstalling MariaDB:

vagrant@vagrant:~$ sudo apt -y install mariadb-server
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following package was automatically installed and is no longer required:
  libevent-core-2.1-6
Use 'sudo apt autoremove' to remove it.
The following additional packages will be installed:
  mariadb-client-10.4 mariadb-server-10.4 mariadb-server-core-10.4
Suggested packages:
  mailx mariadb-test tinyca
The following packages will be REMOVED:
  mysql-client-5.7 mysql-server mysql-server-5.7 mysql-server-core-5.7
The following NEW packages will be installed:
  mariadb-client-10.4 mariadb-server mariadb-server-10.4 mariadb-server-core-10.4
0 upgraded, 4 newly installed, 4 to remove and 13 not upgraded.
Need to get 0 B/12.2 MB of archives.
After this operation, 11.5 MB of additional disk space will be used.
Preconfiguring packages ...
(Reading database ... 51342 files and directories currently installed.)
Removing mysql-server (5.7.29-0ubuntu0.18.04.1) ...
Removing mysql-server-5.7 (5.7.29-0ubuntu0.18.04.1) ...
update-alternatives: warning: forcing reinstallation of alternative /etc/mysql/my.cnf.fallback because link group my.cnf is broken
update-alternatives: warning: not replacing /etc/mysql/my.cnf with a link
Removing mysql-server-core-5.7 (5.7.29-0ubuntu0.18.04.1) ...
dpkg: warning: while removing mysql-server-core-5.7, directory '/usr/share/mysql' not empty so not removed
dpkg: mysql-client-5.7: dependency problems, but removing anyway as you requested:
 mysql-client depends on mysql-client-5.7; however:
  Package mysql-client-5.7 is to be removed.
  Package mariadb-client-10.4 which provides mysql-client-5.7 is not installed.

Removing mysql-client-5.7 (5.7.29-0ubuntu0.18.04.1) ...
Selecting previously unselected package mariadb-client-10.4.
(Reading database ... 51135 files and directories currently installed.)
Preparing to unpack .../mariadb-client-10.4_1%3a10.4.12+maria~bionic_amd64.deb ...
Unpacking mariadb-client-10.4 (1:10.4.12+maria~bionic) ...
Selecting previously unselected package mariadb-server-core-10.4.
Preparing to unpack .../mariadb-server-core-10.4_1%3a10.4.12+maria~bionic_amd64.deb ...
Unpacking mariadb-server-core-10.4 (1:10.4.12+maria~bionic) ...
Selecting previously unselected package mariadb-server-10.4.
Preparing to unpack .../mariadb-server-10.4_1%3a10.4.12+maria~bionic_amd64.deb ...
/var/lib/mysql: found previous version 10.4
Unpacking mariadb-server-10.4 (1:10.4.12+maria~bionic) ...
Selecting previously unselected package mariadb-server.
Preparing to unpack .../mariadb-server_1%3a10.4.12+maria~bionic_all.deb ...
Unpacking mariadb-server (1:10.4.12+maria~bionic) ...
Setting up mariadb-client-10.4 (1:10.4.12+maria~bionic) ...
Setting up mariadb-server-core-10.4 (1:10.4.12+maria~bionic) ...
Setting up mariadb-server-10.4 (1:10.4.12+maria~bionic) ...
Installing new version of config file /etc/apparmor.d/usr.sbin.mysqld ...
Installing new version of config file /etc/logrotate.d/mysql-server ...
Installing new version of config file /etc/mysql/debian-start ...
Setting up mariadb-server (1:10.4.12+maria~bionic) ...
Processing triggers for man-db (2.8.3-2ubuntu0.1) ...

vagrant@vagrant:~$ mysql -uwordpress -pvagrantpassword wordpress
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 35
Server version: 10.4.12-MariaDB-1:10.4.12+maria~bionic mariadb.org binary distribution

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [wordpress]>

This breaks as soon as I reprovision again, necessitating another cycle of manually running sudo apt -y install mariadb-server.

BronsonQuick commented 4 years ago

@banderon Thanks so much for reporting this. I haven't tested this extension much since I created it so I'll have a look now and see if I can replicate and make a fix for it!

BronsonQuick commented 4 years ago

I started having a look into this issue yesterday and it turns out that none of the class I was trying to inherit in here: https://github.com/Chassis/MariaDB/blob/master/modules/mariadb/manifests/init.pp#L29-L46 was being used at all so I need to reimplement that or handle it in a different way so that the mysql socket is opened correctly and I might also need to pass in some innodb specific settings. I should be able to dig into that again in the next day or two.

BronsonQuick commented 4 years ago

I just spent a few hours trying to get a fix for this but I'm still stuck unfortunately. On an initial provision if I SSH into the box and run sudo find / -type s the socket we need has been created. /run/mysqld/mysqld.sock when we reprovision that socket isn't recreated. I think that when sudo apt -y install mariadb-server runs it purges all the existing sockets and sets them up again. I can't work out why the Puppet MySQL class that has MariaDB doesn't do this.

Chassis / MariaDB

DB failing on reprovision #2