search

codership / galera-manager-support

Galera Manager Support Repository
8 stars 2 forks source link

Galera Manager failing to deploy MariaDB nodes on Centos7 #17

Open ramtastic opened 2 years ago

ramtastic commented 2 years ago

Node deployment fails when creating user-provided hosts running CentOS7. Log information is below. I see that MariaDB does get installed but Galera cannot start it.

outside of Github, I was asked:

1)MYSQL_PWD=MYSQL_ROOT_PASSWORD mysqladmin -u root status - what this command shows

[localadmin@osonlybdc1p0665 ~]$ mysqladmin -u root status mysqladmin: connect to server at 'localhost' failed error: 'Can't connect to local server through socket '/var/lib/mysql/mysql.sock' (2)' Check that mariadbd is running and that the socket: '/var/lib/mysql/mysql.sock' exists! [localadmin@osonlybdc1p0665 ~]$ sudo systemctl start mariadb [sudo] password for localadmin: [localadmin@osonlybdc1p0665 ~]$ mysqladmin -u root status mysqladmin: connect to server at 'localhost' failed error: 'Access denied for user 'root'@'localhost'' [localadmin@osonlybdc1p0665 ~]$ mysqladmin status Uptime: 24 Threads: 3 Questions: 13 Slow queries: 0 Opens: 17 Open tables: 10 Queries per second avg: 0.541 [localadmin@osonlybdc1p0665 ~]$

2)cat /etc/ssh/sshd_config | grep AcceptEnv - what this command shows

[localadmin@osonlybdc1p0665 ~]$ cat /etc/ssh/sshd_config | grep AcceptEnv cat: /etc/ssh/sshd_config: Permission denied [localadmin@osonlybdc1p0665 ~]$ sudo cat /etc/ssh/sshd_config | grep AcceptEnv [sudo] password for localadmin: AcceptEnv LANG LC_CTYPE LC_NUMERIC LC_TIME LC_COLLATE LC_MONETARY LC_MESSAGES AcceptEnv LC_PAPER LC_NAME LC_ADDRESS LC_TELEPHONE LC_MEASUREMENT AcceptEnv LC_IDENTIFICATION LC_ALL LANGUAGE AcceptEnv XMODIFIERS AcceptEnv MYSQL_PWD AcceptEnv MYSQL_PWD AcceptEnv MYSQL_PWD AcceptEnv MYSQL_PWD AcceptEnv MYSQL_PWD AcceptEnv MYSQL_PWD AcceptEnv MYSQL_PWD AcceptEnv MYSQL_PWD AcceptEnv MYSQL_PWD AcceptEnv MYSQL_PWD AcceptEnv MYSQL_PWD

I selected MariaDB 10.4 when creating the Cluster but it appears that 10.6 was installed. That's good for me but 10.6 wasn't an option to select and the GUI notes that 10.5 is not supported.

[localadmin@osonlybdc1p0665 ~]$ sudo mysql Welcome to the MariaDB monitor. Commands end with ; or \g. Your MariaDB connection id is 9 Server version: 10.6.5-MariaDB MariaDB Server

Deployment Log

Jan 04, 2022 06:34:44 | galera-manager | deployment status ok Jan 04, 2022 06:34:57 | galera-manager | root@osonlybdc1p0665# cat /etc/galera-manager-node Jan 04, 2022 06:34:57 | host-stdout | cat: /etc/galera-manager-node: No such file or directory Jan 04, 2022 06:34:57 | galera-manager | Enabling sshd to accept env vars Jan 04, 2022 06:34:57 | host-stdout | 10.96.41.121:22$ bash -c 'echo -e "\nAcceptEnv MYSQL_PWD\n" >> /etc/ssh/sshd_config' Jan 04, 2022 06:34:57 | galera-manager | Reloading sshd Jan 04, 2022 06:34:57 | host-stdout | 10.96.41.121:22$ systemctl restart sshd Jan 04, 2022 06:34:57 | galera-manager | Waiting for sshd to reload Jan 04, 2022 06:34:57 | galera-manager | Sleeping for 2 seconds Jan 04, 2022 06:34:59 | galera-manager | Creating /etc/yum.repos.d Jan 04, 2022 06:34:59 | host-stdout | 10.96.41.121:22$ mkdir -p /etc/yum.repos.d Jan 04, 2022 06:34:59 | galera-manager | Writing /etc/yum.repos.d/galera.repo Jan 04, 2022 06:34:59 | galera-manager | Writing /etc/yum.repos.d/galera-manager.repo Jan 04, 2022 06:34:59 | galera-manager | Writing /etc/yum.repos.d/influxdb.repo Jan 04, 2022 06:34:59 | galera-manager | Installing galera Jan 04, 2022 06:34:59 | host-stdout | 10.96.41.121:22$ yum -y install MariaDB-client MariaDB-server galera Jan 04, 2022 06:34:59 | host-stdout | Loaded plugins: fastestmirror Jan 04, 2022 06:34:59 | host-stdout | Loading mirror speeds from cached hostfile Jan 04, 2022 06:34:59 | host-stdout | base | 3.0 kB 00:00 Jan 04, 2022 06:34:59 | host-stdout | espnxtras | 2.9 kB 00:00 Jan 04, 2022 06:35:00 | host-stdout | galera-manager | 1.4 kB 00:00 Jan 04, 2022 06:35:00 | host-stdout | influxdb | 2.5 kB 00:00 Jan 04, 2022 06:35:00 | host-stdout | mariadb | 3.4 kB 00:00 Jan 04, 2022 06:35:00 | host-stdout | Not using downloaded mariadb/repomd.xml because it is older than what we have: Jan 04, 2022 06:35:00 | host-stdout | Current : Tue Nov 9 16:42:47 2021 Jan 04, 2022 06:35:00 | host-stdout | Downloaded: Tue Nov 9 16:36:09 2021 Jan 04, 2022 06:35:01 | host-stdout | mariadb-main | 3.4 kB 00:00 Jan 04, 2022 06:35:01 | host-stdout | mariadb-maxscale | 2.5 kB 00:00 Jan 04, 2022 06:35:01 | host-stdout | mariadb-tools | 2.9 kB 00:00 Jan 04, 2022 06:35:01 | host-stdout | puppet | 2.5 kB 00:00 Jan 04, 2022 06:35:01 | host-stdout | updates | 3.0 kB 00:00 Jan 04, 2022 06:35:05 | host-stdout | Package galera is obsoleted by galera-4, trying to install galera-4-26.4.9-1.el7.centos.x86_64 instead Jan 04, 2022 06:35:05 | host-stdout | Resolving Dependencies Jan 04, 2022 06:35:05 | host-stdout | --> Running transaction check Jan 04, 2022 06:35:05 | host-stdout | ---> Package MariaDB-client.x86_64 0:10.6.5-1.el7.centos will be installed Jan 04, 2022 06:35:05 | host-stdout | ---> Package MariaDB-server.x86_64 0:10.6.5-1.el7.centos will be installed Jan 04, 2022 06:35:05 | host-stdout | ---> Package galera-4.x86_64 0:26.4.9-1.el7.centos will be installed Jan 04, 2022 06:35:06 | host-stdout | --> Finished Dependency Resolution Jan 04, 2022 06:35:06 | host-stdout | Dependencies Resolved Jan 04, 2022 06:35:06 | host-stdout | ======================================== Jan 04, 2022 06:35:06 | host-stdout | Package Arch Version Jan 04, 2022 06:35:06 | host-stdout | Repository Size Jan 04, 2022 06:35:06 | host-stdout | ======================================== Jan 04, 2022 06:35:06 | host-stdout | Installing: Jan 04, 2022 06:35:06 | host-stdout | MariaDB-client Jan 04, 2022 06:35:06 | host-stdout | x86_64 10.6.5-1.el7.centos Jan 04, 2022 06:35:06 | host-stdout | mariadb-main 14 M Jan 04, 2022 06:35:06 | host-stdout | MariaDB-server Jan 04, 2022 06:35:06 | host-stdout | x86_64 10.6.5-1.el7.centos Jan 04, 2022 06:35:06 | host-stdout | mariadb-main 25 M Jan 04, 2022 06:35:06 | host-stdout | galera-4 x86_64 26.4.9-1.el7.centos Jan 04, 2022 06:35:06 | host-stdout | mariadb-main 9.6 M Jan 04, 2022 06:35:06 | host-stdout | Transaction Summary Jan 04, 2022 06:35:06 | host-stdout | ======================================== Jan 04, 2022 06:35:06 | host-stdout | Install 3 Packages Jan 04, 2022 06:35:06 | host-stdout | Total download size: 49 M Jan 04, 2022 06:35:06 | host-stdout | Installed size: 206 M Jan 04, 2022 06:35:06 | host-stdout | Downloading packages: Jan 04, 2022 06:35:07 | host-stdout | (2/3): MariaDB- 0% | 0 B --:-- ETA Jan 04, 2022 06:35:07 | host-stdout | (2/3): MariaDB 45% | 22 MB --:-- ETA Jan 04, 2022 06:35:07 | host-stdout | (1/3): MariaDB-cli | 14 MB 00:00 Jan 04, 2022 06:35:07 | host-stdout | (2/3): MariaDB-ser | 25 MB 00:01 Jan 04, 2022 06:35:07 | host-stdout | (3/3): galera- 80% | 39 MB 00:00 ETA Jan 04, 2022 06:35:08 | host-stdout | (3/3): galera-4-26 | 9.6 MB 00:00 Jan 04, 2022 06:35:08 | host-stdout | ---------------------------------------- Jan 04, 2022 06:35:08 | host-stdout | Total 30 MB/s | 49 MB 00:01 Jan 04, 2022 06:35:08 | host-stdout | Running transaction check Jan 04, 2022 06:35:08 | host-stdout | Running transaction test Jan 04, 2022 06:35:08 | host-stdout | Transaction test succeeded Jan 04, 2022 06:35:08 | host-stdout | Running transaction Jan 04, 2022 06:35:08 | host-stdout | Installing : MariaDB- [ ] 1/3 Jan 04, 2022 06:35:09 | host-stdout | Installing : MariaDB- [# ] 1/3 Jan 04, 2022 06:35:09 | host-stdout | Installing : MariaDB- [## ] 1/3 Jan 04, 2022 06:35:09 | host-stdout | Installing : MariaDB- [### ] 1/3 Jan 04, 2022 06:35:09 | host-stdout | Installing : MariaDB- [#### ] 1/3 Jan 04, 2022 06:35:10 | host-stdout | Installing : MariaDB- [##### ] 1/3 Jan 04, 2022 06:35:10 | host-stdout | Installing : MariaDB- [###### ] 1/3 Jan 04, 2022 06:35:10 | host-stdout | Installing : MariaDB- [####### ] 1/3 Jan 04, 2022 06:35:11 | host-stdout | Installing : MariaDB- [######## ] 1/3 Jan 04, 2022 06:35:11 | host-stdout | Installing : MariaDB-client-10. 1/3 Jan 04, 2022 06:35:12 | host-stdout | Installing : galera-4 [ ] 2/3 Jan 04, 2022 06:35:12 | host-stdout | Installing : galera-4 [# ] 2/3 Jan 04, 2022 06:35:12 | host-stdout | Installing : galera-4 [## ] 2/3 Jan 04, 2022 06:35:12 | host-stdout | Installing : galera-4 [### ] 2/3 Jan 04, 2022 06:35:12 | host-stdout | Installing : galera-4 [#### ] 2/3 Jan 04, 2022 06:35:13 | host-stdout | Installing : galera-4 [##### ] 2/3 Jan 04, 2022 06:35:13 | host-stdout | Installing : galera-4 [###### ] 2/3 Jan 04, 2022 06:35:13 | host-stdout | Installing : galera-4 [####### ] 2/3 Jan 04, 2022 06:35:13 | host-stdout | Installing : galera-4 [######## ] 2/3 Jan 04, 2022 06:35:13 | host-stdout | Installing : galera-4-26.4.9-1. 2/3 Jan 04, 2022 06:35:14 | host-stdout | Installing : MariaDB- [ ] 3/3 Jan 04, 2022 06:35:14 | host-stdout | Installing : MariaDB- [# ] 3/3 Jan 04, 2022 06:35:15 | host-stdout | Installing : MariaDB- [## ] 3/3 Jan 04, 2022 06:35:15 | host-stdout | Installing : MariaDB- [### ] 3/3 Jan 04, 2022 06:35:16 | host-stdout | Installing : MariaDB- [#### ] 3/3 Jan 04, 2022 06:35:16 | host-stdout | Installing : MariaDB- [##### ] 3/3 Jan 04, 2022 06:35:17 | host-stdout | Installing : MariaDB- [###### ] 3/3 Jan 04, 2022 06:35:17 | host-stdout | Installing : MariaDB- [####### ] 3/3 Jan 04, 2022 06:35:18 | host-stdout | Installing : MariaDB- [######## ] 3/3 Jan 04, 2022 06:35:18 | host-stdout | Installing : MariaDB-server-10. 3/3 Jan 04, 2022 06:35:29 | host-stdout | /usr/bin/my_print_defaults: Can't read dir of '/etc/mysql/wsrep/conf.d/' (Errcode: 2 "No such file or directory") Jan 04, 2022 06:35:29 | host-stdout | Fatal error in defaults handling. Program aborted Jan 04, 2022 06:35:29 | host-stdout | /usr/bin/my_print_defaults: Can't read dir of '/etc/mysql/wsrep/conf.d/' (Errcode: 2 "No such file or directory") Jan 04, 2022 06:35:29 | host-stdout | Fatal error in defaults handling. Program aborted Jan 04, 2022 06:35:29 | host-stdout | Verifying : MariaDB-server-10. 1/3 Jan 04, 2022 06:35:29 | host-stdout | Verifying : galera-4-26.4.9-1. 2/3 Jan 04, 2022 06:35:29 | host-stdout | Verifying : MariaDB-client-10. 3/3 Jan 04, 2022 06:35:29 | host-stdout | Installed: Jan 04, 2022 06:35:29 | host-stdout | MariaDB-client.x86_64 0:10.6.5-1.el7.centos Jan 04, 2022 06:35:29 | host-stdout | MariaDB-server.x86_64 0:10.6.5-1.el7.centos Jan 04, 2022 06:35:29 | host-stdout | galera-4.x86_64 0:26.4.9-1.el7.centos Jan 04, 2022 06:35:29 | host-stdout | Complete! Jan 04, 2022 06:35:29 | galera-manager | Installing filebeat Jan 04, 2022 06:35:29 | host-stdout | 10.96.41.121:22$ yum -y install galera-manager-filebeat Jan 04, 2022 06:35:29 | host-stdout | Loaded plugins: fastestmirror Jan 04, 2022 06:35:29 | host-stdout | Loading mirror speeds from cached hostfile Jan 04, 2022 06:35:30 | host-stdout | Resolving Dependencies Jan 04, 2022 06:35:30 | host-stdout | --> Running transaction check Jan 04, 2022 06:35:30 | host-stdout | ---> Package galera-manager-filebeat.x86_64 0:1.2.1-1 will be installed Jan 04, 2022 06:35:30 | host-stdout | --> Finished Dependency Resolution Jan 04, 2022 06:35:30 | host-stdout | Dependencies Resolved Jan 04, 2022 06:35:30 | host-stdout | ======================================== Jan 04, 2022 06:35:30 | host-stdout | Package Jan 04, 2022 06:35:30 | host-stdout | Arch Version Repository Size Jan 04, 2022 06:35:30 | host-stdout | ======================================== Jan 04, 2022 06:35:30 | host-stdout | Installing: Jan 04, 2022 06:35:30 | host-stdout | galera-manager-filebeat Jan 04, 2022 06:35:30 | host-stdout | x86_64 1.2.1-1 galera-manager 7.2 M Jan 04, 2022 06:35:30 | host-stdout | Transaction Summary Jan 04, 2022 06:35:30 | host-stdout | ======================================== Jan 04, 2022 06:35:30 | host-stdout | Install 1 Package Jan 04, 2022 06:35:30 | host-stdout | Total download size: 7.2 M Jan 04, 2022 06:35:30 | host-stdout | Installed size: 25 M Jan 04, 2022 06:35:30 | host-stdout | Downloading packages: Jan 04, 2022 06:35:31 | host-stdout | galera-manager- 0% | 0 B --:-- ETA Jan 04, 2022 06:35:31 | host-stdout | galera-manager- 5% | 384 kB --:-- ETA Jan 04, 2022 06:35:32 | host-stdout | galera-manager 30% | 2.2 MB 00:02 ETA Jan 04, 2022 06:35:32 | host-stdout | galera-manager 89% | 6.4 MB 00:00 ETA Jan 04, 2022 06:35:32 | host-stdout | galera-manager-fil | 7.2 MB 00:01 Jan 04, 2022 06:35:32 | host-stdout | Running transaction check Jan 04, 2022 06:35:32 | host-stdout | Running transaction test Jan 04, 2022 06:35:32 | host-stdout | Transaction test succeeded Jan 04, 2022 06:35:32 | host-stdout | Running transaction Jan 04, 2022 06:35:32 | host-stdout | Installing : galera-m [ ] 1/1 Jan 04, 2022 06:35:33 | host-stdout | Installing : galera-m [# ] 1/1 Jan 04, 2022 06:35:33 | host-stdout | Installing : galera-m [## ] 1/1 Jan 04, 2022 06:35:33 | host-stdout | Installing : galera-m [### ] 1/1 Jan 04, 2022 06:35:33 | host-stdout | Installing : galera-m [#### ] 1/1 Jan 04, 2022 06:35:33 | host-stdout | Installing : galera-m [##### ] 1/1 Jan 04, 2022 06:35:33 | host-stdout | Installing : galera-m [###### ] 1/1 Jan 04, 2022 06:35:33 | host-stdout | Installing : galera-m [####### ] 1/1 Jan 04, 2022 06:35:33 | host-stdout | Installing : galera-m [######## ] 1/1 Jan 04, 2022 06:35:34 | host-stdout | Installing : galera-manager-fil 1/1 Jan 04, 2022 06:35:34 | host-stdout | Verifying : galera-manager-fil 1/1 Jan 04, 2022 06:35:34 | host-stdout | Installed: Jan 04, 2022 06:35:34 | host-stdout | galera-manager-filebeat.x86_64 0:1.2.1-1 Jan 04, 2022 06:35:34 | host-stdout | Complete! Jan 04, 2022 06:35:34 | galera-manager | Installing telegraf Jan 04, 2022 06:35:34 | host-stdout | 10.96.41.121:22$ yum -y install telegraf Jan 04, 2022 06:35:34 | host-stdout | Loaded plugins: fastestmirror Jan 04, 2022 06:35:34 | host-stdout | Loading mirror speeds from cached hostfile Jan 04, 2022 06:35:35 | host-stdout | Package telegraf-1.21.1-1.x86_64 already installed and latest version Jan 04, 2022 06:35:35 | host-stdout | Nothing to do Jan 04, 2022 06:35:35 | galera-manager | Add filebeat user to the mysql group Jan 04, 2022 06:35:35 | host-stdout | 10.96.41.121:22$ usermod -aG mysql filebeat Jan 04, 2022 06:35:35 | galera-manager | Creating /etc/mysql/wsrep/ssl Jan 04, 2022 06:35:35 | host-stdout | 10.96.41.121:22$ mkdir -p /etc/mysql/wsrep/ssl Jan 04, 2022 06:35:35 | galera-manager | Saving /etc/mysql/wsrep/ssl/gmnode.pem Jan 04, 2022 06:35:35 | galera-manager | Saving /etc/mysql/wsrep/ssl/gmnode.key Jan 04, 2022 06:35:35 | galera-manager | Saving /etc/mysql/wsrep/ssl/gmca.pem Jan 04, 2022 06:35:35 | galera-manager | Creating /etc/mysql/wsrep/conf.d Jan 04, 2022 06:35:35 | host-stdout | 10.96.41.121:22$ mkdir -p /etc/mysql/wsrep/conf.d Jan 04, 2022 06:35:35 | galera-manager | Adding sst ssl config Jan 04, 2022 06:35:35 | galera-manager | Creating /etc/mysql/wsrep/conf.d Jan 04, 2022 06:35:35 | host-stdout | 10.96.41.121:22$ mkdir -p /etc/mysql/wsrep/conf.d Jan 04, 2022 06:35:35 | galera-manager | Including /etc/mysql/wsrep/conf.d in my.cnf Jan 04, 2022 06:35:35 | host-stdout | 10.96.41.121:22$ bash -c 'echo "!includedir /etc/mysql/wsrep/conf.d/" >> /etc/my.cnf' Jan 04, 2022 06:35:35 | galera-manager | Including /etc/mysql/wsrep/conf.d in my.cnf Jan 04, 2022 06:35:35 | host-stdout | 10.96.41.121:22$ yum -y install wget Jan 04, 2022 06:35:35 | host-stdout | Loaded plugins: fastestmirror Jan 04, 2022 06:35:35 | host-stdout | Loading mirror speeds from cached hostfile Jan 04, 2022 06:35:35 | host-stdout | Package wget-1.14-18.el7_6.1.x86_64 already installed and latest version Jan 04, 2022 06:35:35 | host-stdout | Nothing to do Jan 04, 2022 06:35:35 | galera-manager | Adding wsrep_provider_options_value Jan 04, 2022 06:35:35 | galera-manager | Installing telegraf mysql_wsrep plugin Jan 04, 2022 06:35:35 | host-stdout | 10.96.41.121:22$ bash -c 'wget https://github.com/go-extras/telegraf-mysql-wsrep/files/5233298/latest-linux-amd64.tar.gz && tar -xzvf latest-linux-amd64.tar.gz && mv mysql_wsrep /usr/local/bin/ && chmod +x /usr/local/bin/mysql_wsrep' Jan 04, 2022 06:35:35 | host-stdout | --2022-01-04 06:35:35-- https://github.com/go-extras/telegraf-mysql-wsrep/files/5233298/latest-linux-amd64.tar.gz Jan 04, 2022 06:35:35 | host-stdout | Resolving github.com (github.com)... Jan 04, 2022 06:35:35 | host-stdout | 140.82.113.4 Jan 04, 2022 06:35:35 | host-stdout | Connecting to github.com (github.com)|140.82.113.4|:443... Jan 04, 2022 06:35:35 | host-stdout | connected. Jan 04, 2022 06:35:35 | host-stdout | Unable to establish SSL connection. Jan 04, 2022 06:35:35 | galera-manager | Got an error and attepts = 0 Jan 04, 2022 06:35:35 | galera-manager | failed to execute cluster config script (RunScriptWithConn): command failed (stepName=__step_no_017, commandId=17, commandType=IncludeCommand): command failed (stepName=install_telegraf_mysql_wsrep_plugin, commandId=1, commandType=ExecCommand): Process exited with status 4 Jan 04, 2022 06:35:35 | galera-manager | Saving Galera Manager node data

ramtastic commented 2 years ago

We found some firewall rules that needed to be tweaked but are now seeing a new failure.

Jan 04, 2022 12:10:39 | galera-manager | deployment status ok Jan 04, 2022 12:10:40 | galera-manager | root@osonlybdc1p0665# cat /etc/galera-manager-node Jan 04, 2022 12:10:41 | host-stdout | cat: /etc/galera-manager-node: No such file or directory Jan 04, 2022 12:10:41 | galera-manager | Enabling sshd to accept env vars Jan 04, 2022 12:10:41 | host-stdout | 10.96.41.121:22$ bash -c 'echo -e "\nAcceptEnv MYSQL_PWD\n" >> /etc/ssh/sshd_config' Jan 04, 2022 12:10:41 | galera-manager | Reloading sshd Jan 04, 2022 12:10:41 | host-stdout | 10.96.41.121:22$ systemctl restart sshd Jan 04, 2022 12:10:41 | galera-manager | Waiting for sshd to reload Jan 04, 2022 12:10:41 | galera-manager | Sleeping for 2 seconds Jan 04, 2022 12:10:43 | galera-manager | Creating /etc/yum.repos.d Jan 04, 2022 12:10:43 | host-stdout | 10.96.41.121:22$ mkdir -p /etc/yum.repos.d Jan 04, 2022 12:10:43 | galera-manager | Writing /etc/yum.repos.d/galera.repo Jan 04, 2022 12:10:43 | galera-manager | Writing /etc/yum.repos.d/galera-manager.repo Jan 04, 2022 12:10:43 | galera-manager | Writing /etc/yum.repos.d/influxdb.repo Jan 04, 2022 12:10:43 | galera-manager | Installing galera Jan 04, 2022 12:10:43 | host-stdout | 10.96.41.121:22$ yum -y install MariaDB-client MariaDB-server galera Jan 04, 2022 12:10:43 | host-stdout | Loaded plugins: fastestmirror Jan 04, 2022 12:10:43 | host-stdout | Loading mirror speeds from cached hostfile Jan 04, 2022 12:10:43 | host-stdout | galera-manager | 1.4 kB 00:00 Jan 04, 2022 12:10:43 | host-stdout | influxdb | 2.5 kB 00:00 Jan 04, 2022 12:10:44 | host-stdout | mariadb | 3.4 kB 00:00 Jan 04, 2022 12:10:44 | host-stdout | Not using downloaded mariadb/repomd.xml because it is older than what we have: Jan 04, 2022 12:10:44 | host-stdout | Current : Tue Nov 9 16:42:47 2021 Jan 04, 2022 12:10:44 | host-stdout | Downloaded: Tue Nov 9 16:36:09 2021 Jan 04, 2022 12:10:44 | host-stdout | Package galera-25.3.34-1.el7.centos.x86_64 is obsoleted by galera-4-26.4.9-1.el7.centos.x86_64 which is already installed Jan 04, 2022 12:10:44 | host-stdout | Resolving Dependencies Jan 04, 2022 12:10:44 | host-stdout | --> Running transaction check Jan 04, 2022 12:10:44 | host-stdout | ---> Package MariaDB-client.x86_64 0:10.6.5-1.el7.centos will be installed Jan 04, 2022 12:10:44 | host-stdout | ---> Package MariaDB-server.x86_64 0:10.6.5-1.el7.centos will be installed Jan 04, 2022 12:10:45 | host-stdout | --> Finished Dependency Resolution Jan 04, 2022 12:10:45 | host-stdout | Dependencies Resolved Jan 04, 2022 12:10:45 | host-stdout | ======================================== Jan 04, 2022 12:10:45 | host-stdout | Package Jan 04, 2022 12:10:45 | host-stdout | Arch Version Jan 04, 2022 12:10:45 | host-stdout | Repository Size Jan 04, 2022 12:10:45 | host-stdout | ======================================== Jan 04, 2022 12:10:45 | host-stdout | Installing: Jan 04, 2022 12:10:45 | host-stdout | MariaDB-client Jan 04, 2022 12:10:45 | host-stdout | x86_64 10.6.5-1.el7.centos Jan 04, 2022 12:10:45 | host-stdout | mariadb-main 14 M Jan 04, 2022 12:10:45 | host-stdout | MariaDB-server Jan 04, 2022 12:10:45 | host-stdout | x86_64 10.6.5-1.el7.centos Jan 04, 2022 12:10:45 | host-stdout | mariadb-main 25 M Jan 04, 2022 12:10:45 | host-stdout | Transaction Summary Jan 04, 2022 12:10:45 | host-stdout | ======================================== Jan 04, 2022 12:10:45 | host-stdout | Install 2 Packages Jan 04, 2022 12:10:45 | host-stdout | Total download size: 39 M Jan 04, 2022 12:10:45 | host-stdout | Installed size: 196 M Jan 04, 2022 12:10:45 | host-stdout | Downloading packages: Jan 04, 2022 12:10:47 | host-stdout | (2/2): MariaDB- 0% | 0 B --:-- ETA Jan 04, 2022 12:10:47 | host-stdout | (1/2): MariaDB-cli | 14 MB 00:02 Jan 04, 2022 12:10:47 | host-stdout | (2/2): MariaDB-ser | 25 MB 00:02 Jan 04, 2022 12:10:47 | host-stdout | ---------------------------------------- Jan 04, 2022 12:10:47 | host-stdout | Total 16 MB/s | 39 MB 00:02 Jan 04, 2022 12:10:47 | host-stdout | Running transaction check Jan 04, 2022 12:10:48 | host-stdout | Running transaction test Jan 04, 2022 12:10:48 | host-stdout | Transaction test succeeded Jan 04, 2022 12:10:48 | host-stdout | Running transaction Jan 04, 2022 12:10:48 | host-stdout | Installing : MariaDB- [ ] 1/2 Jan 04, 2022 12:10:48 | host-stdout | Installing : MariaDB- [# ] 1/2 Jan 04, 2022 12:10:49 | host-stdout | Installing : MariaDB- [## ] 1/2 Jan 04, 2022 12:10:49 | host-stdout | Installing : MariaDB- [### ] 1/2 Jan 04, 2022 12:10:49 | host-stdout | Installing : MariaDB- [#### ] 1/2 Jan 04, 2022 12:10:49 | host-stdout | Installing : MariaDB- [##### ] 1/2 Jan 04, 2022 12:10:50 | host-stdout | Installing : MariaDB- [###### ] 1/2 Jan 04, 2022 12:10:50 | host-stdout | Installing : MariaDB- [####### ] 1/2 Jan 04, 2022 12:10:50 | host-stdout | Installing : MariaDB- [######## ] 1/2 Jan 04, 2022 12:10:51 | host-stdout | Installing : MariaDB-client-10. 1/2 Jan 04, 2022 12:10:51 | host-stdout | Installing : MariaDB- [ ] 2/2 Jan 04, 2022 12:10:52 | host-stdout | Installing : MariaDB- [# ] 2/2 Jan 04, 2022 12:10:52 | host-stdout | Installing : MariaDB- [## ] 2/2 Jan 04, 2022 12:10:52 | host-stdout | Installing : MariaDB- [### ] 2/2 Jan 04, 2022 12:10:53 | host-stdout | Installing : MariaDB- [#### ] 2/2 Jan 04, 2022 12:10:53 | host-stdout | Installing : MariaDB- [##### ] 2/2 Jan 04, 2022 12:10:54 | host-stdout | Installing : MariaDB- [###### ] 2/2 Jan 04, 2022 12:10:54 | host-stdout | Installing : MariaDB- [####### ] 2/2 Jan 04, 2022 12:10:55 | host-stdout | Installing : MariaDB- [######## ] 2/2 Jan 04, 2022 12:10:55 | host-stdout | Installing : MariaDB-server-10. 2/2 Jan 04, 2022 12:11:05 | host-stdout | /usr/bin/my_print_defaults: Can't read dir of '/etc/mysql/wsrep/conf.d/' (Errcode: 2 "No such file or directory") Jan 04, 2022 12:11:05 | host-stdout | Fatal error in defaults handling. Program aborted Jan 04, 2022 12:11:05 | host-stdout | /usr/bin/my_print_defaults: Can't read dir of '/etc/mysql/wsrep/conf.d/' (Errcode: 2 "No such file or directory") Jan 04, 2022 12:11:05 | host-stdout | Fatal error in defaults handling. Program aborted Jan 04, 2022 12:11:06 | host-stdout | Verifying : MariaDB-server-10. 1/2 Jan 04, 2022 12:11:06 | host-stdout | Verifying : MariaDB-client-10. 2/2 Jan 04, 2022 12:11:06 | host-stdout | Installed: Jan 04, 2022 12:11:06 | host-stdout | MariaDB-client.x86_64 0:10.6.5-1.el7.centos Jan 04, 2022 12:11:06 | host-stdout | MariaDB-server.x86_64 0:10.6.5-1.el7.centos Jan 04, 2022 12:11:06 | host-stdout | Complete! Jan 04, 2022 12:11:06 | galera-manager | Installing filebeat Jan 04, 2022 12:11:06 | host-stdout | 10.96.41.121:22$ yum -y install galera-manager-filebeat Jan 04, 2022 12:11:06 | host-stdout | Loaded plugins: fastestmirror Jan 04, 2022 12:11:06 | host-stdout | Loading mirror speeds from cached hostfile Jan 04, 2022 12:11:06 | host-stdout | Package galera-manager-filebeat-1.2.1-1.x86_64 already installed and latest version Jan 04, 2022 12:11:06 | host-stdout | Nothing to do Jan 04, 2022 12:11:06 | galera-manager | Installing telegraf Jan 04, 2022 12:11:06 | host-stdout | 10.96.41.121:22$ yum -y install telegraf Jan 04, 2022 12:11:07 | host-stdout | Loaded plugins: fastestmirror Jan 04, 2022 12:11:07 | host-stdout | Loading mirror speeds from cached hostfile Jan 04, 2022 12:11:07 | host-stdout | Package telegraf-1.21.1-1.x86_64 already installed and latest version Jan 04, 2022 12:11:07 | host-stdout | Nothing to do Jan 04, 2022 12:11:07 | galera-manager | Add filebeat user to the mysql group Jan 04, 2022 12:11:07 | host-stdout | 10.96.41.121:22$ usermod -aG mysql filebeat Jan 04, 2022 12:11:07 | galera-manager | Creating /etc/mysql/wsrep/ssl Jan 04, 2022 12:11:07 | host-stdout | 10.96.41.121:22$ mkdir -p /etc/mysql/wsrep/ssl Jan 04, 2022 12:11:07 | galera-manager | Saving /etc/mysql/wsrep/ssl/gmnode.pem Jan 04, 2022 12:11:07 | galera-manager | Saving /etc/mysql/wsrep/ssl/gmnode.key Jan 04, 2022 12:11:07 | galera-manager | Saving /etc/mysql/wsrep/ssl/gmca.pem Jan 04, 2022 12:11:07 | galera-manager | Creating /etc/mysql/wsrep/conf.d Jan 04, 2022 12:11:07 | host-stdout | 10.96.41.121:22$ mkdir -p /etc/mysql/wsrep/conf.d Jan 04, 2022 12:11:07 | galera-manager | Adding sst ssl config Jan 04, 2022 12:11:07 | galera-manager | Creating /etc/mysql/wsrep/conf.d Jan 04, 2022 12:11:07 | host-stdout | 10.96.41.121:22$ mkdir -p /etc/mysql/wsrep/conf.d Jan 04, 2022 12:11:07 | galera-manager | Including /etc/mysql/wsrep/conf.d in my.cnf Jan 04, 2022 12:11:07 | host-stdout | 10.96.41.121:22$ bash -c 'echo "!includedir /etc/mysql/wsrep/conf.d/" >> /etc/my.cnf' Jan 04, 2022 12:11:07 | galera-manager | Including /etc/mysql/wsrep/conf.d in my.cnf Jan 04, 2022 12:11:07 | host-stdout | 10.96.41.121:22$ yum -y install wget Jan 04, 2022 12:11:07 | host-stdout | Loaded plugins: fastestmirror Jan 04, 2022 12:11:07 | host-stdout | Loading mirror speeds from cached hostfile Jan 04, 2022 12:11:08 | host-stdout | Package wget-1.14-18.el7_6.1.x86_64 already installed and latest version Jan 04, 2022 12:11:08 | host-stdout | Nothing to do Jan 04, 2022 12:11:08 | galera-manager | Adding wsrep_provider_options_value Jan 04, 2022 12:11:08 | galera-manager | Installing telegraf mysql_wsrep plugin Jan 04, 2022 12:11:08 | host-stdout | 10.96.41.121:22$ bash -c 'wget https://github.com/go-extras/telegraf-mysql-wsrep/files/5233298/latest-linux-amd64.tar.gz && tar -xzvf latest-linux-amd64.tar.gz && mv mysql_wsrep /usr/local/bin/ && chmod +x /usr/local/bin/mysql_wsrep' Jan 04, 2022 12:11:08 | host-stdout | --2022-01-04 12:11:08-- https://github.com/go-extras/telegraf-mysql-wsrep/files/5233298/latest-linux-amd64.tar.gz Jan 04, 2022 12:11:08 | host-stdout | Resolving github.com (github.com)... Jan 04, 2022 12:11:08 | host-stdout | 140.82.114.4 Jan 04, 2022 12:11:08 | host-stdout | Connecting to github.com (github.com)|140.82.114.4|:443... Jan 04, 2022 12:11:08 | host-stdout | connected. Jan 04, 2022 12:11:08 | host-stdout | HTTP request sent, awaiting response... Jan 04, 2022 12:11:08 | host-stdout | 302 Found Jan 04, 2022 12:11:08 | host-stdout | Location: https://github-repository-files.githubusercontent.com/295720570/5233298?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20220104%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20220104T171108Z&X-Amz-Expires=300&X-Amz-Signature=e14a0b809e82c8e1e2b6cc7dc03d87949736aa1f2c8a0423c69f8f61d4648c65&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=295720570&response-content-disposition=attachment%3Bfilename%3Dlatest-linux-amd64.tar.gz&response-content-type=application%2Fx-gzip [following] Jan 04, 2022 12:11:08 | host-stdout | --2022-01-04 12:11:08-- https://github-repository-files.githubusercontent.com/295720570/5233298?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20220104%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20220104T171108Z&X-Amz-Expires=300&X-Amz-Signature=e14a0b809e82c8e1e2b6cc7dc03d87949736aa1f2c8a0423c69f8f61d4648c65&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=295720570&response-content-disposition=attachment%3Bfilename%3Dlatest-linux-amd64.tar.gz&response-content-type=application%2Fx-gzip Jan 04, 2022 12:11:08 | host-stdout | Resolving github-repository-files.githubusercontent.com (github-repository-files.githubusercontent.com)... Jan 04, 2022 12:11:08 | host-stdout | 185.199.108.154, 185.199.109.154, 185.199.110.154, ... Jan 04, 2022 12:11:08 | host-stdout | Connecting to github-repository-files.githubusercontent.com (github-repository-files.githubusercontent.com)|185.199.108.154|:443... Jan 04, 2022 12:11:08 | host-stdout | connected. Jan 04, 2022 12:11:08 | host-stdout | HTTP request sent, awaiting response... Jan 04, 2022 12:11:08 | host-stdout | 200 OK Jan 04, 2022 12:11:08 | host-stdout | Length: 5730735 (5.5M) [application/x-gzip] Jan 04, 2022 12:11:08 | host-stdout | Saving to: 'latest-linux-amd64.tar.gz.2' Jan 04, 2022 12:11:08 | host-stdout | 0% 0 --.-K/s Jan 04, 2022 12:11:08 | host-stdout | 100% 5,730,735 --.-K/s in 0.1s Jan 04, 2022 12:11:08 | host-stdout | 2022-01-04 12:11:08 (56.6 MB/s) - 'latest-linux-amd64.tar.gz.2' saved [5730735/5730735] Jan 04, 2022 12:11:08 | host-stdout | mysql_wsrep Jan 04, 2022 12:11:08 | galera-manager | Adding telegraf.conf Jan 04, 2022 12:11:08 | galera-manager | Adding mysql_wsrep-telegraf-plugin.conf Jan 04, 2022 12:11:08 | galera-manager | Saving Galera Manager node data Jan 04, 2022 12:11:08 | galera-manager | Creating filebeat configuration Jan 04, 2022 12:11:08 | galera-manager | Temporarily Disabling SELinux Jan 04, 2022 12:11:08 | host-stdout | 10.96.41.121:22$ setenforce Permissive Jan 04, 2022 12:11:08 | host-stdout | setenforce: SELinux is disabled Jan 04, 2022 12:11:08 | galera-manager | Got an error and attepts = 0 Jan 04, 2022 12:11:08 | galera-manager | failed to execute cluster config script (RunScriptWithConn): command failed (stepName=disable_selinux_temp, commandId=20, commandType=ExecCommand): Process exited with status 1 Jan 04, 2022 12:11:08 | galera-manager | Saving Galera Manager node data

ramtastic commented 2 years ago

Not a Linux expert here but I noticed that GM was attempting to set SELinux to permissive but failed - it being disabled. I manually set it to permissive and rebooted. Trying again succeeded in the deploy. I now have three nodes that claim "success" upon deploy but still cannot start from GM. I was able to start them from a PuTTY session. The individual systems do not recognize eachother. GM reports 0/3 joined/total nodes and Monitor view shows "No data"

denisgcm commented 2 years ago

Thank you for your report and troubleshooting. We'll fix the SELinux issue in the next release.

As for the problem between individual systems and "No data", this is most likely because you need to allow access from the nodes to the GM host (ports 80, 443, 9091 for ssl installations, ports 80, 8081 for non-ssl installations). Nodes constantly report to the metrics and logs database through which Galera Manager then reads all the node data.

Please let me know, if this information helps.

ramtastic commented 2 years ago

We did find some 8081 traffic being blocked and opened that up but I don't think that was our problem (yet). When the nodes deploy, I can see that MariaDB gets installed on the servers but they do not start from GM. I can start them individually from SSH but they don't run as a cluster. It seems that the Galera settings aren't all being configured on deploy. I saw issue #16 in GitHub and saw that the user had to add a mysqld_exporter user account. I manually started one of my DB servers and checked the user list and also didn't see that account listed.

denisgcm commented 2 years ago

Could you please log in via ssh to the galera manager host and run the following commands:

cd ~
# admin or if you replaced the user, then your gm user name
export GM_USER=admin
export GM_PASSWORD=your_gm_password
export CLUSTER_NAME=your_gm_cluster_name
gmc job list 1>gmd.log 2>&1
gmc cluster logs get-all --name=${CLUSTER_NAME} --save-to=cluster.log  1>>gmd.log 2>&1

You should get two files: cluster.log and gmd.log. Would you please send them to our email: info@galeracluster.com? We will analyze the output an come with the solution.

ramtastic commented 2 years ago

Logs sent. Thank you.

denisgcm commented 2 years ago

We received the logs and we are now examining them. I will get back to you soon.

denisgcm commented 2 years ago

We detected a minor issue in the logs, but it shouldn't affect the node connectivity.

There are several options why you might have this issue:

Also, you can try running the following command, which will show if the nodes can actually see each other:

gmc cluster status --name=your_cluster_name

If unsure about the results, please post them here.

ramtastic commented 2 years ago

I'll have my security folks double-check those ports. It will probably be next week. I can say, however, that I can run a Galera cluster successfully on those hosts when not deploying via Galera Manager so I believe they can communicate with each other. Running the cluster status command, I see all hosts "healthy" and all nodes "failed."
Could this be related to the version of MariaDB that's running? I noticed that regardless of what I select in the GM interface, I end up with MariaDB 10.6.5 installed but that's not an option in the GUI drop-down. I am running GUI: v1.4.3

denisgcm commented 2 years ago

We'll check the issue about the version of MariaDB being installed and I'll get back to you.

As of the failed status, it shouldn't be an issue (even with MariaDB 10.6, which is just currently not supported by the gui, but is supported by the backend). I believe, the problem is because the hosts don't see each other. Please let me know about the ports once your engineers check it.

ramtastic commented 2 years ago

Security team isn't seeing any blocks from the firewalls. I tore it down and tried again this morning. It looks the same but when I went to manually start the MariaDB services (outside of GM), I noticed that servers 1 & 3 started as a cluster, though server 2 would not. It started as stand-alone. Looking at the file structure, I found that servers 1 & 3 both have a 99.galera.cnf file in the /etc/mysql/wsrep/conf.d/ folder. Server 2 does not have such a file.

ramtastic commented 2 years ago

So, I manually created the 99.galera.cnf file (and edited the same on the other two to include this server's name) and had to edit the my.cnf file to match the other servers and now all three seem to be running as a functional cluster. For some reason, every time I try to deploy using GM, this one node doesn't get fully configured properly. For what it's worth, this node is also the only one reporting anything back to GM but is reporting CPU data only.

byte commented 2 years ago

So, I manually created the 99.galera.cnf file (and edited the same on the other two to include this server's name) and had to edit the my.cnf file to match the other servers and now all three seem to be running as a functional cluster. For some reason, every time I try to deploy using GM, this one node doesn't get fully configured properly. For what it's worth, this node is also the only one reporting anything back to GM but is reporting CPU data only.

This means it is not reporting data from mysql. Manual intervention should not be required in any way, so this is clearly a bug

waltervanrooyen commented 2 years ago

Has there been any more on this?

We are seeing similar results in that GM creates the nodes fine but never manages to get them to join the cluster.

halindrome commented 2 years ago

I have also run into a similar issue. In my case gmc cluster status reports that nodes and hosts are health, but the cluster has just not started up. This is using mariabd 10.3. What I noticed in the logs is that it is complaining that the ARIA storage engine is not available? I am going to re-attempt a build with 10.4

waltervanrooyen commented 2 years ago

Yeah we are using maraidb as well. However what doesnt make sense is that im pretty sure it shouldnt need ARIA anymore after 10.1