[SOLVED] Newly GaleraManager deployed cluster looses/drops/deletes database? (mariadb10.11,Ubuntu22.04.3)

[brianr-dk]

Hi

I finally managed - with GM to create a running cluster on Ubuntu22.04 and was all smiles. I have 3 nodes and a 4. node to run galera manager on.

Used the GM to deploy Mariadb10.11 on the 3 nodes. No problems. I didn't try educating GM on the settings for anything Galera or MariaDB, just "deploy" and run.

Created a "Test" database and stuffed some data in it. It was replicated to the other nodes. Still smiles.

Then, suddenly I started experiencing the stop/start/restart of the various nodes. And the "Test" database was gone.

Tried creating it again, and it suddenly deletes itself again.

I am not yet sure exactly WHAT restart makes the database go "puff", but I have tried (I think) every kind of restart sequences I dould think of.

Can anyone suggest what might cause this rather unfortunate feature in Galera? I haven't seen it before - and really do not need it ;)

Does GM include some parameter during deployment, unbeknownst to me, that makes new databases volatile :)

Best regards

[ayurchen]

Hi Brian, it appears that GM may drop specifically 'test' database in some circumstances. Could you try a different database name for now? E.g. 'test1'?

[ayurchen]

Could you elaborate, what you mean by

Then, suddenly I started experiencing the stop/start/restart of the various nodes.

Did that happen without your involvement?

[brianr-dk]

Ayurchen - no I am phrasing it wrong.

I myself decided to start testing how smart GM would be - getting cluster up again, so I started stopping service/rebooting servers to see how it acts up.

And ended up wondering why that "test" database would just vanish :)

Thanks for the heads up that this might be a GM "feature" somehow. I have no problem naming that dummy something else - even if it actually is just a test :D

[ayurchen]

Ok, then we need a sequence of your actions to reproduce it - to decide whether it is a bug or we just follow what mysql_secure_installation does.

[brianr-dk]

Hi

Reproducing should be easy:

VirtualBox: create 4 Ubuntu 22.04 nodes, freshly installed. Install your GM with 1.11.0 installer on node4.

Connect to GM on node4. Deploy MariaDB for Ubuntu22.04 (here that ended with MariaDB 10.11. Do so on all 3 nodes. No extra settings than 'deploy'

In MySQL on either node: Create database Test Create som table/data in 'Test' database.

Then you start experiencing with various restarts of db service/nodes from cmdline or GM.

I watched the 'Test' disappear multiple times and was sure I did something very wrong the first few times.

If the database is not called 'Test' it seems to survive.

I did not run any MySQL_secure and did not check if your deploy/recover does?

Thanks.

On Tue, 9 Jan 2024, 14.04 Alexey Yurchenko, @.***> wrote:

Ok, then we need a way to reproduce it - to decide whether it is a bug or we just follow what mysql_secure_installation does.

— Reply to this email directly, view it on GitHub https://github.com/codership/galera-manager-support/issues/84#issuecomment-1883029746, or unsubscribe https://github.com/notifications/unsubscribe-auth/AKVZ2CTZJRRQ7EGIEUJW7VTYNU56VAVCNFSM6AAAAABBS2DTE2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQOBTGAZDSNZUGY . You are receiving this because you authored the thread.Message ID: @.***>

[brianr-dk]

It seems that this is a "feature" with GaleraManager, that it somehow rips away any databases called "Test". Maybe you ought to warn about this, in a way that doesn't scare the crap out of us ignorant users :D

After I renamed my test database to "somethingelse", it is present even after multiple weird attempts to accidentally remove it, lihe what happened to the one called "test" :D

Galera - Feel free to explain why "test" needs to go away and when you ensure it does that. I can live with it, but even better when I know the rationale behind it.

And thanks anyway for an otherwise great tool!!

/Brian

[byte]

@brianr-dk can you please test the latest version (gm-installer 1.12.0, Galera Manager GUI 1.8.3). It should have fixed this for you, i.e. the "test" database does not get dropped.

[brianr-dk]

Thanks a bundle.

I'm having issues installing/updating GM, please see the following.

First is output from gm-installer install:

◈INFO[0008] add-apt-repository universe
INFO[0008] Executing: add-apt-repository universe
The installation log is located at /tmp/gm-installer.log
ERRO[0008] Failed to complete installation: command failed (stepName=add_apt_repository_universe, commandId=3, commandType=ExecCommand): exec: "add-apt-repository": executable file not found in $PATH

And this is from the /tmp/gminstaller.log file:

Hit:2 http://dk.archive.ubuntu.com/ubuntu jammy InRelease
Hit:3 https://repo.galera-manager.com/nexus/repository/galera-manager-release-1804 bionic InRelease
Hit:4 https://repos.influxdata.com/ubuntu jammy InRelease
Hit:5 http://dk.archive.ubuntu.com/ubuntu jammy-updates InRelease
Hit:6 http://dk.archive.ubuntu.com/ubuntu jammy-backports InRelease
Hit:7 http://dk.archive.ubuntu.com/ubuntu jammy-security InRelease
Hit:1 https://downloads.mariadb.com/MariaDB/mariadb-10.11/repo/ubuntu jammy InRelease
Reading package lists...
W: https://repo.galera-manager.com/nexus/repository/galera-manager-release-1804/dists/bionic/InRelease: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details.
W: https://repos.influxdata.com/ubuntu/dists/jammy/InRelease: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details.
W: http://downloads.mariadb.com/MariaDB/mariadb-10.11/repo/ubuntu/dists/jammy/InRelease: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details.

[byte]

Thanks a bundle.

I'm having issues installing/updating GM, please see the following.

First is output from gm-installer install:

◈INFO[0008] add-apt-repository universe
INFO[0008] Executing: add-apt-repository universe
The installation log is located at /tmp/gm-installer.log
ERRO[0008] Failed to complete installation: command failed (stepName=add_apt_repository_universe, commandId=3, commandType=ExecCommand): exec: "add-apt-repository": executable file not found in $PATH

And this is from the /tmp/gminstaller.log file:

Hit:2 http://dk.archive.ubuntu.com/ubuntu jammy InRelease
Hit:3 https://repo.galera-manager.com/nexus/repository/galera-manager-release-1804 bionic InRelease
Hit:4 https://repos.influxdata.com/ubuntu jammy InRelease
Hit:5 http://dk.archive.ubuntu.com/ubuntu jammy-updates InRelease
Hit:6 http://dk.archive.ubuntu.com/ubuntu jammy-backports InRelease
Hit:7 http://dk.archive.ubuntu.com/ubuntu jammy-security InRelease
Hit:1 https://downloads.mariadb.com/MariaDB/mariadb-10.11/repo/ubuntu jammy InRelease
Reading package lists...
W: https://repo.galera-manager.com/nexus/repository/galera-manager-release-1804/dists/bionic/InRelease: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details.
W: https://repos.influxdata.com/ubuntu/dists/jammy/InRelease: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details.
W: http://downloads.mariadb.com/MariaDB/mariadb-10.11/repo/ubuntu/dists/jammy/InRelease: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details.

worth noting that Galera Cluster itself changed the signing key, and this is what seems to be failing here. you have an old key.

can you start with a fresh install? there are of course ways to update keys, see: https://wiki.debian.org/SecureApt

new key announcement re: Galera Manager - https://galeracluster.com/2024/01/galera-manager-january-2024-release/ new key announcement in general for Galera Cluster - https://galeracluster.com/2024/01/galera-cluster-for-mysql-5-7-44-and-mysql-8-0-35-released/

[brianr-dk]

Thanks.

I see that somethings changed since my initial install, to the apt-key things in Ubuntu/Debian. I think I've got most of the issues sorted out now, but your install still fails:

Attempts to clean up:

apt remove galera-manager-filebeat galera-manager
apt purge galera-manager-filebeat galera-manager
apt-ket list
sudo rm -fR /var/lib/apt/lists/*

apt-key list |grep -C2 -i galera
apt-key del BC19DDBA
apt-key del 7DF8B07E
apt-key del c74CD1D8
apt-key del BC19DDBA

curl -fsSL https://repos.influxdata.com/influxdata-archive_compat.key |    gpg --dearmor |    sudo tee  /etc/apt/trusted.gpg.d/influxdata-archive_compat.gpg

sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 467B942D3A79BD29
apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 605C66F00D6C9793     0E98404D386FA1D9 648ACFD622F3D138 D669017EBC19DDBA
curl -LsS https://r.mariadb.com/downloads/mariadb_repo_setup | sudo bash -s --   --mariadb-server-version="mariadb-10.11" --os-type="ubuntu" --os-version="jammy"
apt update

./gm-installer install

root@ubuntu4:/sharedfolder# ./gm-installer install
INFO[0000] OS Detected: Debian / Ubuntu / Linux / jammy / 22.04
Previous installation detected. Do you want to reconfigure the application? [y/N] y
Change GMD package repository URL? [y/N] n
Change GMD package repository user? [y/N] n
Change GMD package repository password? [y/N] n
Change domain name or IP of the server? [y/N] n
By what domain name or an IP address this service will be reached?
(Note that an externally resolvable domain name is needed to use an external
Certificate Authority, otherwise we will have to resort to self-signed
certificates for SSL if encryption is required):
Since you entered an IP address, SSL won't be available.
INFO[0008] Stopping gmd service
INFO[0008] Executing: systemctl stop gmd
▁INFO[0008] Stopping nginx service
INFO[0008] Executing: systemctl stop nginx
INFO[0008] Stopping InfluxDB service
INFO[0008] Executing: systemctl stop influxdb
INFO[0008] Updating the package index
INFO[0008] Executing: apt-get update
▃INFO[0011] add-apt-repository universe
INFO[0011] Executing: add-apt-repository universe
The installation log is located at /tmp/gm-installer.log
ERRO[0011] Failed to complete installation: command failed (stepName=add_apt_repository_universe, commandId=3, commandType=ExecCommand): exec: "add-apt-repository": executable file not found in $PATH

root@ubuntu4:/sharedfolder# view /tmp/gm-installer.log
Failed to stop gmd.service: Unit gmd.service not loaded.
Hit:1 http://dk.archive.ubuntu.com/ubuntu jammy InRelease
Hit:2 http://dk.archive.ubuntu.com/ubuntu jammy-updates InRelease
Hit:3 http://dk.archive.ubuntu.com/ubuntu jammy-backports InRelease
Hit:4 http://dk.archive.ubuntu.com/ubuntu jammy-security InRelease
Hit:6 https://repos.influxdata.com/ubuntu jammy InRelease
Hit:5 https://downloads.mariadb.com/MariaDB/mariadb-10.11/repo/ubuntu jammy InRelease
Hit:7 https://downloads.mariadb.com/Tools/ubuntu jammy InRelease
Get:8 https://dlm.mariadb.com/repo/mariadb-server/10.11/repo/ubuntu jammy InRelease [7767 B]
Get:9 https://dlm.mariadb.com/repo/maxscale/latest/apt jammy InRelease [9344 B]
Fetched 17.1 kB in 1s (12.3 kB/s)
Reading package lists...

Oh, my apt is set up like this - also "as is" from the initial install from a fresh Ubuntu22 download.

root@ubuntu4:/sharedfolder# cat /etc/apt/sources.list|grep -v ^#
deb http://dk.archive.ubuntu.com/ubuntu jammy main restricted
deb http://dk.archive.ubuntu.com/ubuntu jammy-updates main restricted
deb http://dk.archive.ubuntu.com/ubuntu jammy universe
deb http://dk.archive.ubuntu.com/ubuntu jammy-updates universe
deb http://dk.archive.ubuntu.com/ubuntu jammy multiverse
deb http://dk.archive.ubuntu.com/ubuntu jammy-updates multiverse
deb http://dk.archive.ubuntu.com/ubuntu jammy-backports main restricted universe multiverse
deb http://dk.archive.ubuntu.com/ubuntu jammy-security main restricted
deb http://dk.archive.ubuntu.com/ubuntu jammy-security universe
deb http://dk.archive.ubuntu.com/ubuntu jammy-security multiverse

So, still no luck really :)

[brianr-dk]

Solved.

It looks as if you've created an installer with the following requirement on at least Ubuntu 22? :

sudo apt-get install software-properties-common

with that added, gm-installer doesn't complain about "ERRO[0011] Failed to complete installation: command failed (stepName=add_apt_repository_universe, commandId=3, commandType=ExecCommand): exec: "add-apt-repository": executable file not found in $PATH"

[brianr-dk]

Forgot to close :)