mariaDB crashing - Githubissues

Joulinar commented 4 months ago

Discussed in https://github.com/MichaIng/DietPi/discussions/7158

^{Originally posted by **pedrom34** July 13, 2024} Hey everyone! Since latest dietpi update, mariadb won't start on one of my SBC (the other is fine). Can you give a hint on how to debug this? google won't help as I don't know if the answers I found are related to my issue... Here's dietpi-services status command. ``` [FAILED] DietPi-Services | × mariadb.service - MariaDB 10.11.6 database server Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; preset: enabled) Drop-In: /etc/systemd/system/mariadb.service.d └─dietpi.conf Active: failed (Result: exit-code) since Sat 2024-07-13 06:11:53 CEST; 1h 8min ago Docs: man:mariadbd(8) https://mariadb.com/kb/en/library/systemd/ Process: 7436 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (code=exited, status=0/SUCCESS) Process: 7437 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS) Process: 7439 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= || VAR=`cd /usr/bin/..; /usr/bin/galera_recovery`; [ $? -eq 0 ] && systemctl set-environment _WSREP_START_POSITION=$VAR || exit 1 (code=exited, status=0/SUCCESS) Process: 7492 ExecStart=/usr/sbin/mariadbd $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WSREP_START_POSITION (code=exited, status=1/FAILURE) Main PID: 7492 (code=exited, status=1/FAILURE) Status: "MariaDB server is down" CPU: 699ms Jul 13 06:11:53 DietPi2 mariadbd[7492]: 2024-07-13 6:11:53 0 [Note] InnoDB: Retry with innodb_force_recovery=5 Jul 13 06:11:53 DietPi2 mariadbd[7492]: 2024-07-13 6:11:53 0 [ERROR] InnoDB: Plugin initialization aborted with error Page read from tablespace is corrupted. Jul 13 06:11:53 DietPi2 mariadbd[7492]: 2024-07-13 6:11:53 0 [Note] InnoDB: Starting shutdown... Jul 13 06:11:53 DietPi2 mariadbd[7492]: 2024-07-13 6:11:53 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed. Jul 13 06:11:53 DietPi2 mariadbd[7492]: 2024-07-13 6:11:53 0 [Note] Plugin 'FEEDBACK' is disabled. Jul 13 06:11:53 DietPi2 mariadbd[7492]: 2024-07-13 6:11:53 0 [ERROR] Unknown/unsupported storage engine: InnoDB Jul 13 06:11:53 DietPi2 mariadbd[7492]: 2024-07-13 6:11:53 0 [ERROR] Aborting Jul 13 06:11:53 DietPi2 systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE Jul 13 06:11:53 DietPi2 systemd[1]: mariadb.service: Failed with result 'exit-code'. Jul 13 06:11:53 DietPi2 systemd[1]: Failed to start mariadb.service - MariaDB 10.11.6 database server. ``` Please tell me it's not a disk issue. I'm at 800km from my server right now...

Joulinar commented 4 months ago

@pedrom34 moved your topic into issue section

can you share following

journalctl -u mariadb
cat /var/log/mysql/error.log
readlink /var/lib/mysql
readlink -f /var/lib/mysql

pedrom34 commented 4 months ago

Thanks. Here are the results of the commands:

cat /var/log/mysql/error.log

cat: /var/log/mysql/error.log: no such file or folder (translated from french)

readlink /var/lib/mysql

/mnt/dietpi_userdata/mysql

readlink -f /var/lib/mysql

/mnt/129253b6-b521-42d9-b8d8-d9fb5f212d28/dietpi_userdata/mysql

journalctl -u mariadb

juil. 13 06:11:52 DietPi2 systemd[1]: Starting mariadb.service - MariaDB 10.11.6 database server...
juil. 13 06:11:52 DietPi2 mariadbd[7492]: /usr/sbin/mariadbd: Can't create file '/var/log/mysql/error.log' (errno: 2 "No such file or directory")
juil. 13 06:11:52 DietPi2 mariadbd[7492]: 2024-07-13  6:11:52 0 [Note] Starting MariaDB 10.11.6-MariaDB-0+deb12u1 source revision  as process 7492
juil. 13 06:11:52 DietPi2 mariadbd[7492]: 2024-07-13  6:11:52 0 [Note] InnoDB: Compressed tables use zlib 1.2.13      
juil. 13 06:11:52 DietPi2 mariadbd[7492]: 2024-07-13  6:11:52 0 [Note] InnoDB: Number of transaction pools: 1         
juil. 13 06:11:52 DietPi2 mariadbd[7492]: 2024-07-13  6:11:52 0 [Note] InnoDB: Using ARMv8 crc32 instructions         
juil. 13 06:11:52 DietPi2 mariadbd[7492]: 2024-07-13  6:11:52 0 [Note] InnoDB: Using liburing
juil. 13 06:11:52 DietPi2 mariadbd[7492]: 2024-07-13  6:11:52 0 [Note] InnoDB: Initializing buffer pool, total size = 128.000MiB, chunk size = 2.000MiB
juil. 13 06:11:52 DietPi2 mariadbd[7492]: 2024-07-13  6:11:52 0 [Note] InnoDB: Completed initialization of buffer pooljuil. 13 06:11:52 DietPi2 mariadbd[7492]: 2024-07-13  6:11:52 0 [Note] InnoDB: File system buffers for log disabled (block size=512 bytes)
juil. 13 06:11:52 DietPi2 mariadbd[7492]: 2024-07-13  6:11:52 0 [Note] InnoDB: End of log at LSN=6007983
juil. 13 06:11:53 DietPi2 mariadbd[7492]: 2024-07-13  6:11:53 0 [ERROR] InnoDB: Database page corruption on disk or a failed read of file './ibdata1' page [page id: space=0, page number=293]. You may have to recover from a backup.      
juil. 13 06:11:53 DietPi2 mariadbd[7492]: 2024-07-13  6:11:53 0 [Note] InnoDB: Page dump (16384 bytes):
juil. 13 06:11:53 DietPi2 mariadbd[7492]: 2024-07-13  6:11:53 0 [Note] InnoDB: 0000000000000125ffffffffffffffff00000000004e51a70006000000000000
juil. 13 06:11:53 DietPi2 mariadbd[7492]: 2024-07-13  6:11:53 0 [Note] InnoDB: 000000000000000000000000000000000000ffffffff0000ffffffff00000000
juil. 13 06:11:53 DietPi2 mariadbd[7492]: 2024-07-13  6:11:53 0 [Note] InnoDB: 0000000000f324f2ffffffffffffffffffffffffffffffffffffffffffffffff
juil. 13 06:11:53 DietPi2 mariadbd[7492]: 2024-07-13  6:11:53 0 [Note] InnoDB: ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff
juil. 13 06:11:53 DietPi2 mariadbd[7492]: 2024-07-13  6:11:53 0 [Note] InnoDB: ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff
juil. 13 06:11:53 DietPi2 mariadbd[7492]: 2024-07-13  6:11:53 0 [Note] InnoDB: ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff
juil. 13 06:11:53 DietPi2 mariadbd[7492]: 2024-07-13  6:11:53 0 [Note] InnoDB: ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff
juil. 13 06:11:53 DietPi2 mariadbd[7492]: 2024-07-13  6:11:53 0 [Note] InnoDB: ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff
juil. 13 06:11:53 DietPi2 mariadbd[7492]: 2024-07-13  6:11:53 0 [Note] InnoDB: ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff
juil. 13 06:11:53 DietPi2 mariadbd[7492]: 2024-07-13  6:11:53 0 [Note] InnoDB: ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff
juil. 13 06:11:53 DietPi2 mariadbd[7492]: 2024-07-13  6:11:53 0 [Note] InnoDB: ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff
juil. 13 06:11:53 DietPi2 mariadbd[7492]: 2024-07-13  6:11:53 0 [Note] InnoDB: ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff
juil. 13 06:11:53 DietPi2 mariadbd[7492]: 2024-07-13  6:11:53 0 [Note] InnoDB: ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff
juil. 13 06:11:53 DietPi2 mariadbd[7492]: 2024-07-13  6:11:53 0 [Note] InnoDB: ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff
juil. 13 06:11:53 DietPi2 mariadbd[7492]: 2024-07-13  6:11:53 0 [Note] InnoDB: ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff
juil. 13 06:11:53 DietPi2 mariadbd[7492]: 2024-07-13  6:11:53 0 [Note] InnoDB: ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff
juil. 13 06:11:53 DietPi2 mariadbd[7492]: 2024-07-13  6:11:53 0 [Note] InnoDB: 0000000000000000000000000000000000000000000000000000000000000000
juil. 13 06:11:53 DietPi2 mariadbd[7492]: 2024-07-13  6:11:53 0 [Note] InnoDB: 0000000000000000000000000000000000000000000000000000000000000000
juil. 13 06:11:53 DietPi2 mariadbd[7492]: 2024-07-13  6:11:53 0 [Note] InnoDB: 0000000000000000000000000000000000000000000000000000000000000000
juil. 13 06:11:53 DietPi2 mariadbd[7492]: 2024-07-13  6:11:53 0 [Note] InnoDB: 0000000000000000000000000000000000000000000000000000000000000000
juil. 13 06:11:53 DietPi2 mariadbd[7492]: 2024-07-13  6:11:53 0 [Note] InnoDB: 0000000000000000000000000000000000000000000000000000000000000000
juil. 13 06:11:53 DietPi2 mariadbd[7492]: 2024-07-13  6:11:53 0 [Note] InnoDB: 0000000000000000000000000000000000000000000000000000000000000000
juil. 13 06:11:53 DietPi2 mariadbd[7492]: 2024-07-13  6:11:53 0 [Note] InnoDB: 0000000000000000000000000000000000000000000000000000000000000000
juil. 13 06:11:53 DietPi2 mariadbd[7492]: 2024-07-13  6:11:53 0 [Note] InnoDB: 000000000000000000000000000000000000000000000000004e51a7012fde99
juil. 13 06:11:53 DietPi2 mariadbd[7492]: 2024-07-13  6:11:53 0 [Note] InnoDB: End of page dump
juil. 13 06:11:53 DietPi2 mariadbd[7492]: 2024-07-13  6:11:53 0 [ERROR] InnoDB: File './ibdata1' is corrupted         
juil. 13 06:11:53 DietPi2 mariadbd[7492]: 2024-07-13  6:11:53 0 [Note] InnoDB:  You can use CHECK TABLE to scan your table for corruption. Please refer to https://mariadb.com/kb/en/library/innodb-recovery-modes/ for information about forcing recovery.
juil. 13 06:11:53 DietPi2 mariadbd[7492]: 2024-07-13  6:11:53 0 [Note] InnoDB: Retry with innodb_force_recovery=5     
juil. 13 06:11:53 DietPi2 mariadbd[7492]: 2024-07-13  6:11:53 0 [ERROR] InnoDB: Plugin initialization aborted with error Page read from tablespace is corrupted.
juil. 13 06:11:53 DietPi2 mariadbd[7492]: 2024-07-13  6:11:53 0 [Note] InnoDB: Starting shutdown...
juil. 13 06:11:53 DietPi2 mariadbd[7492]: 2024-07-13  6:11:53 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
juil. 13 06:11:53 DietPi2 mariadbd[7492]: 2024-07-13  6:11:53 0 [Note] Plugin 'FEEDBACK' is disabled.
juil. 13 06:11:53 DietPi2 mariadbd[7492]: 2024-07-13  6:11:53 0 [ERROR] Unknown/unsupported storage engine: InnoDB    
juil. 13 06:11:53 DietPi2 mariadbd[7492]: 2024-07-13  6:11:53 0 [ERROR] Aborting
juil. 13 06:11:53 DietPi2 systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
juil. 13 06:11:53 DietPi2 systemd[1]: mariadb.service: Failed with result 'exit-code'.
juil. 13 06:11:53 DietPi2 systemd[1]: Failed to start mariadb.service - MariaDB 10.11.6 database server.

Joulinar commented 4 months ago

database file seems to be corrupted. Can you check for kernel error messages

dmesg -l 0,1,2,3

pedrom34 commented 4 months ago

[    0.966015] genirq: Setting trigger mode 3 for irq 28 failed (meson_gpio_irq_set_type+0x0/0x60)
[    4.226864] lima d00c0000.gpu: error -ENODEV: _opp_set_regulators: no regulator (mali) found
[    4.293319] hub 1-1:1.0: Using single TT (err -71)
[    4.293368] dwc2 c9100000.usb: Not connected
[    4.293383] hub 1-1:1.0: config failed, can't get hub status (err -5)
[   28.522901] critical medium error, dev sda, sector 38250984 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 2
[ 5306.965709] critical medium error, dev sda, sector 33554808 op 0x0:(READ) flags 0x80700 phys_seg 6 prio class 2
[ 6806.998118] critical medium error, dev sda, sector 33554744 op 0x0:(READ) flags 0x80700 phys_seg 18 prio class 2

A lot of redflags, right?

MichaIng commented 4 months ago

At least the drive seems to have some bad blocks. Is this an SSD, HDD or flash drive?

Joulinar commented 4 months ago

What type of SBC you are using and how is your external drive powered?

pedrom34 commented 4 months ago

Odroid C2. System is on eMMC, userdata is on an USB flashdrive.

MichaIng commented 4 months ago

If you have another flash drive at hand, please try to move the userdata over to that one. It might fail when it runs into I/O errors while reading the affected blocks, but worth an attempt before we try to repair the filesystem and mask the bad blocks, and serves as backup as well.

Then try to repair the filesystem with fsck. Can be done with dietpi-drive_manager as well.

pedrom34 commented 4 months ago

Thanks, I'll try as soon as I have physical access to the server. I'll update this issue, but it doesn't seem dietpi related, right?

Joulinar commented 4 months ago

Usually it's not.

pedrom34 commented 3 months ago

Sorry for the delay, I just came back from vacation, bought a new usb drive, and did what @MichaIng told me. Unfortunately it doesn't solve the issue, as mariaDB refuse to start:

dietpi-services status:

[FAILED] DietPi-Services | × mariadb.service - MariaDB 10.11.6 database server
     Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; preset: enabled)
    Drop-In: /etc/systemd/system/mariadb.service.d
             └─dietpi.conf
     Active: failed (Result: exit-code) since Sat 2024-08-03 14:26:17 CEST; 1min 51s ago
       Docs: man:mariadbd(8)
             https://mariadb.com/kb/en/library/systemd/
    Process: 18077 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (code=exited, status=0/SUCCESS)
    Process: 18078 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
    Process: 18080 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= ||   VAR=`cd /usr/bin/..; /usr/bin/galera_recovery`; [ $? -eq 0 ]   && systemctl set-environment _WSREP_START_POSITION=$VAR || exit 1 (code=exited, status=0/SUCCESS)
    Process: 18133 ExecStart=/usr/sbin/mariadbd $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WSREP_START_POSITION (code=exited, status=1/FAILURE)
   Main PID: 18133 (code=exited, status=1/FAILURE)
     Status: "MariaDB server is down"
        CPU: 605ms

Aug 03 14:26:17 DietPi2 systemd[1]: Starting mariadb.service - MariaDB 10.11.6 database server...
Aug 03 14:26:17 DietPi2 mariadbd[18133]: 2024-08-03 14:26:17 0 [Warning] Can't create test file '/var/lib/mysql/DietPi2.lower-test' (Errcode: 13 "Permission denied")
Aug 03 14:26:17 DietPi2 mariadbd[18133]: [92B blob data]
Aug 03 14:26:17 DietPi2 mariadbd[18133]: 2024-08-03 14:26:17 0 [ERROR] Aborting
Aug 03 14:26:17 DietPi2 systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
Aug 03 14:26:17 DietPi2 systemd[1]: mariadb.service: Failed with result 'exit-code'.
Aug 03 14:26:17 DietPi2 systemd[1]: Failed to start mariadb.service - MariaDB 10.11.6 database server.

And I cannot uninstall/reinstall as dietpi-software tries to backup the DB and fails.

Can you help me with that? I can drop the DB, I don't really care.

Joulinar commented 3 months ago

The issue changed.

Can't create test file '/var/lib/mysql/DietPi2.lower-test' (Errcode: 13 "Permission denied") Aug 03 14:26:17 DietPi2 mariadbd[18133]: [92B blob data]

Now it's a permission issue on your new drive.

Can you share following

ls -la /var/lib/mysql/

pedrom34 commented 3 months ago

ls -la /var/lib/mysql/

total 176632
drwx------  7 root   root        4096 13 juil. 20:08 .
drwxr-xr-x 10 dietpi dietpi      4096 13 juil. 20:00 ..
-rwx------  1 root   root      458752 13 juil. 06:11 aria_log.00000001
-rwx------  1 root   root          52 13 juil. 06:11 aria_log_control
drwx------  2 root   root        4096 13 juil. 20:08 authelia
-rwx------  1 root   root           9 28 mai   18:17 ddl_recovery-backup.log
-rwx------  1 root   root           0 12 févr. 20:17 debian-10.11.flag
-rwx------  1 root   root        2102  9 juil. 08:53 ib_buffer_pool
-rwx------  1 root   root    79691776  9 juil. 08:53 ibdata1
-rwx------  1 root   root   100663296  9 juil. 08:53 ib_logfile0
-rwx------  1 root   root           0 31 août   2023 multi-master.info
drwx------  2 root   root        4096 13 juil. 20:08 mysql
-rwx------  1 root   root          15 31 août   2023 mysql_upgrade_info
drwx------  2 root   root        4096 13 juil. 20:08 performance_schema
drwx------  2 root   root        4096 13 juil. 20:08 phpmyadmin
drwx------  2 root   root       12288 13 juil. 20:08 sys

Joulinar commented 3 months ago

permissions are incorrectly set. How did you copy these files? Or was this a fresh install?

Usually, it should like this


root@DietPiVM:~# ls -la /var/lib/mysql/
total 123332
drwxr-xr-x 5 mysql  mysql       4096 Aug  4 11:51 .
drwxrwxr-x 7 dietpi dietpi      4096 Aug  4 11:34 ..
-rw-rw---- 1 mysql  mysql     417792 Aug  4 11:51 aria_log.00000001
-rw-rw---- 1 mysql  mysql         52 Aug  4 11:51 aria_log_control
-rw-rw---- 1 mysql  mysql          9 Aug  4 11:51 ddl_recovery.log
-rw-r--r-- 1 root   root           0 Aug  4 11:34 debian-10.11.flag
-rw-rw---- 1 mysql  mysql        910 Aug  4 11:51 ib_buffer_pool
-rw-rw---- 1 mysql  mysql  100663296 Aug  4 11:34 ib_logfile0
-rw-rw---- 1 mysql  mysql   12582912 Aug  4 11:34 ibdata1
-rw-rw---- 1 mysql  mysql   12582912 Aug  4 11:51 ibtmp1
-rw-rw---- 1 mysql  mysql          0 Aug  4 11:34 multi-master.info
drwx------ 2 mysql  mysql       4096 Aug  4 11:34 mysql
-rw-r--r-- 1 root   root          15 Aug  4 11:34 mysql_upgrade_info
drwx------ 2 mysql  mysql       4096 Aug  4 11:34 performance_schema
drwx------ 2 mysql  mysql      12288 Aug  4 11:34 sys
root@DietPiVM:~#

pedrom34 commented 3 months ago

Well, the copy was made by the dietpi utility to move dietpi_userdata.

I'll adjust permissions and test.

-------- Message d'origine -------- Le 04/08/2024 12:56, Joulinar a écrit :

permissions are incorrectly set. How did you copy these files? Or was this a fresh install?

Usually, it should like this

@.:~# ls -la /var/lib/mysql/ total 123332 drwxr-xr-x 5 mysql mysql 4096 Aug 4 11:51 . drwxrwxr-x 7 dietpi dietpi 4096 Aug 4 11:34 .. -rw-rw---- 1 mysql mysql 417792 Aug 4 11:51 aria_log.00000001 -rw-rw---- 1 mysql mysql 52 Aug 4 11:51 aria_log_control -rw-rw---- 1 mysql mysql 9 Aug 4 11:51 ddl_recovery.log -rw-r--r-- 1 root root 0 Aug 4 11:34 debian-10.11.flag -rw-rw---- 1 mysql mysql 910 Aug 4 11:51 ib_buffer_pool -rw-rw---- 1 mysql mysql 100663296 Aug 4 11:34 ib_logfile0 -rw-rw---- 1 mysql mysql 12582912 Aug 4 11:34 ibdata1 -rw-rw---- 1 mysql mysql 12582912 Aug 4 11:51 ibtmp1 -rw-rw---- 1 mysql mysql 0 Aug 4 11:34 multi-master.info drwx------ 2 mysql mysql 4096 Aug 4 11:34 mysql -rw-r--r-- 1 root root 15 Aug 4 11:34 mysql_upgrade_info drwx------ 2 mysql mysql 4096 Aug 4 11:34 performance_schema drwx------ 2 mysql mysql 12288 Aug 4 11:34 sys @.:~#

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.***>

pedrom34 commented 3 months ago

Ok, I managed to run mariaDB again by adjusting permissions as per your install, and by restoring an old backup of the 3 roles_mapping files, because it seems the files were damaged.

So it was a usb key issue all this time. Thanks a lot for your time 👍

MichaIng / DietPi

mariaDB crashing #7159

Discussed in https://github.com/MichaIng/DietPi/discussions/7158