11.0 - MDEV-31443 InnoDB: Unable to find charset-collation for X

[MGREMY]

Hello,

My docker auto updated mariadb an hour ago and I had some logs that show me the server is not starting properly.. I have to set the tag on "10.11.3" or the server is not starting.

Here is the log : 2023-06-09 23:26:52+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:11.0.2+maria~ubu2204 started. 2023-06-09 23:26:52+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql' 2023-06-09 23:26:52+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:11.0.2+maria~ubu2204 started. 2023-06-09 23:26:52+00:00 [Note] [Entrypoint]: MariaDB upgrade information missing, assuming required 2023-06-09 23:26:52+00:00 [Note] [Entrypoint]: MariaDB upgrade (mariadb-upgrade) required, but skipped due to $MARIADB_AUTO_UPGRADE setting 2023-06-09 23:26:52 0 [Note] Starting MariaDB 11.0.2-MariaDB-1:11.0.2+maria~ubu2204 source revision 0005f2f06c8e1aea4915887decad67885108a929 as process 1 2023-06-09 23:26:53 0 [Note] InnoDB: Compressed tables use zlib 1.2.11 2023-06-09 23:26:53 0 [Note] InnoDB: Number of transaction pools: 1 2023-06-09 23:26:53 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions 2023-06-09 23:26:53 0 [Note] mariadbd: O_TMPFILE is not supported on /tmp (disabling future attempts) 2023-06-09 23:26:53 0 [Note] InnoDB: Using liburing 2023-06-09 23:26:53 0 [Note] InnoDB: Initializing buffer pool, total size = 128.000MiB, chunk size = 2.000MiB 2023-06-09 23:26:53 0 [Note] InnoDB: Completed initialization of buffer pool 2023-06-09 23:26:53 0 [Note] InnoDB: Buffered log writes (block size=512 bytes) 2023-06-09 23:26:53 0 [Note] InnoDB: Upgrading the change buffer 2023-06-09 23:26:53 0 [ERROR] [FATAL] InnoDB: Unable to find charset-collation for 15 230609 23:26:53 [ERROR] mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 11.0.2-MariaDB-1:11.0.2+maria~ubu2204 source revision: 0005f2f06c8e1aea4915887decad67885108a929 key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=0 max_threads=153 thread_count=0 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 468023 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x0 thread_stack 0x49000 Printing to addr2line failed mariadbd(my_print_stacktrace+0x32)[0x55a74e557a62] mariadbd(handle_fatal_signal+0x488)[0x55a74e02ae28] /lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7f8a77372520] /lib/x86_64-linux-gnu/libc.so.6(pthread_kill+0x12c)[0x7f8a773c6a7c] /lib/x86_64-linux-gnu/libc.so.6(raise+0x16)[0x7f8a77372476] /lib/x86_64-linux-gnu/libc.so.6(abort+0xd3)[0x7f8a773587f3] mariadbd(+0x69753b)[0x55a74dc3f53b] mariadbd(+0x6812f7)[0x55a74dc292f7] mariadbd(+0xdf703f)[0x55a74e39f03f] mariadbd(+0xdd3c92)[0x55a74e37bc92] mariadbd(+0x6b2ebc)[0x55a74dc5aebc] mariadbd(+0x691449)[0x55a74dc39449] mariadbd(+0xd7aad9)[0x55a74e322ad9] mariadbd(_Z24ha_initialize_handlertonP13st_plugin_int+0x86)[0x55a74e02e0b6] mariadbd(+0x83d686)[0x55a74dde5686] mariadbd(_Z11plugin_initPiPPci+0x91d)[0x55a74dde684d] mariadbd(+0x70bb91)[0x55a74dcb3b91] mariadbd(_Z11mysqld_mainiPPc+0x424)[0x55a74dcb9324] /lib/x86_64-linux-gnu/libc.so.6(+0x29d90)[0x7f8a77359d90] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80)[0x7f8a77359e40] mariadbd(_start+0x25)[0x55a74dcadb05] The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /var/lib/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size unlimited unlimited bytes Max resident set unlimited unlimited bytes Max processes unlimited unlimited processes Max open files 1048576 1048576 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 62450 62450 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us Core pattern: core Kernel version: Linux version 5.10.0-21-cloud-amd64 (debian-kernel@lists.debian.org) (gcc-10 (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2) #1 SMP Debian 5.10.162-1 (2023-01-21) Fatal signal 11 while backtracing

[grooverdan]

Hi, I found a reported bug MDEV-31443 with the same stack trace as you have.

Are you willing temporary shutdown your 10.11 instance (and make a copy to be 100% safe) and run the following in docker/podman/k8s and record the full output:

podman run --rm  -v m10.11:/var/lib/mysql --name m11.0 --user mysql quay.io/mariadb-foundation/mariadb-debug:11.0 gdb  --ex r --ex 'bt -frame-arguments all full' --args mariadbd

Are you willing to share a datadir to resolve this?

note there is now a :lts tag to follow the last LTS release.

[kyoshiro]

Ran into the same problem. Using 'mariadb:latest'. which points to 'mariadb:11.0.2'. Going down to 10.11.3 works fine as mentioned by @MGREMY .

Error message was: 2023-06-13 18:31:21 0 [ERROR] [FATAL] InnoDB: Unable to find charset-collation for 15

I run docker on arch: Docker version 24.0.2

[MGREMY]

Hello @grooverdan Sorry for the late answer... Here are the logs for the command you asked me :

GNU gdb (Ubuntu 12.1-0ubuntu1~22.04) 12.1
Copyright (C) 2022 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from mariadbd...
Reading symbols from /usr/lib/debug/.build-id/41/d59f933f8e2b57f9de1bee36e781917f26e828.debug...
Starting program: /usr/sbin/mariadbd
warning: Error disabling address space randomization: Operation not permitted
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
2023-06-19 19:03:27 0 [Warning] Can't create test file /var/lib/mysql/2bbd18f4e5dd.lower-test
2023-06-19 19:03:27 0 [Note] Starting MariaDB 11.0.3-MariaDB-1:11.0.3+maria~ubu2204 source revision f25a74c0b094f23e1600ed9499a81420cfab663b as process 12
[New Thread 0x7fa3079df640 (LWP 15)]
2023-06-19 19:03:27 0 [ERROR] mariadbd: Can't create/write to file './ddl_recovery.log' (Errcode: 13 "Permission denied")
2023-06-19 19:03:27 0 [ERROR] DDL_LOG: Failed to create ddl log file: ./ddl_recovery.log
2023-06-19 19:03:27 0 [ERROR] Aborting
[Thread 0x7fa3079df640 (LWP 15) exited]
[Inferior 1 (process 12) exited with code 01]
No stack.
(gdb) quit

For the DataDir, I know it can help a lot but this is my nextcloud database storage, and I don't want to share this sorry...

[grooverdan]

@MGREMY thanks for trying a backtrace. The datadir permissions need fixing before attempting to start:

podman run --rm  -v m10.11:/var/lib/mysql --name m11.0 quay.io/mariadb-foundation/mariadb-debug:11.0 chown -R mysql: /var/lib/mysql

This should reset them all back to the mysql user as it appears within the container.

Ensure the gdb run includes --user mysql as the container runtime option.

No worries about not sharing data, its always a tough ask and I understand why.

[detece]

Ran into the same problem. Using 'mariadb:latest'. which points to 'mariadb:11.0.2'. Going down to 10.11.3 works fine as mentioned by @MGREMY .

Error message was: 2023-06-13 18:31:21 0 [ERROR] [FATAL] InnoDB: Unable to find charset-collation for 15

I run docker on arch: Docker version 24.0.2

Hello, Same probleme here, complete logs below :

2023-06-20 07:55:10+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:11.0.2+maria~ubu2204 started. 2023-06-20 07:55:11+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql' 2023-06-20 07:55:11+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:11.0.2+maria~ubu2204 started. 2023-06-20 07:55:11+00:00 [Note] [Entrypoint]: MariaDB upgrade information missing, assuming required 2023-06-20 07:55:11+00:00 [Note] [Entrypoint]: Starting temporary server 2023-06-20 07:55:11+00:00 [Note] [Entrypoint]: Waiting for server startup 2023-06-20 7:55:11 0 [Note] Starting MariaDB 11.0.2-MariaDB-1:11.0.2+maria~ubu2204 source revision 0005f2f06c8e1aea4915887decad67885108a929 as process 47 2023-06-20 7:55:11 0 [Note] InnoDB: Compressed tables use zlib 1.2.11 2023-06-20 7:55:11 0 [Note] InnoDB: Number of transaction pools: 1 2023-06-20 7:55:11 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions 2023-06-20 7:55:11 0 [Note] mariadbd: O_TMPFILE is not supported on /tmp (disabling future attempts) 2023-06-20 7:55:11 0 [Note] InnoDB: Using liburing 2023-06-20 7:55:11 0 [Note] InnoDB: Initializing buffer pool, total size = 128.000MiB, chunk size = 2.000MiB 2023-06-20 7:55:11 0 [Note] InnoDB: Completed initialization of buffer pool 2023-06-20 7:55:11 0 [Note] InnoDB: File system buffers for log disabled (block size=4096 bytes) 2023-06-20 7:55:11 0 [Note] InnoDB: Upgrading the change buffer 2023-06-20 7:55:11 0 [ERROR] [FATAL] InnoDB: Unable to find charset-collation for 15 230620 7:55:11 [ERROR] mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see https://mariadb.com/kb/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail.

Server version: 11.0.2-MariaDB-1:11.0.2+maria~ubu2204 source revision: 0005f2f06c8e1aea4915887decad67885108a929 key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=0 max_threads=153 thread_count=0 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 468023 K bytes of memory Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x0 thread_stack 0x49000 Printing to addr2line failed mariadbd(my_print_stacktrace+0x32)[0x55fdf5008a62] mariadbd(handle_fatal_signal+0x488)[0x55fdf4adbe28] /lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7fdb04296520] /lib/x86_64-linux-gnu/libc.so.6(pthread_kill+0x12c)[0x7fdb042eaa7c] /lib/x86_64-linux-gnu/libc.so.6(raise+0x16)[0x7fdb04296476] /lib/x86_64-linux-gnu/libc.so.6(abort+0xd3)[0x7fdb0427c7f3] mariadbd(+0x69753b)[0x55fdf46f053b] mariadbd(+0x6812f7)[0x55fdf46da2f7] mariadbd(+0xdf703f)[0x55fdf4e5003f] mariadbd(+0xdd3c92)[0x55fdf4e2cc92] mariadbd(+0x6b2ebc)[0x55fdf470bebc] mariadbd(+0x691449)[0x55fdf46ea449] mariadbd(+0xd7aad9)[0x55fdf4dd3ad9] mariadbd(_Z24ha_initialize_handlertonP13st_plugin_int+0x86)[0x55fdf4adf0b6] mariadbd(+0x83d686)[0x55fdf4896686] mariadbd(_Z11plugin_initPiPPci+0x91d)[0x55fdf489784d] mariadbd(+0x70bb91)[0x55fdf4764b91] mariadbd(_Z11mysqld_mainiPPc+0x424)[0x55fdf476a324] /lib/x86_64-linux-gnu/libc.so.6(+0x29d90)[0x7fdb0427dd90] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80)[0x7fdb0427de40] mariadbd(_start+0x25)[0x55fdf475eb05] The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /var/lib/mysql Resource Limits: Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size unlimited unlimited bytes
Max resident set unlimited unlimited bytes
Max processes unlimited unlimited processes Max open files 1048576 1048576 files
Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 256793 256793 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us
Core pattern: core

Kernel version: Linux version 5.10.0-23-amd64 (debian-kernel@lists.debian.org) (gcc-10 (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2) #1 SMP Debian 5.10.179-1 (2023-05-12)

/usr/local/bin/docker-entrypoint.sh: line 135: 47 Aborted (core dumped) "$@" --skip-networking --default-time-zone=SYSTEM --socket="${SOCKET}" --wsrep_on=OFF --expire-logs-days=0 --loose-innodb_buffer_pool_load_at_startup=0 2023-06-20 07:55:42+00:00 [ERROR] [Entrypoint]: Unable to start server.

[MGREMY]

@grooverdan

@MGREMY thanks for trying a backtrace. The datadir permissions need fixing before attempting to start:

podman run --rm  -v m10.11:/var/lib/mysql --name m11.0 quay.io/mariadb-foundation/mariadb-debug:11.0 chown -R mysql: /var/lib/mysql

This should reset them all back to the mysql user as it appears within the container.

Ensure the gdb run includes --user mysql as the container runtime option.

No worries about not sharing data, its always a tough ask and I understand why.

Is that OK to do so even if that's in a NFS share ? If so, I'll do it quickly.

[grooverdan]

@grooverdan

@MGREMY thanks for trying a backtrace. The datadir permissions need fixing before attempting to start:

podman run --rm  -v m10.11:/var/lib/mysql --name m11.0 quay.io/mariadb-foundation/mariadb-debug:11.0 chown -R mysql: /var/lib/mysql

Is that OK to do so even if that's in a NFS share ?

I think it should be (though I don't have a test environment at the moment).

[grooverdan]

Folks, on MDEV-31443 there is a work around to upgrading, though we'd dearly like someone to provide a backtrace so we can resolve this seamlessly for everyone.

The theory is that its a very old <MySQL-4.1.2 datadir that everyone has upgraded in place from. Does that sound right to everyone? If not can you tell us a little about your upgrade history of your datadir.

[eworm-de]

The theory is that its a very old <MySQL-4.1.2 datadir that everyone has upgraded in place from. Does that sound right to everyone? If not can you tell us a little about your upgrade history of your datadir.

No, it does not sound right... Already told my story in upstream issue.

[chrispage1]

I've just run into the same problem myself and was able to upgrade from 10.5.19 to 11.02 by running SET GLOBAL innodb_fast_shutdown=0; before stopping my MySQL instance and running the upgrade.

However, is this a safe workaround?

[dr-m]

@chrispage1 Yes, a prior shutdown with innodb_fast_shutdown=0 should be safe. Originally I was thinking to avoid going the "extra mile" and not implement any upgrade procedure, just tell the user to downgrade and execute a slow shutdown first, and then come back with an empty change buffer. In fact, for some time we used to recommend a slow shutdown before any upgrade, but in MDEV-23755 it was deemed unnecessary.

The built-in upgrade procedure in 11.0 (MariaDB/server@f27e9c894779a4c7ebe6446ba9aa408f1771c114) is supposed to work, but apparently I missed something in it. I would need a copy of an affected database or some form of remote access for some debugging, to fix this. @eworm-de has provided quite a bit of information in MDEV-31443 already, and I hope that soon we can have a debugging session for the final breakthrough.

[grooverdan]

@dr-m has written some fixes. These have been built into the image quay.io/mariadb-foundation/mariadb-devel:11.0-mdev-31443-pkgtest along with other fixes due to come out in the next 11.0 release. If those affected here can test with that image and report back if successful or not that would be much appreciated.

[eworm-de]

Anybody interested in Arch Linux packages for testing?

[eworm-de]

Fixed in https://github.com/MariaDB/server/commit/313c5a1dfb744aaef10586526dda89d2b4a50651

[grooverdan]

Fixed in server code. Next release is currently scheduled for the 27th July.

quay.io/mariadb-foundation/mariadb-devel:11.0-mdev-31443-pkgtest contains the fix and verification would be most useful.

[hpvb]

I ran into the same issue, I started with quay.io/mariadb-foundation/mariadb-devel:11.0-mdev-31443-pkgtest once and afterwards was able to use the normal mariadb:latest package again. Thanks!

[grooverdan]

@hpvb thanks for confirming and testing.

[kevinpapst]

Homebrew users that were auto-upgraded and now can't use MariaDB:

brew uninstall mariadb
brew install mariadb@10.10
brew services start mariadb@10.10