search

codership / mysql-wsrep

wsrep API patch for MySQL server
Other
65 stars 34 forks source link

mysqld got signal 11 #352

Closed xXxSPYxXx closed 5 years ago

xXxSPYxXx commented 5 years ago

Hello! My GaleraCluster MySQL 5.7 nodes restarts time to time with error:

11:40:21 UTC - mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. Attempting to collect some information that could help diagnose the problem. As this is a crash and something is definitely wrong, the information collection process might fail.

key_buffer_size=16777216 read_buffer_size=4194304 max_used_connections=281 max_threads=1000 thread_count=51 connection_count=26 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 8222321 K bytes of memory Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong...

stack_bottom = 0 thread_stack 0x80000 /usr/sbin/mysqld(my_print_stacktrace+0x2c)[0x55f48c960d9c] /usr/sbin/mysqld(handle_fatal_signal+0x479)[0x55f48c268309] /lib/x86_64-linux-gnu/libpthread.so.0(+0x110e0)[0x7fe1f80f50e0] /usr/sbin/mysqld(handle_connection+0x17e)[0x55f48c7e8f5e] /usr/sbin/mysqld(pfs_spawn_thread+0x1b4)[0x55f48cccfb64] /lib/x86_64-linux-gnu/libpthread.so.0(+0x74a4)[0x7fe1f80eb4a4] /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7fe1f6937d0f]

The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash.

I am make coredump files, but it didnt help, because need debug symbols

[New LWP 5857] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". Core was generated by `/usr/sbin/mysqld --daemonize --pid-file=/var/run/mysqld/mysqld.pid --wsrep_star'. Program terminated with signal SIGSEGV, Segmentation fault.

0 0x00007faea87fa3b4 in pthread_kill () from /lib/x86_64-linux-gnu/libpthread.so.0

[Current thread is 1 (Thread 0x7faa43173700 (LWP 5637))] (gdb) bt full

0 0x00007faea87fa3b4 in pthread_kill () from /lib/x86_64-linux-gnu/libpthread.so.0

No symbol table info available.

1 0x000055cf4bde5487 in my_write_core (sig=) at /home/galera/mysql-wsrep-5.7.24-25.16/mysys/stacktrace.c:249

No locals.

2 0x000055cf4b6ed275 in handle_fatal_signal (sig=11) at /home/galera/mysql-wsrep-5.7.24-25.16/sql/signal_handler.cc:220

    curr_time = 1555041055
    tmins = 25917350
    thrs = 431955
    hrs = 3
    mins = 50
    secs = 55
    hrs_buf = "03"
    mins_buf = "50"
    secs_buf = "55"
    max_threads = <optimized out>

3

No symbol table info available.

4 0x000055cf4bc6df5e in handle_connection (arg=arg@entry=0x55cf67e5e710) at /home/galera/mysql-wsrep-5.7.24-25.16/sql/conn_handler/connection_handler_per_thread.cc:332

    psi = <optimized out>
    channel_info = <optimized out>
    pthread_reused = <optimized out>

5 0x000055cf4c154b64 in pfs_spawn_thread (arg=0x55cf67f527c0) at /home/galera/mysql-wsrep-5.7.24-25.16/storage/perfschema/pfs.cc:2190

    typed_arg = 0x55cf67f527c0
    user_arg = 0x55cf67e5e710
    user_start_routine = 0x55cf4bc6dde0 <handle_connection(void*)>
    pfs = <optimized out>
    klass = <optimized out>

6 0x00007faea87f34a4 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0

No symbol table info available.

7 0x00007faea703fd0f in clone () from /lib/x86_64-linux-gnu/libc.so.6

No symbol table info available. (gdb) bt

0 0x00007faea87fa3b4 in pthread_kill () from /lib/x86_64-linux-gnu/libpthread.so.0

1 0x000055cf4bde5487 in my_write_core (sig=) at /home/galera/mysql-wsrep-5.7.24-25.16/mysys/stacktrace.c:249

2 0x000055cf4b6ed275 in handle_fatal_signal (sig=11) at /home/galera/mysql-wsrep-5.7.24-25.16/sql/signal_handler.cc:220

3

4 0x000055cf4bc6df5e in handle_connection (arg=arg@entry=0x55cf67e5e710) at /home/galera/mysql-wsrep-5.7.24-25.16/sql/conn_handler/connection_handler_per_thread.cc:332

5 0x000055cf4c154b64 in pfs_spawn_thread (arg=0x55cf67f527c0) at /home/galera/mysql-wsrep-5.7.24-25.16/storage/perfschema/pfs.cc:2190

6 0x00007faea87f34a4 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0

7 0x00007faea703fd0f in clone () from /lib/x86_64-linux-gnu/libc.so.6

Do you provied packet like mysql-wsrep-5.7-dbg with debug symbols?

xXxSPYxXx commented 5 years ago

From chat in telegram canal i am get information:

thd object is sued after free (after delete)

https://github.com/codership/mysql-wsrep/blob/5.7/sql/conn_handler/connection_handler_per_thread.cc#L331

mkaruza commented 5 years ago

Thank you for bug report and narrowing where is the problem. Problem will be fixed in next version.

mkaruza commented 5 years ago

Hi, this should be fixed in upcoming 5.7.26-25.18