search

sysown / proxysql

High-performance MySQL proxy with a GPL license.
http://www.proxysql.com
GNU General Public License v3.0
5.95k stars 970 forks source link

ProxySQL 2.0.7 crashing due to signal 11 #3733

Closed kishmu419 closed 1 year ago

kishmu419 commented 2 years ago

ProxySQL 2.0.7 crashed with Signal 11.

Proxysql version : 2.0.7-80-g4dd4ef5 OS Version : CentOS Linux 7

Proxysql error log :

2021-12-20 14:36:19 MySQL_Thread.cpp:4292:process_all_sessions(): [WARNING] Closing unhealthy client connection 172.16.7.41:37265 2021-12-20 14:36:19 [INFO] Starting new mysql event log file /data/proxysql/proxy_audit.log.00012543 Error: signal 11: proxysql(_Z13crash_handleri+0x1a)[0x4992ba] /lib64/libc.so.6(+0x36450)[0x7f128e87b450] /lib64/libstdc++.so.6(_ZNSo6sentryC2ERSo+0xf)[0x7f128f1c129f] /lib64/libstdc++.so.6(_ZNSo5writeEPKcl+0x31)[0x7f128f1c16a1] proxysql(_ZN11MySQL_Event20write_query_format_1EPSt13basic_fstreamIcSt11char_traitsIcEE+0x18b)[0x54be7b] proxysql(_ZN12MySQL_Logger11log_requestEP13MySQL_SessionP17MySQL_Data_Stream+0x253)[0x550473] proxysql(_ZN13MySQL_Session10RequestEndEP17MySQL_Data_Stream+0x9d)[0x4e6f4d] proxysql(_ZN13MySQL_Session7handlerEv+0x281b)[0x4f942b] proxysql(_ZN12MySQL_Thread20process_all_sessionsEv+0x34c)[0x4d489c] proxysql(_ZN12MySQL_Thread3runEv+0xe06)[0x4de536] proxysql(_Z24mysql_worker_thread_funcPv+0x6c)[0x495b9c] /lib64/libpthread.so.0(+0x7ea5)[0x7f128fa5cea5] /lib64/libc.so.6(clone+0x6d)[0x7f128e9439fd] 2021-12-20 14:36:28 main.cpp:1395:ProxySQL_daemonize_phase3(): [ERROR] ProxySQL crashed. Restarting! 2021-12-20 14:36:28 [INFO] ProxySQL version 2.0.7-80-g4dd4ef5 2021-12-20 14:36:28 [INFO] ProxySQL SHA1 checksum: 7fe183ee42e9f9afa5766fa11c9901d14db766bc 2021-12-20 14:36:28 [INFO] Angel process started ProxySQL process 43312 Standard ProxySQL MySQL Logger rev. 2.0.0714 -- MySQL_Logger.cpp -- Fri Sep 27 09:10:35 2019 Standard ProxySQL Cluster rev. 0.4.0906 -- ProxySQL_Cluster.cpp -- Fri Sep 27 09:10:35 2019 Standard ProxySQL Statistics rev. 1.4.1027 -- ProxySQL_Statistics.cpp -- Fri Sep 27 09:10:35 2019 Standard ProxySQL HTTP Server Handler rev. 1.4.1031 -- ProxySQL_HTTP_Server.cpp -- Fri Sep 27 09:10:35 2019 Standard ProxySQL Admin rev. 2.0.6.0805 -- ProxySQL_Admin.cpp -- Fri Sep 27 09:10:35 2019 2021-12-20 14:36:29 [INFO] ProxySQL SHA1 checksum: 7fe183ee42e9f9afa5766fa11c9901d14db766bc Standard MySQL Threads Handler rev. 0.2.0902 -- MySQL_Thread.cpp -- Mon Sep 30 19:01:56 2019 Standard MySQL Authentication rev. 0.2.0902 -- MySQL_Authentication.cpp -- Fri Sep 27 09:10:35 2019 2021-12-20 14:36:29 [INFO] Dumping mysql_servers_incoming

Core dump :

GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-120.el7 Copyright (C) 2013 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". For bug reporting instructions, please see: http://www.gnu.org/software/gdb/bugs/... Reading symbols from /usr/bin/proxysql...done. [New LWP 37834] [New LWP 37838] [New LWP 37835] [New LWP 37832] [New LWP 37837] [New LWP 37830] [New LWP 37833] [New LWP 37841] [New LWP 37845] [New LWP 37843] [New LWP 37840] [New LWP 37839] [New LWP 37844] [New LWP 37827] [New LWP 37842] [New LWP 37828] [New LWP 37846] [New LWP 37826] [New LWP 37831] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `proxysql -c /etc/proxysql.cnf -D /data/proxysql'. Program terminated with signal 11, Segmentation fault.

0 0x00007f128f1c129f in std::ostream::sentry::sentry(std::ostream&) () from /lib64/libstdc++.so.6

Missing separate debuginfos, use: debuginfo-install proxysql-2.0.7-1.x86_64 (gdb) btfull Undefined command: "btfull". Try "help". (gdb) bt full

0 0x00007f128f1c129f in std::ostream::sentry::sentry(std::ostream&) () from /lib64/libstdc++.so.6

No symbol table info available.

1 0x00007f128f1c16a1 in std::ostream::write(char const*, long) () from /lib64/libstdc++.so.6

No symbol table info available.

2 0x000000000054be7b in MySQL_Event::write_query_format_1 (this=0x7f128ba7a9d0, f=0x0) at MySQL_Logger.cpp:265

    total_bytes = 192
    len = <optimized out>

3 0x0000000000550473 in MySQL_Logger::log_request (this=0x7f128e419680, sess=0x7f1289e3de00, myds=0x7f12842fcd80) at MySQL_Logger.cpp:711

    cl = 18
    me = {thread_id = 91570325, username = 0x7f12843422c0 "b2bappuser", schemaname = 0x7f11c27cb568 "yatra", username_len = 10, schemaname_len = 5, start_time = 1639991179796571, 
      end_time = 1639991179796905, query_digest = 10278670706449541542, query_ptr = 0x7f11c02ae4d5 "SELECT code, value FROM b2b_config_services_authentication WHERE channel='DEFAULT' ORDER BY id ASC", 
      query_len = 98, server = 0x7f11c1979f40 "172.16.3.15:3307", client = 0x7f11c1cff5c0 "172.16.7.212:59364", server_len = 16, client_len = 18, buf = "\000\000\000\000\000\000\000\000\000\027", 
      et = PROXYSQL_COM_QUERY, hid = 11, extra_info = 0x0, have_affected_rows = false, have_rows_sent = true, affected_rows = 0, rows_sent = 34}
    sl = <optimized out>
    ca = 0x7f11c1cff5c0 "172.16.7.212:59364"
    sa = <optimized out>
    curpos = <optimized out>
    ui = <optimized out>
    curtime_mono = <optimized out>
    let = PROXYSQL_COM_QUERY
    c = <optimized out>
    ql = <optimized out>

4 0x00000000004e6f4d in MySQL_Session::RequestEnd (this=this@entry=0x7f1289e3de00, myds=myds@entry=0x7f12842fcd80) at MySQL_Session.cpp:6608

No locals.

5 0x00000000004f942b in MySQL_Session::handler (this=this@entry=0x7f1289e3de00) at MySQL_Session.cpp:4035

    qdt = <optimized out>
    myconn = 0x7f127d221700
    myds = 0x7f12842fcd80
    rc = <optimized out>
    begint = {tv_sec = 139717629164640, tv_nsec = 139717629164144}
    endt = {tv_sec = 6521908909941356953, tv_nsec = 6521908909941356953}
    prepared_stmt_with_no_params = false
    pkt = {ptr = 0x7f11c02ae4d0, size = 103}
    c = <optimized out>
    __PRETTY_FUNCTION__ = "int MySQL_Session::handler()"
    __func__ = "handler"
    wrong_pass = false

6 0x00000000004d489c in MySQL_Thread::process_all_sessions (this=this@entry=0x7f1289e00000) at MySQL_Thread.cpp:4301

    sess = 0x7f1289e3de00
    total_active_transactions_ = 0
    idle_maintenance_thread = false
    sess_sort = <optimized out>
    __func__ = "process_all_sessions"
    n = 47
    rc = <optimized out>

7 0x00000000004de536 in MySQL_Thread::run (this=this@entry=0x7f1289e00000) at MySQL_Thread.cpp:4048

    num_idles = <optimized out>
    ttw = <optimized out>
    maintenance_interval = <optimized out>
renecannao commented 2 years ago

You have a crash here: https://github.com/sysown/proxysql/blob/v2.0.7/lib/MySQL_Logger.cpp#L265 You probably ran out of disk space, and the write failed.

Ashik128 commented 2 years ago

Hi Rene,

There was no issue with the disk, also we got the same issue on the same instance today as well. We checked the disk part as well, there is no issue with the disk.

Error: signal 11: proxysql(_Z13crash_handleri+0x1a)[0x4992ba] /lib64/libc.so.6(+0x36450)[0x7f128e87b450] /lib64/libstdc++.so.6(_ZNSo6sentryC2ERSo+0xf)[0x7f128f1c129f] /lib64/libstdc++.so.6(_ZNSo5writeEPKcl+0x31)[0x7f128f1c16a1] proxysql(_ZN11MySQL_Event20write_query_format_1EPSt13basic_fstreamIcSt11char_traitsIcEE+0x18b)[0x54be7b] proxysql(_ZN12MySQL_Logger11log_requestEP13MySQL_SessionP17MySQL_Data_Stream+0x253)[0x550473] proxysql(_ZN13MySQL_Session10RequestEndEP17MySQL_Data_Stream+0x9d)[0x4e6f4d] proxysql(_ZN13MySQL_Session7handlerEv+0x281b)[0x4f942b] proxysql(_ZN12MySQL_Thread20process_all_sessionsEv+0x34c)[0x4d489c] proxysql(_ZN12MySQL_Thread3runEv+0xe06)[0x4de536] proxysql(_Z24mysql_worker_thread_funcPv+0x6c)[0x495b9c] /lib64/libpthread.so.0(+0x7ea5)[0x7f128fa5cea5] /lib64/libc.so.6(clone+0x6d)[0x7f128e9439fd] 2022-02-08 09:19:39 main.cpp:1395:ProxySQL_daemonize_phase3(): [ERROR] ProxySQL crashed. Restarting! 2022-02-08 09:19:39 [INFO] ProxySQL version 2.0.7-80-g4dd4ef5 2022-02-08 09:19:39 [INFO] ProxySQL SHA1 checksum: 7fe183ee42e9f9afa5766fa11c9901d14db766bc 2022-02-08 09:19:39 [INFO] Angel process started ProxySQL process 21561

Coredump:

No symbol table info available.
#0  0x00007f128f1c129f in std::ostream::sentry::sentry(std::ostream&) () from /lib64/libstdc++.so.6
No symbol table info available.
#1  0x00007f128f1c16a1 in std::ostream::write(char const*, long) () from /lib64/libstdc++.so.6
No symbol table info available.
#2  0x000000000054be7b in MySQL_Event::write_query_format_1 (this=0x7f128ba7a9d0, f=0x0) at MySQL_Logger.cpp:265
        total_bytes = 244
        len = <optimized out>
#3  0x0000000000550473 in MySQL_Logger::log_request (this=0x7f128e419680, sess=0x7f1284016900, myds=0x7f1283474680) at MySQL_Logger.cpp:711
        cl = 18
        me = {thread_id = 2900239, username = 0x7f1289e0bc90 "b2bappuser", schemaname = 0x7f127fceccb8 "suniper", username_len = 10, schemaname_len = 5, start_time = 1644292177825250, 
          end_time = 1644292177825584, query_digest = 16236000353717457753, 
          query_ptr = 0x7f1289e18045 "delete from supervison where agency_email = '1209648653' and user_email = 'crazyfr0g@gmail.com' and product_type = 'OPEN'", 
          query_len = 155, server = 0x7f127e52ef40 "10.16.3.14:3307", client = 0x7f127fcc1e80 "10.16.7.212:21001", server_len = 16, client_len = 18, buf = "\000i\001\204\022\177\000\000\000\\", 
          et = PROXYSQL_COM_QUERY, hid = 10, extra_info = 0x0, have_affected_rows = true, have_rows_sent = true, affected_rows = 0, rows_sent = 0}
        sl = <optimized out>
        ca = 0x7f127fcc1e80 "10.16.7.212:21001"
        sa = <optimized out>
        curpos = <optimized out>
        ui = <optimized out>
        curtime_mono = <optimized out>
        let = PROXYSQL_COM_QUERY
        c = <optimized out>
        ql = <optimized out>
#4  0x00000000004e6f4d in MySQL_Session::RequestEnd (this=this@entry=0x7f1284016900, myds=myds@entry=0x7f1283474680) at MySQL_Session.cpp:6608
No locals.
#5  0x00000000004f942b in MySQL_Session::handler (this=this@entry=0x7f1284016900) at MySQL_Session.cpp:4035
        qdt = <optimized out>
        myconn = 0x7f128a25e600
        myds = 0x7f1283474680
        rc = <optimized out>
        begint = {tv_sec = 139717502033664, tv_nsec = 5586748}
        endt = {tv_sec = 32768, tv_nsec = 139717629160448}
        prepared_stmt_with_no_params = false
        pkt = {ptr = 0x7f1289e18040, size = 160}
        c = <optimized out>
        __PRETTY_FUNCTION__ = "int MySQL_Session::handler()"
        __func__ = "handler"
        wrong_pass = false
#6  0x00000000004d489c in MySQL_Thread::process_all_sessions (this=this@entry=0x7f1289e00000) at MySQL_Thread.cpp:4301
        sess = 0x7f1284016900
        total_active_transactions_ = 0
        idle_maintenance_thread = false
        sess_sort = <optimized out>
        __func__ = "process_all_sessions"
        n = 9
        rc = <optimized out>
#7  0x00000000004de536 in MySQL_Thread::run (this=this@entry=0x7f1289e00000) at MySQL_Thread.cpp:4048
        num_idles = <optimized out>
        ttw = <optimized out>
        maintenance_interval = <optimized out>
        idle_maintenance_thread = <optimized out>
        __PRETTY_FUNCTION__ = "void MySQL_Thread::run()"
        __func__ = "run"
        n = <optimized out>
        rc = <optimized out>
#8  0x0000000000495b9c in mysql_worker_thread_func (arg=0x7f128e45c4f0) at main.cpp:648
        thread_attr = {__size = '\000' <repeats 17 times>, "\020", '\000' <repeats 37 times>, __align = 0}
        tmp_stack_size = 8388608
        mysql_thread = 0x7f128e45c4f0
        worker = 0x7f1289e00000
#9  0x00007f128fa5cea5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#10 0x00007f128e9439fd in clone () from /lib64/libc.so.6
No symbol table info available.