search

sysown / proxysql

High-performance MySQL proxy with a GPL license.
http://www.proxysql.com
GNU General Public License v3.0
5.96k stars 972 forks source link

admin-web_enabled=true ProxySQL crashed #2568

Closed nigel889 closed 4 years ago

nigel889 commented 4 years ago

ProxySQL version 2.0.10-27-g5b31997: OS: cat /etc/redhat-release CentOS Linux release 7.5.1804 (Core)

when enable web and Visit website it, ProxySQL crashed!

logs:

2020-02-27 18:36:35 [INFO] Received command set global admin-web_enabled='true' 2020-02-27 18:36:41 [INFO] Received command set admin-web_enabled='true' 2020-02-27 18:37:00 [INFO] Received command set global admin-web_enabled='true' 2020-02-27 18:37:36 [INFO] Received LOAD ADMIN VARIABLES TO RUNTIME command Error: signal 11: /usr/bin/proxysql(_Z13crash_handleri+0x1a)[0x4ae55a] /lib64/libc.so.6(+0x362f0)[0x7f7a96bc22f0] /lib64/libgnutls.so.28(gnutls_priority_set+0x16d)[0x7f7a97ff0eed] /usr/bin/proxysql[0x8dbcf8] /usr/bin/proxysql[0x8dc7f5] /usr/bin/proxysql[0x8de512] /usr/bin/proxysql[0x8df32a] /lib64/libpthread.so.0(+0x7e25)[0x7f7a97da1e25] /lib64/libc.so.6(clone+0x6d)[0x7f7a96c8abad] 2020-02-27 18:38:54 main.cpp:1538:ProxySQL_daemonize_phase3(): [ERROR] ProxySQL crashed. Restarting! 2020-02-27 18:38:54 [INFO] ProxySQL version 2.0.10-27-g5b31997 2020-02-27 18:38:54 [INFO] ProxySQL SHA1 checksum: e1982906ea874ad66ba381b267abd085720dd934 2020-02-27 18:38:54 [INFO] Angel process started ProxySQL process 10414 Standard ProxySQL MySQL Logger rev. 2.0.0714 -- MySQL_Logger.cpp -- Mon Feb 24 16:38:57 2020 Standard ProxySQL Cluster rev. 0.4.0906 -- ProxySQL_Cluster.cpp -- Mon Feb 24 16:38:57 2020 Standard ProxySQL Statistics rev. 1.4.1027 -- ProxySQL_Statistics.cpp -- Mon Feb 24 16:38:57 2020 Standard ProxySQL HTTP Server Handler rev. 1.4.1031 -- ProxySQL_HTTP_Server.cpp -- Mon Feb 24 16:38:57 2020 Standard ProxySQL Admin rev. 2.0.6.0805 -- ProxySQL_Admin.cpp -- Mon Feb 24 16:38:57 2020 2020-02-27 18:38:54 [INFO] ProxySQL SHA1 checksum: e1982906ea874ad66ba381b267abd085720dd934 Standard MySQL Threads Handler rev. 0.2.0902 -- MySQL_Thread.cpp -- Mon Feb 24 16:38:57 2020 Standard MySQL Authentication rev. 0.2.0902 -- MySQL_Authentication.cpp -- Mon Feb 24 16:38:57 2020 2020-02-27 18:38:54 [INFO] Dumping mysql_servers_incoming +--------------+------------+------+-----------+--------+--------+-------------+-----------------+---

alpes214 commented 4 years ago

@nigel889 I was not able to reproduce crash with centos7.5.1804 docker image and proxysql-2.0.10-1-centos7.x86_64.rpm. Can you please provide more details about:

nigel889 commented 4 years ago

1,had a basic configuration settings, include : mysql_replication_hostgroups mysql_servers mysql_users; mysql_query_rules;

and mysql-monitor_username it work fine; but if admin-web_enabled =true , start port 6080 , send HTTP request to the server port 6080, ProxySQL crashed. 2,how did you send HTTP request to the server; yes

nigel889 commented 4 years ago

@val214

1,had a basic configuration settings, include : mysql_replication_hostgroups mysql_servers mysql_users; mysql_query_rules;

and mysql-monitor_username it work fine; but if admin-web_enabled =true , start port 6080 , send HTTP request to the server port 6080, ProxySQL crashed. 2,how did you send HTTP request to the server; yes

3, core.zip

alpes214 commented 4 years ago

Thank you @nigel889 for core dump. It helps to understand that the issue is related to gnutls.

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/usr/bin/proxysql -c /etc/proxysql.cnf'.
Program terminated with signal 11, Segmentation fault.
#0  0x00007f7a97ff0eed in gnutls_priority_set () from /lib64/libgnutls.so.28
Missing separate debuginfos, use: debuginfo-install proxysql-2.0.10-1.x86_64
(gdb) bt
#0  0x00007f7a97ff0eed in gnutls_priority_set () from /lib64/libgnutls.so.28
#1  0x00000000008dbcf8 in internal_add_connection (daemon=daemon@entry=0x7f7a8f6de7b0, client_socket=client_socket@entry=36, addr=addr@entry=0x7f7a893fa590, addrlen=16, external_add=external_add@entry=false,
    non_blck=non_blck@entry=true) at daemon.c:2579
#2  0x00000000008dc7f5 in MHD_accept_connection (daemon=daemon@entry=0x7f7a8f6de7b0) at daemon.c:3303
#3  0x00000000008de512 in MHD_epoll (daemon=daemon@entry=0x7f7a8f6de7b0, may_block=may_block@entry=1) at daemon.c:4516
#4  0x00000000008df32a in MHD_polling_thread (cls=0x7f7a8f6de7b0) at daemon.c:4746
#5  0x00007f7a97da1e25 in start_thread () from /lib64/libpthread.so.0
#6  0x00007f7a96c8abad in clone () from /lib64/libc.so.6

It is still not clear how do you do a request:

alpes214 commented 4 years ago

@nigel889 and one more thing, please provide this information from the host where proxysql is installed

[root@8949dc188301 /]# yum info gnutls
Loaded plugins: fastestmirror, ovl
Loading mirror speeds from cached hostfile
 * base: mirror.eu.oneandone.net
 * extras: ftp.antilo.de
 * updates: mirror.imt-systems.com
Installed Packages
Name        : gnutls
Arch        : x86_64
Version     : 3.3.29
Release     : 9.el7_6
Size        : 2.0 M
Repo        : installed
From repo   : base
Summary     : A TLS protocol implementation
URL         : http://www.gnutls.org/
License     : GPLv3+ and LGPLv2+
Description : GnuTLS is a secure communications library implementing the SSL, TLS and DTLS
            : protocols and technologies around them. It provides a simple C language
            : application programming interface (API) to access the secure communications
            : protocols as well as APIs to parse and write X.509, PKCS #12, OpenPGP and
            : other required structures.

Available Packages
Name        : gnutls
Arch        : i686
Version     : 3.3.29
Release     : 9.el7_6
Size        : 649 k
Repo        : base/7/x86_64
Summary     : A TLS protocol implementation
URL         : http://www.gnutls.org/
License     : GPLv3+ and LGPLv2+
Description : GnuTLS is a secure communications library implementing the SSL, TLS and DTLS
            : protocols and technologies around them. It provides a simple C language
            : application programming interface (API) to access the secure communications
            : protocols as well as APIs to parse and write X.509, PKCS #12, OpenPGP and
            : other required structures.

[root@8949dc188301 /]# ll /usr/lib64/*gnutls*
lrwxrwxrwx 1 root root      20 Feb 28 11:09 /usr/lib64/libgnutls.so.28 -> libgnutls.so.28.43.3
-rwxr-xr-x 1 root root 1300504 Mar 14  2019 /usr/lib64/libgnutls.so.28.43.3
nigel889 commented 4 years ago

@val214 ![Uploading image.png…]()

nigel889 commented 4 years ago

Ãû³Æ £ºgnutls ¼Ü¹¹ £ºx86_64 °æ±¾ £º3.3.29 ·¢²¼ £º9.el7_6 ´óС £º2.0 M Ô´ £ºinstalled À´×ÔÔ´£ºbase ¼ò½é £º A TLS protocol implementation ÍøÖ· £ºhttp://www.gnutls.org/ ЭÒé £º GPLv3+ and LGPLv2+ ÃèÊö £º GnuTLS is a secure communications library implementing the SSL, TLS and DTLS : protocols and technologies around them. It provides a simple C language : application programming interface (API) to access the secure communications : protocols as well as APIs to parse and write X.509, PKCS #12, OpenPGP and : other required structures.

¿É°²×°µÄÈí¼þ°ü Ãû³Æ £ºgnutls ¼Ü¹¹ £ºi686 °æ±¾ £º3.3.29 ·¢²¼ £º9.el7_6 ´óС £º649 k Ô´ £ºbase/7/x86_64 ¼ò½é £º A TLS protocol implementation ÍøÖ· £ºhttp://www.gnutls.org/ ЭÒé £º GPLv3+ and LGPLv2+ ÃèÊö £º GnuTLS is a secure communications library implementing the SSL, TLS and DTLS : protocols and technologies around them. It provides a simple C language : application programming interface (API) to access the secure communications : protocols as well as APIs to parse and write X.509, PKCS #12, OpenPGP and : other required structures.

jianghua1213 commented 4 years ago

2020-03-12 10:03:04 [INFO] Received command SET admin-web_enabled='true' 2020-03-12 10:03:14 [INFO] Received LOAD ADMIN VARIABLES TO RUNTIME command Error: signal 11: /usr/bin/proxysql(_Z13crash_handleri+0x1d)[0x4b2ecd] /lib64/libc.so.6(+0x33b50)[0x7f86ef752b50] /lib64/libgnutls.so.28(gnutls_priority_set+0x1a)[0x7f86f0c1a7fa] /usr/bin/proxysql[0x930320] /usr/bin/proxysql[0x930ae1] /usr/bin/proxysql[0x932b35] /usr/bin/proxysql[0x9337e7] /lib64/libpthread.so.0(+0x740b)[0x7f86f09ca40b] /lib64/libc.so.6(clone+0x3f)[0x7f86ef80be7f] 2020-03-12 10:03:25 main.cpp:1538:ProxySQL_daemonize_phase3(): [ERROR] ProxySQL crashed. Restarting! 2020-03-12 10:03:25 [INFO] ProxySQL version 2.0.10-53-g3f06f73e 2020-03-12 10:03:25 [INFO] ProxySQL SHA1 checksum: e84be566f2457f4141ad03cbf8bbc5d84f439a16 2020-03-12 10:03:25 [INFO] Angel process started ProxySQL process 9023 Standard ProxySQL MySQL Logger rev. 2.0.0714 -- MySQL_Logger.cpp -- Thu Mar 12 08:45:00 2020 Standard ProxySQL Cluster rev. 0.4.0906 -- ProxySQL_Cluster.cpp -- Thu Mar 12 08:45:00 2020 Standard ProxySQL Statistics rev. 1.4.1027 -- ProxySQL_Statistics.cpp -- Thu Mar 12 08:45:00 2020 Standard ProxySQL HTTP Server Handler rev. 1.4.1031 -- ProxySQL_HTTP_Server.cpp -- Thu Mar 12 08:45:00 2020 Standard ProxySQL Admin rev. 2.0.6.0805 -- ProxySQL_Admin.cpp -- Thu Mar 12 08:45:00 2020 2020-03-12 10:03:25 [INFO] ProxySQL SHA1 checksum: e84be566f2457f4141ad03cbf8bbc5d84f439a16 Standard MySQL Threads Handler rev. 0.2.0902 -- MySQL_Thread.cpp -- Thu Mar 12 08:45:00 2020 Standard MySQL Authentication rev. 0.2.0902 -- MySQL_Authentication.cpp -- Thu Mar 12 08:45:00 2020 2020-03-12 10:03:25 [INFO] Dumping mysql_servers_incoming

renecannao commented 4 years ago

@nigel889 :

Standard ProxySQL MySQL Logger rev. 2.0.0714 -- MySQL_Logger.cpp -- Mon Feb 24 16:38:57 2020
Standard ProxySQL Cluster rev. 0.4.0906 -- ProxySQL_Cluster.cpp -- Mon Feb 24 16:38:57 2020
Standard ProxySQL Statistics rev. 1.4.1027 -- ProxySQL_Statistics.cpp -- Mon Feb 24 16:38:57 2020
Standard ProxySQL HTTP Server Handler rev. 1.4.1031 -- ProxySQL_HTTP_Server.cpp -- Mon Feb 24 16:38:57 2020

@jianghua1213:

Standard ProxySQL MySQL Logger rev. 2.0.0714 -- MySQL_Logger.cpp -- Thu Mar 12 08:45:00 2020
Standard ProxySQL Cluster rev. 0.4.0906 -- ProxySQL_Cluster.cpp -- Thu Mar 12 08:45:00 2020
Standard ProxySQL Statistics rev. 1.4.1027 -- ProxySQL_Statistics.cpp -- Thu Mar 12 08:45:00 2020
Standard ProxySQL HTTP Server Handler rev. 1.4.1031 -- ProxySQL_HTTP_Server.cpp -- Thu Mar 12 08:45:00 2020
Standard ProxySQL Admin rev. 2.0.6.0805 -- ProxySQL_Admin.cpp -- Thu Mar 12 08:45:00 2020

2.0.10 was released on Feb 17 , and timestamp newer that that it means you aren't using our binaries. Closing