Upgrade from 1.4.13 to 1.4.14/1.4.15 results in crashing

[mgale]

Hello,

We are having an issue where if we try and upgrade our working ProxySQL setup (1.4.13) to either 1.4.14 or 1.4.15 it results in ProxySQL constantly crashing almost every minute.

Working Environment: OS: Ubuntu 14.04 LTS Update 5 ProxySQL 1.4.13 AMD64 / EC2 instances in AWS 2 ProxySQL instances configured as a cluster

Broken environment 1: Upgraded ProxySQL to 1.4.14

Broken environment 2: Upgraded ProxySQL to 1.4.15

Here are the logs from two different environments: Env1: Standard MySQL Monitor (StdMyMon) rev. 1.2.0723 -- MySQL_Monitor.cpp -- Thu Feb 14 22:38:24 2019 Error: signal 11: proxysql(_Z13crash_handleri+0x1a)[0x44d8aa] /lib/x86_64-linux-gnu/libc.so.6(+0x36cb0)[0x7fbfe2684cb0] /lib/x86_64-linux-gnu/libc.so.6(+0x9ac36)[0x7fbfe26e8c36] proxysql(vio_read+0xa8)[0x628598] proxysql[0x628a55] proxysql(my_net_read+0x18d)[0x62961d] proxysql(net_safe_read+0x32)[0x60ef82] proxysql(mthd_my_read_one_row+0x22)[0x60f5b2] proxysql(mysql_fetch_row+0x64)[0x610164] proxysql[0x615288] proxysql(my_context_spawn+0x41)[0x614621] 2019-03-04 15:57:29 main.cpp:910:ProxySQL_daemonize_phase3(): [ERROR] ProxySQL crashed. Restarting! 2019-03-04 15:57:29 [INFO] Angel process started ProxySQL process 16155

and Env2: Standard MySQL Monitor (StdMyMon) rev. 1.2.0723 -- MySQL_Monitor.cpp -- Thu Feb 14 22:38:24 2019 Error: signal 11: proxysql(_Z13crash_handleri+0x1a)[0x44d8aa] /lib/x86_64-linux-gnu/libc.so.6(+0x36cb0)[0x7f229905ecb0] proxysql(_ZN19StmtLongDataHandlerD1Ev+0xa)[0x4cd46a] proxysql(_ZN13MySQL_Session5resetEv+0x118)[0x486408] proxysql(_ZN13MySQL_SessionD2Ev+0xe)[0x48723e] proxysql(_ZN12MySQL_Thread20process_all_sessionsEv+0x208)[0x473f78] proxysql(_ZN12MySQL_Thread3runEv+0xdde)[0x47d8fe] proxysql(_Z24mysql_worker_thread_funcPv+0x7c)[0x44b6fc] /lib/x86_64-linux-gnu/libpthread.so.0(+0x8184)[0x7f229a660184] /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f229912603d] 2019-03-04 11:58:30 main.cpp:910:ProxySQL_daemonize_phase3(): [ERROR] ProxySQL crashed. Restarting! 2019-03-04 11:58:30 [INFO] Angel process started ProxySQL process 24769 core-hub-ca-pxwelldb1-tst-03-50v.gz core-hub-ca-pxhubdb2-tst-01-50v.gz

[mgale]

The core files are from ProxySQL version 1.4.15.

[shrestha-s]

I can reproduce crashes in v1.4.15 by logging into the backend db through proxysql, using a user that has FAST_FORWARD enabled.

I essentially ran a bash script which logs into the backend db through proxysql, using mysql cli and runs a command. E.g.

#!/bin/bash

for  i in {1..50}
do
    echo try $i
    mysql -h <host> -u<user> -p<password> -e "show databases"
    sleep 5
done

This would caush ProxySQL v1.4.15 to crash about every 4-5 iterations. I changed the query being run as well to no avail (e.g. using a SELECT statement instead of SHOW DATABASE)

If I disable FAST_FORWARD for the same user, the crashes seemed to disappear.

My configuration: Proxysql version: 1.4.15-1-g61bc777, codename Truls MySQL version: 5.6.37

[renecannao]

@mgale : which proxysql binary are you using? A link to the package you installed will be useful.

@shrestha-s : your issue should be unrelated to what @mgale reported. I just fixed it in 1.4.16.

[mgale]

@renecannao We pull the debians from here: http://repo.proxysql.com/ProxySQL/proxysql-1.4.x/trusty/

apt-cache proxysql output: `Package: proxysql Versions: 1.4.15 (/var/lib/apt/lists/repo.proxysql.com_ProxySQL_proxysql-1.4.xtrusty._Packages) Description Language: File: /var/lib/apt/lists/repo.proxysql.com_ProxySQL_proxysql-1.4.xtrusty._Packages MD5: 927896cf559e9a147e66de11872db9da

1.4.14 (/var/lib/apt/lists/repo.proxysql.com_ProxySQL_proxysql-1.4.xtrusty._Packages) Description Language: File: /var/lib/apt/lists/repo.proxysql.com_ProxySQL_proxysql-1.4.xtrusty._Packages MD5: 927896cf559e9a147e66de11872db9da

1.4.13 (/var/lib/apt/lists/repo.proxysql.com_ProxySQL_proxysql-1.4.xtrusty._Packages) (/var/lib/dpkg/status) Description Language: File: /var/lib/apt/lists/repo.proxysql.com_ProxySQL_proxysql-1.4.xtrusty._Packages MD5: 927896cf559e9a147e66de11872db9da `

dpkg-query output: Package: proxysql Status: install ok installed Priority: optional Section: misc Installed-Size: 23023 Maintainer: Rene Cannao rene.cannao@gmail.com Architecture: amd64 Version: 1.4.13 Conffiles: /etc/proxysql.cnf e2bbf9a04697db9076e499d457777578 /etc/init.d/proxysql f1a97545da29f291cc27a9035ec14aea Description: High performance MySQL proxy ProxySQL is a fast, reliable MySQL proxy with advanced runtime configuration management (virtually no configuration change requires a restart). . It features query routing, query caching, query rewriting (for queries generated by ORMs, for example) and is most of the time a drop-in replacement for mysqld from the point of view of the application. It can be configured and remote controlled through an SQL-compatible admin interface. Homepage: http://www.proxysql.com