proxysql crashed when processing stmt_execute packet

btGuo commented 4 months ago

OS version: Debian 10; kernel: 4.19.117.bsk.business.1-amd64

ProxySQL version: 2.5.1-90-gbedaa6c

Mysql version: 5.7.37

Mysql client sdk: go-sql-driver/mysql

logs:

Error: signal 11:
/opt/tiger/proxysql/proxysql(_Z13crash_handleri+0x2a)[0x5596565e739a]
/lib/x86_64-linux-gnu/libc.so.6(+0x37970)[0x7f8651d5b970]
/opt/tiger/proxysql/proxysql(_Z19mysql_decode_lengthPhPm+0x5)[0x55965666fe55]
/opt/tiger/proxysql/proxysql(_ZN14MySQL_Protocol18get_binds_from_pktEPvjP22MySQL_STMT_Global_infoPP23stmt_execute_metadata_t+0x439)[0x5596566722f9]
/opt/tiger/proxysql/proxysql(_ZN13MySQL_Session75handler___status_WAITING_CLIENT_DATA___STATE_SLEEP___MYSQL_COM_STMT_EXECUTEER10_PtrSize_t+0x1fc)[0x55965665e81c]
/opt/tiger/proxysql/proxysql(_ZN13MySQL_Session20get_pkts_from_clientERbR10_PtrSize_t+0xce9)[0x559656668df9]
/opt/tiger/proxysql/proxysql(_ZN13MySQL_Session7handlerEv+0xb8)[0x5596566698b8]
/opt/tiger/proxysql/proxysql(_ZN12MySQL_Thread20process_all_sessionsEv+0x3bc)[0x559656649eec]
/opt/tiger/proxysql/proxysql(_ZN12MySQL_Thread3runEv+0x50a)[0x55965664a68a]
/opt/tiger/proxysql/proxysql(_Z24mysql_worker_thread_funcPv+0x74)[0x5596565dec44]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x7fa3)[0x7f8652245fa3]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f8651e1d06f]
 ---- /opt/tiger/proxysql/proxysql(_Z13crash_handleri+0x2a) [0x5596565e739a] : crash_handler(int)
 ---- /opt/tiger/proxysql/proxysql(_Z19mysql_decode_lengthPhPm+0x5) [0x55965666fe55] : mysql_decode_length(unsigned char*, unsigned long*)
 ---- /opt/tiger/proxysql/proxysql(_ZN14MySQL_Protocol18get_binds_from_pktEPvjP22MySQL_STMT_Global_infoPP23stmt_execute_metadata_t+0x439) [0x5596566722f9] : MySQL_Protocol::get_binds_from_pkt(
void*, unsigned int, MySQL_STMT_Global_info*, stmt_execute_metadata_t**)
 ---- /opt/tiger/proxysql/proxysql(_ZN13MySQL_Session20get_pkts_from_clientERbR10_PtrSize_t+0xce9) [0x559656668df9] : MySQL_Session::get_pkts_from_client(bool&, _PtrSize_t&)
 ---- /opt/tiger/proxysql/proxysql(_ZN13MySQL_Session7handlerEv+0xb8) [0x5596566698b8] : MySQL_Session::handler()
 ---- /opt/tiger/proxysql/proxysql(_ZN12MySQL_Thread20process_all_sessionsEv+0x3bc) [0x559656649eec] : MySQL_Thread::process_all_sessions()
 ---- /opt/tiger/proxysql/proxysql(_ZN12MySQL_Thread3runEv+0x50a) [0x55965664a68a] : MySQL_Thread::run()
 ---- /opt/tiger/proxysql/proxysql(_Z24mysql_worker_thread_funcPv+0x74) [0x5596565dec44] : mysql_worker_thread_func(void*)
To report a crashing bug visit: https://github.com/sysown/proxysql/issues
For support visit: https://proxysql.com/services/support/

proxysql crashed when processing stmt_execute packet; this problem happens twice in our production environment; finally I set fast_forward=1 and this problem did not occur again.

renecannao commented 4 months ago

Hi @btGuo . Thank you for the report. Unless you provide a reproducible test case and/or a core dump, there is little I can suggest right now other than upgrading to the latest version. Are you able to provide a core dump?

Please note that setting fast_forward=1 disables the vast majority of the features of proxysql.

btGuo commented 4 months ago

Hi @btGuo . Thank you for the report. Unless you provide a reproducible test case and/or a core dump, there is little I can suggest right now other than upgrading to the latest version. Are you able to provide a core dump?

Please note that setting fast_forward=1 disables the vast majority of the features of proxysql.

Thanks for your reply, unfortunately the core dump file was lost. If I get a core file next time, I will try provide it. I have read the bug fix between 2.5.1 and the newest version, but I can't find something related to this problem. These days I reading the source code and I found that mysql_decode_length was called in case MYSQL_TYPE_GEOMETRY, but our program never use this mysql data type, I think that there maybe some dirty memory. I understand that enable fast_forward will lost many features (such as stats_mysql_query_digests metrics we currently used).

renecannao commented 4 months ago

Hi @btGuo .

Yes please, if you try to reproduce it, please let us know.

Please note that mysql_decode_length is used for many data type, specifically any data type that do not have a fixed length.

btGuo commented 4 months ago

Hi @btGuo .

Yes please, if you try to reproduce it, please let us know.

Please note that mysql_decode_length is used for many data type, specifically any data type that do not have a fixed length.

Oh, I got it, there are multi 'case'.

sysown / proxysql