ProxySQL 2 is Crashing on free/delete

[altmannmarcelo]

ProxySQL is crashing (on what it looks) when terminating connections. The issue has been observed in 2.0.4 and is still occurring on 2.0.6:

2.0.6

(gdb) bt
#0  atomic_load_p (mo=atomic_memory_order_acquire, a=0x48) at include/jemalloc/internal/atomic.h:62
#1  extent_prof_tctx_get (extent=0x0) at include/jemalloc/internal/extent_inlines.h:185
#2  je_large_prof_tctx_get (tsdn=tsdn@entry=0x7f7d2317aad0, extent=0x0) at src/large.c:374
#3  0x00000000007de2a7 in arena_prof_tctx_get (alloc_ctx=<synthetic pointer>, ptr=0x7f7d1b003630, tsdn=0x7f7d2317aad0) at include/jemalloc/internal/arena_inlines_b.h:50
#4  prof_tctx_get (alloc_ctx=<synthetic pointer>, ptr=0x7f7d1b003630, tsdn=0x7f7d2317aad0) at include/jemalloc/internal/prof_inlines_b.h:44
#5  prof_free (alloc_ctx=<synthetic pointer>, usize=<optimized out>, ptr=0x7f7d1b003630, tsd=0x7f7d2317aad0) at include/jemalloc/internal/prof_inlines_b.h:239
#6  ifree (slow_path=false, tcache=0x7f7d2317acc0, ptr=0x7f7d1b003630, tsd=0x7f7d2317aad0) at src/jemalloc.c:2495
#7  je_free_default (ptr=0x7f7d1b003630) at src/jemalloc.c:2708
#8  0x00000000005eea2c in MySQL_Connection::~MySQL_Connection (this=0x7f7d21621480, __in_chrg=<optimized out>) at mysql_connection.cpp:254
#9  0x00000000005f2408 in MySQL_Data_Stream::~MySQL_Data_Stream (this=0x7f7d216d0c00, __in_chrg=<optimized out>) at mysql_data_stream.cpp:289
#10 0x00000000004f0088 in MySQL_Session::~MySQL_Session (this=0x7f7d216c4100, __in_chrg=<optimized out>) at MySQL_Session.cpp:506
#11 0x00000000004dc79b in MySQL_Thread::process_all_sessions (this=this@entry=0x7f7d21600000) at MySQL_Thread.cpp:3836
#12 0x00000000004e5e16 in MySQL_Thread::run (this=this@entry=0x7f7d21600000) at MySQL_Thread.cpp:3565
#13 0x000000000049fbac in mysql_worker_thread_func (arg=0x7f7d26c2bc70) at main.cpp:648
#14 0x00007f7d2836cdd5 in start_thread () from /lib64/libpthread.so.0
#15 0x00007f7d27254ead in clone () from /lib64/libc.so.6
(gdb) frame 8
#8  0x00000000005eea2c in MySQL_Connection::~MySQL_Connection (this=0x7f7d21621480, __in_chrg=<optimized out>) at mysql_connection.cpp:254
254 mysql_connection.cpp: No such file or directory.
(gdb) p local_stmts
$1 = (MySQL_STMTs_local_v14 *) 0x7f7d21622200
(gdb) p *local_stmts
$2 = {is_client_ = 32, free_client_ids = std::stack wrapping: std::deque with 0 elements, local_max_stmt_id = 0, client_stmt_to_global_ids = std::map with 0 elements, global_stmt_to_client_ids = std::multimap with 0 elements,
  backend_stmt_to_global_ids = std::map with 0 elements, global_stmt_to_backend_ids = std::map with 0 elements, global_stmt_to_backend_stmt = std::map with 0 elements, sess = 0x7f7d216c4100}

2.0.4:

(gdb) bt
#0  atomic_load_p (mo=atomic_memory_order_relaxed, a=0x1b1818) at include/jemalloc/internal/atomic.h:62
#1  rtree_leaf_elm_bits_read (tsdn=<optimized out>, rtree=<optimized out>, dependent=true, elm=0x1b1818) at include/jemalloc/internal/rtree.h:175
#2  rtree_szind_slab_read (r_slab=<synthetic pointer>, r_szind=<synthetic pointer>, dependent=true, key=139638885856051, rtree_ctx=0x7f9f35be8b20, rtree=<optimized out>, tsdn=0x7f9f35be8af0) at include/jemalloc/internal/rtree.h:500
#3  ifree (slow_path=false, tcache=0x7f9f35be8ce0, ptr=0x7f0036303333, tsd=0x7f9f35be8af0) at src/jemalloc.c:2488
#4  je_free_default (ptr=0x7f0036303333) at src/jemalloc.c:2708
#5  0x00000000005e38ac in MySQL_Connection::~MySQL_Connection (this=0x7f9f32790700, __in_chrg=<optimized out>) at mysql_connection.cpp:265
#6  0x00000000005e7398 in MySQL_Data_Stream::destroy_queues (this=0x7f9f2b02e880) at mysql_data_stream.cpp:1382
#7  0x00000000004b4348 in MySQL_Authentication::decrease_frontend_user_connections (this=0x7f9f32790650, username=0x0) at MySQL_Authentication.cpp:328
#8  0x00000000004ed43f in MySQL_Session::~MySQL_Session (this=0x7f9f2b02e880, __in_chrg=<optimized out>) at MySQL_Session.cpp:391
#9  0x00000000004dabb8 in MySQL_Thread::process_all_sessions (this=this@entry=0x7f9f326ff000) at MySQL_Thread.cpp:3689
#10 0x00000000004e3bb6 in MySQL_Thread::run (this=this@entry=0x7f9f326ff000) at MySQL_Thread.cpp:3427
#11 0x000000000049eb2c in mysql_worker_thread_func (arg=0x7f9f37258f90) at main.cpp:627
#12 0x00007f9f388abdd5 in start_thread () from /lib64/libpthread.so.0
#13 0x00007f9f3779402d in gnu_dev_makedev () from /lib64/libc.so.6
#14 0x0000000000000000 in ?? ()

OS: Red Hat Enterprise Linux Server release 7.6 (Maipo) Kernel: 3.10.0-957.1.3.el7.x86_64

I will be sending core dump via email

[renecannao]

@altmannmarcelo : thank you for the core dump. Are you able to reproduce it? I would also like to have a look at error log

[altmannmarcelo]

Hi @renecannao . We were unable to reproduce it. The crash happened a few seconds after the writer of a GR cluster missed a health check.

I've sent the error log via Email.