Recurring FreeSwitch Crash

[vpaul7]

Describe the bug We recently upgraded FreeSwitch to version 1.10.9 and it is crashing almost twice a week or sometime more than twice.

To Reproduce Steps to reproduce the behavior: FreeSWITCH crashes most of the time during business hours.

Expected behavior FreeSwitch crash on its own, and restart. All active calls drop.

Package version or git hash FreeSWITCH Version 1.10.9-release.15~64bit (-release.15 64bit)

Trace logs We are getting below error: segfault at 1 ip 00002b2ad7cd08c1 sp 00002b2bdfad75a8 error 4 in libc-2.17.so[2b2ad7b61000+1c4000]

backtrace from core file

#0  0x00002b0409776387 in raise () from /usr/lib64/libc.so.6
#1  0x00002b0409777a78 in abort () from /usr/lib64/libc.so.6
#2  0x00002b04097b8f67 in __libc_message () from /usr/lib64/libc.so.6
#3  0x00002b04097c1329 in _int_free () from /usr/lib64/libc.so.6
#4  0x00002b040687c6be in sqlite3_free () from /usr/lib64/libsqlite3.so.0
#5  0x00002b040687cb95 in pcache1Free () from /usr/lib64/libsqlite3.so.0
#6  0x00002b040687cc42 in pcache1FreePage () from /usr/lib64/libsqlite3.so.0
#7  0x00002b040687cd2d in pcache1TruncateUnsafe () from /usr/lib64/libsqlite3.so.0
#8  0x00002b040687cd58 in pcache1Truncate () from /usr/lib64/libsqlite3.so.0
#9  0x00002b04068af13c in sqlite3PagerSharedLock () from /usr/lib64/libsqlite3.so.0
#10 0x00002b04068afaa9 in sqlite3BtreeBeginTrans () from /usr/lib64/libsqlite3.so.0
#11 0x00002b04068d0df8 in sqlite3VdbeExec () from /usr/lib64/libsqlite3.so.0
#12 0x00002b04068d7cff in sqlite3_step () from /usr/lib64/libsqlite3.so.0
#13 0x00002b04068cd7fa in sqlite3_exec () from /usr/lib64/libsqlite3.so.0
#14 0x00002b04064835fa in switch_core_db_exec (db=0x2b0480037f98, sql=0x2b048020c320 "delete from limit_data where uuid='34f60d0b-6699-46f5-8559-c22a58a7f281'", callback=0x0, data=0x0, errmsg=0x2b0518882960) at src/switch_core_db.c:90
#15 0x00002b04063f060e in switch_cache_db_execute_sql_real (dbh=0x2b048049ea10, sql=0x2b048020c320 "delete from limit_data where uuid='34f60d0b-6699-46f5-8559-c22a58a7f281'", err=0x0) at src/switch_core_sqldb.c:710
#16 0x00002b04063f07f7 in switch_cache_db_execute_sql_chunked (dbh=0x2b048049ea10, sql=0x2b048020c320 "delete from limit_data where uuid='34f60d0b-6699-46f5-8559-c22a58a7f281'", chunk_size=32768, err=0x0) at src/switch_core_sqldb.c:757
#17 0x00002b04063f094d in switch_cache_db_execute_sql (dbh=0x2b048049ea10, sql=0x2b048020c320 "delete from limit_data where uuid='34f60d0b-6699-46f5-8559-c22a58a7f281'", err=0x0) at src/switch_core_sqldb.c:806
#18 0x00002b040f39650d in limit_execute_sql (sql=0x2b048020c320 "delete from limit_data where uuid='34f60d0b-6699-46f5-8559-c22a58a7f281'") at mod_db.c:118
#19 0x00002b040f396ac2 in limit_release_db (session=0x2b04b85afe58, realm=0x0, resource=0x0) at mod_db.c:230
#20 0x00002b0406535b75 in switch_limit_release (backend=0x2b048025d530 "db", session=0x2b04b85afe58, realm=0x0, resource=0x0) at src/switch_limit.c:156
#21 0x00002b040653584e in limit_state_handler (session=0x2b04b85afe58) at src/switch_limit.c:96
#22 0x00002b04063ffbaf in switch_core_session_signal_state_change (session=0x2b04b85afe58) at src/switch_core_session.c:1490
#23 0x00002b04063c8111 in switch_channel_perform_set_state (channel=0x2b04b8266ed0, file=0x2b040678a848 "src/switch_core_state_machine.c", func=0x2b040678b420 <__func__.8> "switch_core_session_run", line=616, state=CS_REPORTING) at src/switch_channel.c:2606
#24 0x00002b0406407db1 in switch_core_session_run (session=0x2b04b85afe58) at src/switch_core_state_machine.c:616
#25 0x00002b0406400773 in switch_core_session_thread (thread=0x2b04f81c46b0, obj=0x2b04b85afe58) at src/switch_core_session.c:1726
#26 0x00002b0406400bbf in switch_core_session_thread_pool_worker (thread=0x2b04f81c46b0, obj=0x2b04f81c4540) at src/switch_core_session.c:1790
#27 0x00002b040676b21c in dummy_worker () from /usr/lib64/libfreeswitch.so.1.0.0
#28 0x00002b0408e56ea5 in start_thread () from /usr/lib64/libpthread.so.0
#29 0x00002b040983eb0d in clone () from /usr/lib64/libc.so.6

(gdb) set pagination off
(gdb) set logging file /tmp/backtrace.log
(gdb) set logging on
Copying output to /tmp/backtrace.log.
(gdb) bt
(gdb) bt full
(gdb) info threads
(gdb) thread apply all bt
(gdb) thread apply all bt full
(gdb) set logging off
Done logging to /tmp/backtrace.log.
(gdb) quit

[tal952]

Happens to us also, please help 🙏

[00Asgaroth00]

+1, we're experiencing the same issue, if need be I can also upload some backtraces