search

signalwire / freeswitch

FreeSWITCH is a Software Defined Telecom Stack enabling the digital transformation from proprietary telecom switches to a versatile software implementation that runs on any commodity hardware. From a Raspberry PI to a multi-core server, FreeSWITCH can unlock the telecommunications potential of any device.
https://freeswitch.com/#getting-started
Other
3.49k stars 1.4k forks source link

FreeSwitch crashs every day #1644

Open dxhgq-github opened 2 years ago

dxhgq-github commented 2 years ago

Hello,

FreeSwitch crashs one or two times every 24 hours , the FS version is 1.10.7 ,the OS is centos 7 ,the only error messages in /var/log/messages is : Apr 25 12:19:43 localhost systemd: freeswitch.service: main process exited, code=killed, status=8/FPE Apr 25 12:19:43 localhost systemd: Unit freeswitch.service entered failed state. Apr 25 12:19:43 localhost systemd: freeswitch.service failed.

Backtrace:

Core was generated by `/usr/local/fs1.10.7/bin/freeswitch -nonat'.
Program terminated with signal SIGFPE, Arithmetic exception.
#0  0x00007f1651eb87e8 in su_block_add () from /usr/local/lib/libsofia-sip-ua.so.0
[Current thread is 1 (LWP 45853)]
(gdb) where
#0  0x00007f36d8db1c3d in su_block_add (b=0x7f36945a3ce0, p=0x7f3694a73170) at su_alloc.c:339
#1  0x00007f36d8db21f2 in sub_alloc (home=0x7f36944eea20, sub=0x7f36945a3ce0, size=80, 
    zero=do_calloc) at su_alloc.c:527
#2  0x00007f36d8db3d2f in su_zalloc (home=0x7f36944eea20, size=80) at su_alloc.c:1574
#3  0x00007f36d8d3dcf9 in outgoing_make_a_aaaa_query (orq=0x7f36945a4a40) at nta.c:10343
#4  0x00007f36d8d3d32e in outgoing_resolve_next (orq=0x7f36945a4a40) at nta.c:10124
#5  0x00007f36d8d3eda7 in outgoing_answer_srv (orq=0x7f36945a4a40, q=0x7f36945a4c20, 
    answers=0x7f369411dfa0) at nta.c:10707
#6  0x00007f36d8da5fd9 in sres_query_report_error (q=0x7f36945a4c20, answers=0x7f369411dfa0)
    at sres.c:2990
#7  0x00007f36d8da6368 in sres_resend_dns_query (res=0x7f3694002960, q=0x7f36945a4c20, 
    timeout=0) at sres.c:3090
#8  0x00007f36d8da725b in sres_resolver_report_error (res=0x7f3694002960, socket=129, 
    errcode=111, remote=0x7f36ab318450, remotelen=16, 
    info=0x7f36ab3183d0 "icmp type=3 code=3 reported by 127.0.0.1") at sres.c:3439
#9  0x00007f36d8da6f29 in sres_resolver_error (res=0x7f3694002960, socket=129) at sres.c:3353
#10 0x00007f36d8db078d in sres_sofia_poll (magic=0x7f36ac001180, w=0x7f3694000ff4, 
    reg=0x7f3694003790) at sresolv.c:357
#11 0x00007f36d8db69ac in su_epoll_port_wait_events (self=0x7f36940008c0, tout=0)
    at su_epoll_port.c:510
#12 0x00007f36d8dbc17f in su_port_wait_events (self=0x7f36940008c0, timeout=0) at su_port.h:318
#13 0x00007f36d8dbd481 in su_root_yield (self=0x7f3694001130) at su_root.c:904
#14 0x00007f36d8d71ffd in nua_stack_timer (nua=0x7f36ac001180, t=0x7f36940011e0, a=0x0)
    at nua_stack.c:739
#15 0x00007f36d8dc69f9 in su_timer_expire (timers=0x7f3694000908, timeout=0x7f36ab318cd8, 
    now=...) at su_timer.c:585
#16 0x00007f36d8db4ee2 in su_base_port_run (self=0x7f36940008c0) at su_base_port.c:339
#17 0x00007f36d8dbc1aa in su_port_run (self=0x7f36940008c0) at su_port.h:326
#18 0x00007f36d8dbd284 in su_root_run (self=0x7f3694001130) at su_root.c:819
#19 0x00007f36d8dbb933 in su_pthread_port_clone_main (varg=0x7f36d8abb4a0)
    at su_pthread_port.c:343
#20 0x00007f36e6ec9e65 in start_thread () from /lib64/libpthread.so.0
#21 0x00007f36e651e88d in __libc_ifunc_impl_list () from /lib64/libc.so.6
#22 0x0000000000000000 in ?? ()

Thanks!

andywolk commented 2 years ago

Please provide a proper gdb log. See Creating A Backtrace With gdb section: https://freeswitch.org/confluence/display/FREESWITCH/Debugging

andywolk commented 2 years ago

What version of the sofia-sip library are you using?

dxhgq-github commented 2 years ago

sofia-sip 1.13.7

dxhgq-github commented 2 years ago

Thread 146 (LWP 63539):

0 0x00007f1650ef571d in write () from /lib64/libpthread.so.0

1 0x0000000000000000 in ?? ()

Thread 145 (LWP 45867):

0 0x00007f1650543e63 in arch_prctl () from /lib64/libc.so.6

1 0x0000000000000000 in ?? ()

Thread 144 (LWP 64468):

0 0x00007f1650538bed in fts_read () from /lib64/libc.so.6

1 0x00007f16542a3607 in rtp_common_read (rtp_session=<error reading variable: Cannot access memory at address 0xfffffe08>, payload_type=<error reading variable: Cannot access memory at address 0xfffffe00>, pmapP=<error reading variable: Cannot access memory at address 0xfffffdf8>, flags=<error reading variable: Cannot access memory at address 0xfffffdf0>, io_flags=<error reading variable: Cannot access memory at address 0xfffffdec>) at src/switch_rtp.c:7571

Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 143 (LWP 45890):

0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007f158e8aafe1 in boost::asio::detail::task_io_service::run(boost::system::error_code&) () from /lib64/librocketmq.so

2 0x00007f158e8ac136 in boost::asio::io_service::run() () from /lib64/librocketmq.so

3 0x00007f158e98a7cf in thread_proxy () from /lib64/librocketmq.so

4 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0

5 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6

6 0x0000000000000000 in ?? ()

Thread 142 (LWP 45893):

0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007f158e8aafe1 in boost::asio::detail::task_io_service::run(boost::system::error_code&) () from /lib64/librocketmq.so

2 0x00007f158e8ac136 in boost::asio::io_service::run() () from /lib64/librocketmq.so

3 0x00007f158e98a7cf in thread_proxy () from /lib64/librocketmq.so

4 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0

5 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6

6 0x0000000000000000 in ?? ()

Thread 141 (LWP 65096):

0 0x00007f165053a953 in setegid () from /lib64/libc.so.6

1 0x000000000007a120 in ?? ()

2 0x00007f152a359126 in ?? () from /usr/local/fs1.10.7/lib/freeswitch/mod/mod_callcenter.so

3 0x0000000002a1be40 in ?? ()

4 0x00007f1493e53ca0 in ?? ()

5 0x00007f165431193c in switch_sleep (t=44154432) at src/switch_time.c:644

Backtrace stopped: frame did not save the PC

Thread 140 (LWP 45895):

0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007f158e8aafe1 in boost::asio::detail::task_io_service::run(boost::system::error_code&) () from /lib64/librocketmq.so

2 0x00007f158e8ac136 in boost::asio::io_service::run() () from /lib64/librocketmq.so

3 0x00007f158e98a7cf in thread_proxy () from /lib64/librocketmq.so

4 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0

5 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6

6 0x0000000000000000 in ?? ()

Thread 139 (LWP 45905):

0 0x00007f165050a80d in sleep () from /lib64/libc.so.6

1 0x0000000000000000 in ?? ()

Thread 138 (LWP 45876):

0 0x00007f165053a953 in setegid () from /lib64/libc.so.6

1 0x00000000000f4240 in ?? ()

2 0x0000000000000000 in ?? ()

Thread 137 (LWP 45879):

0 0x00007f1650ef571d in write () from /lib64/libpthread.so.0

1 0x0000000000000000 in ?? ()

Thread 136 (LWP 45880):

0 0x00007f1650ef571d in write () from /lib64/libpthread.so.0

1 0x0000000000000000 in ?? ()

Thread 135 (LWP 45883):

0 0x00007f1650543e63 in arch_prctl () from /lib64/libc.so.6

1 0x0000000000000000 in ?? ()

Thread 134 (LWP 64687):

0 0x00007f1650ef571d in write () from /lib64/libpthread.so.0

1 0x0000000000000000 in ?? ()

Thread 133 (LWP 45886):

0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007f158e8aafe1 in boost::asio::detail::task_io_service::run(boost::system::error_code&) () from /lib64/librocketmq.so

2 0x00007f158e8ac136 in boost::asio::io_service::run() () from /lib64/librocketmq.so

3 0x00007f158e98a7cf in thread_proxy () from /lib64/librocketmq.so

4 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0

5 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6

6 0x0000000000000000 in ?? ()

Thread 132 (LWP 63929):

0 0x00007f1650ef2da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007f1654549702 in apr_thread_cond_timedwait (cond=0x7f1654c38f48, mutex=0x7f1654c38ef8, timeout=5000000) at locks/unix/thread_cond.c:89

2 0x00007f1654540dd9 in apr_queue_pop_timeout (queue=0x7f1654c38eb8, data=0x7f1493fc1d38, timeout=5000000) at misc/apr_queue.c:339

3 0x00007f16541a8c64 in switch_queue_pop_timeout (queue=0x7f1654c38eb8, data=0x7f1493fc1d38, timeout=5000000) at src/switch_apr.c:1225

4 0x00007f16541e8709 in switch_core_session_thread_pool_worker (thread=0x7f1628382dc0, obj=0x7f1628382c50) at src/switch_core_session.c:1792

5 0x00007f165454f770 in dummy_worker (opaque=0x7f1628382dc0) at threadproc/unix/thread.c:151

6 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0

7 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6

8 0x0000000000000000 in ?? ()

Thread 131 (LWP 64389):

0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007f16541a740f in switch_thread_cond_wait (cond=0x7f16180a1210, mutex=0x7f16180a11c0) at src/switch_apr.c:371

2 0x00007f16542cf5d2 in recording_thread (thread=0x7f16180a13d0, obj=0x7f16180af708) at src/switch_ivr_async.c:1329

3 0x00007f165454f770 in dummy_worker (opaque=0x7f16180a13d0) at threadproc/unix/thread.c:151

4 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0

5 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6

6 0x0000000000000000 in ?? ()

Thread 130 (LWP 64075):

0 0x00007f1650ef571d in write () from /lib64/libpthread.so.0

1 0x0000000000000000 in ?? ()

Thread 129 (LWP 64456):

0 0x00007f16505598c4 in __get_clockfreq () from /lib64/libc.so.6

1 0x0000000000000000 in ?? ()

Thread 128 (LWP 64839):

0 0x00007f165053a953 in setegid () from /lib64/libc.so.6

1 0x000000000007a120 in ?? ()

2 0x00007f152a359126 in ?? () from /usr/local/fs1.10.7/lib/freeswitch/mod/mod_callcenter.so

3 0x0000000002a1be40 in ?? ()

4 0x00007f1492dacca0 in ?? ()

5 0x00007f165431193c in switch_sleep (t=44154432) at src/switch_time.c:644

Backtrace stopped: frame did not save the PC

Thread 127 (LWP 64774):

0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007f16541a740f in switch_thread_cond_wait (cond=0x7f15ac05ccc8, mutex=0x7f15ac05cc78) at src/switch_apr.c:371

2 0x00007f16542cf5d2 in recording_thread (thread=0x7f15ac05ce88, obj=0x7f15ac032d78) at src/switch_ivr_async.c:1329

3 0x00007f165454f770 in dummy_worker (opaque=0x7f15ac05ce88) at threadproc/unix/thread.c:151

4 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0

5 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6

6 0x0000000000000000 in ?? ()

Thread 126 (LWP 64469):

0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007f16541a740f in switch_thread_cond_wait (cond=0x7f149c02aff0, mutex=0x7f149c02afa0) at src/switch_apr.c:371

2 0x00007f16542cf5d2 in recording_thread (thread=0x7f149c02b1b0, obj=0x7f149cb4cbe8) at src/switch_ivr_async.c:1329

3 0x00007f165454f770 in dummy_worker (opaque=0x7f149c02b1b0) at threadproc/unix/thread.c:151

4 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0

5 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6

6 0x0000000000000000 in ?? ()

Thread 125 (LWP 65053):

0 0x00007f1650ef571d in write () from /lib64/libpthread.so.0

1 0x0000000000000000 in ?? ()

Thread 124 (LWP 65073):

0 0x00007f1650ef571d in write () from /lib64/libpthread.so.0

1 0x0000000000000000 in ?? ()

Thread 123 (LWP 65095):

0 0x00007f165053a953 in setegid () from /lib64/libc.so.6

1 0x0000000003473bc0 in ?? ()

2 0x0000000000000000 in ?? ()

Thread 122 (LWP 65104):

0 0x00007f165053a953 in setegid () from /lib64/libc.so.6

1 0x000000000007a120 in ?? ()

2 0x00007f152a359126 in ?? () from /usr/local/fs1.10.7/lib/freeswitch/mod/mod_callcenter.so

3 0x0000000002a1be40 in ?? ()

4 0x00007f155c145ca0 in ?? ()

5 0x00007f165431193c in switch_sleep (t=44154432) at src/switch_time.c:644

Backtrace stopped: frame did not save the PC

Thread 121 (LWP 45898):

0 0x00007f1650543e63 in arch_prctl () from /lib64/libc.so.6

1 0x0000000000000000 in ?? ()

Thread 120 (LWP 64225):

0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007f16541a740f in switch_thread_cond_wait (cond=0x7f150c041948, mutex=0x7f150c0418f8) at src/switch_apr.c:371

2 0x00007f16542cf5d2 in recording_thread (thread=0x7f150c041b08, obj=0x7f150c0d0098) at src/switch_ivr_async.c:1329

3 0x00007f165454f770 in dummy_worker (opaque=0x7f150c041b08) at threadproc/unix/thread.c:151

4 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0

5 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6

6 0x0000000000000000 in ?? ()

Thread 119 (LWP 64452):

0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007f16541a740f in switch_thread_cond_wait (cond=0x7f1498ab2a80, mutex=0x7f1498ab2a30) at src/switch_apr.c:371

2 0x00007f16542cf5d2 in recording_thread (thread=0x7f1498ab2c40, obj=0x7f1498730808) at src/switch_ivr_async.c:1329

3 0x00007f165454f770 in dummy_worker (opaque=0x7f1498ab2c40) at threadproc/unix/thread.c:151

4 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0

5 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6

6 0x0000000000000000 in ?? ()

Thread 118 (LWP 64451):

0 0x00007f1650538bed in fts_read () from /lib64/libc.so.6

1 0x00007f16542a3607 in rtp_common_read (rtp_session=<error reading variable: Cannot access memory at address 0xfffffe08>, payload_type=<error reading variable: Cannot access memory at address 0xfffffe00>, pmapP=<error reading variable: Cannot access memory at address 0xfffffdf8>, flags=<error reading variable: Cannot access memory at address 0xfffffdf0>, io_flags=<error reading variable: Cannot access memory at address 0xfffffdec>) at src/switch_rtp.c:7571

Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 117 (LWP 65040):

0 0x00007f1650ef571d in write () from /lib64/libpthread.so.0

1 0x0000000000000000 in ?? ()

Thread 116 (LWP 65059):

0 0x00007f165053a953 in setegid () from /lib64/libc.so.6

1 0x0000000003473bc0 in ?? ()

2 0x0000000000000000 in ?? ()

Thread 115 (LWP 45877):

0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007f16541a740f in switch_thread_cond_wait (cond=0x2af4f58, mutex=0x2af4f08) at src/switch_apr.c:371

2 0x00007f15ccec65cd in timer_thread_run (thread=0x2af5118, obj=0x0) at mod_spandsp_fax.c:223

3 0x00007f165454f770 in dummy_worker (opaque=0x2af5118) at threadproc/unix/thread.c:151

4 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0

5 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6

6 0x0000000000000000 in ?? ()

Thread 114 (LWP 45882):

0 0x00007f1650ef571d in write () from /lib64/libpthread.so.0

1 0x0000000000000000 in ?? ()

Thread 113 (LWP 65214):

0 0x00007f1650ef571d in write () from /lib64/libpthread.so.0

1 0x0000000000000000 in ?? ()

Thread 112 (LWP 45884):

0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007f158e8aafe1 in boost::asio::detail::task_io_service::run(boost::system::error_code&) () from /lib64/librocketmq.so

2 0x00007f158e8ac136 in boost::asio::io_service::run() () from /lib64/librocketmq.so

3 0x00007f158e98a7cf in thread_proxy () from /lib64/librocketmq.so

4 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0

5 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6

6 0x0000000000000000 in ?? ()

Thread 111 (LWP 45885):

0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007f158e8aafe1 in boost::asio::detail::task_io_service::run(boost::system::error_code&) () from /lib64/librocketmq.so

2 0x00007f158e8ac136 in boost::asio::io_service::run() () from /lib64/librocketmq.so

3 0x00007f158e98a7cf in thread_proxy () from /lib64/librocketmq.so

4 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0

5 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6

6 0x0000000000000000 in ?? ()

Thread 110 (LWP 65071):

0 0x00007f1650ef2da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007f1654549702 in apr_thread_cond_timedwait (cond=0x7f1654c38f48, mutex=0x7f1654c38ef8, timeout=5000000) at locks/unix/thread_cond.c:89

2 0x00007f1654540dd9 in apr_queue_pop_timeout (queue=0x7f1654c38eb8, data=0x7f1493123d38, timeout=5000000) at misc/apr_queue.c:339

3 0x00007f16541a8c64 in switch_queue_pop_timeout (queue=0x7f1654c38eb8, data=0x7f1493123d38, timeout=5000000) at src/switch_apr.c:1225

4 0x00007f16541e8709 in switch_core_session_thread_pool_worker (thread=0x7f14f002fb50, obj=0x7f14f002f9e0) at src/switch_core_session.c:1792

5 0x00007f165454f770 in dummy_worker (opaque=0x7f14f002fb50) at threadproc/unix/thread.c:151

6 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0

7 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6

8 0x0000000000000000 in ?? ()

Thread 109 (LWP 65077):

0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007f16541a740f in switch_thread_cond_wait (cond=0x7f14f893b140, mutex=0x7f14f893b0f0) at src/switch_apr.c:371

2 0x00007f16542cf5d2 in recording_thread (thread=0x7f14f893b300, obj=0x7f14f806ec08) at src/switch_ivr_async.c:1329

3 0x00007f165454f770 in dummy_worker (opaque=0x7f14f893b300) at threadproc/unix/thread.c:151

4 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0

5 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6

6 0x0000000000000000 in ?? ()

Thread 108 (LWP 45887):

0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007f158e8aafe1 in boost::asio::detail::task_io_service::run(boost::system::error_code&) () from /lib64/librocketmq.so

2 0x00007f158e8ac136 in boost::asio::io_service::run() () from /lib64/librocketmq.so

3 0x00007f158e98a7cf in thread_proxy () from /lib64/librocketmq.so

4 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0

5 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6

6 0x0000000000000000 in ?? ()

Thread 107 (LWP 45888):

0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007f158e8aafe1 in boost::asio::detail::task_io_service::run(boost::system::error_code&) () from /lib64/librocketmq.so

2 0x00007f158e8ac136 in boost::asio::io_service::run() () from /lib64/librocketmq.so

3 0x00007f158e98a7cf in thread_proxy () from /lib64/librocketmq.so

4 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0

5 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6

6 0x0000000000000000 in ?? ()

Thread 106 (LWP 45889):

0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007f158e8aafe1 in boost::asio::detail::task_io_service::run(boost::system::error_code&) () from /lib64/librocketmq.so

2 0x00007f158e8ac136 in boost::asio::io_service::run() () from /lib64/librocketmq.so

3 0x00007f158e98a7cf in thread_proxy () from /lib64/librocketmq.so

4 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0

5 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6

6 0x0000000000000000 in ?? ()

Thread 105 (LWP 45891):

0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007f158e8aafe1 in boost::asio::detail::task_io_service::run(boost::system::error_code&) () from /lib64/librocketmq.so

2 0x00007f158e8ac136 in boost::asio::io_service::run() () from /lib64/librocketmq.so

3 0x00007f158e98a7cf in thread_proxy () from /lib64/librocketmq.so

4 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0

5 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6

6 0x0000000000000000 in ?? ()

Thread 104 (LWP 45892):

0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007f158e8aafe1 in boost::asio::detail::task_io_service::run(boost::system::error_code&) () from /lib64/librocketmq.so

2 0x00007f158e8ac136 in boost::asio::io_service::run() () from /lib64/librocketmq.so

3 0x00007f158e98a7cf in thread_proxy () from /lib64/librocketmq.so

4 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0

5 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6

6 0x0000000000000000 in ?? ()

Thread 103 (LWP 45894):

0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007f158e8aafe1 in boost::asio::detail::task_io_service::run(boost::system::error_code&) () from /lib64/librocketmq.so

2 0x00007f158e8ac136 in boost::asio::io_service::run() () from /lib64/librocketmq.so

3 0x00007f158e98a7cf in thread_proxy () from /lib64/librocketmq.so

4 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0

5 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6

6 0x0000000000000000 in ?? ()

Thread 102 (LWP 45896):

0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007f158e8aafe1 in boost::asio::detail::task_io_service::run(boost::system::error_code&) () from /lib64/librocketmq.so

2 0x00007f158e8ac136 in boost::asio::io_service::run() () from /lib64/librocketmq.so

3 0x00007f158e98a7cf in thread_proxy () from /lib64/librocketmq.so

4 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0

5 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6

6 0x0000000000000000 in ?? ()

Thread 101 (LWP 45897):

0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007f158e8aafe1 in boost::asio::detail::task_io_service::run(boost::system::error_code&) () from /lib64/librocketmq.so

2 0x00007f158e932b15 in rocketmq::TcpRemotingClient::boost_asio_work() () from /lib64/librocketmq.so

3 0x00007f158e98a7cf in thread_proxy () from /lib64/librocketmq.so

4 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0

5 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6

6 0x0000000000000000 in ?? ()

Thread 100 (LWP 65207):

0 0x00007f165053a953 in setegid () from /lib64/libc.so.6

1 0x0000000003473bc0 in ?? ()

2 0x0000000000000000 in ?? ()

Thread 99 (LWP 65166):

0 0x00007f165053a953 in setegid () from /lib64/libc.so.6

1 0x0000000003473bc0 in ?? ()

2 0x0000000000000000 in ?? ()

Thread 98 (LWP 65060):

0 0x00007f165053a953 in setegid () from /lib64/libc.so.6

1 0x000000000007a120 in ?? ()

2 0x00007f152a359126 in ?? () from /usr/local/fs1.10.7/lib/freeswitch/mod/mod_callcenter.so

3 0x0000000002a1be40 in ?? ()

4 0x00007f1529095ca0 in ?? ()

5 0x00007f165431193c in switch_sleep (t=44154432) at src/switch_time.c:644

Backtrace stopped: frame did not save the PC

Thread 97 (LWP 45944):

0 0x00007f1650538bed in fts_read () from /lib64/libc.so.6

1 0x00007f1648f81b49 in read_packet (listener=<error reading variable: Cannot access memory at address 0x15c08d818>, event=<error reading variable: Cannot access memory at address 0x15c08d810>, timeout=<error reading variable: Cannot access memory at address 0x15c08d80c>) at mod_event_socket.c:1496

Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 96 (LWP 45986):

0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007f165454969d in apr_thread_cond_wait (cond=, mutex=) at locks/unix/thread_cond.c:68

2 0x00007f1654540ce1 in apr_queue_pop (queue=0x7f164c6008e8, data=0x7f15296abcf0) at misc/apr_queue.c:276

3 0x00007f16541a8c37 in switch_queue_pop (queue=0x7f164c6008e8, data=0x7f15296abcf0) at src/switch_apr.c:1220

4 0x00007f1654288b9e in switch_event_channel_deliver_thread (thread=0x7f1640010b10, obj=0x7f164c6008e8) at src/switch_event.c:3001

5 0x00007f16541e8786 in switch_core_session_thread_pool_worker (thread=0x7f1640010b10, obj=0x7f16400109a0) at src/switch_core_session.c:1800

6 0x00007f165454f770 in dummy_worker (opaque=0x7f1640010b10) at threadproc/unix/thread.c:151

7 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0

8 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6

9 0x0000000000000000 in ?? ()

Thread 95 (LWP 61545):

0 0x00007f1650ef571d in write () from /lib64/libpthread.so.0

1 0x0000000000000000 in ?? ()

Thread 94 (LWP 63430):

0 0x00007f1650ef571d in write () from /lib64/libpthread.so.0

1 0x0000000000000000 in ?? ()

Thread 93 (LWP 63542):

0 0x00007f1650ef571d in write () from /lib64/libpthread.so.0

1 0x0000000000000000 in ?? ()

Thread 92 (LWP 63867):

0 0x00007f1650ef571d in write () from /lib64/libpthread.so.0

1 0x0000000000000000 in ?? ()

Thread 91 (LWP 64379):

0 0x00007f1650ef571d in write () from /lib64/libpthread.so.0

1 0x0000000000000000 in ?? ()

Thread 90 (LWP 64450):

0 0x00007f16505598c4 in __get_clockfreq () from /lib64/libc.so.6

1 0x0000000000000000 in ?? ()

Thread 89 (LWP 63971):

0 0x00007f1650ef571d in write () from /lib64/libpthread.so.0

1 0x0000000000000000 in ?? ()

Thread 88 (LWP 63972):

0 0x00007f1650ef571d in write () from /lib64/libpthread.so.0

1 0x0000000000000000 in ?? ()

Thread 87 (LWP 64092):

0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007f16541a740f in switch_thread_cond_wait (cond=0x7f15bc4bf400, mutex=0x7f15bc4bf3b0) at src/switch_apr.c:371

2 0x00007f16542cf5d2 in recording_thread (thread=0x7f15bc4bf5c0, obj=0x7f15bc196e08) at src/switch_ivr_async.c:1329

3 0x00007f165454f770 in dummy_worker (opaque=0x7f15bc4bf5c0) at threadproc/unix/thread.c:151

4 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0

5 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6

6 0x0000000000000000 in ?? ()

Thread 86 (LWP 65167):

0 0x00007f165053a953 in setegid () from /lib64/libc.so.6

1 0x000000000007a120 in ?? ()

2 0x00007f152a359126 in ?? () from /usr/local/fs1.10.7/lib/freeswitch/mod/mod_callcenter.so

3 0x0000000002a1be40 in ?? ()

4 0x00007f1493282ca0 in ?? ()

5 0x00007f165431193c in switch_sleep (t=44154432) at src/switch_time.c:644

Backtrace stopped: frame did not save the PC

Thread 85 (LWP 65151):

0 0x00007f165053a953 in setegid () from /lib64/libc.so.6

1 0x000000000007a120 in ?? ()

2 0x00007f152a359126 in ?? () from /usr/local/fs1.10.7/lib/freeswitch/mod/mod_callcenter.so

3 0x0000000002a1be40 in ?? ()

4 0x00007f14935bcca0 in ?? ()

5 0x00007f165431193c in switch_sleep (t=44154432) at src/switch_time.c:644

Backtrace stopped: frame did not save the PC

Thread 84 (LWP 65212):

0 0x00007f1650ef571d in write () from /lib64/libpthread.so.0

1 0x0000000000000000 in ?? ()

Thread 83 (LWP 64561):

0 0x00007f1650ef571d in write () from /lib64/libpthread.so.0

1 0x0000000000000000 in ?? ()

Thread 82 (LWP 64700):

0 0x00007f1650ef571d in write () from /lib64/libpthread.so.0

1 0x0000000000000000 in ?? ()

Thread 81 (LWP 64703):

0 0x00007f1650ef571d in write () from /lib64/libpthread.so.0

1 0x0000000000000000 in ?? ()

Thread 80 (LWP 64838):

0 0x00007f165053a953 in setegid () from /lib64/libc.so.6

1 0x0000000003473bc0 in ?? ()

2 0x0000000000000000 in ?? ()

Thread 79 (LWP 64855):

0 0x00007f165053a953 in setegid () from /lib64/libc.so.6

1 0x0000000003473bc0 in ?? ()

2 0x0000000000000000 in ?? ()

Thread 78 (LWP 64856):

0 0x00007f165053a953 in setegid () from /lib64/libc.so.6

1 0x000000000007a120 in ?? ()

2 0x00007f152a359126 in ?? () from /usr/local/fs1.10.7/lib/freeswitch/mod/mod_callcenter.so

3 0x0000000002dcc570 in ?? ()

4 0x00007f14939f7ca0 in ?? ()

5 0x00007f165431193c in switch_sleep (t=48022896) at src/switch_time.c:644

Backtrace stopped: frame did not save the PC

Thread 77 (LWP 64928):

0 0x00007f1650ef571d in write () from /lib64/libpthread.so.0

1 0x0000000000000000 in ?? ()

Thread 76 (LWP 64945):

0 0x00007f165053a953 in setegid () from /lib64/libc.so.6

1 0x0000000003473bc0 in ?? ()

2 0x0000000000000000 in ?? ()

Thread 75 (LWP 65048):

0 0x00007f1650ef571d in write () from /lib64/libpthread.so.0

1 0x0000000000000000 in ?? ()

Thread 74 (LWP 65062):

0 0x00007f165053a953 in setegid () from /lib64/libc.so.6

1 0x0000000003473bc0 in ?? ()

2 0x0000000000000000 in ?? ()

Thread 73 (LWP 65066):

0 0x00007f165053a953 in setegid () from /lib64/libc.so.6

1 0x0000000003473bc0 in ?? ()

2 0x0000000000000000 in ?? ()

Thread 72 (LWP 65072):

0 0x00007f16505598c4 in __get_clockfreq () from /lib64/libc.so.6

1 0x0000000000000000 in ?? ()

Thread 71 (LWP 65063):

0 0x00007f165053a953 in setegid () from /lib64/libc.so.6

1 0x000000000007a120 in ?? ()

2 0x00007f152a359126 in ?? () from /usr/local/fs1.10.7/lib/freeswitch/mod/mod_callcenter.so

3 0x0000000002a1be40 in ?? ()

4 0x00007f1528f3fca0 in ?? ()

5 0x00007f165431193c in switch_sleep (t=44154432) at src/switch_time.c:644

Backtrace stopped: frame did not save the PC

Thread 70 (LWP 65213):

0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007f16541a740f in switch_thread_cond_wait (cond=0x7f157000c780, mutex=0x7f157000c730) at src/switch_apr.c:371

2 0x00007f16542cf5d2 in recording_thread (thread=0x7f157000c940, obj=0x7f1570a7f758) at src/switch_ivr_async.c:1329

3 0x00007f165454f770 in dummy_worker (opaque=0x7f157000c940) at threadproc/unix/thread.c:151

4 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0

5 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6

6 0x0000000000000000 in ?? ()

Thread 69 (Thread 0x7f1654c9e8c0 (LWP 45827)):

0 0x00007f165053a953 in setegid () from /lib64/libc.so.6

1 0x00000000000f4240 in ?? ()

2 0x00007fff556c0910 in ?? ()

3 0x0000000002df4ca8 in ?? ()

4 0x00007fff556c0910 in ?? ()

5 0x00007f165431193c in switch_sleep (t=48188584) at src/switch_time.c:644

Backtrace stopped: frame did not save the PC

Thread 68 (LWP 45836):

0 0x00007f1650538bed in fts_read () from /lib64/libc.so.6

1 0x0000000000000005 in ?? ()

2 0xffffffffffffffff in ?? ()

3 0x0000000000000000 in ?? ()

Thread 67 (LWP 65087):

0 0x00007f165053a953 in setegid () from /lib64/libc.so.6

1 0x0000000003473bc0 in ?? ()

2 0x0000000000000000 in ?? ()

Thread 66 (LWP 65101):

0 0x00007f1650ef571d in write () from /lib64/libpthread.so.0

1 0x0000000000000000 in ?? ()

Thread 65 (LWP 65103):

0 0x00007f165053a953 in setegid () from /lib64/libc.so.6

1 0x0000000003473bc0 in ?? ()

2 0x0000000000000000 in ?? ()

Thread 64 (LWP 65108):

0 0x00007f165053a953 in setegid () from /lib64/libc.so.6

1 0x0000000003473bc0 in ?? ()

2 0x0000000000000000 in ?? ()

Thread 63 (LWP 65113):

0 0x00007f1650ef571d in write () from /lib64/libpthread.so.0

1 0x0000000000000000 in ?? ()

Thread 62 (LWP 65161):

0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007f16541a740f in switch_thread_cond_wait (cond=0x7f14d0004300, mutex=0x7f14d00042b0) at src/switch_apr.c:371

2 0x00007f16542cf5d2 in recording_thread (thread=0x7f14d00044c0, obj=0x7f14d000ca28) at src/switch_ivr_async.c:1329

3 0x00007f165454f770 in dummy_worker (opaque=0x7f14d00044c0) at threadproc/unix/thread.c:151

4 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0

5 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6

6 0x0000000000000000 in ?? ()

Thread 61 (LWP 65208):

0 0x00007f165053a953 in setegid () from /lib64/libc.so.6

1 0x000000000007a120 in ?? ()

2 0x00007f152a359126 in ?? () from /usr/local/fs1.10.7/lib/freeswitch/mod/mod_callcenter.so

3 0x0000000002a1be40 in ?? ()

4 0x00007f1529b42ca0 in ?? ()

5 0x00007f165431193c in switch_sleep (t=44154432) at src/switch_time.c:644

Backtrace stopped: frame did not save the PC

Thread 60 (LWP 65150):

0 0x00007f165053a953 in setegid () from /lib64/libc.so.6

1 0x0000000003473bc0 in ?? ()

2 0x0000000000000000 in ?? ()

Thread 59 (LWP 65160):

0 0x00007f1650ef571d in write () from /lib64/libpthread.so.0

1 0x0000000000000000 in ?? ()

Thread 58 (LWP 65162):

0 0x00007f1650ef571d in write () from /lib64/libpthread.so.0

1 0x0000000000000000 in ?? ()

Thread 57 (LWP 45830):

0 0x00007f165053a953 in setegid () from /lib64/libc.so.6

1 0x00000000000f4240 in ?? ()

2 0x000000000003d000 in ?? ()

3 0x00007f1654c38c40 in ?? ()

4 0x00007f1654bd5d20 in ?? ()

5 0x00007f165431193c in switch_sleep (t=139733888109632) at src/switch_time.c:644

Backtrace stopped: frame did not save the PC

Thread 56 (LWP 45832):

0 0x00007f1650538bed in fts_read () from /lib64/libc.so.6

1 0x0000000000000000 in ?? ()

Thread 55 (LWP 65215):

0 0x00007f1650ef571d in write () from /lib64/libpthread.so.0

1 0x0000000000000000 in ?? ()

Thread 54 (LWP 65216):

0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007f16541a740f in switch_thread_cond_wait (cond=0x7f15840402e0, mutex=0x7f1584040290) at src/switch_apr.c:371

2 0x00007f16542cf5d2 in recording_thread (thread=0x7f15840404a0, obj=0x7f1584b08c28) at src/switch_ivr_async.c:1329

3 0x00007f165454f770 in dummy_worker (opaque=0x7f15840404a0) at threadproc/unix/thread.c:151

4 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0

5 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6

6 0x0000000000000000 in ?? ()

Thread 53 (LWP 65217):

0 0x00007f1650ef571d in write () from /lib64/libpthread.so.0

1 0x0000000000000000 in ?? ()

Thread 52 (LWP 45833):

0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007f165454969d in apr_thread_cond_wait (cond=, mutex=) at locks/unix/thread_cond.c:68

2 0x00007f1654540ce1 in apr_queue_pop (queue=0x7f164c7dbec8, data=0x7f164c53bd50) at misc/apr_queue.c:276

3 0x00007f16541a8c37 in switch_queue_pop (queue=0x7f164c7dbec8, data=0x7f164c53bd50) at src/switch_apr.c:1220

4 0x00007f16543029b3 in log_thread (t=0x7f164c6005a8, obj=0x0) at src/switch_log.c:501

5 0x00007f165454f770 in dummy_worker (opaque=0x7f164c6005a8) at threadproc/unix/thread.c:151

6 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0

7 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6

8 0x0000000000000000 in ?? ()

Thread 51 (LWP 45851):

0 0x00007f1650ef2da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007f1654549702 in apr_thread_cond_timedwait (cond=0x7f1648cb3b68, mutex=0x7f1648cb3b18, timeout=100000) at locks/unix/thread_cond.c:89

2 0x00007f1654540dd9 in apr_queue_pop_timeout (queue=0x7f1648cb3ad8, data=0x7f162cf08d18, timeout=100000) at misc/apr_queue.c:339

3 0x00007f16541a8c64 in switch_queue_pop_timeout (queue=0x7f1648cb3ad8, data=0x7f162cf08d18, timeout=100000) at src/switch_apr.c:1225

4 0x00007f1648cf08fe in sofia_profile_worker_thread_run (thread=0x7f1648295b40, obj=0x29b4ff0) at sofia.c:3010

5 0x00007f165454f770 in dummy_worker (opaque=0x7f1648295b40) at threadproc/unix/thread.c:151

6 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0

7 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6

8 0x0000000000000000 in ?? ()

Thread 50 (LWP 45870):

0 0x00007f1650543e63 in arch_prctl () from /lib64/libc.so.6

1 0x0000000000000000 in ?? ()

Thread 49 (LWP 64464):

0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007f16541a740f in switch_thread_cond_wait (cond=0x7f14a4020d78, mutex=0x7f14a4020d28) at src/switch_apr.c:371

2 0x00007f16542cf5d2 in recording_thread (thread=0x7f14a4020f38, obj=0x7f14a4132318) at src/switch_ivr_async.c:1329

3 0x00007f165454f770 in dummy_worker (opaque=0x7f14a4020f38) at threadproc/unix/thread.c:151

4 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0

5 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6

6 0x0000000000000000 in ?? ()

Thread 48 (LWP 45834):

0 0x00007f1650ef2da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007f1654549702 in apr_thread_cond_timedwait (cond=0x2733118, mutex=0x27330c8, timeout=500000) at locks/unix/thread_cond.c:89

2 0x00007f1654540dd9 in apr_queue_pop_timeout (queue=0x2733088, data=0x7f164c314d58, timeout=500000) at misc/apr_queue.c:339

3 0x00007f16541a8c64 in switch_queue_pop_timeout (queue=0x2733088, data=0x7f164c314d58, timeout=500000) at src/switch_apr.c:1225

4 0x00007f16542690a9 in switch_scheduler_task_thread (thread=0x7f164c4fe5d8, obj=0x0) at src/switch_scheduler.c:191

5 0x00007f165454f770 in dummy_worker (opaque=0x7f164c4fe5d8) at threadproc/unix/thread.c:151

6 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0

7 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6

8 0x0000000000000000 in ?? ()

Thread 47 (LWP 45850):

0 0x00007f1650ef2da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007f1654549702 in apr_thread_cond_timedwait (cond=0x7f1648cb3b68, mutex=0x7f1648cb3b18, timeout=100000) at locks/unix/thread_cond.c:89

2 0x00007f1654540dd9 in apr_queue_pop_timeout (queue=0x7f1648cb3ad8, data=0x7f162cf45d18, timeout=100000) at misc/apr_queue.c:339

3 0x00007f16541a8c64 in switch_queue_pop_timeout (queue=0x7f1648cb3ad8, data=0x7f162cf45d18, timeout=100000) at src/switch_apr.c:1225

4 0x00007f1648cf08fe in sofia_profile_worker_thread_run (thread=0x7f1648232b40, obj=0x29c2170) at sofia.c:3010

5 0x00007f165454f770 in dummy_worker (opaque=0x7f1648232b40) at threadproc/unix/thread.c:151

6 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0

7 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6

8 0x0000000000000000 in ?? ()

Thread 46 (LWP 64308):

0 0x00007f1650ef571d in write () from /lib64/libpthread.so.0

1 0x0000000000000000 in ?? ()

Thread 45 (LWP 45852):

0 0x00007f1650ef2da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007f1654549702 in apr_thread_cond_timedwait (cond=0x7f1648cb3b68, mutex=0x7f1648cb3b18, timeout=100000) at locks/unix/thread_cond.c:89

2 0x00007f1654540dd9 in apr_queue_pop_timeout (queue=0x7f1648cb3ad8, data=0x7f162cecbd18, timeout=100000) at misc/apr_queue.c:339

3 0x00007f16541a8c64 in switch_queue_pop_timeout (queue=0x7f1648cb3ad8, data=0x7f162cecbd18, timeout=100000) at src/switch_apr.c:1225

4 0x00007f1648cf08fe in sofia_profile_worker_thread_run (thread=0x29be5f8, obj=0x29bb7d0) at sofia.c:3010

5 0x00007f165454f770 in dummy_worker (opaque=0x29be5f8) at threadproc/unix/thread.c:151

6 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0

7 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6

8 0x0000000000000000 in ?? ()

Thread 44 (LWP 45878):

0 0x00007f1650ef571d in write () from /lib64/libpthread.so.0

1 0x0000000000000000 in ?? ()

Thread 43 (LWP 45881):

0 0x00007f1650ef571d in write () from /lib64/libpthread.so.0

1 0x0000000000000000 in ?? ()

Thread 42 (LWP 45932):

0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007f165454969d in apr_thread_cond_wait (cond=, mutex=) at locks/unix/thread_cond.c:68

2 0x00007f1654540ce1 in apr_queue_pop (queue=0x2dd3720, data=0x7f155c1bfd50) at misc/apr_queue.c:276

3 0x00007f16541a8c37 in switch_queue_pop (queue=0x2dd3720, data=0x7f155c1bfd50) at src/switch_apr.c:1220

4 0x00007f165426e3b4 in chat_thread_run (thread=0x2de3250, obj=0x2dd3720) at src/switch_loadable_module.c:911

5 0x00007f165454f770 in dummy_worker (opaque=0x2de3250) at threadproc/unix/thread.c:151

6 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0

7 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6

8 0x0000000000000000 in ?? ()

Thread 41 (LWP 45933):

0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007f165454969d in apr_thread_cond_wait (cond=, mutex=) at locks/unix/thread_cond.c:68

2 0x00007f1654540ce1 in apr_queue_pop (queue=0x2de3280, data=0x7f155c182d50) at misc/apr_queue.c:276

3 0x00007f16541a8c37 in switch_queue_pop (queue=0x2de3280, data=0x7f155c182d50) at src/switch_apr.c:1220

4 0x00007f165426e3b4 in chat_thread_run (thread=0x2def5e0, obj=0x2de3280) at src/switch_loadable_module.c:911

5 0x00007f165454f770 in dummy_worker (opaque=0x2def5e0) at threadproc/unix/thread.c:151

6 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0

7 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6

8 0x0000000000000000 in ?? ()

Thread 40 (LWP 45938):

0 0x00007f16505598c4 in __get_clockfreq () from /lib64/libc.so.6

1 0x0000000000000000 in ?? ()

Thread 39 (LWP 45928):

0 0x00007f16505598c4 in __get_clockfreq () from /lib64/libc.so.6

1 0x0000000000000000 in ?? ()

Thread 38 (LWP 45859):

0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007f165454969d in apr_thread_cond_wait (cond=, mutex=) at locks/unix/thread_cond.c:68

2 0x00007f1654540ce1 in apr_queue_pop (queue=0x29d5558, data=0x7f161fd7bd10) at misc/apr_queue.c:276

3 0x00007f161fdb0ac2 in apt_consumer_task_run () from /usr/local/fs1.10.7/lib/freeswitch/mod/mod_unimrcp.so

4 0x00007f161fdb41a7 in apt_task_run (thread_handle=0x29ec528, data=0x29d93e8) at src/apt_task.c:591

5 0x00007f165454f770 in dummy_worker (opaque=0x29ec528) at threadproc/unix/thread.c:151

6 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0

7 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6

8 0x0000000000000000 in ?? ()

Thread 37 (LWP 45863):

0 0x00007f1650543e63 in arch_prctl () from /lib64/libc.so.6

1 0x0000000000000000 in ?? ()

Thread 36 (LWP 64592):

0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007f16541a740f in switch_thread_cond_wait (cond=0x7f14e0029ab8, mutex=0x7f14e0029a68) at src/switch_apr.c:371

2 0x00007f16542cf5d2 in recording_thread (thread=0x7f14e0029c78, obj=0x7f14e00eed18) at src/switch_ivr_async.c:1329

3 0x00007f165454f770 in dummy_worker (opaque=0x7f14e0029c78) at threadproc/unix/thread.c:151

4 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0

5 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6

6 0x0000000000000000 in ?? ()

Thread 35 (LWP 45860):

0 0x00007f1650543e63 in arch_prctl () from /lib64/libc.so.6

1 0x0000000000000000 in ?? ()

Thread 34 (LWP 64949):

0 0x00007f1650ef571d in write () from /lib64/libpthread.so.0

1 0x0000000000000000 in ?? ()

Thread 33 (LWP 45856):

0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007f165454969d in apr_thread_cond_wait (cond=, mutex=) at locks/unix/thread_cond.c:68

2 0x00007f1654540ce1 in apr_queue_pop (queue=0x7f1648c50ad8, data=0x7f162c426d48) at misc/apr_queue.c:276

3 0x00007f16541a8c37 in switch_queue_pop (queue=0x7f1648c50ad8, data=0x7f162c426d48) at src/switch_apr.c:1220

4 0x00007f1648cecd58 in sofia_msg_thread_run (thread=0x29b40d0, obj=0x7f1648c50ad8) at sofia.c:2295

5 0x00007f165454f770 in dummy_worker (opaque=0x29b40d0) at threadproc/unix/thread.c:151

6 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0

7 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6

8 0x0000000000000000 in ?? ()

Thread 32 (LWP 45855):

0 0x00007f1650ef2da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007f1654549702 in apr_thread_cond_timedwait (cond=0x7f1648cb3b68, mutex=0x7f1648cb3b18, timeout=100000) at locks/unix/thread_cond.c:89

2 0x00007f1654540dd9 in apr_queue_pop_timeout (queue=0x7f1648cb3ad8, data=0x7f162c463d18, timeout=100000) at misc/apr_queue.c:339

3 0x00007f16541a8c64 in switch_queue_pop_timeout (queue=0x7f1648cb3ad8, data=0x7f162c463d18, timeout=100000) at src/switch_apr.c:1225

4 0x00007f1648cf08fe in sofia_profile_worker_thread_run (thread=0x7f162c68db50, obj=0x29c8950) at sofia.c:3010

5 0x00007f165454f770 in dummy_worker (opaque=0x7f162c68db50) at threadproc/unix/thread.c:151

6 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0

7 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6

8 0x0000000000000000 in ?? ()

Thread 31 (LWP 65067):

0 0x00007f165053a953 in setegid () from /lib64/libc.so.6

1 0x000000000007a120 in ?? ()

2 0x00007f152a359126 in ?? () from /usr/local/fs1.10.7/lib/freeswitch/mod/mod_callcenter.so

3 0x0000000002a1be40 in ?? ()

4 0x00007f14936b0ca0 in ?? ()

5 0x00007f165431193c in switch_sleep (t=44154432) at src/switch_time.c:644

Backtrace stopped: frame did not save the PC

Thread 30 (LWP 65076):

0 0x00007f1650538bed in fts_read () from /lib64/libc.so.6

1 0x00007f16542a3607 in rtp_common_read (rtp_session=<error reading variable: Cannot access memory at address 0xfffffe08>, payload_type=<error reading variable: Cannot access memory at address 0xfffffe00>, pmapP=<error reading variable: Cannot access memory at address 0xfffffdf8>, flags=<error reading variable: Cannot access memory at address 0xfffffdf0>, io_flags=<error reading variable: Cannot access memory at address 0xfffffdec>) at src/switch_rtp.c:7571

Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 29 (LWP 45837):

0 0x00007f16505598c4 in __get_clockfreq () from /lib64/libc.so.6

1 0x0000000000000000 in ?? ()

Thread 28 (LWP 45849):

0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007f16541a740f in switch_thread_cond_wait (cond=0x7f1628028ce8, mutex=0x7f1628028bf8) at src/switch_apr.c:371

2 0x00007f16541dceeb in switch_user_sql_thread (thread=0x7f162d0846c0, obj=0x7f1628028b30) at src/switch_core_sqldb.c:2296

3 0x00007f165454f770 in dummy_worker (opaque=0x7f162d0846c0) at threadproc/unix/thread.c:151

4 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0

5 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6

6 0x0000000000000000 in ?? ()

Thread 27 (LWP 45854):

0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007f16541a740f in switch_thread_cond_wait (cond=0x7f1624002598, mutex=0x7f16240024a8) at src/switch_apr.c:371

2 0x00007f16541dceeb in switch_user_sql_thread (thread=0x7f162c5656c0, obj=0x7f16240023e0) at src/switch_core_sqldb.c:2296

3 0x00007f165454f770 in dummy_worker (opaque=0x7f162c5656c0) at threadproc/unix/thread.c:151

4 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0

5 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6

6 0x0000000000000000 in ?? ()

Thread 26 (LWP 45929):

0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007f16541a740f in switch_thread_cond_wait (cond=0x29f1548, mutex=0x29f14a8) at src/switch_apr.c:371

2 0x00007f161c1698f4 in mod_verto_runtime () at mod_verto.c:6907

3 0x00007f165426ac27 in switch_loadable_module_exec (thread=0x273ca38, obj=0x273c618) at src/switch_loadable_module.c:123

4 0x00007f165454f770 in dummy_worker (opaque=0x273ca38) at threadproc/unix/thread.c:151

5 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0

6 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6

7 0x0000000000000000 in ?? ()

Thread 25 (LWP 45865):

0 0x00007f1650543e63 in arch_prctl () from /lib64/libc.so.6

1 0x0000000000000000 in ?? ()

Thread 24 (LWP 45846):

0 0x00007f1650543e63 in arch_prctl () from /lib64/libc.so.6

1 0x0000000000000000 in ?? ()

Thread 23 (LWP 45838):

0 0x00007f165053a953 in setegid () from /lib64/libc.so.6

1 0x00000000000f4240 in ?? ()

2 0x00007f164ad7dd30 in ?? ()

3 0x00007f16541e1f7d in switch_core_expire_registration (force=0) at src/switch_core_sqldb.c:3557

4 0x00007f165431193c in switch_sleep (t=1000000) at src/switch_time.c:644

5 0x00007f16541db054 in switch_core_sql_db_thread (thread=0x7f164c600750, obj=0x0) at src/switch_core_sqldb.c:1541

6 0x00007f165454f770 in dummy_worker (opaque=0x7f164c600750) at threadproc/unix/thread.c:151

7 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0

8 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6

9 0x0000000000000000 in ?? ()

Thread 22 (LWP 45930):

0 0x00007f1650ef599d in fcntl () from /lib64/libpthread.so.0

1 0x00007f15180029a0 in ?? ()

2 0x0000000000002687 in ?? ()

3 0x0000000000000000 in ?? ()

Thread 21 (LWP 45840):

0 0x00007f1650543e63 in arch_prctl () from /lib64/libc.so.6

1 0x0000000000000000 in ?? ()

Thread 20 (LWP 45841):

0 0x00007f1650543e63 in arch_prctl () from /lib64/libc.so.6

1 0x0000000000000000 in ?? ()

Thread 19 (LWP 45931):

0 0x00007f1650ef571d in write () from /lib64/libpthread.so.0

1 0x0000000000000000 in ?? ()

Thread 18 (LWP 45842):

0 0x00007f1650543e63 in arch_prctl () from /lib64/libc.so.6

1 0x0000000000000000 in ?? ()

Thread 17 (LWP 45927):

0 0x00007f16505598c4 in __get_clockfreq () from /lib64/libc.so.6

1 0x0000000000000000 in ?? ()

Thread 16 (LWP 45845):

0 0x00007f1650543e63 in arch_prctl () from /lib64/libc.so.6

1 0x0000000000000000 in ?? ()

Thread 15 (LWP 45844):

0 0x00007f1650543e63 in arch_prctl () from /lib64/libc.so.6

1 0x0000000000000000 in ?? ()

Thread 14 (LWP 45847):

0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007f16541a740f in switch_thread_cond_wait (cond=0x7f1638028d08, mutex=0x7f1638028c18) at src/switch_apr.c:371

2 0x00007f16541dceeb in switch_user_sql_thread (thread=0x7f162d20e6c0, obj=0x7f1638028b40) at src/switch_core_sqldb.c:2296

3 0x00007f165454f770 in dummy_worker (opaque=0x7f162d20e6c0) at threadproc/unix/thread.c:151

4 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0

5 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6

6 0x0000000000000000 in ?? ()

Thread 13 (LWP 45873):

0 0x00007f1650ef2da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007f15a0fa7939 in ks_cond_timedwait (cond=0x2a1fb58, ms=10000) at /home/freeswitch/tmp/libks/src/ks_mutex.c:563

2 0x00007f15cf1a7c4e in __manager_loop () at /home/freeswitch/tmp/20220405/signalwire-c/src/handle_manager.c:255

3 0x00007f15cf1a7d0f in __manager_thread_wrapper (thread=0x2a1fcc8, data=0x0) at /home/freeswitch/tmp/20220405/signalwire-c/src/handle_manager.c:275

4 0x00007f15a0fb5036 in thread_launch (args=0x2a1fcc8) at /home/freeswitch/tmp/libks/src/ks_thread.c:248

5 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0

6 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6

7 0x0000000000000000 in ?? ()

Thread 12 (LWP 45872):

0 0x00007f1650543e63 in arch_prctl () from /lib64/libc.so.6

1 0x0000000000000000 in ?? ()

Thread 11 (LWP 45843):

0 0x00007f1650543e63 in arch_prctl () from /lib64/libc.so.6

1 0x0000000000000000 in ?? ()

Thread 10 (LWP 65206):

0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007f16541a740f in switch_thread_cond_wait (cond=0x7f14a812e0f0, mutex=0x7f14a812e0a0) at src/switch_apr.c:371

2 0x00007f16542cf5d2 in recording_thread (thread=0x7f14a812e2b0, obj=0x7f14a80a8bc8) at src/switch_ivr_async.c:1329

3 0x00007f165454f770 in dummy_worker (opaque=0x7f14a812e2b0) at threadproc/unix/thread.c:151

4 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0

5 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6

6 0x0000000000000000 in ?? ()

Thread 9 (LWP 45831):

0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007f165454969d in apr_thread_cond_wait (cond=, mutex=) at locks/unix/thread_cond.c:68

2 0x00007f1654540ce1 in apr_queue_pop (queue=0x24a1d98, data=0x7f164c751d48) at misc/apr_queue.c:276

3 0x00007f16541a8c37 in switch_queue_pop (queue=0x24a1d98, data=0x7f164c751d48) at src/switch_apr.c:1220

4 0x00007f16542819cd in switch_event_dispatch_thread (thread=0x7f164c7dbc40, obj=0x24a1d98) at src/switch_event.c:331

5 0x00007f165454f770 in dummy_worker (opaque=0x7f164c7dbc40) at threadproc/unix/thread.c:151

6 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0

7 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6

8 0x0000000000000000 in ?? ()

Thread 8 (LWP 45848):

0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007f16541a740f in switch_thread_cond_wait (cond=0x7f16200280a8, mutex=0x7f1620027fb8) at src/switch_apr.c:371

2 0x00007f16541dceeb in switch_user_sql_thread (thread=0x7f162d1496c0, obj=0x7f1620027ee0) at src/switch_core_sqldb.c:2296

3 0x00007f165454f770 in dummy_worker (opaque=0x7f162d1496c0) at threadproc/unix/thread.c:151

4 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0

5 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6

6 0x0000000000000000 in ?? ()

Thread 7 (LWP 45861):

0 0x00007f165053a953 in setegid () from /lib64/libc.so.6

1 0x0000000000000000 in ?? ()

Thread 6 (LWP 45874):

0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007f165454969d in apr_thread_cond_wait (cond=, mutex=) at locks/unix/thread_cond.c:68

2 0x00007f1654540ce1 in apr_queue_pop (queue=0x29a4628, data=0x7f162c113d50) at misc/apr_queue.c:276

3 0x00007f16541a8c37 in switch_queue_pop (queue=0x29a4628, data=0x7f162c113d50) at src/switch_apr.c:1220

4 0x00007f1648d308fa in sofia_presence_event_thread_run (thread=0x29a54e0, obj=0x0) at sofia_presence.c:1625

5 0x00007f165454f770 in dummy_worker (opaque=0x29a54e0) at threadproc/unix/thread.c:151

6 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0

7 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6

8 0x0000000000000000 in ?? ()

Thread 5 (LWP 45862):

0 0x00007f1650543e63 in arch_prctl () from /lib64/libc.so.6

1 0x0000000000000000 in ?? ()

Thread 4 (LWP 45900):

0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007f165454969d in apr_thread_cond_wait (cond=, mutex=) at locks/unix/thread_cond.c:68

2 0x00007f1654540ce1 in apr_queue_pop (queue=0x7f153001b958, data=0x7f15c406d468) at misc/apr_queue.c:276

3 0x00007f16541a8c37 in switch_queue_pop (queue=0x7f153001b958, data=0x7f15c406d468) at src/switch_apr.c:1220

4 0x00007f165433b0ee in EventConsumer::pop (this=, block=block@entry=1, timeout=) at src/switch_cpp.cpp:110

5 0x00007f155c1ebaf5 in _wrap_EventConsumer_pop (L=0x7f15300008c8) at mod_lua_wrap.cpp:5547

6 0x00007f155c1f48a0 in luaD_precall () from /usr/local/fs1.10.7/lib/freeswitch/mod/mod_lua.so

7 0x00007f155c1ff29f in luaV_execute () from /usr/local/fs1.10.7/lib/freeswitch/mod/mod_lua.so

8 0x00007f155c1f4c9c in luaD_call () from /usr/local/fs1.10.7/lib/freeswitch/mod/mod_lua.so

9 0x00007f155c1ff0ac in luaV_execute () from /usr/local/fs1.10.7/lib/freeswitch/mod/mod_lua.so

10 0x00007f155c1f4c9c in luaD_call () from /usr/local/fs1.10.7/lib/freeswitch/mod/mod_lua.so

11 0x00007f155c1f425f in luaD_rawrunprotected () from /usr/local/fs1.10.7/lib/freeswitch/mod/mod_lua.so

12 0x00007f155c1f4f0d in luaD_pcall () from /usr/local/fs1.10.7/lib/freeswitch/mod/mod_lua.so

13 0x00007f155c1f275e in lua_pcallk () from /usr/local/fs1.10.7/lib/freeswitch/mod/mod_lua.so

14 0x00007f155c1d1b2d in docall (L=L@entry=0x7f15300008c8, narg=narg@entry=0, nresults=nresults@entry=0, perror=perror@entry=0, fatal=fatal@entry=1) at mod_lua.cpp:92

15 0x00007f155c1d213f in lua_parse_and_execute (L=L@entry=0x7f15300008c8, input_code=, session=session@entry=0x0) at mod_lua.cpp:194

16 0x00007f155c1d21d7 in lua_thread_run (thread=, obj=0x2c06050) at mod_lua.cpp:221

17 0x00007f165454f770 in dummy_worker (opaque=0x2c061d8) at threadproc/unix/thread.c:151

18 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0

19 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6

20 0x0000000000000000 in ?? ()

Thread 3 (LWP 45875):

0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007f16541a740f in switch_thread_cond_wait (cond=0x2a8d800, mutex=0x2a8d710) at src/switch_apr.c:371

2 0x00007f16541dceeb in switch_user_sql_thread (thread=0x7f161c1496c0, obj=0x2a8d660) at src/switch_core_sqldb.c:2296

3 0x00007f165454f770 in dummy_worker (opaque=0x7f161c1496c0) at threadproc/unix/thread.c:151

4 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0

5 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6

6 0x0000000000000000 in ?? ()

Thread 2 (LWP 45864):

0 0x00007f1650543e63 in arch_prctl () from /lib64/libc.so.6

1 0x0000000000000000 in ?? ()

Thread 1 (LWP 45853):

0 0x00007f1651eb87e8 in su_block_add () from /usr/local/lib/libsofia-sip-ua.so.0

1 0x00007f1651eb8d9d in sub_alloc () from /usr/local/lib/libsofia-sip-ua.so.0

2 0x00007f1651eba8b1 in su_zalloc () from /usr/local/lib/libsofia-sip-ua.so.0

3 0x00007f1651e42d4b in outgoing_make_a_aaaa_query () from /usr/local/lib/libsofia-sip-ua.so.0

4 0x00007f1651e42350 in outgoing_resolve_next () from /usr/local/lib/libsofia-sip-ua.so.0

5 0x00007f1651e43e88 in outgoing_answer_srv () from /usr/local/lib/libsofia-sip-ua.so.0

6 0x00007f1651ead296 in sres_query_report_error () from /usr/local/lib/libsofia-sip-ua.so.0

7 0x00007f1651ead643 in sres_resend_dns_query () from /usr/local/lib/libsofia-sip-ua.so.0

8 0x00007f1651eae590 in sres_resolver_report_error () from /usr/local/lib/libsofia-sip-ua.so.0

9 0x00007f1651eae24f in sres_resolver_error () from /usr/local/lib/libsofia-sip-ua.so.0

10 0x00007f1651eb3eee in sres_sofia_poll () from /usr/local/lib/libsofia-sip-ua.so.0

11 0x00007f1651ec5768 in su_epoll_port_wait_events () from /usr/local/lib/libsofia-sip-ua.so.0

12 0x00007f1651ec1b68 in su_base_port_run () from /usr/local/lib/libsofia-sip-ua.so.0

13 0x00007f1651ebde18 in su_port_run () from /usr/local/lib/libsofia-sip-ua.so.0

14 0x00007f1651ebeef2 in su_root_run () from /usr/local/lib/libsofia-sip-ua.so.0

15 0x00007f1651ec2924 in su_pthread_port_clone_main () from /usr/local/lib/libsofia-sip-ua.so.0

16 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0

17 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6

18 0x0000000000000000 in ?? ()

andywolk commented 2 years ago

You need to install sofia sip debugging symbols. Also I don't think your sofia is the latest. And please attach the log, do not paste it.

andywolk commented 2 years ago

Sounds like it is not a bug but some memory issues on your box. Information you provided does not show me that a crash happened. Line numbers do not match sofia-sip version you mentioned.

dxhgq-github commented 2 years ago

Ok, I will take a check at sofia-sip version , Thanks!

dxhgq-github commented 2 years ago

Hello I have downloaded and rebuilt the latest version 1.13.7 of sofia-sip , after running 4 days , the FreeSwitch still crashs. error message is : Core was generated by `/usr/local/fs1.10.7/bin/freeswitch -nonat'. Program terminated with signal SIGFPE, Arithmetic exception.

0 0x00007f2c9e58381a in su_block_add (b=0x7f2c542282f0, p=0x7f2c542547f0) at su_alloc.c:342

342 h = (size_t)((uintptr_t)p % b->sub_n); [Current thread is 1 (LWP 92647)] (gdb) (gdb) bt

0 0x00007f2c9e58381a in su_block_add (b=0x7f2c542282f0, p=0x7f2c542547f0) at su_alloc.c:342

1 0x00007f2c9e583dcf in sub_alloc (home=0x7f2c54250ef0, sub=0x7f2c542282f0, size=80,

zero=do_calloc) at su_alloc.c:530

2 0x00007f2c9e5858e3 in su_zalloc (home=0x7f2c54250ef0, size=80) at su_alloc.c:1577

3 0x00007f2c9e50dd4b in outgoing_make_a_aaaa_query (orq=0x7f2c54254690) at nta.c:10463

4 0x00007f2c9e50d350 in outgoing_resolve_next (orq=0x7f2c54254690) at nta.c:10244

5 0x00007f2c9e50ee88 in outgoing_answer_srv (orq=0x7f2c54254690, q=0x7f2c5410b2c0,

answers=0x7f2c541167b0) at nta.c:10830

6 0x00007f2c9e5782c8 in sres_query_report_error (q=0x7f2c5410b2c0, answers=0x7f2c541167b0)

at sres.c:2991

7 0x00007f2c9e578675 in sres_resend_dns_query (res=0x7f2c54002990, q=0x7f2c5410b2c0,

timeout=0) at sres.c:3091

8 0x00007f2c9e5795c2 in sres_resolver_report_error (res=0x7f2c54002990, socket=590,

errcode=111, remote=0x7f2c7d58c560, remotelen=16, 
info=0x7f2c7d58c4e0 "icmp type=3 code=3 reported by 127.0.0.1") at sres.c:3440

9 0x00007f2c9e579281 in sres_resolver_error (res=0x7f2c54002990, socket=590) at sres.c:3354

10 0x00007f2c9e57ef20 in sres_sofia_poll (magic=0x7f2c68001180, w=0x7f2c54001084,

reg=0x7f2c540037c0) at sresolv.c:357

11 0x00007f2c9e59079a in su_epoll_port_wait_events (self=0x7f2c540008c0, tout=690)

at su_epoll_port.c:510

12 0x00007f2c9e58cb9a in su_base_port_run (self=0x7f2c540008c0) at su_base_port.c:349

13 0x00007f2c9e588e4a in su_port_run (self=0x7f2c540008c0) at su_port.h:326

14 0x00007f2c9e589f24 in su_root_run (self=0x7f2c54001130) at su_root.c:819

15 0x00007f2c9e58d956 in su_pthread_port_clone_main (varg=0x7f2c91233460)

at su_pthread_port.c:343

16 0x00007f2c9d5b9e65 in start_thread () from /lib64/libpthread.so.0

17 0x00007f2c9cc0e88d in __libc_ifunc_impl_list () from /lib64/libc.so.6

18 0x0000000000000000 in ?? ()

(gdb)

backtrace1.log

dxhgq-github commented 2 years ago

Please help , thanks

andywolk commented 2 years ago

Please wrap backtraces into a code block when you comment.

andywolk commented 2 years ago

Based on your last backtrace please print some debugging information:

frame 2
p *home
dxhgq-github commented 2 years ago

thank you for replay !

dxhgq-github commented 2 years ago

(gdb) (gdb) (gdb) frame 2

2 0x00007ff892b5992d in su_zalloc (home=0x7ff86028d060, size=80) at su_alloc.c:1577

1577 data = sub_alloc(home, MEMLOCK(home), size, (enum sub_zero)1); (gdb) p *home $1 = {suh_size = 1620230464, suh_blocks = 0x7ff8613c3900, suh_lock = 0x0} (gdb)

thank you !

dxhgq-github commented 2 years ago

whether it is a IPv6 resolve problem ? but I do not configure any call flow about local resolve, and ping 127.0.0.1 is alive , and /etc/resolv.conf is empty.

dxhgq-github commented 2 years ago

backtrace.log

dxhgq-github commented 2 years ago

last week ,i disabled the ipv6 profile ,mv internal-ipv6.xml internal-ipv6.xml.bak ,mv external-ipv6 external-ipv6.bak , but problem is still unsolved。

dxhgq-github commented 2 years ago

yesterday ,i disabled the ipv6 on box ,add tow lines : net.ipv6.conf.all.disable_ipv6 =1 net.ipv6.conf.default.disable_ipv6 =1 to /etc/sysctl.conf. by the end of this time ,i have tested for about 30 hours.

andywolk commented 2 years ago

This frame 2 is not from a backtrace mentioned in this github issue. From another crash?

dragos-oancea commented 2 years ago

@dxhgq-github , https://github.com/signalwire/freeswitch/issues/1659 - you should not open the same issue in two places, sofia-sip and FS, just keep the discussion in one place.

dxhgq-github commented 2 years ago

ok , This frame 2 is from another crash, same problem ,above attachment is backtrace