search

signalwire / freeswitch

FreeSWITCH is a Software Defined Telecom Stack enabling the digital transformation from proprietary telecom switches to a versatile software implementation that runs on any commodity hardware. From a Raspberry PI to a multi-core server, FreeSWITCH can unlock the telecommunications potential of any device.
https://freeswitch.com/#getting-started
Other
3.6k stars 1.42k forks source link

WRONG_CALL_STATE & wss port dead & freeSWITCH Process exits #1513

Open yzh1127116449 opened 2 years ago

yzh1127116449 commented 2 years ago

(1)WRONG_CALL_STATE invite ---> 407 <----- ack ---> invite ---> the second invite,FreeSWITCH no have log print，10 seconds later, WRONG_CALL_STATE appears, By capturing the packet file, it can be seen that FreeSWITCH has received the invite for the second time. (2)WSS port feigned death problem After the problem occurs, I can Telnet, but the WSS handshake fails，Then I connect to fs_cli and execute command: reload mod_sofia relaod fails, and the process exits. here the error logs and core file (3) The FreeSWITCH process exits

Expected behavior I used FreeSWITCH to convert webRTC to UDP, and I developed a WebrTC-based client bulk call myself using JSSIP.

Package version or git hash

ae24ace0bb13f37d331c5fadd0b9f0ce

Trace logs Provide freeswitch logs w/ DEBUG and UUID logging enabled 1641346984(1)

1641347095(1)

1641347137(1)

4c1dd984-03aa-4428-8ed0-07288d923699 2021-12-31 09:13:14.308671 98.93% [NOTICE] switch_channel.c:1123 New Channel sofia/internal/8010054@172.31.100.169:17443 [4c1dd984-03aa-4428-8ed0-07288d923699]
4c1dd984-03aa-4428-8ed0-07288d923699 2021-12-31 09:13:14.308671 98.93% [DEBUG] switch_core_state_machine.c:581 (sofia/internal/8010054@172.31.100.169:17443) Running State Change CS_NEW (Cur 2 Tot 251890)
4c1dd984-03aa-4428-8ed0-07288d923699 2021-12-31 09:13:14.308671 98.93% [INFO] sofia.c:10462 sofia/internal/8010054@172.31.100.169:17443 receiving invite from 172.31.100.169:61854 version: 1.10.7 -release 64bit call-id: 7nr1siisue78nrotie51
4c1dd984-03aa-4428-8ed0-07288d923699 2021-12-31 09:13:14.308671 98.93% [DEBUG] switch_core_state_machine.c:600 (sofia/internal/8010054@172.31.100.169:17443) State NEW
2021-12-31 09:13:14.308671 98.93% [DEBUG] sofia.c:2419 detaching session 4c1dd984-03aa-4428-8ed0-07288d923699
4c1dd984-03aa-4428-8ed0-07288d923699 2021-12-31 09:13:24.348691 99.07% [WARNING] switch_core_state_machine.c:684 4c1dd984-03aa-4428-8ed0-07288d923699 sofia/internal/8010054@172.31.100.169:17443 Abandoned
4c1dd984-03aa-4428-8ed0-07288d923699 2021-12-31 09:13:24.348691 99.07% [NOTICE] switch_core_state_machine.c:687 Hangup sofia/internal/8010054@172.31.100.169:17443 [CS_NEW] [WRONG_CALL_STATE]
4c1dd984-03aa-4428-8ed0-07288d923699 2021-12-31 09:13:24.348691 99.07% [DEBUG] switch_core_state_machine.c:581 (sofia/internal/8010054@172.31.100.169:17443) Running State Change CS_HANGUP (Cur 2 Tot 251890)
4c1dd984-03aa-4428-8ed0-07288d923699 2021-12-31 09:13:24.348691 99.07% [DEBUG] switch_core_state_machine.c:844 (sofia/internal/8010054@172.31.100.169:17443) Callstate Change DOWN -> HANGUP
4c1dd984-03aa-4428-8ed0-07288d923699 2021-12-31 09:13:24.348691 99.07% [DEBUG] switch_core_state_machine.c:846 (sofia/internal/8010054@172.31.100.169:17443) State HANGUP
4c1dd984-03aa-4428-8ed0-07288d923699 2021-12-31 09:13:24.348691 99.07% [DEBUG] mod_sofia.c:468 Channel sofia/internal/8010054@172.31.100.169:17443 hanging up, cause: WRONG_CALL_STATE
4c1dd984-03aa-4428-8ed0-07288d923699 2021-12-31 09:13:24.348691 99.07% [DEBUG] switch_core_state_machine.c:59 sofia/internal/8010054@172.31.100.169:17443 Standard HANGUP, cause: WRONG_CALL_STATE
4c1dd984-03aa-4428-8ed0-07288d923699 2021-12-31 09:13:24.348691 99.07% [DEBUG] switch_core_state_machine.c:846 (sofia/internal/8010054@172.31.100.169:17443) State HANGUP going to sleep
4c1dd984-03aa-4428-8ed0-07288d923699 2021-12-31 09:13:24.348691 99.07% [DEBUG] switch_core_state_machine.c:616 (sofia/internal/8010054@172.31.100.169:17443) State Change CS_HANGUP -> CS_REPORTING
4c1dd984-03aa-4428-8ed0-07288d923699 2021-12-31 09:13:24.348691 99.07% [DEBUG] switch_core_state_machine.c:581 (sofia/internal/8010054@172.31.100.169:17443) Running State Change CS_REPORTING (Cur 2 Tot 251890)
4c1dd984-03aa-4428-8ed0-07288d923699 2021-12-31 09:13:24.348691 99.07% [DEBUG] switch_core_state_machine.c:932 (sofia/internal/8010054@172.31.100.169:17443) State REPORTING
4c1dd984-03aa-4428-8ed0-07288d923699 2021-12-31 09:13:24.348691 99.07% [DEBUG] switch_core_state_machine.c:168 sofia/internal/8010054@172.31.100.169:17443 Standard REPORTING, cause: WRONG_CALL_STATE
4c1dd984-03aa-4428-8ed0-07288d923699 2021-12-31 09:13:24.348691 99.07% [DEBUG] switch_core_state_machine.c:932 (sofia/internal/8010054@172.31.100.169:17443) State REPORTING going to sleep
4c1dd984-03aa-4428-8ed0-07288d923699 2021-12-31 09:13:24.348691 99.07% [DEBUG] switch_core_state_machine.c:607 (sofia/internal/8010054@172.31.100.169:17443) State Change CS_REPORTING -> CS_DESTROY
4c1dd984-03aa-4428-8ed0-07288d923699 2021-12-31 09:13:24.348691 99.07% [DEBUG] switch_core_session.c:1753 Session 251890 (sofia/internal/8010054@172.31.100.169:17443) Locked, Waiting on external entities
4c1dd984-03aa-4428-8ed0-07288d923699 2021-12-31 09:13:24.348691 99.07% [NOTICE] switch_core_session.c:1771 Session 251890 (sofia/internal/8010054@172.31.100.169:17443) Ended
4c1dd984-03aa-4428-8ed0-07288d923699 2021-12-31 09:13:24.348691 99.07% [NOTICE] switch_core_session.c:1775 Close Channel sofia/internal/8010054@172.31.100.169:17443 [CS_DESTROY]
4c1dd984-03aa-4428-8ed0-07288d923699 2021-12-31 09:13:24.348691 99.07% [DEBUG] switch_core_state_machine.c:735 (sofia/internal/8010054@172.31.100.169:17443) Running State Change CS_DESTROY (Cur 1 Tot 251890)
4c1dd984-03aa-4428-8ed0-07288d923699 2021-12-31 09:13:24.348691 99.07% [DEBUG] switch_core_state_machine.c:745 (sofia/internal/8010054@172.31.100.169:17443) State DESTROY
4c1dd984-03aa-4428-8ed0-07288d923699 2021-12-31 09:13:24.348691 99.07% [DEBUG] mod_sofia.c:379 sofia/internal/8010054@172.31.100.169:17443 SOFIA DESTROY
4c1dd984-03aa-4428-8ed0-07288d923699 2021-12-31 09:13:24.348691 99.07% [DEBUG] switch_core_state_machine.c:175 sofia/internal/8010054@172.31.100.169:17443 Standard DESTROY
4c1dd984-03aa-4428-8ed0-07288d923699 2021-12-31 09:13:24.348691 99.07% [DEBUG] switch_core_state_machine.c:745 (sofia/internal/8010054@172.31.100.169:17443) State DESTROY going to sleep

backtrace from core file If applicable, provide the full backtrace from the core file.

No stack.
#0  0x00007f004b2b9bc7 in abort () from /lib64/libc.so.6
#1  0x00007f004b2b11a6 in __assert_fail_base () from /lib64/libc.so.6
#2  0x00007f004b2b1252 in __assert_fail () from /lib64/libc.so.6
#3  0x00007f004ca5e72f in nua_destroy () from /usr/local/lib/libsofia-sip-ua.so.0
#4  0x00007f0044e96d64 in sofia_profile_thread_run (thread=<optimized out>, obj=0x7effec469650) at sofia.c:3536
#5  0x00007f004ee46630 in dummy_worker (opaque=0x7effec46bb60) at threadproc/unix/thread.c:151
#6  0x00007f004bd2cea5 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f004b380b0d in clone () from /lib64/libc.so.6
#0  0x00007f004b2b9bc7 in abort () from /lib64/libc.so.6
No symbol table info available.
#1  0x00007f004b2b11a6 in __assert_fail_base () from /lib64/libc.so.6
No symbol table info available.
#2  0x00007f004b2b1252 in __assert_fail () from /lib64/libc.so.6
No symbol table info available.
#3  0x00007f004ca5e72f in nua_destroy () from /usr/local/lib/libsofia-sip-ua.so.0
No symbol table info available.
#4  0x00007f0044e96d64 in sofia_profile_thread_run (thread=<optimized out>, obj=0x7effec469650) at sofia.c:3536
        profile = 0x7effec469650
        node = <optimized out>
        s_event = 0x0
        use_100rel = <optimized out>
        use_timer = <optimized out>
        use_rfc_5626 = <optimized out>
        supported = 0x7effec46bc08 "timer, path, replaces"
        sanity = 4
        attempts = <optimized out>
        worker_thread = 0x7f00280de690
        st = SWITCH_STATUS_SUCCESS
        qname = "sofia:internal", '\000' <repeats 113 times>
        key = <optimized out>
        cert = <optimized out>
        chain = <optimized out>
        ssl_ctx = <optimized out>
        ssl_method = <optimized out>
        __func__ = "sofia_profile_thread_run"
        __PRETTY_FUNCTION__ = "sofia_profile_thread_run"
#5  0x00007f004ee46630 in dummy_worker (opaque=0x7effec46bb60) at threadproc/unix/thread.c:151
        thread = 0x7effec46bb60
#6  0x00007f004bd2cea5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#7  0x00007f004b380b0d in clone () from /lib64/libc.so.6
No symbol table info available.
  Id   Target Id         Frame 
  42   Thread 0x7f0044cdb700 (LWP 9575) 0x00007f004b3810e3 in epoll_wait () from /lib64/libc.so.6
  41   Thread 0x7f0044dde700 (LWP 6441) 0x00007f004bd30a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  40   Thread 0x7f002407c700 (LWP 44) 0x00007f004b396b44 in clock_nanosleep () from /lib64/libc.so.6
  39   Thread 0x7effb65fb700 (LWP 30812) 0x00007f004b375ddd in poll () from /lib64/libc.so.6
  38   Thread 0x7f0026235700 (LWP 33) 0x00007f004b375ddd in poll () from /lib64/libc.so.6
  37   Thread 0x7f0046b51700 (LWP 12) 0x00007f004b375ddd in poll () from /lib64/libc.so.6
  36   Thread 0x7f00442e7700 (LWP 204) 0x00007f004bd30a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  35   Thread 0x7f0044e58700 (LWP 425) 0x00007f004b375ddd in poll () from /lib64/libc.so.6
  34   Thread 0x7f00240f6700 (LWP 48) 0x00007f004bd30a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  33   Thread 0x7f0047699700 (LWP 198) 0x00007f004b396b44 in clock_nanosleep () from /lib64/libc.so.6
  32   Thread 0x7f0047713700 (LWP 8) 0x00007f004bd30a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  31   Thread 0x7effb567e700 (LWP 19926) 0x00007f004b375ddd in poll () from /lib64/libc.so.6
  30   Thread 0x7effe404f700 (LWP 49) 0x00007f004bd30a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  29   Thread 0x7f002403f700 (LWP 45) 0x00007f004bd30a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  28   Thread 0x7f0024ea2700 (LWP 36) 0x00007f004bd30a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  27   Thread 0x7effcc14c700 (LWP 50) 0x00007f004bd3375d in read () from /lib64/libpthread.so.0
  26   Thread 0x7f0044d18700 (LWP 9587) 0x00007f004bd30de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  25   Thread 0x7f0024133700 (LWP 47) 0x00007f004bd3375d in read () from /lib64/libpthread.so.0
  24   Thread 0x7f0046b14700 (LWP 6440) 0x00007f004bd3375d in read () from /lib64/libpthread.so.0
  23   Thread 0x7f0045dbc700 (LWP 15) 0x00007f004b377b43 in select () from /lib64/libc.so.6
  22   Thread 0x7f0025db6700 (LWP 34) 0x00007f004bd30de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  21   Thread 0x7f004f5a68c0 (LWP 1) 0x00007f004b377b43 in select () from /lib64/libc.so.6
  20   Thread 0x7f00258b8700 (LWP 9580) 0x00007f004bd30a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  19   Thread 0x7f00240b9700 (LWP 46) 0x00007f004bd339dd in accept () from /lib64/libpthread.so.0
  18   Thread 0x7effb57ec700 (LWP 23283) 0x00007f004b375ddd in poll () from /lib64/libc.so.6
  17   Thread 0x7f0044e1b700 (LWP 9573) 0x00007f004b3810e3 in epoll_wait () from /lib64/libc.so.6
  16   Thread 0x7f00476d6700 (LWP 9577) 0x00007f004bd30a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  15   Thread 0x7f0024170700 (LWP 38) 0x00007f004bd30a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  14   Thread 0x7f0047352700 (LWP 11) 0x00007f004bd30de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  13   Thread 0x7f004763e700 (LWP 10) apr_queue_trypush (queue=0x7effb059fa08, data=0x7f0030038870) at misc/apr_queue.c:229
  12   Thread 0x7f0044d55700 (LWP 9585) 0x00007f004bd30de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  11   Thread 0x7f0024e65700 (LWP 37) 0x00007f004b377b43 in select () from /lib64/libc.so.6
  10   Thread 0x7f00277fe700 (LWP 9579) 0x00007f004b3810e3 in epoll_wait () from /lib64/libc.so.6
  9    Thread 0x7f0044c61700 (LWP 9582) 0x00007f004b3810e3 in epoll_wait () from /lib64/libc.so.6
  8    Thread 0x7f0026be7700 (LWP 9581) 0x00007f004bd30de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  7    Thread 0x7f0045df9700 (LWP 14) 0x00007f004b396b44 in clock_nanosleep () from /lib64/libc.so.6
  6    Thread 0x7f004f4de700 (LWP 7) 0x00007f004b377b43 in select () from /lib64/libc.so.6
  5    Thread 0x7f0026baa700 (LWP 9586) 0x00007f004bd30a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  4    Thread 0x7f0026b30700 (LWP 9574) 0x00007f004b3810e3 in epoll_wait () from /lib64/libc.so.6
  3    Thread 0x7f0027fff700 (LWP 9583) 0x00007f004b3810e3 in epoll_wait () from /lib64/libc.so.6
  2    Thread 0x7f0044085700 (LWP 9584) 0x00007f004bd30a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
* 1    Thread 0x7f0026ce9700 (LWP 6429) 0x00007f004b2b9bc7 in abort () from /lib64/libc.so.6

Thread 42 (Thread 0x7f0044cdb700 (LWP 9575)):
#0  0x00007f004b3810e3 in epoll_wait () from /lib64/libc.so.6
#1  0x00007f004cad86ae in su_epoll_port_wait_events () from /usr/local/lib/libsofia-sip-ua.so.0
#2  0x00007f004cad4d35 in su_base_port_step () from /usr/local/lib/libsofia-sip-ua.so.0
#3  0x00007f004cad0edd in su_port_step () from /usr/local/lib/libsofia-sip-ua.so.0
#4  0x00007f004cad2019 in su_root_step () from /usr/local/lib/libsofia-sip-ua.so.0
#5  0x00007f0044e96ae3 in sofia_profile_thread_run (thread=<optimized out>, obj=0x7effec0182a0) at sofia.c:3465
#6  0x00007f004ee46630 in dummy_worker (opaque=0x7effec35e540) at threadproc/unix/thread.c:151
#7  0x00007f004bd2cea5 in start_thread () from /lib64/libpthread.so.0
#8  0x00007f004b380b0d in clone () from /lib64/libc.so.6

Thread 41 (Thread 0x7f0044dde700 (LWP 6441)):
#0  0x00007f004bd30a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f004eb235bb in switch_user_sql_thread (thread=<optimized out>, obj=0x7f00280dec80) at src/switch_core_sqldb.c:2296
#2  0x00007f004ee46630 in dummy_worker (opaque=0x7f00284fc160) at threadproc/unix/thread.c:151
#3  0x00007f004bd2cea5 in start_thread () from /lib64/libpthread.so.0
#4  0x00007f004b380b0d in clone () from /lib64/libc.so.6

Thread 40 (Thread 0x7f002407c700 (LWP 44)):
#0  0x00007f004b396b44 in clock_nanosleep () from /lib64/libc.so.6
#1  0x00007effe4fb9da0 in ks_sleep (microsec=1000000) at /home/depends/libks/src/ks_time.c:226
#2  0x00007f0025ff361a in mod_signalwire_runtime () at mod_signalwire.c:1467
#3  0x00007f004eb8f042 in switch_loadable_module_exec (thread=0xb20948, obj=0xb20428) at src/switch_loadable_module.c:123
#4  0x00007f004ee46630 in dummy_worker (opaque=0xb20948) at threadproc/unix/thread.c:151
#5  0x00007f004bd2cea5 in start_thread () from /lib64/libpthread.so.0
#6  0x00007f004b380b0d in clone () from /lib64/libc.so.6

Thread 39 (Thread 0x7effb65fb700 (LWP 30812)):
#0  0x00007f004b375ddd in poll () from /lib64/libc.so.6
#1  0x00007f004ee46413 in apr_poll (aprset=aprset@entry=0x7effb04bb0a0, num=num@entry=1, nsds=nsds@entry=0x7effb65fa0a0, timeout=<optimized out>, timeout@entry=20000) at poll/unix/poll.c:117
#2  0x00007f004eafbc1a in switch_poll (aprset=0x7effb04bb0a0, numsock=numsock@entry=1, nsds=nsds@entry=0x7effb65fa0a0, timeout=timeout@entry=20000) at src/switch_apr.c:1096
#3  0x00007f00451066bd in read_packet (listener=listener@entry=0x7effb000cd48, event=event@entry=0x7effb65fa708, timeout=timeout@entry=0) at mod_event_socket.c:1496
#4  0x00007f004510ccef in listener_run (thread=<optimized out>, obj=0x7effb000cd48) at mod_event_socket.c:2756
#5  0x00007f004ee46630 in dummy_worker (opaque=0x7effb04bb258) at threadproc/unix/thread.c:151
#6  0x00007f004bd2cea5 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f004b380b0d in clone () from /lib64/libc.so.6

Thread 38 (Thread 0x7f0026235700 (LWP 33)):
#0  0x00007f004b375ddd in poll () from /lib64/libc.so.6
#1  0x00007f004eb9d697 in switch_wait_socklist (waitlist=waitlist@entry=0x7f0026232b20, len=len@entry=2, ms=ms@entry=100) at src/switch_utils.c:3130
#2  0x00007f00264937e3 in profile_one_loop (profile=profile@entry=0x7f0026236038) at mod_verto.c:4873
#3  0x00007f00264941a0 in runtime (profile=0x7f0026236038) at mod_verto.c:4996
#4  profile_thread (thread=<optimized out>, obj=0x7f0026236038) at mod_verto.c:5792
#5  0x00007f004eb29f5b in switch_core_session_thread_pool_worker (thread=0xc1c310, obj=<optimized out>) at src/switch_core_session.c:1800
#6  0x00007f004ee46630 in dummy_worker (opaque=0xc1c310) at threadproc/unix/thread.c:151
#7  0x00007f004bd2cea5 in start_thread () from /lib64/libpthread.so.0
#8  0x00007f004b380b0d in clone () from /lib64/libc.so.6

Thread 37 (Thread 0x7f0046b51700 (LWP 12)):
#0  0x00007f004b375ddd in poll () from /lib64/libc.so.6
#1  0x00007f004eb9d697 in switch_wait_socklist (waitlist=waitlist@entry=0x7f0046b4eb20, len=len@entry=3, ms=ms@entry=100) at src/switch_utils.c:3130
#2  0x00007f00264937e3 in profile_one_loop (profile=profile@entry=0x7f002625a038) at mod_verto.c:4873
#3  0x00007f00264941a0 in runtime (profile=0x7f002625a038) at mod_verto.c:4996
#4  profile_thread (thread=<optimized out>, obj=0x7f002625a038) at mod_verto.c:5792
#5  0x00007f004eb29f5b in switch_core_session_thread_pool_worker (thread=0x7f0040000d10, obj=<optimized out>) at src/switch_core_session.c:1800
#6  0x00007f004ee46630 in dummy_worker (opaque=0x7f0040000d10) at threadproc/unix/thread.c:151
#7  0x00007f004bd2cea5 in start_thread () from /lib64/libpthread.so.0
#8  0x00007f004b380b0d in clone () from /lib64/libc.so.6

Thread 36 (Thread 0x7f00442e7700 (LWP 204)):
#0  0x00007f004bd30a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f004ee37bb1 in apr_queue_pop (queue=0x7f0047600908, data=0x7f00442e6cc0) at misc/apr_queue.c:276
#2  0x00007f004eafbe15 in switch_queue_pop (queue=<optimized out>, data=<optimized out>) at src/switch_apr.c:1220
#3  0x00007f004eb9fc29 in switch_event_channel_deliver_thread (thread=<optimized out>, obj=0x7f0047600908) at src/switch_event.c:2998
#4  0x00007f004eb29f5b in switch_core_session_thread_pool_worker (thread=0x7f003c043a20, obj=<optimized out>) at src/switch_core_session.c:1800
#5  0x00007f004ee46630 in dummy_worker (opaque=0x7f003c043a20) at threadproc/unix/thread.c:151
#6  0x00007f004bd2cea5 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f004b380b0d in clone () from /lib64/libc.so.6

Thread 35 (Thread 0x7f0044e58700 (LWP 425)):
#0  0x00007f004b375ddd in poll () from /lib64/libc.so.6
#1  0x00007f004ee46413 in apr_poll (aprset=aprset@entry=0x7effb0663b70, num=num@entry=1, nsds=nsds@entry=0x7f0044e570a0, timeout=<optimized out>, timeout@entry=20000) at poll/unix/poll.c:117
#2  0x00007f004eafbc1a in switch_poll (aprset=0x7effb0663b70, numsock=numsock@entry=1, nsds=nsds@entry=0x7f0044e570a0, timeout=timeout@entry=20000) at src/switch_apr.c:1096
#3  0x00007f00451066bd in read_packet (listener=listener@entry=0x7effb0006d18, event=event@entry=0x7f0044e57708, timeout=timeout@entry=0) at mod_event_socket.c:1496
#4  0x00007f004510ccef in listener_run (thread=<optimized out>, obj=0x7effb0006d18) at mod_event_socket.c:2756
#5  0x00007f004ee46630 in dummy_worker (opaque=0x7effb0663d28) at threadproc/unix/thread.c:151
#6  0x00007f004bd2cea5 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f004b380b0d in clone () from /lib64/libc.so.6

Thread 34 (Thread 0x7f00240f6700 (LWP 48)):
#0  0x00007f004bd30a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f004ee37bb1 in apr_queue_pop (queue=0xd7da48, data=0x7f00240f5d40) at misc/apr_queue.c:276
#2  0x00007f004eafbe15 in switch_queue_pop (queue=<optimized out>, data=<optimized out>) at src/switch_apr.c:1220
#3  0x00007f004eb965de in chat_thread_run (thread=<optimized out>, obj=0xd7da48) at src/switch_loadable_module.c:914
#4  0x00007f004ee46630 in dummy_worker (opaque=0xd896d0) at threadproc/unix/thread.c:151
#5  0x00007f004bd2cea5 in start_thread () from /lib64/libpthread.so.0
#6  0x00007f004b380b0d in clone () from /lib64/libc.so.6

Thread 33 (Thread 0x7f0047699700 (LWP 198)):
#0  0x00007f004b396b44 in clock_nanosleep () from /lib64/libc.so.6
#1  0x00007f004ec0f2ab in do_sleep (t=<optimized out>) at src/switch_time.c:173
#2  do_sleep (t=<optimized out>) at src/switch_time.c:150
#3  0x00007f00249ffc9a in vm_event_thread_run (thread=<optimized out>, obj=<optimized out>) at mod_voicemail.c:4082
#4  vm_event_thread_run (thread=<optimized out>, obj=<optimized out>) at mod_voicemail.c:4047
#5  0x00007f004ee46630 in dummy_worker (opaque=0xd023d8) at threadproc/unix/thread.c:151
#6  0x00007f004bd2cea5 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f004b380b0d in clone () from /lib64/libc.so.6

Thread 32 (Thread 0x7f0047713700 (LWP 8)):
#0  0x00007f004bd30a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f004ee37bb1 in apr_queue_pop (queue=0x94cd68, data=0x7f0047712d20) at misc/apr_queue.c:276
#2  0x00007f004eafbe15 in switch_queue_pop (queue=<optimized out>, data=<optimized out>) at src/switch_apr.c:1220
#3  0x00007f004eba19b6 in switch_event_dispatch_thread (thread=<optimized out>, obj=0x94cd68) at src/switch_event.c:331
#4  0x00007f004ee46630 in dummy_worker (opaque=0x7f0047811f40) at threadproc/unix/thread.c:151
#5  0x00007f004bd2cea5 in start_thread () from /lib64/libpthread.so.0
#6  0x00007f004b380b0d in clone () from /lib64/libc.so.6

Thread 31 (Thread 0x7effb567e700 (LWP 19926)):
#0  0x00007f004b375ddd in poll () from /lib64/libc.so.6
#1  0x00007f004ee46413 in apr_poll (aprset=aprset@entry=0x7effb0333080, num=num@entry=1, nsds=nsds@entry=0x7effb567d0a0, timeout=<optimized out>, timeout@entry=20000) at poll/unix/poll.c:117
#2  0x00007f004eafbc1a in switch_poll (aprset=0x7effb0333080, numsock=numsock@entry=1, nsds=nsds@entry=0x7effb567d0a0, timeout=timeout@entry=20000) at src/switch_apr.c:1096
#3  0x00007f00451066bd in read_packet (listener=listener@entry=0x7effb0010d68, event=event@entry=0x7effb567d708, timeout=timeout@entry=0) at mod_event_socket.c:1496
#4  0x00007f004510ccef in listener_run (thread=<optimized out>, obj=0x7effb0010d68) at mod_event_socket.c:2756
#5  0x00007f004ee46630 in dummy_worker (opaque=0x7effb0333238) at threadproc/unix/thread.c:151
#6  0x00007f004bd2cea5 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f004b380b0d in clone () from /lib64/libc.so.6

Thread 30 (Thread 0x7effe404f700 (LWP 49)):
#0  0x00007f004bd30a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f004ee37bb1 in apr_queue_pop (queue=0xd89700, data=0x7effe404ed40) at misc/apr_queue.c:276
#2  0x00007f004eafbe15 in switch_queue_pop (queue=<optimized out>, data=<optimized out>) at src/switch_apr.c:1220
#3  0x00007f004eb965de in chat_thread_run (thread=<optimized out>, obj=0xd89700) at src/switch_loadable_module.c:914
#4  0x00007f004ee46630 in dummy_worker (opaque=0xd956f0) at threadproc/unix/thread.c:151
#5  0x00007f004bd2cea5 in start_thread () from /lib64/libpthread.so.0
#6  0x00007f004b380b0d in clone () from /lib64/libc.so.6

Thread 29 (Thread 0x7f002403f700 (LWP 45)):
#0  0x00007f004bd30a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f00264901f3 in mod_verto_runtime () at mod_verto.c:6907
#2  0x00007f004eb8f042 in switch_loadable_module_exec (thread=0xb20e00, obj=0xb209e0) at src/switch_loadable_module.c:123
#3  0x00007f004ee46630 in dummy_worker (opaque=0xb20e00) at threadproc/unix/thread.c:151
#4  0x00007f004bd2cea5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f004b380b0d in clone () from /lib64/libc.so.6

Thread 28 (Thread 0x7f0024ea2700 (LWP 36)):
#0  0x00007f004bd30a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f004eb235bb in switch_user_sql_thread (thread=<optimized out>, obj=0xccec20) at src/switch_core_sqldb.c:2296
#2  0x00007f004ee46630 in dummy_worker (opaque=0x7f0024f666c0) at threadproc/unix/thread.c:151
#3  0x00007f004bd2cea5 in start_thread () from /lib64/libpthread.so.0
#4  0x00007f004b380b0d in clone () from /lib64/libc.so.6

Thread 27 (Thread 0x7effcc14c700 (LWP 50)):
#0  0x00007f004bd3375d in read () from /lib64/libpthread.so.0
#1  0x00007f004da98cf8 in read_char () from /lib64/libedit.so.0
#2  0x00007f004da990ae in el_wgetc () from /lib64/libedit.so.0
#3  0x00007f004da9934f in el_wgets () from /lib64/libedit.so.0
#4  0x00007f004daa974d in el_gets () from /lib64/libedit.so.0
#5  0x00007f004eb0dcd7 in console_thread (thread=<optimized out>, obj=<optimized out>) at src/switch_console.c:1065
#6  0x00007f004ee46630 in dummy_worker (opaque=0xd99ca8) at threadproc/unix/thread.c:151
#7  0x00007f004bd2cea5 in start_thread () from /lib64/libpthread.so.0
#8  0x00007f004b380b0d in clone () from /lib64/libc.so.6

Thread 26 (Thread 0x7f0044d18700 (LWP 9587)):
#0  0x00007f004bd30de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f004ee405d2 in apr_thread_cond_timedwait (cond=0x7effec2e1de8, mutex=0x7effec2e1d98, timeout=100000) at locks/unix/thread_cond.c:89
#2  0x00007f004ee37ca9 in apr_queue_pop_timeout (queue=0x7effec2e1d58, data=0x7f0044d17d20, timeout=100000) at misc/apr_queue.c:339
#3  0x00007f004eafbe25 in switch_queue_pop_timeout (queue=<optimized out>, data=<optimized out>, timeout=<optimized out>) at src/switch_apr.c:1225
#4  0x00007f0044e7f2bc in sofia_profile_worker_thread_run (thread=<optimized out>, obj=0x7effec3728c0) at sofia.c:3010
#5  0x00007f004ee46630 in dummy_worker (opaque=0x7f00140e47f8) at threadproc/unix/thread.c:151
#6  0x00007f004bd2cea5 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f004b380b0d in clone () from /lib64/libc.so.6

Thread 25 (Thread 0x7f0024133700 (LWP 47)):
#0  0x00007f004bd3375d in read () from /lib64/libpthread.so.0
#1  0x00007f004ec10ca2 in softtimer_runtime () at src/switch_time.c:1176
#2  0x00007f004eb8f042 in switch_loadable_module_exec (thread=0xb21770, obj=0xb21350) at src/switch_loadable_module.c:123
#3  0x00007f004ee46630 in dummy_worker (opaque=0xb21770) at threadproc/unix/thread.c:151
#4  0x00007f004bd2cea5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f004b380b0d in clone () from /lib64/libc.so.6

Thread 24 (Thread 0x7f0046b14700 (LWP 6440)):
#0  0x00007f004bd3375d in read () from /lib64/libpthread.so.0
#1  0x00007f004b76d76b in sock_read () from /lib64/libcrypto.so.10
#2  0x00007f004b76b7ab in BIO_read () from /lib64/libcrypto.so.10
#3  0x00007f004bae38ab in ssl23_read_bytes () from /lib64/libssl.so.10
#4  0x00007f004bae1f2a in ssl23_get_client_hello () from /lib64/libssl.so.10
#5  0x00007f004bae2598 in ssl23_accept () from /lib64/libssl.so.10
#6  0x00007f004bae3782 in ssl23_write () from /lib64/libssl.so.10
#7  0x00007f004cafb73c in ws_raw_write () from /usr/local/lib/libsofia-sip-ua.so.0
#8  0x00007f004cafcb61 in ws_write_frame () from /usr/local/lib/libsofia-sip-ua.so.0
#9  0x00007f004caf97b1 in tport_send_stream_ws () from /usr/local/lib/libsofia-sip-ua.so.0
#10 0x00007f004caee9fe in tport_vsend () from /usr/local/lib/libsofia-sip-ua.so.0
#11 0x00007f004caee49a in tport_send_msg () from /usr/local/lib/libsofia-sip-ua.so.0
#12 0x00007f004caee213 in tport_prepare_and_send () from /usr/local/lib/libsofia-sip-ua.so.0
#13 0x00007f004caee055 in tport_tsend () from /usr/local/lib/libsofia-sip-ua.so.0
#14 0x00007f004ca4f539 in outgoing_send () from /usr/local/lib/libsofia-sip-ua.so.0
#15 0x00007f004ca4f1aa in outgoing_send_via () from /usr/local/lib/libsofia-sip-ua.so.0
#16 0x00007f004ca4edd5 in outgoing_prepare_send () from /usr/local/lib/libsofia-sip-ua.so.0
#17 0x00007f004ca4ebe0 in outgoing_create () from /usr/local/lib/libsofia-sip-ua.so.0
#18 0x00007f004ca4ca21 in nta_outgoing_mcreate () from /usr/local/lib/libsofia-sip-ua.so.0
#19 0x00007f004ca6e907 in nua_base_client_request () from /usr/local/lib/libsofia-sip-ua.so.0
#20 0x00007f004ca6e5ed in nua_base_client_trequest () from /usr/local/lib/libsofia-sip-ua.so.0
#21 0x00007f004ca8aafa in nua_bye_client_request () from /usr/local/lib/libsofia-sip-ua.so.0
#22 0x00007f004ca6e3bb in nua_client_request_sendmsg () from /usr/local/lib/libsofia-sip-ua.so.0
#23 0x00007f004ca6d907 in nua_client_request_try () from /usr/local/lib/libsofia-sip-ua.so.0
#24 0x00007f004ca6d6b8 in nua_client_init_request0 () from /usr/local/lib/libsofia-sip-ua.so.0
#25 0x00007f004ca6cf88 in nua_client_init_request () from /usr/local/lib/libsofia-sip-ua.so.0
#26 0x00007f004ca6c736 in nua_client_create () from /usr/local/lib/libsofia-sip-ua.so.0
#27 0x00007f004ca83e01 in nua_invite_client_report () from /usr/local/lib/libsofia-sip-ua.so.0
#28 0x00007f004ca7039c in nua_client_report () from /usr/local/lib/libsofia-sip-ua.so.0
#29 0x00007f004ca70121 in nua_base_client_response () from /usr/local/lib/libsofia-sip-ua.so.0
#30 0x00007f004ca837ee in nua_session_client_response () from /usr/local/lib/libsofia-sip-ua.so.0
#31 0x00007f004ca826b8 in nua_invite_client_response () from /usr/local/lib/libsofia-sip-ua.so.0
#32 0x00007f004ca6ed29 in nua_client_response () from /usr/local/lib/libsofia-sip-ua.so.0
#33 0x00007f004ca6e9bc in nua_client_orq_response () from /usr/local/lib/libsofia-sip-ua.so.0
#34 0x00007f004ca53478 in outgoing_recv () from /usr/local/lib/libsofia-sip-ua.so.0
#35 0x00007f004ca54d00 in outgoing_delayed_recv () from /usr/local/lib/libsofia-sip-ua.so.0
#36 0x00007f004cad4a31 in su_base_port_execute_msgs () from /usr/local/lib/libsofia-sip-ua.so.0
#37 0x00007f004cad47bf in su_base_port_getmsgs () from /usr/local/lib/libsofia-sip-ua.so.0
#38 0x00007f004cad4b26 in su_base_port_run () from /usr/local/lib/libsofia-sip-ua.so.0
#39 0x00007f004cad0e7c in su_port_run () from /usr/local/lib/libsofia-sip-ua.so.0
#40 0x00007f004cad1f56 in su_root_run () from /usr/local/lib/libsofia-sip-ua.so.0
#41 0x00007f004cad5988 in su_pthread_port_clone_main () from /usr/local/lib/libsofia-sip-ua.so.0
#42 0x00007f004bd2cea5 in start_thread () from /lib64/libpthread.so.0
#43 0x00007f004b380b0d in clone () from /lib64/libc.so.6

Thread 23 (Thread 0x7f0045dbc700 (LWP 15)):
#0  0x00007f004b377b43 in select () from /lib64/libc.so.6
#1  0x00007f004ee47c75 in apr_sleep (t=<optimized out>) at time/unix/time.c:246
#2  0x00007f004ec0f24c in do_sleep (t=<optimized out>) at src/switch_time.c:164
#3  0x00007f004eb2895d in switch_core_sql_db_thread (thread=<optimized out>, obj=<optimized out>) at src/switch_core_sqldb.c:1541
#4  0x00007f004ee46630 in dummy_worker (opaque=0x7f0047600750) at threadproc/unix/thread.c:151
#5  0x00007f004bd2cea5 in start_thread () from /lib64/libpthread.so.0
#6  0x00007f004b380b0d in clone () from /lib64/libc.so.6

Thread 22 (Thread 0x7f0025db6700 (LWP 34)):
#0  0x00007f004bd30de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007effe4fa79ef in ks_cond_timedwait (cond=0xc23138, ms=10000) at /home/depends/libks/src/ks_mutex.c:563
#2  0x00007f0025dc8c7e in __manager_loop () at /home/depends/signalwire-c/src/handle_manager.c:255
#3  0x00007f0025dc8d3f in __manager_thread_wrapper (thread=0xc232a8, data=0x0) at /home/depends/signalwire-c/src/handle_manager.c:275
#4  0x00007effe4fb5109 in thread_launch (args=0xc232a8) at /home/depends/libks/src/ks_thread.c:248
#5  0x00007f004bd2cea5 in start_thread () from /lib64/libpthread.so.0
#6  0x00007f004b380b0d in clone () from /lib64/libc.so.6

Thread 21 (Thread 0x7f004f5a68c0 (LWP 1)):
#0  0x00007f004b377b43 in select () from /lib64/libc.so.6
#1  0x00007f004ee47c75 in apr_sleep (t=<optimized out>) at time/unix/time.c:246
#2  0x00007f004ec0f24c in do_sleep (t=<optimized out>) at src/switch_time.c:164
#3  0x00007f004eb0f4ea in switch_console_loop () at src/switch_console.c:1186
#4  0x00007f004eb3cb25 in switch_core_runtime_loop (bg=bg@entry=0) at src/switch_core.c:1205
#5  0x000000000040258c in main (argc=1, argv=<optimized out>) at src/switch.c:1227

Thread 20 (Thread 0x7f00258b8700 (LWP 9580)):
#0  0x00007f004bd30a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f004eb235bb in switch_user_sql_thread (thread=<optimized out>, obj=0x7f0010040580) at src/switch_core_sqldb.c:2296
#2  0x00007f004ee46630 in dummy_worker (opaque=0x7f00101389e0) at threadproc/unix/thread.c:151
#3  0x00007f004bd2cea5 in start_thread () from /lib64/libpthread.so.0
#4  0x00007f004b380b0d in clone () from /lib64/libc.so.6

Thread 19 (Thread 0x7f00240b9700 (LWP 46)):
#0  0x00007f004bd339dd in accept () from /lib64/libpthread.so.0
#1  0x00007f004ee45166 in apr_socket_accept (new=new@entry=0x7f00240b8cf0, sock=0x7effb0002bd8, connection_context=0x7effb001ff98) at network_io/unix/sockets.c:195
#2  0x00007f004eafb585 in switch_socket_accept (new_sock=new_sock@entry=0x7f00240b8cf0, sock=<optimized out>, pool=<optimized out>) at src/switch_apr.c:745
#3  0x00007f0045109e10 in mod_event_socket_runtime () at mod_event_socket.c:3010
#4  0x00007f004eb8f042 in switch_loadable_module_exec (thread=0xb212b8, obj=0xb20e98) at src/switch_loadable_module.c:123
#5  0x00007f004ee46630 in dummy_worker (opaque=0xb212b8) at threadproc/unix/thread.c:151
#6  0x00007f004bd2cea5 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f004b380b0d in clone () from /lib64/libc.so.6

Thread 18 (Thread 0x7effb57ec700 (LWP 23283)):
#0  0x00007f004b375ddd in poll () from /lib64/libc.so.6
#1  0x00007f004ee46413 in apr_poll (aprset=aprset@entry=0x7effb01ab060, num=num@entry=1, nsds=nsds@entry=0x7effb57eb0a0, timeout=<optimized out>, timeout@entry=20000) at poll/unix/poll.c:117
#2  0x00007f004eafbc1a in switch_poll (aprset=0x7effb01ab060, numsock=numsock@entry=1, nsds=nsds@entry=0x7effb57eb0a0, timeout=timeout@entry=20000) at src/switch_apr.c:1096
#3  0x00007f00451066bd in read_packet (listener=listener@entry=0x7effb0008d28, event=event@entry=0x7effb57eb708, timeout=timeout@entry=0) at mod_event_socket.c:1496
#4  0x00007f004510ccef in listener_run (thread=<optimized out>, obj=0x7effb0008d28) at mod_event_socket.c:2756
#5  0x00007f004ee46630 in dummy_worker (opaque=0x7effb01ab218) at threadproc/unix/thread.c:151
#6  0x00007f004bd2cea5 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f004b380b0d in clone () from /lib64/libc.so.6

Thread 17 (Thread 0x7f0044e1b700 (LWP 9573)):
#0  0x00007f004b3810e3 in epoll_wait () from /lib64/libc.so.6
#1  0x00007f004cad86ae in su_epoll_port_wait_events () from /usr/local/lib/libsofia-sip-ua.so.0
#2  0x00007f004cad4d35 in su_base_port_step () from /usr/local/lib/libsofia-sip-ua.so.0
#3  0x00007f004cad0edd in su_port_step () from /usr/local/lib/libsofia-sip-ua.so.0
#4  0x00007f004cad2019 in su_root_step () from /usr/local/lib/libsofia-sip-ua.so.0
#5  0x00007f0044e96ae3 in sofia_profile_thread_run (thread=<optimized out>, obj=0x7effec378a30) at sofia.c:3465
#6  0x00007f004ee46630 in dummy_worker (opaque=0x7effec100738) at threadproc/unix/thread.c:151
#7  0x00007f004bd2cea5 in start_thread () from /lib64/libpthread.so.0
#8  0x00007f004b380b0d in clone () from /lib64/libc.so.6

Thread 16 (Thread 0x7f00476d6700 (LWP 9577)):
#0  0x00007f004bd30a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f004ee37bb1 in apr_queue_pop (queue=0x7effec343d68, data=0x7f00476d5d30) at misc/apr_queue.c:276
#2  0x00007f004eafbe15 in switch_queue_pop (queue=<optimized out>, data=<optimized out>) at src/switch_apr.c:1220
#3  0x00007f0044eac476 in sofia_msg_thread_run (thread=<optimized out>, obj=0x7effec343d68) at sofia.c:2295
#4  sofia_msg_thread_run (thread=<optimized out>, obj=0x7effec343d68) at sofia.c:2273
#5  0x00007f004ee46630 in dummy_worker (opaque=0x7effec35daa0) at threadproc/unix/thread.c:151
#6  0x00007f004bd2cea5 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f004b380b0d in clone () from /lib64/libc.so.6

Thread 15 (Thread 0x7f0024170700 (LWP 38)):
#0  0x00007f004bd30a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007effe4868c91 in timer_thread_run (thread=<optimized out>, obj=<optimized out>) at mod_spandsp_fax.c:223
#2  0x00007f004ee46630 in dummy_worker (opaque=0xd25a08) at threadproc/unix/thread.c:151
#3  0x00007f004bd2cea5 in start_thread () from /lib64/libpthread.so.0
#4  0x00007f004b380b0d in clone () from /lib64/libc.so.6

Thread 14 (Thread 0x7f0047352700 (LWP 11)):
#0  0x00007f004bd30de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f004ee405d2 in apr_thread_cond_timedwait (cond=0xb15908, mutex=0xb158b8, timeout=500000) at locks/unix/thread_cond.c:89
#2  0x00007f004ee37ca9 in apr_queue_pop_timeout (queue=0xb15878, data=0x7f0047351d40, timeout=500000) at misc/apr_queue.c:339
#3  0x00007f004eafbe25 in switch_queue_pop_timeout (queue=<optimized out>, data=<optimized out>, timeout=<optimized out>) at src/switch_apr.c:1225
#4  0x00007f004eb8dbaf in switch_scheduler_task_thread (thread=<optimized out>, obj=<optimized out>) at src/switch_scheduler.c:191
#5  0x00007f004ee46630 in dummy_worker (opaque=0x7f004753b5d8) at threadproc/unix/thread.c:151
#6  0x00007f004bd2cea5 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f004b380b0d in clone () from /lib64/libc.so.6

Thread 13 (Thread 0x7f004763e700 (LWP 10)):
#0  apr_queue_trypush (queue=0x7effb059fa08, data=0x7f0030038870) at misc/apr_queue.c:229
#1  0x00007f004eafbeab in switch_queue_trypush (queue=0x7effb059fa08, data=0x7f0030038870) at src/switch_apr.c:1259
#2  0x00007f00451076ac in socket_logger (node=0x7f0028028aa0, level=<optimized out>) at mod_event_socket.c:178
#3  0x00007f004ec046b2 in log_thread (t=<optimized out>, obj=<optimized out>) at src/switch_log.c:515
#4  0x00007f004ee46630 in dummy_worker (opaque=0x7f00476005a8) at threadproc/unix/thread.c:151
#5  0x00007f004bd2cea5 in start_thread () from /lib64/libpthread.so.0
#6  0x00007f004b380b0d in clone () from /lib64/libc.so.6

Thread 12 (Thread 0x7f0044d55700 (LWP 9585)):
#0  0x00007f004bd30de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f004ee405d2 in apr_thread_cond_timedwait (cond=0x7effec2e1de8, mutex=0x7effec2e1d98, timeout=100000) at locks/unix/thread_cond.c:89
#2  0x00007f004ee37ca9 in apr_queue_pop_timeout (queue=0x7effec2e1d58, data=0x7f0044d54d20, timeout=100000) at misc/apr_queue.c:339
#3  0x00007f004eafbe25 in switch_queue_pop_timeout (queue=<optimized out>, data=<optimized out>, timeout=<optimized out>) at src/switch_apr.c:1225
#4  0x00007f0044e7f2bc in sofia_profile_worker_thread_run (thread=<optimized out>, obj=0x7effec378a30) at sofia.c:3010
#5  0x00007f004ee46630 in dummy_worker (opaque=0x7efff0181ce0) at threadproc/unix/thread.c:151
#6  0x00007f004bd2cea5 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f004b380b0d in clone () from /lib64/libc.so.6

Thread 11 (Thread 0x7f0024e65700 (LWP 37)):
#0  0x00007f004b377b43 in select () from /lib64/libc.so.6
#1  0x00007f004ee47c75 in apr_sleep (t=<optimized out>) at time/unix/time.c:246
#2  0x00007f004ec0f24c in do_sleep (t=<optimized out>) at src/switch_time.c:164
#3  0x00007f002503cbaa in node_thread_run (thread=<optimized out>, obj=<optimized out>) at mod_fifo.c:2183
#4  0x00007f004ee46630 in dummy_worker (opaque=0xcbbc68) at threadproc/unix/thread.c:151
#5  0x00007f004bd2cea5 in start_thread () from /lib64/libpthread.so.0
#6  0x00007f004b380b0d in clone () from /lib64/libc.so.6

Thread 10 (Thread 0x7f00277fe700 (LWP 9579)):
#0  0x00007f004b3810e3 in epoll_wait () from /lib64/libc.so.6
#1  0x00007f004cad86ae in su_epoll_port_wait_events () from /usr/local/lib/libsofia-sip-ua.so.0
#2  0x00007f004cad4bcc in su_base_port_run () from /usr/local/lib/libsofia-sip-ua.so.0
#3  0x00007f004cad0e7c in su_port_run () from /usr/local/lib/libsofia-sip-ua.so.0
#4  0x00007f004cad1f56 in su_root_run () from /usr/local/lib/libsofia-sip-ua.so.0
#5  0x00007f004cad5988 in su_pthread_port_clone_main () from /usr/local/lib/libsofia-sip-ua.so.0
#6  0x00007f004bd2cea5 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f004b380b0d in clone () from /lib64/libc.so.6

Thread 9 (Thread 0x7f0044c61700 (LWP 9582)):
#0  0x00007f004b3810e3 in epoll_wait () from /lib64/libc.so.6
#1  0x00007f004cad86ae in su_epoll_port_wait_events () from /usr/local/lib/libsofia-sip-ua.so.0
#2  0x00007f004cad4bcc in su_base_port_run () from /usr/local/lib/libsofia-sip-ua.so.0
#3  0x00007f004cad0e7c in su_port_run () from /usr/local/lib/libsofia-sip-ua.so.0
#4  0x00007f004cad1f56 in su_root_run () from /usr/local/lib/libsofia-sip-ua.so.0
#5  0x00007f004cad5988 in su_pthread_port_clone_main () from /usr/local/lib/libsofia-sip-ua.so.0
#6  0x00007f004bd2cea5 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f004b380b0d in clone () from /lib64/libc.so.6

Thread 8 (Thread 0x7f0026be7700 (LWP 9581)):
#0  0x00007f004bd30de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f004ee405d2 in apr_thread_cond_timedwait (cond=0x7effec2e1de8, mutex=0x7effec2e1d98, timeout=100000) at locks/unix/thread_cond.c:89
#2  0x00007f004ee37ca9 in apr_queue_pop_timeout (queue=0x7effec2e1d58, data=0x7f0026be6d20, timeout=100000) at misc/apr_queue.c:339
#3  0x00007f004eafbe25 in switch_queue_pop_timeout (queue=<optimized out>, data=<optimized out>, timeout=<optimized out>) at src/switch_apr.c:1225
#4  0x00007f0044e7f2bc in sofia_profile_worker_thread_run (thread=<optimized out>, obj=0x7effec0182a0) at sofia.c:3010
#5  0x00007f004ee46630 in dummy_worker (opaque=0x7f0010bcbbe0) at threadproc/unix/thread.c:151
#6  0x00007f004bd2cea5 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f004b380b0d in clone () from /lib64/libc.so.6

Thread 7 (Thread 0x7f0045df9700 (LWP 14)):
#0  0x00007f004b396b44 in clock_nanosleep () from /lib64/libc.so.6
#1  0x00007f004ec0f2ab in do_sleep (t=<optimized out>) at src/switch_time.c:173
#2  do_sleep (t=<optimized out>) at src/switch_time.c:150
#3  0x00007f004eb234ec in switch_user_sql_thread (thread=<optimized out>, obj=0xb9e290) at src/switch_core_sqldb.c:2303
#4  0x00007f004ee46630 in dummy_worker (opaque=0x7f0045ebd6c0) at threadproc/unix/thread.c:151
#5  0x00007f004bd2cea5 in start_thread () from /lib64/libpthread.so.0
#6  0x00007f004b380b0d in clone () from /lib64/libc.so.6

Thread 6 (Thread 0x7f004f4de700 (LWP 7)):
#0  0x00007f004b377b43 in select () from /lib64/libc.so.6
#1  0x00007f004ee47c75 in apr_sleep (t=<optimized out>) at time/unix/time.c:246
#2  0x00007f004ec0f24c in do_sleep (t=<optimized out>) at src/switch_time.c:164
#3  0x00007f004eb18a87 in pool_thread (thread=<optimized out>, obj=<optimized out>) at src/switch_core_memory.c:636
#4  0x00007f004ee46630 in dummy_worker (opaque=0x7f004f540c40) at threadproc/unix/thread.c:151
#5  0x00007f004bd2cea5 in start_thread () from /lib64/libpthread.so.0
#6  0x00007f004b380b0d in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x7f0026baa700 (LWP 9586)):
#0  0x00007f004bd30a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f004eb235bb in switch_user_sql_thread (thread=<optimized out>, obj=0x7f0014045da0) at src/switch_core_sqldb.c:2296
#2  0x00007f004ee46630 in dummy_worker (opaque=0x7f001426bf50) at threadproc/unix/thread.c:151
#3  0x00007f004bd2cea5 in start_thread () from /lib64/libpthread.so.0
#4  0x00007f004b380b0d in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x7f0026b30700 (LWP 9574)):
#0  0x00007f004b3810e3 in epoll_wait () from /lib64/libc.so.6
#1  0x00007f004cad86ae in su_epoll_port_wait_events () from /usr/local/lib/libsofia-sip-ua.so.0
#2  0x00007f004cad4d35 in su_base_port_step () from /usr/local/lib/libsofia-sip-ua.so.0
#3  0x00007f004cad0edd in su_port_step () from /usr/local/lib/libsofia-sip-ua.so.0
#4  0x00007f004cad2019 in su_root_step () from /usr/local/lib/libsofia-sip-ua.so.0
#5  0x00007f0044e96ae3 in sofia_profile_thread_run (thread=<optimized out>, obj=0x7effec3728c0) at sofia.c:3465
#6  0x00007f004ee46630 in dummy_worker (opaque=0x7effec270bf0) at threadproc/unix/thread.c:151
#7  0x00007f004bd2cea5 in start_thread () from /lib64/libpthread.so.0
#8  0x00007f004b380b0d in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x7f0027fff700 (LWP 9583)):
#0  0x00007f004b3810e3 in epoll_wait () from /lib64/libc.so.6
#1  0x00007f004cad86ae in su_epoll_port_wait_events () from /usr/local/lib/libsofia-sip-ua.so.0
#2  0x00007f004cad4bcc in su_base_port_run () from /usr/local/lib/libsofia-sip-ua.so.0
#3  0x00007f004cad0e7c in su_port_run () from /usr/local/lib/libsofia-sip-ua.so.0
#4  0x00007f004cad1f56 in su_root_run () from /usr/local/lib/libsofia-sip-ua.so.0
#5  0x00007f004cad5988 in su_pthread_port_clone_main () from /usr/local/lib/libsofia-sip-ua.so.0
#6  0x00007f004bd2cea5 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f004b380b0d in clone () from /lib64/libc.so.6

Thread 2 (Thread 0x7f0044085700 (LWP 9584)):
#0  0x00007f004bd30a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f004eb235bb in switch_user_sql_thread (thread=<optimized out>, obj=0x7efff00fc550) at src/switch_core_sqldb.c:2296
#2  0x00007f004ee46630 in dummy_worker (opaque=0x7efff0309880) at threadproc/unix/thread.c:151
#3  0x00007f004bd2cea5 in start_thread () from /lib64/libpthread.so.0
#4  0x00007f004b380b0d in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x7f0026ce9700 (LWP 6429)):
#0  0x00007f004b2b9bc7 in abort () from /lib64/libc.so.6
#1  0x00007f004b2b11a6 in __assert_fail_base () from /lib64/libc.so.6
#2  0x00007f004b2b1252 in __assert_fail () from /lib64/libc.so.6
#3  0x00007f004ca5e72f in nua_destroy () from /usr/local/lib/libsofia-sip-ua.so.0
#4  0x00007f0044e96d64 in sofia_profile_thread_run (thread=<optimized out>, obj=0x7effec469650) at sofia.c:3536
#5  0x00007f004ee46630 in dummy_worker (opaque=0x7effec46bb60) at threadproc/unix/thread.c:151
#6  0x00007f004bd2cea5 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f004b380b0d in clone () from /lib64/libc.so.6

Thread 42 (Thread 0x7f0044cdb700 (LWP 9575)):
#0  0x00007f004b3810e3 in epoll_wait () from /lib64/libc.so.6
No symbol table info available.
#1  0x00007f004cad86ae in su_epoll_port_wait_events () from /usr/local/lib/libsofia-sip-ua.so.0
No symbol table info available.
#2  0x00007f004cad4d35 in su_base_port_step () from /usr/local/lib/libsofia-sip-ua.so.0
No symbol table info available.
#3  0x00007f004cad0edd in su_port_step () from /usr/local/lib/libsofia-sip-ua.so.0
No symbol table info available.
#4  0x00007f004cad2019 in su_root_step () from /usr/local/lib/libsofia-sip-ua.so.0
No symbol table info available.
#5  0x00007f0044e96ae3 in sofia_profile_thread_run (thread=<optimized out>, obj=0x7effec0182a0) at sofia.c:3465
        profile = 0x7effec0182a0
        node = <optimized out>
        s_event = 0x0
        use_100rel = <optimized out>
        use_timer = <optimized out>
        use_rfc_5626 = <optimized out>
        supported = 0x7effec35e5e8 "timer, path, replaces"
        sanity = <optimized out>
        attempts = <optimized out>
        worker_thread = 0x7f0010bcbbe0
        st = SWITCH_STATUS_SUCCESS
        qname = "sofia:internal-ipv6", '\000' <repeats 108 times>
        key = <optimized out>
        cert = <optimized out>
        chain = <optimized out>
        ssl_ctx = <optimized out>
        ssl_method = <optimized out>
        __func__ = "sofia_profile_thread_run"
        __PRETTY_FUNCTION__ = "sofia_profile_thread_run"
#6  0x00007f004ee46630 in dummy_worker (opaque=0x7effec35e540) at threadproc/unix/thread.c:151
        thread = 0x7effec35e540
#7  0x00007f004bd2cea5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#8  0x00007f004b380b0d in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 41 (Thread 0x7f0044dde700 (LWP 6441)):
#0  0x00007f004bd30a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x00007f004eb235bb in switch_user_sql_thread (thread=<optimized out>, obj=0x7f00280dec80) at src/switch_core_sqldb.c:2296
        i = <optimized out>
        lc = <optimized out>
        written = <optimized out>
        iterations = <optimized out>
        sanity = <optimized out>
        qm = 0x7f00280dec80
        i = <optimized out>
        __func__ = "switch_user_sql_thread"
#2  0x00007f004ee46630 in dummy_worker (opaque=0x7f00284fc160) at threadproc/unix/thread.c:151
        thread = 0x7f00284fc160
#3  0x00007f004bd2cea5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#4  0x00007f004b380b0d in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 40 (Thread 0x7f002407c700 (LWP 44)):
#0  0x00007f004b396b44 in clock_nanosleep () from /lib64/libc.so.6
No symbol table info available.
#1  0x00007effe4fb9da0 in ks_sleep (microsec=1000000) at /home/depends/libks/src/ks_time.c:226
        ts = {tv_sec = 1, tv_nsec = 0}
#2  0x00007f0025ff361a in mod_signalwire_runtime () at mod_signalwire.c:1467
        __func__ = "mod_signalwire_runtime"
        __PRETTY_FUNCTION__ = "mod_signalwire_runtime"
#3  0x00007f004eb8f042 in switch_loadable_module_exec (thread=0xb20948, obj=0xb20428) at src/switch_loadable_module.c:123
        status = SWITCH_STATUS_SUCCESS
        ts = 0xb20428
        module = 0xc21090
        __PRETTY_FUNCTION__ = "switch_loadable_module_exec"
        __func__ = "switch_loadable_module_exec"
#4  0x00007f004ee46630 in dummy_worker (opaque=0xb20948) at threadproc/unix/thread.c:151
        thread = 0xb20948
#5  0x00007f004bd2cea5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#6  0x00007f004b380b0d in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 39 (Thread 0x7effb65fb700 (LWP 30812)):
#0  0x00007f004b375ddd in poll () from /lib64/libc.so.6
No symbol table info available.
#1  0x00007f004ee46413 in apr_poll (aprset=aprset@entry=0x7effb04bb0a0, num=num@entry=1, nsds=nsds@entry=0x7effb65fa0a0, timeout=<optimized out>, timeout@entry=20000) at poll/unix/poll.c:117
        i = <optimized out>
        tmp_pollset = {{fd = 91, events = 9, revents = 0}, {fd = 1323581131, events = 32512, revents = 0}, {fd = 0, events = 0, revents = 0}, {fd = 77, events = 0, revents = 0}, {fd = -1235247128, events = 32511, revents = 0}, {fd = 1320138253, events = 32512, revents = 0}, {fd = -1235245360, events = 32511, revents = 0}, {fd = 805535056, events = 32512, revents = 0}}
        pollset = 0x7effb65f9fa0
#2  0x00007f004eafbc1a in switch_poll (aprset=0x7effb04bb0a0, numsock=numsock@entry=1, nsds=nsds@entry=0x7effb65fa0a0, timeout=timeout@entry=20000) at src/switch_apr.c:1096
        st = 1
#3  0x00007f00451066bd in read_packet (listener=listener@entry=0x7effb000cd48, event=event@entry=0x7effb65fa708, timeout=timeout@entry=0) at mod_event_socket.c:1496
        fdr = 0
        do_sleep = <optimized out>
        mlen = 0
        bytes = <optimized out>
        mbuf = 0x7efe5000a050 ""
        buf = "Content-Type: log/data\nContent-Length: 77\nLog-Level: 7\nText-Channel: 0\nLog-File: sofia.c\nLog-Func: sofia_profile_thread_run\nLog-Line: 3589\nUser-Data: \n\n\000\000ne: 681\nUser-Data: \n\n\000\n\n\000\066a1d886e-bf15-42e9-b7"...
        len = 77
        status = <optimized out>
        count = 0
        elapsed = <optimized out>
        start = 1641268197
        pop = 0x7f0030037d50
        ptr = 0x7efe5000a050 ""
        crcount = <optimized out>
        buf_len = 2048
        channel = <optimized out>
        clen = 0
        __PRETTY_FUNCTION__ = "read_packet"
        __func__ = "read_packet"
#4  0x00007f004510ccef in listener_run (thread=<optimized out>, obj=0x7effb000cd48) at mod_event_socket.c:2756
        listener = 0x7effb000cd48
        buf = '\000' <repeats 1023 times>
        len = 1024
        status = <optimized out>
        event = 0x0
        reply = "\000OK log level  [7]", '\000' <repeats 493 times>
        session = 0x0
        channel = 0x0
        revent = 0x0
        var = <optimized out>
        locked = 1
        __PRETTY_FUNCTION__ = "listener_run"
        __func__ = "listener_run"
#5  0x00007f004ee46630 in dummy_worker (opaque=0x7effb04bb258) at threadproc/unix/thread.c:151
        thread = 0x7effb04bb258
#6  0x00007f004bd2cea5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#7  0x00007f004b380b0d in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 38 (Thread 0x7f0026235700 (LWP 33)):
#0  0x00007f004b375ddd in poll () from /lib64/libc.so.6
No symbol table info available.
#1  0x00007f004eb9d697 in switch_wait_socklist (waitlist=waitlist@entry=0x7f0026232b20, len=len@entry=2, ms=ms@entry=100) at src/switch_utils.c:3130
        pfds = 0x7effe000dff0
        s = 0
        r = 0
        i = <optimized out>
        __PRETTY_FUNCTION__ = "switch_wait_socklist"
#2  0x00007f00264937e3 in profile_one_loop (profile=profile@entry=0x7f0026236038) at mod_verto.c:4873
        pfds = {{sock = 57, events = 5, revents = 0}, {sock = 58, events = 5, revents = 0}, {sock = 0, events = 0, revents = 0} <repeats 24 times>, {sock = 0, events = 1, revents = 0}, {sock = 0, events = 0, revents = 0}, {sock = -1843462134, events = 0, revents = 0}}
        res = <optimized out>
        x = 0
        i = 2
        max = 2
        __func__ = "profile_one_loop"
#3  0x00007f00264941a0 in runtime (profile=0x7f0026236038) at mod_verto.c:4996
        i = <optimized out>
        listeners = 2
        __func__ = "runtime"
#4  profile_thread (thread=<optimized out>, obj=0x7f0026236038) at mod_verto.c:5792
        profile = 0x7f0026236038
        sanity = 50
        __func__ = "profile_thread"
#5  0x00007f004eb29f5b in switch_core_session_thread_pool_worker (thread=0xc1c310, obj=<optimized out>) at src/switch_core_session.c:1800
        td = 0x7f0026258718
        pop = 0x7f0026258718
        check_status = <optimized out>
        node = <optimized out>
        pool = 0xc1c0b8
        __func__ = "switch_core_session_thread_pool_worker"
#6  0x00007f004ee46630 in dummy_worker (opaque=0xc1c310) at threadproc/unix/thread.c:151
        thread = 0xc1c310
#7  0x00007f004bd2cea5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#8  0x00007f004b380b0d in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 37 (Thread 0x7f0046b51700 (LWP 12)):
#0  0x00007f004b375ddd in poll () from /lib64/libc.so.6
No symbol table info available.
#1  0x00007f004eb9d697 in switch_wait_socklist (waitlist=waitlist@entry=0x7f0046b4eb20, len=len@entry=3, ms=ms@entry=100) at src/switch_utils.c:3130
        pfds = 0x7effd80058a0
        s = 0
        r = 0
        i = <optimized out>
        __PRETTY_FUNCTION__ = "switch_wait_socklist"
#2  0x00007f00264937e3 in profile_one_loop (profile=profile@entry=0x7f002625a038) at mod_verto.c:4873
        pfds = {{sock = 59, events = 5, revents = 0}, {sock = 60, events = 5, revents = 0}, {sock = 61, events = 5, revents = 0}, {sock = 0, events = 0, revents = 0} <repeats 23 times>, {sock = 0, events = 1, revents = 2451505154}, {sock = -1453056084, events = 0, revents = 0}, {sock = 0, events = 0, revents = 0}}
        res = <optimized out>
        x = 0
        i = 3
        max = 3
        __func__ = "profile_one_loop"
#3  0x00007f00264941a0 in runtime (profile=0x7f002625a038) at mod_verto.c:4996
        i = <optimized out>
        listeners = 2
        __func__ = "runtime"
#4  profile_thread (thread=<optimized out>, obj=0x7f002625a038) at mod_verto.c:5792
        profile = 0x7f002625a038
        sanity = 50
        __func__ = "profile_thread"
#5  0x00007f004eb29f5b in switch_core_session_thread_pool_worker (thread=0x7f0040000d10, obj=<optimized out>) at src/switch_core_session.c:1800
        td = 0x7f002627c740
        pop = 0x7f002627c740
        check_status = <optimized out>
        node = <optimized out>
        pool = 0x7f0040000ab8
        __func__ = "switch_core_session_thread_pool_worker"
#6  0x00007f004ee46630 in dummy_worker (opaque=0x7f0040000d10) at threadproc/unix/thread.c:151
        thread = 0x7f0040000d10
#7  0x00007f004bd2cea5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#8  0x00007f004b380b0d in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 36 (Thread 0x7f00442e7700 (LWP 204)):
#0  0x00007f004bd30a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x00007f004ee37bb1 in apr_queue_pop (queue=0x7f0047600908, data=0x7f00442e6cc0) at misc/apr_queue.c:276
        rv = <optimized out>
#2  0x00007f004eafbe15 in switch_queue_pop (queue=<optimized out>, data=<optimized out>) at src/switch_apr.c:1220
No locals.
#3  0x00007f004eb9fc29 in switch_event_channel_deliver_thread (thread=<optimized out>, obj=0x7f0047600908) at src/switch_event.c:2998
        queue = 0x7f0047600908
        pop = 0x7f003c0878b0
        ecd = 0x0
        __func__ = "switch_event_channel_deliver_thread"
#4  0x00007f004eb29f5b in switch_core_session_thread_pool_worker (thread=0x7f003c043a20, obj=<optimized out>) at src/switch_core_session.c:1800
        td = 0x7f003c057740
        pop = 0x7f003c057740
        check_status = <optimized out>
        node = <optimized out>
        pool = 0x7f003c0437c8
        __func__ = "switch_core_session_thread_pool_worker"
#5  0x00007f004ee46630 in dummy_worker (opaque=0x7f003c043a20) at threadproc/unix/thread.c:151
        thread = 0x7f003c043a20
#6  0x00007f004bd2cea5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#7  0x00007f004b380b0d in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 35 (Thread 0x7f0044e58700 (LWP 425)):
#0  0x00007f004b375ddd in poll () from /lib64/libc.so.6
No symbol table info available.
#1  0x00007f004ee46413 in apr_poll (aprset=aprset@entry=0x7effb0663b70, num=num@entry=1, nsds=nsds@entry=0x7f0044e570a0, timeout=<optimized out>, timeout@entry=20000) at poll/unix/poll.c:117
        i = <optimized out>
        tmp_pollset = {{fd = 86, events = 9, revents = 0}, {fd = 1323581131, events = 32512, revents = 0}, {fd = 0, events = 0, revents = 0}, {fd = 77, events = 0, revents = 0}, {fd = 1155887080, events = 32512, revents = 0}, {fd = 1320138253, events = 32512, revents = 0}, {fd = 1155888848, events = 32512, revents = 0}, {fd = 805533648, events = 32512, revents = 0}}
        pollset = 0x7f0044e56fa0
#2  0x00007f004eafbc1a in switch_poll (aprset=0x7effb0663b70, numsock=numsock@entry=1, nsds=nsds@entry=0x7f0044e570a0, timeout=timeout@entry=20000) at src/switch_apr.c:1096
        st = 1
#3  0x00007f00451066bd in read_packet (listener=listener@entry=0x7effb0006d18, event=event@entry=0x7f0044e57708, timeout=timeout@entry=0) at mod_event_socket.c:1496
        fdr = 0
        do_sleep = <optimized out>
        mlen = 0
        bytes = <optimized out>
        mbuf = 0x7effec2684a0 ""
        buf = "Content-Type: log/data\nContent-Length: 77\nLog-Level: 7\nText-Channel: 0\nLog-File: sofia.c\nLog-Func: sofia_profile_thread_run\nLog-Line: 3589\nUser-Data: \n\n\000\000ne: 681\nUser-Data: \n\n\000\n\n", '\000' <repeats 845 times>
        len = 77
        status = <optimized out>
        count = 0
        elapsed = <optimized out>
        start = 1641345241
        pop = 0x7f00300377d0
        ptr = 0x7effec2684a0 ""
        crcount = <optimized out>
        buf_len = 2048
        channel = <optimized out>
        clen = 0
        __PRETTY_FUNCTION__ = "read_packet"
        __func__ = "read_packet"
#4  0x00007f004510ccef in listener_run (thread=<optimized out>, obj=0x7effb0006d18) at mod_event_socket.c:2756
        listener = 0x7effb0006d18
        buf = '\000' <repeats 1023 times>
        len = 1024
        status = <optimized out>
        event = 0x0
        reply = "\000OK log level  [7]", '\000' <repeats 493 times>
        session = 0x0
        channel = 0x0
        revent = 0x0
        var = <optimized out>
        locked = 1
        __PRETTY_FUNCTION__ = "listener_run"
        __func__ = "listener_run"
#5  0x00007f004ee46630 in dummy_worker (opaque=0x7effb0663d28) at threadproc/unix/thread.c:151
        thread = 0x7effb0663d28
#6  0x00007f004bd2cea5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#7  0x00007f004b380b0d in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 34 (Thread 0x7f00240f6700 (LWP 48)):
#0  0x00007f004bd30a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x00007f004ee37bb1 in apr_queue_pop (queue=0xd7da48, data=0x7f00240f5d40) at misc/apr_queue.c:276
        rv = <optimized out>
#2  0x00007f004eafbe15 in switch_queue_pop (queue=<optimized out>, data=<optimized out>) at src/switch_apr.c:1220
No locals.
#3  0x00007f004eb965de in chat_thread_run (thread=<optimized out>, obj=0xd7da48) at src/switch_loadable_module.c:914
        pop = 0x0
        q = 0xd7da48
        __func__ = "chat_thread_run"
#4  0x00007f004ee46630 in dummy_worker (opaque=0xd896d0) at threadproc/unix/thread.c:151
        thread = 0xd896d0
#5  0x00007f004bd2cea5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#6  0x00007f004b380b0d in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 33 (Thread 0x7f0047699700 (LWP 198)):
#0  0x00007f004b396b44 in clock_nanosleep () from /lib64/libc.so.6
No symbol table info available.
#1  0x00007f004ec0f2ab in do_sleep (t=<optimized out>) at src/switch_time.c:173
        ts = {tv_sec = 0, tv_nsec = 100000000}
#2  do_sleep (t=<optimized out>) at src/switch_time.c:150
No locals.
#3  0x00007f00249ffc9a in vm_event_thread_run (thread=<optimized out>, obj=<optimized out>) at mod_voicemail.c:4082
        count = 0
        pop = 0x7f003c024670
        __func__ = "vm_event_thread_run"
#4  vm_event_thread_run (thread=<optimized out>, obj=<optimized out>) at mod_voicemail.c:4047
        done = 0
        __func__ = "vm_event_thread_run"
#5  0x00007f004ee46630 in dummy_worker (opaque=0xd023d8) at threadproc/unix/thread.c:151
        thread = 0xd023d8
#6  0x00007f004bd2cea5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#7  0x00007f004b380b0d in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 32 (Thread 0x7f0047713700 (LWP 8)):
#0  0x00007f004bd30a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x00007f004ee37bb1 in apr_queue_pop (queue=0x94cd68, data=0x7f0047712d20) at misc/apr_queue.c:276
        rv = <optimized out>
#2  0x00007f004eafbe15 in switch_queue_pop (queue=<optimized out>, data=<optimized out>) at src/switch_apr.c:1220
No locals.
#3  0x00007f004eba19b6 in switch_event_dispatch_thread (thread=<optimized out>, obj=0x94cd68) at src/switch_event.c:331
        pop = 0x0
        event = 0x0
        queue = 0x94cd68
        my_id = 0
        __func__ = "switch_event_dispatch_thread"
#4  0x00007f004ee46630 in dummy_worker (opaque=0x7f0047811f40) at threadproc/unix/thread.c:151
        thread = 0x7f0047811f40
#5  0x00007f004bd2cea5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#6  0x00007f004b380b0d in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 31 (Thread 0x7effb567e700 (LWP 19926)):
#0  0x00007f004b375ddd in poll () from /lib64/libc.so.6
No symbol table info available.
#1  0x00007f004ee46413 in apr_poll (aprset=aprset@entry=0x7effb0333080, num=num@entry=1, nsds=nsds@entry=0x7effb567d0a0, timeout=<optimized out>, timeout@entry=20000) at poll/unix/poll.c:117
        i = <optimized out>
        tmp_pollset = {{fd = 56, events = 9, revents = 0}, {fd = 1323581131, events = 32512, revents = 0}, {fd = 0, events = 0, revents = 0}, {fd = 77, events = 0, revents = 0}, {fd = -1251487768, events = 32511, revents = 0}, {fd = 1320138253, events = 32512, revents = 0}, {fd = -1251486000, events = 32511, revents = 0}, {fd = 805536896, events = 32512, revents = 0}}
        pollset = 0x7effb567cfa0
#2  0x00007f004eafbc1a in switch_poll (aprset=0x7effb0333080, numsock=numsock@entry=1, nsds=nsds@entry=0x7effb567d0a0, timeout=timeout@entry=20000) at src/switch_apr.c:1096
        st = 1
#3  0x00007f00451066bd in read_packet (listener=listener@entry=0x7effb0010d68, event=event@entry=0x7effb567d708, timeout=timeout@entry=0) at mod_event_socket.c:1496
        fdr = 0
        do_sleep = <optimized out>
        mlen = 0
        bytes = <optimized out>
        mbuf = 0x7efe1c029080 ""
        buf = "Content-Type: log/data\nContent-Length: 77\nLog-Level: 7\nText-Channel: 0\nLog-File: sofia.c\nLog-Func: sofia_profile_thread_run\nLog-Line: 3589\nUser-Data: \n\n\000\000ne: 681\nUser-Data: \n\n\000\n\n\000\066a1d886e-bf15-42e9-b7"...
        len = 77
        status = <optimized out>
        count = 0
        elapsed = <optimized out>
        start = 1640855651
        pop = 0x7f0030038480
        ptr = 0x7efe1c029080 ""
        crcount = <optimized out>
        buf_len = 2048
        channel = <optimized out>
        clen = 0
        __PRETTY_FUNCTION__ = "read_packet"
        __func__ = "read_packet"
#4  0x00007f004510ccef in listener_run (thread=<optimized out>, obj=0x7effb0010d68) at mod_event_socket.c:2756
        listener = 0x7effb0010d68
        buf = '\000' <repeats 1023 times>
        len = 1024
        status = <optimized out>
        event = 0x0
        reply = "\000OK log level  [7]", '\000' <repeats 493 times>
        session = 0x0
        channel = 0x0
        revent = 0x0
        var = <optimized out>
        locked = 1
        __PRETTY_FUNCTION__ = "listener_run"
        __func__ = "listener_run"
#5  0x00007f004ee46630 in dummy_worker (opaque=0x7effb0333238) at threadproc/unix/thread.c:151
        thread = 0x7effb0333238
#6  0x00007f004bd2cea5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#7  0x00007f004b380b0d in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 30 (Thread 0x7effe404f700 (LWP 49)):
#0  0x00007f004bd30a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x00007f004ee37bb1 in apr_queue_pop (queue=0xd89700, data=0x7effe404ed40) at misc/apr_queue.c:276
        rv = <optimized out>
#2  0x00007f004eafbe15 in switch_queue_pop (queue=<optimized out>, data=<optimized out>) at src/switch_apr.c:1220
No locals.
#3  0x00007f004eb965de in chat_thread_run (thread=<optimized out>, obj=0xd89700) at src/switch_loadable_module.c:914
        pop = 0x0
        q = 0xd89700
        __func__ = "chat_thread_run"
#4  0x00007f004ee46630 in dummy_worker (opaque=0xd956f0) at threadproc/unix/thread.c:151
        thread = 0xd956f0
#5  0x00007f004bd2cea5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#6  0x00007f004b380b0d in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 29 (Thread 0x7f002403f700 (LWP 45)):
#0  0x00007f004bd30a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x00007f00264901f3 in mod_verto_runtime () at mod_verto.c:6907
No locals.
#2  0x00007f004eb8f042 in switch_loadable_module_exec (thread=0xb20e00, obj=0xb209e0) at src/switch_loadable_module.c:123
        status = SWITCH_STATUS_SUCCESS
        ts = 0xb209e0
        module = 0xbef558
        __PRETTY_FUNCTION__ = "switch_loadable_module_exec"
        __func__ = "switch_loadable_module_exec"
#3  0x00007f004ee46630 in dummy_worker (opaque=0xb20e00) at threadproc/unix/thread.c:151
        thread = 0xb20e00
#4  0x00007f004bd2cea5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#5  0x00007f004b380b0d in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 28 (Thread 0x7f0024ea2700 (LWP 36)):
#0  0x00007f004bd30a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x00007f004eb235bb in switch_user_sql_thread (thread=<optimized out>, obj=0xccec20) at src/switch_core_sqldb.c:2296
        i = <optimized out>
        lc = <optimized out>
        written = <optimized out>
        iterations = <optimized out>
        sanity = <optimized out>
        qm = 0xccec20
        i = <optimized out>
        __func__ = "switch_user_sql_thread"
#2  0x00007f004ee46630 in dummy_worker (opaque=0x7f0024f666c0) at threadproc/unix/thread.c:151
        thread = 0x7f0024f666c0
#3  0x00007f004bd2cea5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#4  0x00007f004b380b0d in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 27 (Thread 0x7effcc14c700 (LWP 50)):
#0  0x00007f004bd3375d in read () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x00007f004da98cf8 in read_char () from /lib64/libedit.so.0
No symbol table info available.
#2  0x00007f004da990ae in el_wgetc () from /lib64/libedit.so.0
No symbol table info available.
#3  0x00007f004da9934f in el_wgets () from /lib64/libedit.so.0
No symbol table info available.
#4  0x00007f004daa974d in el_gets () from /lib64/libedit.so.0
No symbol table info available.
#5  0x00007f004eb0dcd7 in console_thread (thread=<optimized out>, obj=<optimized out>) at src/switch_console.c:1065
        arg = 1
        count = 0
        line = <optimized out>
        pool = 0xd99a58
        __func__ = "console_thread"
#6  0x00007f004ee46630 in dummy_worker (opaque=0xd99ca8) at threadproc/unix/thread.c:151
        thread = 0xd99ca8
#7  0x00007f004bd2cea5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#8  0x00007f004b380b0d in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 26 (Thread 0x7f0044d18700 (LWP 9587)):
#0  0x00007f004bd30de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x00007f004ee405d2 in apr_thread_cond_timedwait (cond=0x7effec2e1de8, mutex=0x7effec2e1d98, timeout=100000) at locks/unix/thread_cond.c:89
        rv = <optimized out>
        then = <optimized out>
        abstime = {tv_sec = 1641345276, tv_nsec = 679609000}
#2  0x00007f004ee37ca9 in apr_queue_pop_timeout (queue=0x7effec2e1d58, data=0x7f0044d17d20, timeout=100000) at misc/apr_queue.c:339
        rv = <optimized out>
#3  0x00007f004eafbe25 in switch_queue_pop_timeout (queue=<optimized out>, data=<optimized out>, timeout=<optimized out>) at src/switch_apr.c:1225
No locals.
#4  0x00007f0044e7f2bc in sofia_profile_worker_thread_run (thread=<optimized out>, obj=0x7effec3728c0) at sofia.c:3010
        profile = 0x7effec3728c0
        ireg_loops = 5
        iping_loops = 0
        gateway_loops = <optimized out>
        pop = 0x7f004b645828 <__elf_set___libc_thread_subfreeres_element_res_thread_freeres__>
        tick = 0
        x = 6
        __PRETTY_FUNCTION__ = "sofia_profile_worker_thread_run"
        __func__ = "sofia_profile_worker_thread_run"
#5  0x00007f004ee46630 in dummy_worker (opaque=0x7f00140e47f8) at threadproc/unix/thread.c:151
        thread = 0x7f00140e47f8
#6  0x00007f004bd2cea5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#7  0x00007f004b380b0d in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 25 (Thread 0x7f0024133700 (LWP 47)):
#0  0x00007f004bd3375d in read () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x00007f004ec10ca2 in softtimer_runtime () at src/switch_time.c:1176
        exp = 1
        too_late = 20000000
        current_ms = 1760
        x = <optimized out>
        tick = 18
        sps_interval_ticks = 257
        ts = 1641345276608666
        last = <optimized out>
        fwd_errs = 0
        rev_errs = 0
        profile_tick = 0
        tfd = 68
        time_sync = 4
        last_MICROSECONDS_PER_TICK = 20000
        spec = {it_interval = {tv_sec = 0, tv_nsec = 20000000}, it_value = {tv_sec = 0, tv_nsec = 100000}}
        __func__ = "softtimer_runtime"
#2  0x00007f004eb8f042 in switch_loadable_module_exec (thread=0xb21770, obj=0xb21350) at src/switch_loadable_module.c:123
        status = SWITCH_STATUS_SUCCESS
        ts = 0xb21350
        module = 0xb23d50
        __PRETTY_FUNCTION__ = "switch_loadable_module_exec"
        __func__ = "switch_loadable_module_exec"
#3  0x00007f004ee46630 in dummy_worker (opaque=0xb21770) at threadproc/unix/thread.c:151
        thread = 0xb21770
#4  0x00007f004bd2cea5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#5  0x00007f004b380b0d in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 24 (Thread 0x7f0046b14700 (LWP 6440)):
#0  0x00007f004bd3375d in read () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x00007f004b76d76b in sock_read () from /lib64/libcrypto.so.10
No symbol table info available.
#2  0x00007f004b76b7ab in BIO_read () from /lib64/libcrypto.so.10
No symbol table info available.
#3  0x00007f004bae38ab in ssl23_read_bytes () from /lib64/libssl.so.10
No symbol table info available.
#4  0x00007f004bae1f2a in ssl23_get_client_hello () from /lib64/libssl.so.10
No symbol table info available.
#5  0x00007f004bae2598 in ssl23_accept () from /lib64/libssl.so.10
No symbol table info available.
#6  0x00007f004bae3782 in ssl23_write () from /lib64/libssl.so.10
No symbol table info available.
#7  0x00007f004cafb73c in ws_raw_write () from /usr/local/lib/libsofia-sip-ua.so.0
No symbol table info available.
#8  0x00007f004cafcb61 in ws_write_frame () from /usr/local/lib/libsofia-sip-ua.so.0
No symbol table info available.
#9  0x00007f004caf97b1 in tport_send_stream_ws () from /usr/local/lib/libsofia-sip-ua.so.0
No symbol table info available.
#10 0x00007f004caee9fe in tport_vsend () from /usr/local/lib/libsofia-sip-ua.so.0
No symbol table info available.
#11 0x00007f004caee49a in tport_send_msg () from /usr/local/lib/libsofia-sip-ua.so.0
No symbol table info available.
#12 0x00007f004caee213 in tport_prepare_and_send () from /usr/local/lib/libsofia-sip-ua.so.0
No symbol table info available.
#13 0x00007f004caee055 in tport_tsend () from /usr/local/lib/libsofia-sip-ua.so.0
No symbol table info available.
#14 0x00007f004ca4f539 in outgoing_send () from /usr/local/lib/libsofia-sip-ua.so.0
No symbol table info available.
#15 0x00007f004ca4f1aa in outgoing_send_via () from /usr/local/lib/libsofia-sip-ua.so.0
No symbol table info available.
#16 0x00007f004ca4edd5 in outgoing_prepare_send () from /usr/local/lib/libsofia-sip-ua.so.0
No symbol table info available.
#17 0x00007f004ca4ebe0 in outgoing_create () from /usr/local/lib/libsofia-sip-ua.so.0
No symbol table info available.
#18 0x00007f004ca4ca21 in nta_outgoing_mcreate () from /usr/local/lib/libsofia-sip-ua.so.0
No symbol table info available.
#19 0x00007f004ca6e907 in nua_base_client_request () from /usr/local/lib/libsofia-sip-ua.so.0
No symbol table info available.
#20 0x00007f004ca6e5ed in nua_base_client_trequest () from /usr/local/lib/libsofia-sip-ua.so.0
No symbol table info available.
#21 0x00007f004ca8aafa in nua_bye_client_request () from /usr/local/lib/libsofia-sip-ua.so.0
No symbol table info available.
#22 0x00007f004ca6e3bb in nua_client_request_sendmsg () from /usr/local/lib/libsofia-sip-ua.so.0
No symbol table info available.
#23 0x00007f004ca6d907 in nua_client_request_try () from /usr/local/lib/libsofia-sip-ua.so.0
No symbol table info available.
#24 0x00007f004ca6d6b8 in nua_client_init_request0 () from /usr/local/lib/libsofia-sip-ua.so.0
No symbol table info available.
#25 0x00007f004ca6cf88 in nua_client_init_request () from /usr/local/lib/libsofia-sip-ua.so.0
No symbol table info available.
#26 0x00007f004ca6c736 in nua_client_create () from /usr/local/lib/libsofia-sip-ua.so.0
No symbol table info available.
#27 0x00007f004ca83e01 in nua_invite_client_report () from /usr/local/lib/libsofia-sip-ua.so.0
No symbol table info available.
#28 0x00007f004ca7039c in nua_client_report () from /usr/local/lib/libsofia-sip-ua.so.0
No symbol table info available.
#29 0x00007f004ca70121 in nua_base_client_response () from /usr/local/lib/libsofia-sip-ua.so.0
No symbol table info available.
#30 0x00007f004ca837ee in nua_session_client_response () from /usr/local/lib/libsofia-sip-ua.so.0
No symbol table info available.
#31 0x00007f004ca826b8 in nua_invite_client_response () from /usr/local/lib/libsofia-sip-ua.so.0
No symbol table info available.
#32 0x00007f004ca6ed29 in nua_client_response () from /usr/local/lib/libsofia-sip-ua.so.0
No symbol table info available.
#33 0x00007f004ca6e9bc in nua_client_orq_response () from /usr/local/lib/libsofia-sip-ua.so.0
No symbol table info available.
#34 0x00007f004ca53478 in outgoing_recv () from /usr/local/lib/libsofia-sip-ua.so.0
No symbol table info available.
#35 0x00007f004ca54d00 in outgoing_delayed_recv () from /usr/local/lib/libsofia-sip-ua.so.0
No symbol table info available.
#36 0x00007f004cad4a31 in su_base_port_execute_msgs () from /usr/local/lib/libsofia-sip-ua.so.0
No symbol table info available.
#37 0x00007f004cad47bf in su_base_port_getmsgs () from /usr/local/lib/libsofia-sip-ua.so.0
No symbol table info available.
#38 0x00007f004cad4b26 in su_base_port_run () from /usr/local/lib/libsofia-sip-ua.so.0
No symbol table info available.
#39 0x00007f004cad0e7c in su_port_run () from /usr/local/lib/libsofia-sip-ua.so.0
No symbol table info available.
#40 0x00007f004cad1f56 in su_root_run () from /usr/local/lib/libsofia-sip-ua.so.0
No symbol table info available.
#41 0x00007f004cad5988 in su_pthread_port_clone_main () from /usr/local/lib/libsofia-sip-ua.so.0
No symbol table info available.
#42 0x00007f004bd2cea5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#43 0x00007f004b380b0d in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 23 (Thread 0x7f0045dbc700 (LWP 15)):
#0  0x00007f004b377b43 in select () from /lib64/libc.so.6
No symbol table info available.
#1  0x00007f004ee47c75 in apr_sleep (t=<optimized out>) at time/unix/time.c:246
        tv = {tv_sec = 0, tv_usec = 513204}
#2  0x00007f004ec0f24c in do_sleep (t=<optimized out>) at src/switch_time.c:164
No locals.
#3  0x00007f004eb2895d in switch_core_sql_db_thread (thread=<optimized out>, obj=<optimized out>) at src/switch_core_sqldb.c:1541
        sec = 25
        reg_sec = 10
#4  0x00007f004ee46630 in dummy_worker (opaque=0x7f0047600750) at threadproc/unix/thread.c:151
        thread = 0x7f0047600750
#5  0x00007f004bd2cea5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#6  0x00007f004b380b0d in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 22 (Thread 0x7f0025db6700 (LWP 34)):
#0  0x00007f004bd30de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x00007effe4fa79ef in ks_cond_timedwait (cond=0xc23138, ms=10000) at /home/depends/libks/src/ks_mutex.c:563
        ts = {tv_sec = 1641345276, tv_nsec = 823519000}
        n = 1641345276823519
        r = 0
#2  0x00007f0025dc8c7e in __manager_loop () at /home/depends/signalwire-c/src/handle_manager.c:255
        __func__ = "__manager_loop"
        __PRETTY_FUNCTION__ = "__manager_loop"
#3  0x00007f0025dc8d3f in __manager_thread_wrapper (thread=0xc232a8, data=0x0) at /home/depends/signalwire-c/src/handle_manager.c:275
No locals.
#4  0x00007effe4fb5109 in thread_launch (args=0xc232a8) at /home/depends/libks/src/ks_thread.c:248
        thread = 0xc232a8
        ret = 0x0
        __func__ = "thread_launch"
        __PRETTY_FUNCTION__ = "thread_launch"
#5  0x00007f004bd2cea5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#6  0x00007f004b380b0d in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 21 (Thread 0x7f004f5a68c0 (LWP 1)):
#0  0x00007f004b377b43 in select () from /lib64/libc.so.6
No symbol table info available.
#1  0x00007f004ee47c75 in apr_sleep (t=<optimized out>) at time/unix/time.c:246
        tv = {tv_sec = 0, tv_usec = 537434}
#2  0x00007f004ec0f24c in do_sleep (t=<optimized out>) at src/switch_time.c:164
No locals.
#3  0x00007f004eb0f4ea in switch_console_loop () at src/switch_console.c:1186
        arg = 1
        thread = 0xd99ca8
        thd_attr = 0xd99b40
        pool = 0xd99a58
        __func__ = "switch_console_loop"
        __PRETTY_FUNCTION__ = "switch_console_loop"
#4  0x00007f004eb3cb25 in switch_core_runtime_loop (bg=bg@entry=0) at src/switch_core.c:1205
No locals.
#5  0x000000000040258c in main (argc=1, argv=<optimized out>) at src/switch.c:1227
        pid_path = "/usr/local/freeswitch/run/freeswitch.pid", '\000' <repeats 4055 times>
        pid_buffer = "1", '\000' <repeats 30 times>
        old_pid_buffer = '\000' <repeats 31 times>
        pid_len = 1
        old_pid_len = 0
        err = 0x7f004ee75c39 "Success"
        nf = <optimized out>
        do_wait = <optimized out>
        runas_user = <optimized out>
        runas_group = <optimized out>
        reincarnate = <optimized out>
        reincarnate_reexec = <optimized out>
        fds = {0, 0}
        nc = SWITCH_FALSE
        elegant_term = <optimized out>
        pid = <optimized out>
        i = <optimized out>
        x = <optimized out>
        opts = <optimized out>
        opts_str = '\000' <repeats 1023 times>
        local_argv = {0x7ffc186e0efe "/usr/local/freeswitch/bin/freeswitch", 0x0 <repeats 1023 times>}
        local_argc = <optimized out>
        arg_argv = {0x0 <repeats 128 times>}
        alt_dirs = <optimized out>
        alt_base = <optimized out>
        log_set = <optimized out>
        run_set = <optimized out>
        do_kill = <optimized out>
        priority = <optimized out>
        flags = <optimized out>
        ret = 0
        destroy_status = <optimized out>
        fd = 0x91c6f0
        pool = 0x91c678

Thread 20 (Thread 0x7f00258b8700 (LWP 9580)):
#0  0x00007f004bd30a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x00007f004eb235bb in switch_user_sql_thread (thread=<optimized out>, obj=0x7f0010040580) at src/switch_core_sqldb.c:2296
        i = <optimized out>
        lc = <optimized out>
        written = <optimized out>
        iterations = <optimized out>
        sanity = <optimized out>
        qm = 0x7f0010040580
        i = <optimized out>
        __func__ = "switch_user_sql_thread"
#2  0x00007f004ee46630 in dummy_worker (opaque=0x7f00101389e0) at threadproc/unix/thread.c:151
        thread = 0x7f00101389e0
#3  0x00007f004bd2cea5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#4  0x00007f004b380b0d in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 19 (Thread 0x7f00240b9700 (LWP 46)):
#0  0x00007f004bd339dd in accept () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x00007f004ee45166 in apr_socket_accept (new=new@entry=0x7f00240b8cf0, sock=0x7effb0002bd8, connection_context=0x7effb001ff98) at network_io/unix/sockets.c:195
No locals.
#2  0x00007f004eafb585 in switch_socket_accept (new_sock=new_sock@entry=0x7f00240b8cf0, sock=<optimized out>, pool=<optimized out>) at src/switch_apr.c:745
No locals.
#3  0x00007f0045109e10 in mod_event_socket_runtime () at mod_event_socket.c:3010
        pool = 0x7effb0002a28
        listener_pool = 0x7effb001ff98
        rv = <optimized out>
        sa = 0x7effb0002b08
        inbound_socket = 0x7effb0020078
        listener = 0x7effb0006d18
        x = 0
        errs = 0
        __func__ = "mod_event_socket_runtime"
#4  0x00007f004eb8f042 in switch_loadable_module_exec (thread=0xb212b8, obj=0xb20e98) at src/switch_loadable_module.c:123
        status = SWITCH_STATUS_SUCCESS
        ts = 0xb20e98
        module = 0xbb0710
        __PRETTY_FUNCTION__ = "switch_loadable_module_exec"
        __func__ = "switch_loadable_module_exec"
#5  0x00007f004ee46630 in dummy_worker (opaque=0xb212b8) at threadproc/unix/thread.c:151
        thread = 0xb212b8
#6  0x00007f004bd2cea5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#7  0x00007f004b380b0d in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 18 (Thread 0x7effb57ec700 (LWP 23283)):
#0  0x00007f004b375ddd in poll () from /lib64/libc.so.6
No symbol table info available.
#1  0x00007f004ee46413 in apr_poll (aprset=aprset@entry=0x7effb01ab060, num=num@entry=1, nsds=nsds@entry=0x7effb57eb0a0, timeout=<optimized out>, timeout@entry=20000) at poll/unix/poll.c:117
        i = <optimized out>
        tmp_pollset = {{fd = 87, events = 9, revents = 0}, {fd = 1323581131, events = 32512, revents = 0}, {fd = 0, events = 0, revents = 0}, {fd = 77, events = 0, revents = 0}, {fd = -1249988632, events = 32511, revents = 0}, {fd = 1320138253, events = 32512, revents = 0}, {fd = -1249986864, events = 32511, revents = 0}, {fd = 805466592, events = 32512, revents = 0}}
        pollset = 0x7effb57eafa0
#2  0x00007f004eafbc1a in switch_poll (aprset=0x7effb01ab060, numsock=numsock@entry=1, nsds=nsds@entry=0x7effb57eb0a0, timeout=timeout@entry=20000) at src/switch_apr.c:1096
        st = 1
#3  0x00007f00451066bd in read_packet (listener=listener@entry=0x7effb0008d28, event=event@entry=0x7effb57eb708, timeout=timeout@entry=0) at mod_event_socket.c:1496
        fdr = 0
        do_sleep = <optimized out>
        mlen = 0
        bytes = <optimized out>
        mbuf = 0x7efd34016170 ""
        buf = "Content-Type: log/data\nContent-Length: 77\nLog-Level: 7\nText-Channel: 0\nLog-File: sofia.c\nLog-Func: sofia_profile_thread_run\nLog-Line: 3589\nUser-Data: \n\n\000\000ne: 681\nUser-Data: \n\n\000\n\n\000\066a1d886e-bf15-42e9-b7"...
        len = 77
        status = <optimized out>
        count = 0
        elapsed = <optimized out>
        start = 1640912314
        pop = 0x7f00300271e0
        ptr = 0x7efd34016170 ""
        crcount = <optimized out>
        buf_len = 2048
        channel = <optimized out>
        clen = 0
        __PRETTY_FUNCTION__ = "read_packet"
        __func__ = "read_packet"
#4  0x00007f004510ccef in listener_run (thread=<optimized out>, obj=0x7effb0008d28) at mod_event_socket.c:2756
        listener = 0x7effb0008d28
        buf = '\000' <repeats 1023 times>
        len = 1024
        status = <optimized out>
        event = 0x0
        reply = "\000OK log level  [7]", '\000' <repeats 493 times>
        session = 0x0
        channel = 0x0
        revent = 0x0
        var = <optimized out>
        locked = 1
        __PRETTY_FUNCTION__ = "listener_run"
        __func__ = "listener_run"
#5  0x00007f004ee46630 in dummy_worker (opaque=0x7effb01ab218) at threadproc/unix/thread.c:151
        thread = 0x7effb01ab218
#6  0x00007f004bd2cea5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#7  0x00007f004b380b0d in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 17 (Thread 0x7f0044e1b700 (LWP 9573)):
#0  0x00007f004b3810e3 in epoll_wait () from /lib64/libc.so.6
No symbol table info available.
#1  0x00007f004cad86ae in su_epoll_port_wait_events () from /usr/local/lib/libsofia-sip-ua.so.0
No symbol table info available.
#2  0x00007f004cad4d35 in su_base_port_step () from /usr/local/lib/libsofia-sip-ua.so.0
No symbol table info available.
#3  0x00007f004cad0edd in su_port_step () from /usr/local/lib/libsofia-sip-ua.so.0
No symbol table info available.
#4  0x00007f004cad2019 in su_root_step () from /usr/local/lib/libsofia-sip-ua.so.0
No symbol table info available.
#5  0x00007f0044e96ae3 in sofia_profile_thread_run (thread=<optimized out>, obj=0x7effec378a30) at sofia.c:3465
        profile = 0x7effec378a30
        node = <optimized out>
        s_event = 0x0
        use_100rel = <optimized out>
        use_timer = <optimized out>
        use_rfc_5626 = <optimized out>
        supported = 0x7effec1007e0 "timer, path, replaces"
        sanity = <optimized out>
        attempts = <optimized out>
        worker_thread = 0x7efff0181ce0
        st = 32512
        qname = "sofia:external-ipv6", '\000' <repeats 108 times>
        key = <optimized out>
        cert = <optimized out>
        chain = <optimized out>
        ssl_ctx = <optimized out>
        ssl_method = <optimized out>
        __func__ = "sofia_profile_thread_run"
        __PRETTY_FUNCTION__ = "sofia_profile_thread_run"
#6  0x00007f004ee46630 in dummy_worker (opaque=0x7effec100738) at threadproc/unix/thread.c:151
        thread = 0x7effec100738
#7  0x00007f004bd2cea5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#8  0x00007f004b380b0d in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 16 (Thread 0x7f00476d6700 (LWP 9577)):
#0  0x00007f004bd30a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x00007f004ee37bb1 in apr_queue_pop (queue=0x7effec343d68, data=0x7f00476d5d30) at misc/apr_queue.c:276
        rv = <optimized out>
#2  0x00007f004eafbe15 in switch_queue_pop (queue=<optimized out>, data=<optimized out>) at src/switch_apr.c:1220
No locals.
#3  0x00007f0044eac476 in sofia_msg_thread_run (thread=<optimized out>, obj=0x7effec343d68) at sofia.c:2295
        pop = 0x7f00140561e0
        __func__ = "sofia_msg_thread_run"
#4  sofia_msg_thread_run (thread=<optimized out>, obj=0x7effec343d68) at sofia.c:2273
        q = 0x7effec343d68
        my_id = <optimized out>
        __func__ = "sofia_msg_thread_run"
#5  0x00007f004ee46630 in dummy_worker (opaque=0x7effec35daa0) at threadproc/unix/thread.c:151
        thread = 0x7effec35daa0
#6  0x00007f004bd2cea5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#7  0x00007f004b380b0d in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 15 (Thread 0x7f0024170700 (LWP 38)):
#0  0x00007f004bd30a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x00007effe4868c91 in timer_thread_run (thread=<optimized out>, obj=<optimized out>) at mod_spandsp_fax.c:223
        timer = {interval = 20, flags = 1, samples = 160, samplecount = 160, last_samplecount = 0, timer_interface = 0xb23c40, memory_pool = 0x7effc80009b8, private_info = 0x7effc8000aa0, diff = 0, start = 1640674519130498, tick = 0}
        pvt = <optimized out>
        __func__ = "timer_thread_run"
#2  0x00007f004ee46630 in dummy_worker (opaque=0xd25a08) at threadproc/unix/thread.c:151
        thread = 0xd25a08
#3  0x00007f004bd2cea5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#4  0x00007f004b380b0d in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 14 (Thread 0x7f0047352700 (LWP 11)):
#0  0x00007f004bd30de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x00007f004ee405d2 in apr_thread_cond_timedwait (cond=0xb15908, mutex=0xb158b8, timeout=500000) at locks/unix/thread_cond.c:89
        rv = <optimized out>
        then = <optimized out>
        abstime = {tv_sec = 1641345277, tv_nsec = 18672000}
#2  0x00007f004ee37ca9 in apr_queue_pop_timeout (queue=0xb15878, data=0x7f0047351d40, timeout=500000) at misc/apr_queue.c:339
        rv = <optimized out>
#3  0x00007f004eafbe25 in switch_queue_pop_timeout (queue=<optimized out>, data=<optimized out>, timeout=<optimized out>) at src/switch_apr.c:1225
No locals.
#4  0x00007f004eb8dbaf in switch_scheduler_task_thread (thread=<optimized out>, obj=<optimized out>) at src/switch_scheduler.c:191
        pop = 0x7f0040017c80
        __func__ = "switch_scheduler_task_thread"
#5  0x00007f004ee46630 in dummy_worker (opaque=0x7f004753b5d8) at threadproc/unix/thread.c:151
        thread = 0x7f004753b5d8
#6  0x00007f004bd2cea5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#7  0x00007f004b380b0d in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 13 (Thread 0x7f004763e700 (LWP 10)):
#0  apr_queue_trypush (queue=0x7effb059fa08, data=0x7f0030038870) at misc/apr_queue.c:229
        rv = <optimized out>
#1  0x00007f004eafbeab in switch_queue_trypush (queue=0x7effb059fa08, data=0x7f0030038870) at src/switch_apr.c:1259
        s = 75079
#2  0x00007f00451076ac in socket_logger (node=0x7f0028028aa0, level=<optimized out>) at mod_event_socket.c:178
        dnode = 0x7f0030038870
        l = 0x7effb0006d18
        qstatus = <optimized out>
        __func__ = "socket_logger"
#3  0x00007f004ec046b2 in log_thread (t=<optimized out>, obj=<optimized out>) at src/switch_log.c:515
        pop = 0x7f0028028aa0
        node = 0x7f0028028aa0
        binding = 0x7f00476007b0
        __func__ = "log_thread"
#4  0x00007f004ee46630 in dummy_worker (opaque=0x7f00476005a8) at threadproc/unix/thread.c:151
        thread = 0x7f00476005a8
#5  0x00007f004bd2cea5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#6  0x00007f004b380b0d in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 12 (Thread 0x7f0044d55700 (LWP 9585)):
#0  0x00007f004bd30de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x00007f004ee405d2 in apr_thread_cond_timedwait (cond=0x7effec2e1de8, mutex=0x7effec2e1d98, timeout=100000) at locks/unix/thread_cond.c:89
        rv = <optimized out>
        then = <optimized out>
        abstime = {tv_sec = 1641345276, tv_nsec = 677138000}
#2  0x00007f004ee37ca9 in apr_queue_pop_timeout (queue=0x7effec2e1d58, data=0x7f0044d54d20, timeout=100000) at misc/apr_queue.c:339
        rv = <optimized out>
#3  0x00007f004eafbe25 in switch_queue_pop_timeout (queue=<optimized out>, data=<optimized out>, timeout=<optimized out>) at src/switch_apr.c:1225
No locals.
#4  0x00007f0044e7f2bc in sofia_profile_worker_thread_run (thread=<optimized out>, obj=0x7effec378a30) at sofia.c:3010
        profile = 0x7effec378a30
        ireg_loops = 5
        iping_loops = 0
        gateway_loops = <optimized out>
        pop = 0x7f004b645828 <__elf_set___libc_thread_subfreeres_element_res_thread_freeres__>
        tick = 0
        x = 6
        __PRETTY_FUNCTION__ = "sofia_profile_worker_thread_run"
        __func__ = "sofia_profile_worker_thread_run"
#5  0x00007f004ee46630 in dummy_worker (opaque=0x7efff0181ce0) at threadproc/unix/thread.c:151
        thread = 0x7efff0181ce0
#6  0x00007f004bd2cea5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#7  0x00007f004b380b0d in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 11 (Thread 0x7f0024e65700 (LWP 37)):
#0  0x00007f004b377b43 in select () from /lib64/libc.so.6
No symbol table info available.
#1  0x00007f004ee47c75 in apr_sleep (t=<optimized out>) at time/unix/time.c:246
        tv = {tv_sec = 0, tv_usec = 536816}
#2  0x00007f004ec0f24c in do_sleep (t=<optimized out>) at src/switch_time.c:164
No locals.
#3  0x00007f002503cbaa in node_thread_run (thread=<optimized out>, obj=<optimized out>) at mod_fifo.c:2183
        ppl_waiting = <optimized out>
        consumer_total = <optimized out>
        idle_consumers = <optimized out>
        need_sleep = <optimized out>
        node = 0x0
        last = <optimized out>
        this_node = <optimized out>
        cur_priority = <optimized out>
        __func__ = "node_thread_run"
#4  0x00007f004ee46630 in dummy_worker (opaque=0xcbbc68) at threadproc/unix/thread.c:151
        thread = 0xcbbc68
#5  0x00007f004bd2cea5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#6  0x00007f004b380b0d in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 10 (Thread 0x7f00277fe700 (LWP 9579)):
#0  0x00007f004b3810e3 in epoll_wait () from /lib64/libc.so.6
No symbol table info available.
#1  0x00007f004cad86ae in su_epoll_port_wait_events () from /usr/local/lib/libsofia-sip-ua.so.0
No symbol table info available.
#2  0x00007f004cad4bcc in su_base_port_run () from /usr/local/lib/libsofia-sip-ua.so.0
No symbol table info available.
#3  0x00007f004cad0e7c in su_port_run () from /usr/local/lib/libsofia-sip-ua.so.0
No symbol table info available.
#4  0x00007f004cad1f56 in su_root_run () from /usr/local/lib/libsofia-sip-ua.so.0
No symbol table info available.
#5  0x00007f004cad5988 in su_pthread_port_clone_main () from /usr/local/lib/libsofia-sip-ua.so.0
No symbol table info available.
#6  0x00007f004bd2cea5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#7  0x00007f004b380b0d in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 9 (Thread 0x7f0044c61700 (LWP 9582)):
#0  0x00007f004b3810e3 in epoll_wait () from /lib64/libc.so.6
No symbol table info available.
#1  0x00007f004cad86ae in su_epoll_port_wait_events () from /usr/local/lib/libsofia-sip-ua.so.0
No symbol table info available.
#2  0x00007f004cad4bcc in su_base_port_run () from /usr/local/lib/libsofia-sip-ua.so.0
No symbol table info available.
#3  0x00007f004cad0e7c in su_port_run () from /usr/local/lib/libsofia-sip-ua.so.0
No symbol table info available.
#4  0x00007f004cad1f56 in su_root_run () from /usr/local/lib/libsofia-sip-ua.so.0
No symbol table info available.
#5  0x00007f004cad5988 in su_pthread_port_clone_main () from /usr/local/lib/libsofia-sip-ua.so.0
No symbol table info available.
#6  0x00007f004bd2cea5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#7  0x00007f004b380b0d in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 8 (Thread 0x7f0026be7700 (LWP 9581)):
#0  0x00007f004bd30de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x00007f004ee405d2 in apr_thread_cond_timedwait (cond=0x7effec2e1de8, mutex=0x7effec2e1d98, timeout=100000) at locks/unix/thread_cond.c:89
        rv = <optimized out>
        then = <optimized out>
        abstime = {tv_sec = 1641345276, tv_nsec = 667276000}
#2  0x00007f004ee37ca9 in apr_queue_pop_timeout (queue=0x7effec2e1d58, data=0x7f0026be6d20, timeout=100000) at misc/apr_queue.c:339
        rv = <optimized out>
#3  0x00007f004eafbe25 in switch_queue_pop_timeout (queue=<optimized out>, data=<optimized out>, timeout=<optimized out>) at src/switch_apr.c:1225
No locals.
#4  0x00007f0044e7f2bc in sofia_profile_worker_thread_run (thread=<optimized out>, obj=0x7effec0182a0) at sofia.c:3010
        profile = 0x7effec0182a0
        ireg_loops = 5
        iping_loops = 0
        gateway_loops = <optimized out>
        pop = 0x7f0026be7700
        tick = 0
        x = 6
        __PRETTY_FUNCTION__ = "sofia_profile_worker_thread_run"
        __func__ = "sofia_profile_worker_thread_run"
#5  0x00007f004ee46630 in dummy_worker (opaque=0x7f0010bcbbe0) at threadproc/unix/thread.c:151
        thread = 0x7f0010bcbbe0
#6  0x00007f004bd2cea5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#7  0x00007f004b380b0d in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 7 (Thread 0x7f0045df9700 (LWP 14)):
#0  0x00007f004b396b44 in clock_nanosleep () from /lib64/libc.so.6
No symbol table info available.
#1  0x00007f004ec0f2ab in do_sleep (t=<optimized out>) at src/switch_time.c:173
        ts = {tv_sec = 0, tv_nsec = 5000000}
#2  do_sleep (t=<optimized out>) at src/switch_time.c:150
No locals.
#3  0x00007f004eb234ec in switch_user_sql_thread (thread=<optimized out>, obj=0xb9e290) at src/switch_core_sqldb.c:2303
        i = 36
        lc = <optimized out>
        written = <optimized out>
        iterations = <optimized out>
        sanity = <optimized out>
        qm = 0xb9e290
        i = <optimized out>
        __func__ = "switch_user_sql_thread"
#4  0x00007f004ee46630 in dummy_worker (opaque=0x7f0045ebd6c0) at threadproc/unix/thread.c:151
        thread = 0x7f0045ebd6c0
#5  0x00007f004bd2cea5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#6  0x00007f004b380b0d in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 6 (Thread 0x7f004f4de700 (LWP 7)):
#0  0x00007f004b377b43 in select () from /lib64/libc.so.6
No symbol table info available.
#1  0x00007f004ee47c75 in apr_sleep (t=<optimized out>) at time/unix/time.c:246
        tv = {tv_sec = 0, tv_usec = 761990}
#2  0x00007f004ec0f24c in do_sleep (t=<optimized out>) at src/switch_time.c:164
No locals.
#3  0x00007f004eb18a87 in pool_thread (thread=<optimized out>, obj=<optimized out>) at src/switch_core_memory.c:636
        len = 0
#4  0x00007f004ee46630 in dummy_worker (opaque=0x7f004f540c40) at threadproc/unix/thread.c:151
        thread = 0x7f004f540c40
#5  0x00007f004bd2cea5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#6  0x00007f004b380b0d in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 5 (Thread 0x7f0026baa700 (LWP 9586)):
#0  0x00007f004bd30a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x00007f004eb235bb in switch_user_sql_thread (thread=<optimized out>, obj=0x7f0014045da0) at src/switch_core_sqldb.c:2296
        i = <optimized out>
        lc = <optimized out>
        written = <optimized out>
        iterations = <optimized out>
        sanity = <optimized out>
        qm = 0x7f0014045da0
        i = <optimized out>
        __func__ = "switch_user_sql_thread"
#2  0x00007f004ee46630 in dummy_worker (opaque=0x7f001426bf50) at threadproc/unix/thread.c:151
        thread = 0x7f001426bf50
#3  0x00007f004bd2cea5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#4  0x00007f004b380b0d in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 4 (Thread 0x7f0026b30700 (LWP 9574)):
#0  0x00007f004b3810e3 in epoll_wait () from /lib64/libc.so.6
No symbol table info available.
#1  0x00007f004cad86ae in su_epoll_port_wait_events () from /usr/local/lib/libsofia-sip-ua.so.0
No symbol table info available.
#2  0x00007f004cad4d35 in su_base_port_step () from /usr/local/lib/libsofia-sip-ua.so.0
No symbol table info available.
#3  0x00007f004cad0edd in su_port_step () from /usr/local/lib/libsofia-sip-ua.so.0
No symbol table info available.
#4  0x00007f004cad2019 in su_root_step () from /usr/local/lib/libsofia-sip-ua.so.0
No symbol table info available.
#5  0x00007f0044e96ae3 in sofia_profile_thread_run (thread=<optimized out>, obj=0x7effec3728c0) at sofia.c:3465
        profile = 0x7effec3728c0
        node = <optimized out>
        s_event = 0x0
        use_100rel = <optimized out>
        use_timer = <optimized out>
        use_rfc_5626 = <optimized out>
        supported = 0x7effec270c98 "timer, path, replaces"
        sanity = <optimized out>
        attempts = <optimized out>
        worker_thread = 0x7f00140e47f8
        st = SWITCH_STATUS_SUCCESS
        qname = "sofia:external", '\000' <repeats 113 times>
        key = <optimized out>
        cert = <optimized out>
        chain = <optimized out>
        ssl_ctx = <optimized out>
        ssl_method = <optimized out>
        __func__ = "sofia_profile_thread_run"
        __PRETTY_FUNCTION__ = "sofia_profile_thread_run"
#6  0x00007f004ee46630 in dummy_worker (opaque=0x7effec270bf0) at threadproc/unix/thread.c:151
        thread = 0x7effec270bf0
#7  0x00007f004bd2cea5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#8  0x00007f004b380b0d in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 3 (Thread 0x7f0027fff700 (LWP 9583)):
#0  0x00007f004b3810e3 in epoll_wait () from /lib64/libc.so.6
No symbol table info available.
#1  0x00007f004cad86ae in su_epoll_port_wait_events () from /usr/local/lib/libsofia-sip-ua.so.0
No symbol table info available.
#2  0x00007f004cad4bcc in su_base_port_run () from /usr/local/lib/libsofia-sip-ua.so.0
No symbol table info available.
#3  0x00007f004cad0e7c in su_port_run () from /usr/local/lib/libsofia-sip-ua.so.0
No symbol table info available.
#4  0x00007f004cad1f56 in su_root_run () from /usr/local/lib/libsofia-sip-ua.so.0
No symbol table info available.
#5  0x00007f004cad5988 in su_pthread_port_clone_main () from /usr/local/lib/libsofia-sip-ua.so.0
No symbol table info available.
#6  0x00007f004bd2cea5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#7  0x00007f004b380b0d in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 2 (Thread 0x7f0044085700 (LWP 9584)):
#0  0x00007f004bd30a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x00007f004eb235bb in switch_user_sql_thread (thread=<optimized out>, obj=0x7efff00fc550) at src/switch_core_sqldb.c:2296
        i = <optimized out>
        lc = <optimized out>
        written = <optimized out>
        iterations = <optimized out>
        sanity = <optimized out>
        qm = 0x7efff00fc550
        i = <optimized out>
        __func__ = "switch_user_sql_thread"
#2  0x00007f004ee46630 in dummy_worker (opaque=0x7efff0309880) at threadproc/unix/thread.c:151
        thread = 0x7efff0309880
#3  0x00007f004bd2cea5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#4  0x00007f004b380b0d in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 1 (Thread 0x7f0026ce9700 (LWP 6429)):
#0  0x00007f004b2b9bc7 in abort () from /lib64/libc.so.6
No symbol table info available.
#1  0x00007f004b2b11a6 in __assert_fail_base () from /lib64/libc.so.6
No symbol table info available.
#2  0x00007f004b2b1252 in __assert_fail () from /lib64/libc.so.6
No symbol table info available.
#3  0x00007f004ca5e72f in nua_destroy () from /usr/local/lib/libsofia-sip-ua.so.0
No symbol table info available.
#4  0x00007f0044e96d64 in sofia_profile_thread_run (thread=<optimized out>, obj=0x7effec469650) at sofia.c:3536
        profile = 0x7effec469650
        node = <optimized out>
        s_event = 0x0
        use_100rel = <optimized out>
        use_timer = <optimized out>
        use_rfc_5626 = <optimized out>
        supported = 0x7effec46bc08 "timer, path, replaces"
        sanity = 4
        attempts = <optimized out>
        worker_thread = 0x7f00280de690
        st = SWITCH_STATUS_SUCCESS
        qname = "sofia:internal", '\000' <repeats 113 times>
        key = <optimized out>
        cert = <optimized out>
        chain = <optimized out>
        ssl_ctx = <optimized out>
        ssl_method = <optimized out>
        __func__ = "sofia_profile_thread_run"
        __PRETTY_FUNCTION__ = "sofia_profile_thread_run"
#5  0x00007f004ee46630 in dummy_worker (opaque=0x7effec46bb60) at threadproc/unix/thread.c:151
        thread = 0x7effec46bb60
#6  0x00007f004bd2cea5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#7  0x00007f004b380b0d in clone () from /lib64/libc.so.6
No symbol table info available.
mjerris commented 2 years ago

are you for sure using the most recent sofia-sip library?

yzh1127116449 commented 2 years ago

Dear Engineer

     I confirm that I am using version 1.13，Because I'm compiling , tip me ：sofia-sip-ua >= 1.13… configure: error: no usable sofia-sip; please install sofia-sip-ua devel package or equivalent. Thank you for your help！

1641355197(1)

------------------ 原始邮件 ------------------ 发件人: "signalwire/freeswitch" @.>; 发送时间: 2022年1月5日(星期三) 中午11:33 @.>; @.**@.>; 主题: Re: [signalwire/freeswitch] WRONG_CALL_STATE & wss port dead & freeSWITCH Process exits (Issue #1513)

are you for sure using the most recent sofia-sip library?

Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android. You are receiving this because you authored the thread.Message ID: @.***>

boteman commented 2 years ago

On the rather rare occasions that I see WRONG_CALL_STATE in the FS logs, I note that it's always preceded by "detaching session", but there is never the expected "Re-attaching to session" immediately following it as I see with successful calls.

I also see the same 10 second timeout before the call is abandoned in WRONG_CALL_STATE.

FS does not crash for me, that call simply fails. Then things behave normally after that.

Hope this helps.

yzh1127116449 commented 2 years ago

On the rather rare occasions that I see WRONG_CALL_STATE in the FS logs, I note that it's always preceded by "detaching session", but there is never the expected "Re-attaching to session" immediately following it as I see with successful calls.

I also see the same 10 second timeout before the call is abandoned in WRONG_CALL_STATE.

FS does not crash for me, that call simply fails. Then things behave normally after that.

Hope this helps.

Thank you for your reply. I still don't know how to solve the two problems

  1. As for the issue of WRONG_CALL_STATE, actually I don't understand what you mean
  2. I developed webRTC client with JSSIP, registered 100 users at the same time, made 500 calls, re-registered, and then made another 500 calls, re-registered, and the cycle continued; If the WSS connection fails for a period of time, run reload mod_sofia on the fs_cli, The FS process to exit and produce the core file. The core file analysis log is shown as above.
boteman commented 2 years ago

I saw a small similarity with your issue. I sometimes (rarely) have a problem with Polycom SIP phones initiating a call to FS. When the call fails it shows WRONG_CALL_STATE after a 10 second timeout, same as your issue. I have very little information to guide me, but at first I thought sofia might be the common element.

Now the lead developer, anthm, suggested that my phone is not responding to an authentication challenge from FS and to increase debug logging levels to see if this reveals the problem. Now I'm not sure if this is related to your issue as I have never worked with JSSIP nor web sockets, but maybe this can help you?

Here is a link to his comment on the FreeSWITCH Slack channel: https://signalwire-community.slack.com/archives/CDC8H14PN/p1642461343064100

themsley-voiceflex commented 11 months ago

Lots of webrtc issues will be fixed by #2283 so I'd suggest trying that patch

ChiMuYuan commented 10 months ago

I am still encountering this problem, while using WSS (JsSIP) WebRTC for calls. The error occurs after 10 seconds, returning 'Abandoned' and 'WRONG_CALL_STATE.'

Currently, I'm using FreeSWITCH Version 1.10.10-release+git20230813T165739Z4cb05e7f4a~64bit (git 4cb05e7 2023-08-13 16:57:39Z 64bit).

I saw someone mentioning #2283 fixing a related issue, referencing sofia pull #233 in v1.13.17. I'm unsure about the sofia version I'm currently using, and I couldn't find a command in the documentation to check the sofia version.