Server Hanging when call is initiated by drachtio and answered also by drachtio

[amoonrah]

I have a call that goes out from drachio on one interface to my SBC, sbc routes call back to drachtio on another interface.

below is the last output from drachtio and gdb, it does not crash, it just hangs.

running the version that I last pulled from the repository to test the previous crash, which is the latest i think

if I generate the call from SIPP and it reaches drachtio, it does not hang it processes the call.

2019-10-12 21:58:11.818848 Client::processMessage - got request with 5 tokens 2019-10-12 21:58:11.818965 Client::processMessage - request id 1f218660-c515-4994-bcb6-52981957b7da, request type: sip transaction id: , dialog id: 2019-10-12 21:58:11.819121 Client::processMessage - sending a request outside of a dialog 2019-10-12 21:58:11.819237 ClientController::addAppTransaction: transactionId cc27f661-386d-4110-8987-cf310997b894; size: 1 2019-10-12 21:58:11.819341 ClientController::addApiRequest: clientMsgId 1f218660-c515-4994-bcb6-52981957b7da; size: 1 2019-10-12 21:58:11.819611 SipDialogController::doSendRequestOutsideDialog sending request to route url: sip:10.140.22.12 2019-10-12 21:58:11.819756 DrachtioController::findTportForSubscription: no transport found for 4167617201@ims.bell.ca 2019-10-12 21:58:11.820792 SipTransport::findAppropriateTransport: searching for a transport to reach udp/sip:10.140.22.12 2019-10-12 21:58:11.820956 SipTransport::findAppropriateTransport - after filtering for transport we have 36 candidates 2019-10-12 21:58:11.821102 SipTransport::findAppropriateTransport - after filtering for protocol we have 18 candidates

^C Thread 1 "drachtio" received signal SIGINT, Interrupt. 0x00007ffff7766c79 in inet_pton4 () from /lib64/libc.so.6 Missing separate debuginfos, use: dnf debuginfo-install brotli-1.0.7-3.fc30.x86_64 cyrus-sasl-lib-2.1.27-0.6rc7.fc30.x86_64 keyutils-libs-1.6-2.fc30.x86_64 krb5-libs-1.17-14.fc30.x86_64 libcom_err-1.44.6-1.fc30.x86_64 libcurl-7.65.3-2.fc30.x86_64 libgcc-9.1.1-1.fc30.x86_64 libidn2-2.2.0-1.fc30.x86_64 libnghttp2-1.38.0-1.fc30.x86_64 libpsl-0.20.2-6.fc30.x86_64 libselinux-2.9-1.fc30.x86_64 libssh-0.9.0-5.fc30.x86_64 libstdc++-9.1.1-1.fc30.x86_64 libunistring-0.9.10-5.fc30.x86_64 libxcrypt-4.4.6-2.fc30.x86_64 openldap-2.4.47-1.fc30.x86_64 openssl-libs-1.1.1c-2.fc30.x86_64 pcre2-10.33-9.fc30.x86_64 zlib-1.2.11-17.fc30.x86_64 (gdb) bt

0 0x00007ffff7766c79 in inet_pton4 () from /lib64/libc.so.6

1 0x00007ffff7766d6c in __inet_pton_length () from /lib64/libc.so.6

2 0x000000000050d6b7 in drachtio::SipTransport::isInNetwork (this=0x7ef440, address=)

at ../src/sip-transports.cpp:126

3 0x0000000000512c6b in drachtio::SipTransport::isInNetwork (address=, this=)

at ../src/sip-transports.cpp:123

4 drachtio::SipTransport::<lambda(const std::shared_ptr&, const std::shared_ptr&)>::operator() (pB=<error reading variable: Cannot access memory at address 0xffffffff402eef58>,

pA=std::shared_ptr<drachtio::SipTransport> (use count 2, weak count 1) = {...}, __closure=0x7fffffffd8c0)
at ../src/sip-transports.cpp:463

5 gnu_cxx::__ops::_Iter_comp_iter<drachtio::SipTransport::findAppropriateTransport(char const, char const)::<lambda(const std::shared_ptr&, const std::shared_ptr&)> >::operator()<gnu_cxx::normal_iterator<std::shared_ptr*, std::vector<std::shared_ptr > >, __gnu_cxx::normal_iterator<std::shared_ptr*, std::vector<std::shared_ptr > > > (it2=..., it1=...,

this=0x7fffffffd8c0) at /usr/include/c++/9/bits/predefined_ops.h:143

6 std::unguarded_partition<__gnu_cxx::normal_iterator<std::shared_ptr, std::vector<std::shared_ptr > >, __gnu_cxx::__ops::_Iter_comp_iter<drachtio::SipTransport::findAppropriateTransport(char const, char const*)::<lambda(const std::shared_ptr&, const std::shared_ptr&)> > > (__comp=...,

__pivot=..., __last=<error reading variable: Cannot access memory at address 0xffffffff402eef50>, __first=

std::shared_ptr (use count 2, weak count 1) = {get() = 0x7ec4b0}) at /usr/include/c++/9/bits/stl_algo.h:1894

7 std::unguarded_partition_pivot<gnu_cxx::normal_iterator<std::shared_ptr*, std::vector<std::shared_ptr > >, __gnu_cxx::ops::_Iter_comp_iter<drachtio::SipTransport::findAppropriateTransport(char const, char const)::<lambda(const std::shared_ptr&, const std::shared_ptr&)> > > (__comp=...,

__last=std::shared_ptr<drachtio::SipTransport> (use count 1, weak count 1) = {get() = 0x7eedc0}, __first=

std::shared_ptr (use count 2, weak count 1) = {get() = 0x7ef440}) at /usr/include/c++/9/bits/stl_algo.h:1912

8 std::introsort_loop<__gnu_cxx::normal_iterator<std::shared_ptr, std::vector<std::shared_ptr > >, long int, __gnu_cxx::__ops::_Iter_comp_iter<drachtio::SipTransport::findAppropriateTransport(char const, char const)::<lambda(const std::shared_ptr&, const std::shared_ptr&)> > >(__gnu_cxx::__normal_iterator<std::shared_ptr, std::vector<std::shared_ptr, std::allocator<std::shared_ptr > > >, gnu_cxx::normal_iterator<std::shared_ptr*, std::vector<std::shared_ptr, std::allocator<std::shared_ptr > > >, long, gnu_cxx::ops::_Iter_comp_iter<drachtio::SipTransport::findAppropriateTransport(char const, char const)::<lambda(const std::shared_ptr&, const std::shared_ptr&)> >) (__first=

std::shared_ptr (use count 6868640, weak count -1) = {get() = 0x32312e32}, last=..., depth_limit=, __comp=...) at /usr/include/c++/9/bits/stl_algo.h:1942

9 0x0000000000513f64 in std::sort<gnu_cxx::normal_iterator<std::shared_ptr*, std::vector<std::shared_ptr > >, __gnu_cxx::ops::_Iter_comp_iter<drachtio::SipTransport::findAppropriateTransport(char const, char const)::<lambda(const std::shared_ptr&, const std::shared_ptr&)> > > (__comp=...,

__last=std::shared_ptr<drachtio::SipTransport> (use count 1, weak count 1) = {get() = 0x7eedc0}, __first=

std::shared_ptr (use count 2, weak count 1) = {get() = 0x7ef440}) at /usr/include/c++/9/bits/stl_algobase.h:1029

10 std::sort<gnu_cxx::__normal_iterator<std::shared_ptr, std::vector<std::shared_ptr > >, drachtio::SipTransport::findAppropriateTransport(char const, char const*)::<lambda(const std::shared_ptr&, const std::shared_ptr&)> > (comp=..., __last=

std::shared_ptr (use count 1, weak count 1) = {get() = 0x7eedc0}, __first= std::shared_ptr (use count 2, weak count 1) = {get() = 0x7ef440}) at /usr/include/c++/9/bits/stl_algo.h:4850

11 drachtio::SipTransport::findAppropriateTransport (remoteHost=, proto=proto@entry=0x695ad5 "udp")

at ../src/sip-transports.cpp:461

12 0x00000000004d99f5 in drachtio::SipDialogController::doSendRequestOutsideDialog (this=0x7f17c0, pData=0x7ffff000d0b8)

at /usr/include/c++/9/bits/basic_string.h:2300

13 0x00000000004db6c1 in (anonymous namespace)::cloneSendSipRequest (p=, msg=, arg=)

at /usr/include/c++/9/ext/atomicity.h:96

14 0x00000000005c2cb6 in su_base_port_execute_msgs ()

--Type for more, q to quit, c to continue without paging--c

15 0x00000000005c2a40 in su_base_port_getmsgs ()

16 0x00000000005c2dab in su_base_port_run ()

17 0x0000000000576599 in su_port_run ()

18 0x000000000057767c in su_root_run ()

19 0x0000000000449cc5 in drachtio::DrachtioController::run (this=0x7914b0) at ../src/controller.cpp:1159

20 0x000000000042b516 in main (argc=1, argv=0x7fffffffe488) at ../src/main.cpp:47

[davehorton]

I'm not clear -- are you saying the call arrives to drachtio, but is not processed? Or are you saying that somehow the INVITE was never sent from the drachtio?

[amoonrah]

Hey,

the call arrives to drachtio but is not processed.

[davehorton]

ok, can I see the drachtio server log file for the server receiving the call (at debug log level)

[amoonrah]

hey, I was away for a while. drachtiolog.txt

[davehorton]

That log seems to show a crash, just as the app was trying to send an INVITE. Can you provide the stack trace that goes along with it?

[amoonrah]

you're gonna have to remind me how to do the stack trace