kamailio / kamailio

Kamailio - The Open Source SIP Server for large VoIP and real-time communication platforms -
https://www.kamailio.org
Other
2.27k stars 948 forks source link

Topos CRASH SIGABRT #1842

Closed marekche closed 5 years ago

marekche commented 5 years ago

Description

I'm using sipwise that uses kamailio v5.1.6. When I activate topos function, the service kamailio-lb crashed.

Troubleshooting

Reproduction

Debugging Data

root@sipwise:/var/lib/systemd/coredump# gdb /usr/sbin/kamailio core.kamailio.114.e2e9f644da934aa79dee789c15ca82ec.9647.1548426648000000
GNU gdb (Debian 7.12-6) 7.12.0.20161007-git
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/sbin/kamailio...Reading symbols from /usr/lib/debug/.build-id/9f/2ba82048cf1a75e75010b612f547a8938b4aeb.debug...done.
done.
[New LWP 9647]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/sbin/kamailio -P /run/kamailio/kamailio.lb.pid -f /etc/kamailio/lb/kamaili'.
Program terminated with signal SIGABRT, Aborted.
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
51      ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) backtrace
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
#1  0x00007f66d5aab42a in __GI_abort () at abort.c:89
#2  0x00007f66d5ae7c00 in __libc_message (do_abort=do_abort@entry=2, fmt=fmt@entry=0x7f66d5bdb305 "*** %s ***: %s terminated\n")
    at ../sysdeps/posix/libc_fatal.c:175
#3  0x00007f66d5b701f7 in __GI___fortify_fail (msg=msg@entry=0x7f66d5bdb29c "buffer overflow detected") at fortify_fail.c:30
#4  0x00007f66d5b6e330 in __GI___chk_fail () at chk_fail.c:28
#5  0x00007f66b867e097 in memcpy (__len=<optimized out>, __src=<optimized out>, __dest=0x7f66b8892524 <_tps_redis_cbuf+4>)
    at /usr/include/x86_64-linux-gnu/bits/string3.h:53
#6  tps_redis_load_branch (msg=<optimized out>, md=<optimized out>, sd=0x7ffed3993d30, mode=<optimized out>) at topos_redis_storage.c:744
#7  0x00007f66b88b35c0 in tps_request_received (msg=msg@entry=0x7ffed3995f80, dialog=dialog@entry=1) at tps_msg.c:786
#8  0x00007f66b88b9255 in tps_msg_received (evp=<optimized out>) at topos_mod.c:332
#9  0x000055e1dc07da88 in sr_event_exec (type=<optimized out>, evp=<optimized out>) at core/events.c:211
#10 0x000055e1dc0415f5 in receive_msg (
    buf=buf@entry=0x55e1dc511d40 <buf> "BYE sip:atpsh-5c4b0f6e-25b2-9@10.10.10.2 SIP/2.0\r\nVia: SIP/2.0/UDP 10.10.10.1:5060;branch=z9hG4bK898122669\r\nFrom: <sip:2972426338@10.10.10.1>;tag=3296734298\r\nTo: <sip:414533@10.10.10.2>;tag=03D2CCF0-5"..., len=381, rcv_info=rcv_info@entry=0x7ffed3996980) at core/receive.c:157
#11 0x000055e1dbf5e6e3 in udp_rcv_loop () at core/udp_server.c:554
#12 0x000055e1dbef3a9f in main_loop () at main.c:1619
#13 0x000055e1dbeea8cb in main (argc=<optimized out>, argv=0x7ffed3996f38) at main.c:2638
(gdb) bt full
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
        set = {__val = {0, 3904965243653075002, 2314885530818457913, 2314885530818453536, 3395749441387372576, 6860733171187738988, 8679965255892022326, 
            7091318039360988973, 3329341198525361252, 7221018865173163891, 3256155514114093622, 3689070848804021815, 3276212736508309553, 3473172721157808240, 
            3472339519882145840, 2319698187989365536}}
        pid = <optimized out>
        tid = <optimized out>
#1  0x00007f66d5aab42a in __GI_abort () at abort.c:89
        save_stage = 2
        act = {__sigaction_handler = {sa_handler = 0x343664363666372d, sa_sigaction = 0x343664363666372d}, sa_mask = {__val = {8606977229197423669, 
              3472328296226648109, 3475143045726351408, 3919881042301820976, 3472332715816543286, 3919317182990200112, 8223625903103882548, 3616443484039294253, 
              4207881715281047603, 4050479036606984240, 2314885530818453555, 2314885530818453536, 7795484802351636512, 3917909816998060649, 3276497845987585332, 
              140732448437520}}, sa_flags = 64, sa_restorer = 0x7ffed398f510}
        sigs = {__val = {32, 0 <repeats 15 times>}}
#2  0x00007f66d5ae7c00 in __libc_message (do_abort=do_abort@entry=2, fmt=fmt@entry=0x7f66d5bdb305 "*** %s ***: %s terminated\n")
    at ../sysdeps/posix/libc_fatal.c:175
        ap = {{gp_offset = 32, fp_offset = 0, overflow_arg_area = 0x7ffed398f520, reg_save_area = 0x7ffed398f4b0}}
        fd = 2
        on_2 = <optimized out>
        list = <optimized out>
        nlist = <optimized out>
        cp = <optimized out>
        written = <optimized out>
#3  0x00007f66d5b701f7 in __GI___fortify_fail (msg=msg@entry=0x7f66d5bdb29c "buffer overflow detected") at fortify_fail.c:30
No locals.
#4  0x00007f66d5b6e330 in __GI___chk_fail () at chk_fail.c:28
No locals.
#5  0x00007f66b867e097 in memcpy (__len=<optimized out>, __src=<optimized out>, __dest=0x7f66b8892524 <_tps_redis_cbuf+4>)
    at /usr/include/x86_64-linux-gnu/bits/string3.h:53
No locals.
#6  tps_redis_load_branch (msg=<optimized out>, md=<optimized out>, sd=0x7ffed3993d30, mode=<optimized out>) at topos_redis_storage.c:744
        argv = {0x0 <repeats 48 times>}
        argvlen = {0 <repeats 48 times>}
        argc = 0
        rcmd = {s = 0x7f66b868b914 "HGETALL", len = 7}
        rkey = {s = 0x0, len = 0}
        rp = 0x7f66b8892520 <_tps_redis_cbuf> "b:x:INVITE:240827696@10.10.10.1:03D2CCF0-5C4B1D8B000137E9-D8ACC700"
        i = <optimized out>
        rsrv = 0x7f66d1b27108
        rrpl = 0x0
        skey = {s = 0x0, len = 0}
        sval = {s = 0x0, len = 0}
        xvbranch1 = 0x7ffed3991a08
        id = {cbuf = '\000' <repeats 6224 times>..., cp = 0x4000000 <error: Cannot access memory at address 0x4000000>, a_uuid = {
            s = 0x3d47a30 <error: Cannot access memory at address 0x3d47a30>, len = 2090496}, b_uuid = {
            s = 0x2b85d0 <error: Cannot access memory at address 0x2b85d0>, len = 3688360}, a_callid = {s = 0x8 <error: Cannot access memory at address 0x8>, 
---Type <return> to continue, or q <return> to quit---      
            len = 46}, a_rr = {s = 0x2a4d537a18c0e100 <error: Cannot access memory at address 0x2a4d537a18c0e100>, len = -775182280}, b_rr = {
            s = 0x55e1dc113a3f <qm_malloc+1215> "\353\"\017\037\200", len = 0}, s_rr = {s = 0x55e1dc1df604 "core", len = -601324824}, a_contact = {
            s = 0x44 <error: Cannot access memory at address 0x44>, len = 0}, b_contact = {s = 0x0, len = 0}, as_contact = {s = 0x7f66d1cba7f8 "", 
            len = -775772584}, bs_contact = {s = 0x7ffed3995f80 "", len = -744941040}, a_tag = {
            s = 0x7ffed3991b40 "SIP/2.0/UDP 10.10.10.1:5060;branch=z9hG4bK898122669", len = 10}, b_tag = {
            s = 0x55e1dc1898cd <parse_contacts+93> "H\205\300I\211\307\017\204\265$", len = 3688360}, a_uri = {s = 0x7f66d1cba800 "p\250\313\321f\177", len = 46}, 
          b_uri = {s = 0x2a4d537a18c0e100 <error: Cannot access memory at address 0x2a4d537a18c0e100>, len = -775182400}, r_uri = {
            s = 0x55e1dc113a3f <qm_malloc+1215> "\353\"\017\037\200", len = 0}, a_srcaddr = {s = 0x55e1dc1df604 "core", len = -601323000}, b_srcaddr = {
            s = 0x53 <error: Cannot access memory at address 0x53>, len = 0}, a_socket = {
            s = 0x2a4d537a18c0e100 <error: Cannot access memory at address 0x2a4d537a18c0e100>, len = 0}, b_socket = {s = 0x7f66d1cba7f8 "", len = -775772584}, 
          x_via1 = {s = 0x7ffed3995f80 "", len = 2}, x_via2 = {s = 0x7ffed3991b40 "SIP/2.0/UDP 10.10.10.1:5060;branch=z9hG4bK898122669", len = 10}, x_vbranch1 = {
            s = 0x55e1dc18d034 <parse_contact+548> "\205\300y\300H\215=\305%\005", len = -598663560}, x_via = {s = 0x0, len = 0}, x_tag = {
            s = 0x2a4d537a18c0e100 <error: Cannot access memory at address 0x2a4d537a18c0e100>, len = 0}, x_rr = {s = 0x0, len = 0}, y_rr = {
            s = 0x7ffed3995f80 "", len = 2}, x_uri = {s = 0x2a4d537a18c0e100 <error: Cannot access memory at address 0x2a4d537a18c0e100>, len = 10}, s_method = {
            s = 0x7f66b88ac7e6 <tps_storage_link_msg+2262> "\203\370\002\017\216\207\374\377\377L\213%Jw!", len = 0}, s_cseq = {s = 0x0, len = 0}, iflags = 0, 
          direction = 0, s_method_id = 0}
        __func__ = "tps_redis_load_branch"
#7  0x00007f66b88b35c0 in tps_request_received (msg=msg@entry=0x7ffed3995f80, dialog=dialog@entry=1) at tps_msg.c:786
        mtsd = {cbuf = "SIP/2.0/UDP 10.10.10.1:5060;branch=z9hG4bK898122669", '\000' <repeats 8140 times>, cp = 0x7ffed3991b73 "", a_uuid = {
            s = 0x55e1dc511d48 <buf+8> "atpsh-5c4b0f6e-25b2-9@10.10.10.2 SIP/2.0\r\nVia: SIP/2.0/UDP 10.10.10.1:5060;branch=z9hG4bK898122669\r\nFrom: <sip:2972426338@10.10.10.1>;tag=3296734298\r\nTo: <sip:414533@10.10.10.2>;tag=03D2CCF0-5C4B1D8B0"..., len = 21}, b_uuid = {s = 0x0, len = 0}, a_callid = {
            s = 0x55e1dc511e2b <buf+235> "240827696@10.10.10.1\r\nCSeq: 21 BYE\r\nContact: <sip:2972426338@10.10.10.1:5060>\r\nMax-Forwards: 70\r\nUser-Agent: Synway/5.3.2.1\r\nContent-Length: 0\r\n\r\n", len = 20}, a_rr = {s = 0x0, len = 0}, b_rr = {s = 0x0, len = 0}, s_rr = {s = 0x0, len = 0}, a_contact = {
            s = 0x55e1dc511e59 <buf+281> "sip:2972426338@10.10.10.1:5060>\r\nMax-Forwards: 70\r\nUser-Agent: Synway/5.3.2.1\r\nContent-Length: 0\r\n\r\n", 
            len = 30}, b_contact = {s = 0x0, len = 0}, as_contact = {s = 0x0, len = 0}, bs_contact = {s = 0x0, len = 0}, a_tag = {
            s = 0x55e1dc511dd2 <buf+146> "3296734298\r\nTo: <sip:414533@10.10.10.2>;tag=03D2CCF0-5C4B1D8B000137E9-D8ACC700\r\nCall-ID: 240827696@10.10.10.1\r\nCSeq: 21 BYE\r\nContact: <sip:2972426338@10.10.10.1:5060>\r\nMax-Forwards: 70\r\nUser-Agent: Sy"..., len = 10}, b_tag = {
            s = 0x55e1dc511dfe <buf+190> "03D2CCF0-5C4B1D8B000137E9-D8ACC700\r\nCall-ID: 240827696@10.10.10.1\r\nCSeq: 21 BYE\r\nContact: <sip:2972426338@10.10.10.1:5060>\r\nMax-Forwards: 70\r\nUser-Agent: Synway/5.3.2.1\r\nContent-Length: 0\r\n\r\n", len = 34}, a_uri = {s = 0x0, len = 0}, b_uri = {s = 0x0, len = 0}, 
          r_uri = {s = 0x0, len = 0}, a_srcaddr = {s = 0x0, len = 0}, b_srcaddr = {s = 0x0, len = 0}, a_socket = {s = 0x0, len = 0}, b_socket = {s = 0x0, 
            len = 0}, x_via1 = {s = 0x7ffed3991b40 "SIP/2.0/UDP 10.10.10.1:5060;branch=z9hG4bK898122669", len = 51}, x_via2 = {s = 0x0, len = 0}, x_vbranch1 = {
            s = 0x7ffed3991b63 "z9hG4bK898122669", len = 16}, x_via = {s = 0x0, len = 0}, x_tag = {
            s = 0x55e1dc511dd2 <buf+146> "3296734298\r\nTo: <sip:414533@10.10.10.2>;tag=03D2CCF0-5C4B1D8B000137E9-D8ACC700\r\nCall-ID: 240827696@10.10.10.1\r\nCSeq: 21 BYE\r\nContact: <sip:2972426338@10.10.10.1:5060>\r\nMax-Forwards: 70\r\nUser-Agent: Sy"..., len = 10}, x_rr = {s = 0x0, len = 0}, y_rr = {s = 0x0, len = 0}, 
          x_uri = {s = 0x0, len = 0}, s_method = {
            s = 0x55e1dc511e4a <buf+266> "BYE\r\nContact: <sip:2972426338@10.10.10.1:5060>\r\nMax-Forwards: 70\r\nUser-Agent: Synway/5.3.2.1\r\nContent-Length: 0\r\n\r\n", len = 3}, s_cseq = {
            s = 0x55e1dc511e47 <buf+263> "21 BYE\r\nContact: <sip:2972426338@10.10.10.1:5060>\r\nMax-Forwards: 70\r\nUser-Agent: Synway/5.3.2.1\r\nContent-Length: 0\r\n\r\n", len = 2}, iflags = 0, direction = 0, s_method_id = 8}
        stsd = {cbuf = '\000' <repeats 8191 times>, cp = 0x0, a_uuid = {s = 0x0, len = 0}, b_uuid = {s = 0x0, len = 0}, a_callid = {s = 0x0, len = 0}, a_rr = {
            s = 0x0, len = 0}, b_rr = {s = 0x0, len = 0}, s_rr = {s = 0x0, len = 0}, a_contact = {s = 0x0, len = 0}, b_contact = {s = 0x0, len = 0}, as_contact = {
            s = 0x0, len = 0}, bs_contact = {s = 0x0, len = 0}, a_tag = {s = 0x0, len = 0}, b_tag = {s = 0x0, len = 0}, a_uri = {s = 0x0, len = 0}, b_uri = {
            s = 0x0, len = 0}, r_uri = {s = 0x0, len = 0}, a_srcaddr = {s = 0x0, len = 0}, b_srcaddr = {s = 0x0, len = 0}, a_socket = {s = 0x0, len = 0}, 
          b_socket = {s = 0x0, len = 0}, x_via1 = {s = 0x0, len = 0}, x_via2 = {s = 0x0, len = 0}, x_vbranch1 = {s = 0x0, len = 0}, x_via = {s = 0x0, len = 0}, 
          x_tag = {s = 0x0, len = 0}, x_rr = {s = 0x0, len = 0}, y_rr = {s = 0x0, len = 0}, x_uri = {s = 0x0, len = 0}, s_method = {s = 0x0, len = 0}, s_cseq = {
---Type <return> to continue, or q <return> to quit---
            s = 0x0, len = 0}, iflags = 0, direction = 0, s_method_id = 0}
        lkey = {
          s = 0x55e1dc511e2b <buf+235> "240827696@10.10.10.1\r\nCSeq: 21 BYE\r\nContact: <sip:2972426338@10.10.10.1:5060>\r\nMax-Forwards: 70\r\nUser-Agent: Synway/5.3.2.1\r\nContent-Length: 0\r\n\r\n", len = 20}
        nuri = {s = 0x0, len = 0}
        direction = 0
        ret = <optimized out>
        use_branch = 0
        __func__ = "tps_request_received"
#8  0x00007f66b88b9255 in tps_msg_received (evp=<optimized out>) at topos_mod.c:332
        msg = {id = 0, pid = 0, tval = {tv_sec = 0, tv_usec = 0}, fwd_send_flags = {f = 0, blst_imask = 0}, rpl_send_flags = {f = 0, blst_imask = 0}, 
          first_line = {type = 1, flags = 1, len = 50, u = {request = {method = {
                  s = 0x55e1dc511d40 <buf> "BYE sip:atpsh-5c4b0f6e-25b2-9@10.10.10.2 SIP/2.0\r\nVia: SIP/2.0/UDP 10.10.10.1:5060;branch=z9hG4bK898122669\r\nFrom: <sip:2972426338@10.10.10.1>;tag=3296734298\r\nTo: <sip:414533@10.10.10.2>;tag=03D2CCF0-5"..., len = 3}, uri = {
                  s = 0x55e1dc511d44 <buf+4> "sip:atpsh-5c4b0f6e-25b2-9@10.10.10.2 SIP/2.0\r\nVia: SIP/2.0/UDP 10.10.10.1:5060;branch=z9hG4bK898122669\r\nFrom: <sip:2972426338@10.10.10.1>;tag=3296734298\r\nTo: <sip:414533@10.10.10.2>;tag=03D2CCF0-5C4B1"..., len = 36}, version = {
                  s = 0x55e1dc511d69 <buf+41> "SIP/2.0\r\nVia: SIP/2.0/UDP 10.10.10.1:5060;branch=z9hG4bK898122669\r\nFrom: <sip:2972426338@10.10.10.1>;tag=3296734298\r\nTo: <sip:414533@10.10.10.2>;tag=03D2CCF0-5C4B1D8B000137E9-D8ACC700\r\nCall-ID: 240827"..., len = 7}, method_value = 8}, reply = {version = {
                  s = 0x55e1dc511d40 <buf> "BYE sip:atpsh-5c4b0f6e-25b2-9@10.10.10.2 SIP/2.0\r\nVia: SIP/2.0/UDP 10.10.10.1:5060;branch=z9hG4bK898122669\r\nFrom: <sip:2972426338@10.10.10.1>;tag=3296734298\r\nTo: <sip:414533@10.10.10.2>;tag=03D2CCF0-5"..., len = 3}, status = {
                  s = 0x55e1dc511d44 <buf+4> "sip:atpsh-5c4b0f6e-25b2-9@10.10.10.2 SIP/2.0\r\nVia: SIP/2.0/UDP 10.10.10.1:5060;branch=z9hG4bK898122669\r\nFrom: <sip:2972426338@10.10.10.1>;tag=3296734298\r\nTo: <sip:414533@10.10.10.2>;tag=03D2CCF0-5C4B1"..., len = 36}, reason = {
                  s = 0x55e1dc511d69 <buf+41> "SIP/2.0\r\nVia: SIP/2.0/UDP 10.10.10.1:5060;branch=z9hG4bK898122669\r\nFrom: <sip:2972426338@10.10.10.1>;tag=3296734298\r\nTo: <sip:414533@10.10.10.2>;tag=03D2CCF0-5C4B1D8B000137E9-D8ACC700\r\nCall-ID: 240827"..., len = 7}, statuscode = 8}}}, via1 = 0x7f66d1d04488, via2 = 0x0, 
          headers = 0x7f66d1c2a0b8, last_header = 0x7f66d1cba478, parsed_flag = 18446744073709551615, h_via1 = 0x7f66d1c2a0b8, h_via2 = 0x0, 
          callid = 0x7f66d1d04868, to = 0x7f66d1d04728, cseq = 0x7f66d1c2a518, from = 0x7f66d1d04680, contact = 0x7f66d1c2a658, maxforwards = 0x7f66d1c2a700, 
          route = 0x0, record_route = 0x0, content_type = 0x0, content_length = 0x7f66d1cba478, authorization = 0x0, expires = 0x0, proxy_auth = 0x0, 
          supported = 0x0, require = 0x0, proxy_require = 0x0, unsupported = 0x0, allow = 0x0, event = 0x0, accept = 0x0, accept_language = 0x0, 
          organization = 0x0, priority = 0x0, subject = 0x0, user_agent = 0x7f66d1cba3d0, server = 0x0, content_disposition = 0x0, diversion = 0x0, rpid = 0x0, 
          refer_to = 0x0, session_expires = 0x0, min_se = 0x0, sipifmatch = 0x0, subscription_state = 0x0, date = 0x0, identity = 0x0, identity_info = 0x0, 
          pai = 0x0, ppi = 0x0, path = 0x0, privacy = 0x0, min_expires = 0x0, body = 0x0, eoh = 0x55e1dc511ebb <buf+379> "\r\n", 
          unparsed = 0x55e1dc511ebb <buf+379> "\r\n", rcv = {src_ip = {af = 0, len = 0, u = {addrl = {0, 0}, addr32 = {0, 0, 0, 0}, addr16 = {0, 0, 0, 0, 0, 0, 0, 
                  0}, addr = '\000' <repeats 15 times>}}, dst_ip = {af = 0, len = 0, u = {addrl = {0, 0}, addr32 = {0, 0, 0, 0}, addr16 = {0, 0, 0, 0, 0, 0, 0, 
                  0}, addr = '\000' <repeats 15 times>}}, src_port = 0, dst_port = 0, proto_reserved1 = 0, proto_reserved2 = 0, src_su = {s = {sa_family = 0, 
                sa_data = '\000' <repeats 13 times>}, sin = {sin_family = 0, sin_port = 0, sin_addr = {s_addr = 0}, sin_zero = "\000\000\000\000\000\000\000"}, 
              sin6 = {sin6_family = 0, sin6_port = 0, sin6_flowinfo = 0, sin6_addr = {__in6_u = {__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 
                      0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}}, bind_address = 0x0, proto = 0 '\000'}, 
          buf = 0x55e1dc511d40 <buf> "BYE sip:atpsh-5c4b0f6e-25b2-9@10.10.10.2 SIP/2.0\r\nVia: SIP/2.0/UDP 10.10.10.1:5060;branch=z9hG4bK898122669\r\nFrom: <sip:2972426338@10.10.10.1>;tag=3296734298\r\nTo: <sip:414533@10.10.10.2>;tag=03D2CCF0-5"..., len = 381, new_uri = {s = 0x0, len = 0}, dst_uri = {s = 0x0, len = 0}, 
          parsed_uri_ok = 1, parsed_uri = {user = {
              s = 0x55e1dc511d48 <buf+8> "atpsh-5c4b0f6e-25b2-9@10.10.10.2 SIP/2.0\r\nVia: SIP/2.0/UDP 10.10.10.1:5060;branch=z9hG4bK898122669\r\nFrom: <sip:2972426338@10.10.10.1>;tag=3296734298\r\nTo: <sip:414533@10.10.10.2>;tag=03D2CCF0-5C4B1D8B0"..., len = 21}, passwd = {s = 0x0, len = 0}, host = {
              s = 0x55e1dc511d5e <buf+30> "10.10.10.2 SIP/2.0\r\nVia: SIP/2.0/UDP 10.10.10.1:5060;branch=z9hG4bK898122669\r\nFrom: <sip:2972426338@10.10.10.1>;tag=3296734298\r\nTo: <sip:414533@10.10.10.2>;tag=03D2CCF0-5C4B1D8B000137E9-D8ACC700\r\nCall"..., len = 10}, port = {s = 0x0, len = 0}, params = {s = 0x0, len = 0}, 
            sip_params = {s = 0x0, len = 0}, headers = {s = 0x0, len = 0}, port_no = 0, proto = 0, type = SIP_URI_T, flags = (unknown: 0), transport = {s = 0x0, 
---Type <return> to continue, or q <return> to quit---
              len = 0}, ttl = {s = 0x0, len = 0}, user_param = {s = 0x0, len = 0}, maddr = {s = 0x0, len = 0}, method = {s = 0x0, len = 0}, lr = {s = 0x0, 
              len = 0}, r2 = {s = 0x0, len = 0}, gr = {s = 0x0, len = 0}, transport_val = {s = 0x0, len = 0}, ttl_val = {s = 0x0, len = 0}, user_param_val = {
              s = 0x0, len = 0}, maddr_val = {s = 0x0, len = 0}, method_val = {s = 0x0, len = 0}, lr_val = {s = 0x0, len = 0}, r2_val = {s = 0x0, len = 0}, 
            gr_val = {s = 0x0, len = 0}}, parsed_orig_ruri_ok = 0, parsed_orig_ruri = {user = {s = 0x0, len = 0}, passwd = {s = 0x0, len = 0}, host = {s = 0x0, 
              len = 0}, port = {s = 0x0, len = 0}, params = {s = 0x0, len = 0}, sip_params = {s = 0x0, len = 0}, headers = {s = 0x0, len = 0}, port_no = 0, 
            proto = 0, type = ERROR_URI_T, flags = (unknown: 0), transport = {s = 0x0, len = 0}, ttl = {s = 0x0, len = 0}, user_param = {s = 0x0, len = 0}, 
            maddr = {s = 0x0, len = 0}, method = {s = 0x0, len = 0}, lr = {s = 0x0, len = 0}, r2 = {s = 0x0, len = 0}, gr = {s = 0x0, len = 0}, transport_val = {
              s = 0x0, len = 0}, ttl_val = {s = 0x0, len = 0}, user_param_val = {s = 0x0, len = 0}, maddr_val = {s = 0x0, len = 0}, method_val = {s = 0x0, 
              len = 0}, lr_val = {s = 0x0, len = 0}, r2_val = {s = 0x0, len = 0}, gr_val = {s = 0x0, len = 0}}, add_rm = 0x0, body_lumps = 0x0, reply_lump = 0x0, 
          add_to_branch_s = '\000' <repeats 57 times>, add_to_branch_len = 0, hash_index = 0, msg_flags = 0, flags = 0, set_global_address = {s = 0x0, len = 0}, 
          set_global_port = {s = 0x0, len = 0}, force_send_socket = 0x0, path_vec = {s = 0x0, len = 0}, instance = {s = 0x0, len = 0}, reg_id = 0, ruid = {
            s = 0x0, len = 0}, location_ua = {s = 0x0, len = 0}, ldv = {flow = {decoded = 0, rcv = {src_ip = {af = 0, len = 0, u = {addrl = {0, 0}, addr32 = {0, 
                      0, 0, 0}, addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, addr = '\000' <repeats 15 times>}}, dst_ip = {af = 0, len = 0, u = {addrl = {0, 0}, addr32 = {
                      0, 0, 0, 0}, addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, addr = '\000' <repeats 15 times>}}, src_port = 0, dst_port = 0, proto_reserved1 = 0, 
                proto_reserved2 = 0, src_su = {s = {sa_family = 0, sa_data = '\000' <repeats 13 times>}, sin = {sin_family = 0, sin_port = 0, sin_addr = {
                      s_addr = 0}, sin_zero = "\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 0, sin6_port = 0, sin6_flowinfo = 0, sin6_addr = {__in6_u = {
                        __u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}}, 
                bind_address = 0x0, proto = 0 '\000'}}}}
        obuf = 0x7ffed3996760
        nbuf = 0x0
        dialog = <optimized out>
        ret = 0
        __func__ = "tps_msg_received"
#9  0x000055e1dc07da88 in sr_event_exec (type=<optimized out>, evp=<optimized out>) at core/events.c:211
        ret = 0
#10 0x000055e1dc0415f5 in receive_msg (
    buf=buf@entry=0x55e1dc511d40 <buf> "BYE sip:atpsh-5c4b0f6e-25b2-9@10.10.10.2 SIP/2.0\r\nVia: SIP/2.0/UDP 10.10.10.1:5060;branch=z9hG4bK898122669\r\nFrom: <sip:2972426338@10.10.10.1>;tag=3296734298\r\nTo: <sip:414533@10.10.10.2>;tag=03D2CCF0-5"..., len=381, rcv_info=rcv_info@entry=0x7ffed3996980) at core/receive.c:157
        msg = <optimized out>
        ctx = {rec_lev = 0, run_flags = 0, last_retcode = 1, jmp_env = {{__jmpbuf = {140732448466864, 404251807625349706, 140079583580320, 140079583296384, 904, 
                0, -404875561808065974, -5881834129312609718}, __mask_was_saved = 0, __saved_mask = {__val = {140079190344920, 140079190344872, 140079190344872, 
                  140079190344904, 140079190344904, 4402341479424, 4410931414018, 140079570933248, 0, 140079583502920, 94428548119103, 0, 94428548953604, 
                  94428549057192, 13, 17179869184}}}}}
        bctx = <optimized out>
        ret = <optimized out>
        tvb = {tv_sec = 0, tv_usec = 0}
        tve = {tv_sec = 280375465083135, tv_usec = 0}
        tz = {tz_minuteswest = 0, tz_dsttime = 0}
        diff = 0
        inb = {
          s = 0x55e1dc511d40 <buf> "BYE sip:atpsh-5c4b0f6e-25b2-9@10.10.10.2 SIP/2.0\r\nVia: SIP/2.0/UDP 10.10.10.1:5060;branch=z9hG4bK898122669\r\nFrom: <sip:2972426338@10.10.10.1>;tag=3296734298\r\nTo: <sip:414533@10.10.10.2>;tag=03D2CCF0-5"..., len = 381}
        netinfo = {data = {s = 0xff00000000ff <error: Cannot access memory at address 0xff00000000ff>, len = 0}, rcv = 0x5b5b5b5b5b5b5b5b, 
          dst = 0x5b5b5b5b5b5b5b5b}
        keng = 0x0
---Type <return> to continue, or q <return> to quit---
        evp = {data = 0x7ffed3996760, rcv = 0x7ffed3996980, dst = 0x0}
        errsipmsg = 0
        __func__ = "receive_msg"
#11 0x000055e1dbf5e6e3 in udp_rcv_loop () at core/udp_server.c:554
        len = 381
        buf = "BYE sip:atpsh-5c4b0f6e-25b2-9@10.10.10.2 SIP/2.0\r\nVia: SIP/2.0/UDP 10.10.10.1:5060;branch=z9hG4bK898122669\r\nFrom: <sip:2972426338@10.10.10.1>;tag=3296734298\r\nTo: <sip:414533@10.10.10.2>;tag=03D2CCF0-5"...
        from = <optimized out>
        fromlen = 16
        ri = {src_ip = {af = 2, len = 4, u = {addrl = {206175865354, 140732448467576}, addr32 = {17435146, 48, 3550046840, 32766}, addr16 = {2570, 266, 48, 0, 
                27256, 54169, 32766, 0}, addr = "\n\n\n\001\060\000\000\000xj\231\323\376\177\000"}}, dst_ip = {af = 2, len = 4, u = {addrl = {34212362, 0}, 
              addr32 = {34212362, 0, 0, 0}, addr16 = {2570, 522, 0, 0, 0, 0, 0, 0}, addr = "\n\n\n\002", '\000' <repeats 11 times>}}, src_port = 5060, 
          dst_port = 5060, proto_reserved1 = 0, proto_reserved2 = 0, src_su = {s = {sa_family = 2, sa_data = "\023\304\n\n\n\001\000\000\000\000\000\000\000"}, 
            sin = {sin_family = 2, sin_port = 50195, sin_addr = {s_addr = 17435146}, sin_zero = "\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 2, 
              sin6_port = 50195, sin6_flowinfo = 17435146, sin6_addr = {__in6_u = {__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, 
                  __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}}, bind_address = 0x7f66d1b05a38, proto = 1 '\001'}
        evp = {data = 0x0, rcv = 0x0, dst = 0x0}
        printbuf = '\000' <repeats 48 times>, "\005\000\000\000\000\000\000\000\000\341\300\030zSM*\005\000\000\000\000\000\000\000(_\262\321f\177\000\000\005\000\000\000\000\000\000\000\005\000\000\000\000\000\000\000\005\000\000\000\000\000\000\000\363\033\351\320f\177\000\000-\000\000\000\000\000\000\000\000\341\300\030zSM*-\000\000\000\000\000\000\000Z\035\351\320f\177\000\000\005\000\000\000\000\000\000\000\005\000\000\000\000\000\000\000\005\000\000\000\000\000\000\000\005\000\000\000\000\000\000\000\005\000\000\000\000\000\000\000\005\000\000\000\000\000\000\000\005\000\000\000\000\000\000\000"...
        i = <optimized out>
        j = <optimized out>
        __func__ = "udp_rcv_loop"
#12 0x000055e1dbef3a9f in main_loop () at main.c:1619
        i = <optimized out>
        pid = <optimized out>
        si = <optimized out>
        si_desc = "udp receiver child=4 sock=10.10.10.2:5060\000\000\000\001\000\000\000X\252!\334\341U\000\000\000\000\000\000\000\000\000\000\001\000\000\000\000\000\000\000\063\000\000\000\000\000\000\000#\000\000\000\000\000\000\000\000\341\300\030zSM*\000\000\000\000\000\000\000\000\000\341\300\030zSM*\250\351\\\334\341U\000\000\001\000\000\000\000\000\000"
        nrprocs = <optimized out>
        woneinit = <optimized out>
        __func__ = "main_loop"
#13 0x000055e1dbeea8cb in main (argc=<optimized out>, argv=0x7ffed3996f38) at main.c:2638
        cfg_stream = <optimized out>
        c = <optimized out>
        r = <optimized out>
        tmp = 0x7ffed3998ec6 ""
        tmp_len = -709000473
        port = 32614
        proto = 1700966438
        options = 0x55e1dc1e27b8 ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:x:X:Y:"
        ret = -1
        seed = 1164502589
---Type <return> to continue, or q <return> to quit---
        debug_save = <optimized out>
        debug_flag = <optimized out>
        dont_fork_cnt = <optimized out>
        p = <optimized out>
        st = {st_dev = 18, st_ino = 10861, st_nlink = 2, st_mode = 16877, st_uid = 114, st_gid = 120, __pad0 = 0, st_rdev = 0, st_size = 120, st_blksize = 4096, 
          st_blocks = 0, st_atim = {tv_sec = 1548342456, tv_nsec = 87921300}, st_mtim = {tv_sec = 1548422956, tv_nsec = 858936100}, st_ctim = {
            tv_sec = 1548422956, tv_nsec = 858936100}, __glibc_reserved = {0, 0, 0}}
        __func__ = "main"

Log Messages

kamailio-lb
Jan 24 10:53:36 sipwise lb[5118]: ERROR: ndb_redis [redis_client.c:903]: redisc_exec_argv(): Redis error: Server closed the connection
Jan 24 10:54:20 sipwise lb[11283]: ERROR: topos [tps_storage.c:340]: tps_storage_link_msg(): bad sip message or missing Contact hdr
Jan 24 10:54:27 sipwise lb[11311]: CRITICAL: <core> [core/pass_fd.c:277]: receive_fd(): EOF on 21
Jan 24 10:54:27 sipwise lb[11261]: ALERT: <core> [main.c:738]: handle_sigs(): child process 11267 exited by a signal 6
Jan 24 10:54:27 sipwise lb[11261]: ALERT: <core> [main.c:741]: handle_sigs(): core was generated

SIP Traffic

(paste your sip traffic here)

Possible Solutions

Additional Information

version: kamailio 5.1.6 (x86_64/linux) 
flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144 MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
id: unknown 
compiled with gcc 6.3.0
Welcome to Sipwise NGCP platform version 'mr6.5.3'
     _               _          
 ___(_)_ ____      _(_)___  ___ 
/ __| | '_ \ \ /\ / / / __|/ _ \
\__ \ | |_) \ V  V /| \__ \  __/
|___/_| .__/ \_/\_/ |_|___/\___|
      |_|                       
Linux sipwise 4.9.0-8-amd64 #1 SMP Debian 4.9.130-2 (2018-10-27) x86_64
miconda commented 5 years ago

Was this one time crash, or can it be reproduced?

miconda commented 5 years ago

Btw, I just checked the code and there is a patch in branch 5.1 that likely fixed it, seems similar to your report -- can you try with 74b8fbf0fc820d48a0203b81d2cd41f6e84dc5c5 ? That patch is part of 5.1.7, but you run 5.1.6.

marekche commented 5 years ago

Was this one time crash, or can it be reproduced?

Hi! Yes, I can reproduce it. This happens when the server receive a BYE. The backtrace shows same result.

miconda commented 5 years ago

As written in my previous post, likely this is fixed in 5.1.7, with the patch referenced above. Try it and see the results.

miconda commented 5 years ago

Closing. Reopen if the patch referenced above does not fix it.