search

OpenSIPS / opensips

OpenSIPS is a GPL implementation of a multi-functionality SIP Server that targets to deliver a high-level technical solution (performance, security and quality) to be used in professional SIP server platforms.
https://opensips.org
Other
1.27k stars 578 forks source link

[CRASH] OpenSips Crashes during traffic #1808

Closed syedrizwansrs closed 5 years ago

syedrizwansrs commented 5 years ago

[CRASH] OpenSips Crashes during traffic

OpenSIPS version you are running

version: opensips 3.0.0 (x86_64/linux)
flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, HP_MALLOC, DBG_MALLOC, FAST_LOCK-ADAPTIVE_WAIT
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535
poll method support: poll, epoll, sigio_rt, select.
git revision: 69c12ff9e
main.c compiled on  with gcc 7

Crash Core Dump

https://bit.ly/2OYoI6I

OS/environment information

adamoverbeeke commented 5 years ago

[CRASH] OpenSips Crashes during traffic

We believe we are seeing the same thing.

OpenSIPS version you are running

version: opensips 2.4.6 (x86_64/linux)
flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC, F_MALLOC, FAST_LOCK-ADAPTIVE_WAIT
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535
poll method support: poll, epoll, sigio_rt, select.
git revision: 4e258b17f
main.c compiled on 14:29:57 Aug 26 2019 with gcc 7

Crash Core Dump

#0  0x00007fc9c3677dd5 in fake_req (with_dst=0, uac=<optimized out>, uas=0x7fc9d10aeb30, shm_msg=0x7fc9d32ab548, faked_req=0x7fc9c38f58c0 <faked_req>) at t_msgbuilder.h:246
        ld = 0x7fc9cfc7b9e8
        c = 0x8
        hdr = 0x7fc9d32accf0
        la = 0x7fc9cfc7ba20
#1  run_failure_handlers (t=0x7fc9d10aea40) at t_reply.c:576
        shmem_msg = 0x7fc9d32ab548
        uac = <optimized out>
        on_failure = <optimized out>
        faked_req = {id = 896577, first_line = {type = 1, len = 68, u = {request = {method = {
                  s = 0x7fc9d32abdd0 "INVITE sip:12287281018@testacdload18.byoc.inintca.com:5061 SIP/2.0\r\nMax-Forwards:  9\r\nRecord-Route: <sip:10.32.20.60:5060;du=10.32.255.104:6060;ftag=875073;lr;r2=on>\r\nRecord-Route: <sip:52.206.40.246:"..., len = 6}, uri = {
                  s = 0x7fc9d32abdd7 "sip:12287281018@testacdload18.byoc.inintca.com:5061 SIP/2.0\r\nMax-Forwards:  9\r\nRecord-Route: <sip:10.32.20.60:5060;du=10.32.255.104:6060;ftag=875073;lr;r2=on>\r\nRecord-Route: <sip:52.206.40.246:5061;tr"..., len = 51}, version = {
                  s = 0x7fc9d32abe0b "SIP/2.0\r\nMax-Forwards:  9\r\nRecord-Route: <sip:10.32.20.60:5060;du=10.32.255.104:6060;ftag=875073;lr;r2=on>\r\nRecord-Route: <sip:52.206.40.246:5061;transport=tls;ftag=875073;lr;r2=on>\r\nTo:  <sip:1228728"..., len = 7}, method_value = 1}, reply = {version = {
                  s = 0x7fc9d32abdd0 "INVITE sip:12287281018@testacdload18.byoc.inintca.com:5061 SIP/2.0\r\nMax-Forwards:  9\r\nRecord-Route: <sip:10.32.20.60:5060;du=10.32.255.104:6060;ftag=875073;lr;r2=on>\r\nRecord-Route: <sip:52.206.40.246:"..., len = 6}, status = {
                  s = 0x7fc9d32abdd7 "sip:12287281018@testacdload18.byoc.inintca.com:5061 SIP/2.0\r\nMax-Forwards:  9\r\nRecord-Route: <sip:10.32.20.60:5060;du=10.32.255.104:6060;ftag=875073;lr;r2=on>\r\nRecord-Route: <sip:52.206.40.246:5061;tr"..., len = 51}, reason = {
                  s = 0x7fc9d32abe0b "SIP/2.0\r\nMax-Forwards:  9\r\nRecord-Route: <sip:10.32.20.60:5060;du=10.32.255.104:6060;ftag=875073;lr;r2=on>\r\nRecord-Route: <sip:52.206.40.246:5061;transport=tls;ftag=875073;lr;r2=on>\r\nTo:  <sip:1228728"..., len = 7}, statuscode = 1}}}, via1 = 0x7fc9d32acf30, via2 = 0x7fc9d32ad0e8,
          headers = 0x7fc9d32ac3e8, last_header = 0x7fc9d32ad0a0, parsed_flag = 18446744073709551615, h_via1 = 0x7fc9d32acee8, h_via2 = 0x7fc9d32ad0a0,
          callid = 0x7fc9d32acba0, to = 0x7fc9d32ac4c0, cseq = 0x7fc9d32acc30, from = 0x7fc9d32ac818, contact = 0x7fc9d32accf0, maxforwards = 0x7fc9d32ac3e8, route = 0x0,
          record_route = 0x7fc9d32ac430, path = 0x7fc9d32acbe8, content_type = 0x7fc9d32ace10, content_length = 0x7fc9d32acea0, authorization = 0x0, expires = 0x0,
          proxy_auth = 0x0, supported = 0x0, proxy_require = 0x0, unsupported = 0x0, allow = 0x0, event = 0x0, accept = 0x0, accept_language = 0x0, organization = 0x0,
          priority = 0x0, subject = 0x0, user_agent = 0x0, content_disposition = 0x0, accept_disposition = 0x0, diversion = 0x0, rpid = 0x0, refer_to = 0x0,
          session_expires = 0x0, min_se = 0x0, ppi = 0x0, pai = 0x0, privacy = 0x0, call_info = 0x0, www_authenticate = 0x0, proxy_authenticate = 0x0, min_expires = 0x0,
          body = 0x0,
          eoh = 0x7fc9d32ac1d8 "\r\nv=0\r\no=- 2316033873 3775904106 IN IP4 10.30.182.202\r\ns= \r\nc=IN IP4 10.30.182.202\r\nt=0 0\r\nm=audio 25728 RTP/SAVP 96 0 97 101 102 103\r\na=rtpmap:96 opus/48000/2\r\na=rtpmap:97 vp8/90000\r\na=rtpmap:101 tel"...,
          unparsed = 0x7fc9d32ac1d8 "\r\nv=0\r\no=- 2316033873 3775904106 IN IP4 10.30.182.202\r\ns= \r\nc=IN IP4 10.30.182.202\r\nt=0 0\r\nm=audio 25728 RTP/SAVP 96 0 97 101 102 103\r\na=rtpmap:96 opus/48000/2\r\na=rtpmap:97 vp8/90000\r\na=rtpmap:101 tel"..., rcv = {src_ip = {af = 2, len = 4, u = {addrl = {1007951882, 18446744073709551615},
---Type <return> to continue, or q <return> to quit---
                 = {1007951882, 0, 4294967295, 4294967295}, addr16 = {8202, 15380, 0, 0, 65535, 65535, 65535, 65535}, addr = "\n \024<\000\000\000\000\377\377\377\377\377\377\377\377"}}, dst_ip = {af = 2, len = 4, u = {addrl = {1761550346, 0},
                addr32 = {1761550346, 0, 0, 0}, addr16 = {8202, 26879, 0, 0, 0, 0, 0, 0}, addr = "\n \377h", '\000' <repeats 11 times>}}, src_port = 5060, dst_port = 6060, proto = 1, proto_reserved1 = 0, proto_reserved2 = 0, src_su = {s = {
                sa_family = 2, sa_data = "\023\304\n \024<\000\000\000\000\000\000\000"}, sin = {sin_family = 2, sin_port = 50195, sin_addr = {s_addr = 1007951882}, sin_zero = "\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 2,
                sin6_port = 50195, sin6_flowinfo = 1007951882, sin6_addr = {__in6_u = {__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 4350785}},
            bind_address = 0x7fca08c333a0},
          buf = 0x7fc9d32abdd0 "INVITE sip:12287281018@testacdload18.byoc.inintca.com:5061 SIP/2.0\r\nMax-Forwards:  9\r\nRecord-Route: <sip:10.32.20.60:5060;du=10.32.255.104:6060;ftag=875073;lr;r2=on>\r\nRecord-Route: <sip:52.206.40.246:"...,
          len = 1554, new_uri = {s = 0x7fca0b9090a0 "sip:12287281018@10.32.71.72:5060;transport=tcp", len = 46}, dst_uri = {s = 0x0, len = 0}, ruri_q = -1, ruri_bflags = 0, force_send_socket = 0x0, path_vec = {s = 0x0, len = 0}, parsed_uri_ok = 0,
          parsed_uri = {user = {
              s = 0x7fc9d32abddb "12287281018@testacdload18.byoc.inintca.com:5061 SIP/2.0\r\nMax-Forwards:  9\r\nRecord-Route: <sip:10.32.20.60:5060;du=10.32.255.104:6060;ftag=875073;lr;r2=on>\r\nRecord-Route: <sip:52.206.40.246:5061;transp"...,
              len = 11}, passwd = {s = 0x0, len = 0}, host = {
              s = 0x7fc9d32abde7 "testacdload18.byoc.inintca.com:5061 SIP/2.0\r\nMax-Forwards:  9\r\nRecord-Route: <sip:10.32.20.60:5060;du=10.32.255.104:6060;ftag=875073;lr;r2=on>\r\nRecord-Route: <sip:52.206.40.246:5061;transport=tls;ftag"...,
              len = 30}, port = {
              s = 0x7fc9d32abe06 "5061 SIP/2.0\r\nMax-Forwards:  9\r\nRecord-Route: <sip:10.32.20.60:5060;du=10.32.255.104:6060;ftag=875073;lr;r2=on>\r\nRecord-Route: <sip:52.206.40.246:5061;transport=tls;ftag=875073;lr;r2=on>\r\nTo:  <sip:12"...,
              len = 4}, params = {s = 0x0, len = 0}, headers = {s = 0x0, len = 0}, port_no = 5061, proto = 0, type = SIP_URI_T, transport = {s = 0x0, len = 0}, ttl = {s = 0x0, len = 0}, user_param = {s = 0x0, len = 0}, maddr = {s = 0x0, len = 0},
            method = {s = 0x0, len = 0}, lr = {s = 0x0, len = 0}, r2 = {s = 0x0, len = 0}, gr = {s = 0x0, len = 0}, transport_val = {s = 0x0, len = 0}, ttl_val = {s = 0x0, len = 0}, user_param_val = {s = 0x0, len = 0}, maddr_val = {s = 0x0, len = 0},
            method_val = {s = 0x0, len = 0}, lr_val = {s = 0x0, len = 0}, r2_val = {s = 0x0, len = 0}, gr_val = {s = 0x0, len = 0}, u_name = {{s = 0x0, len = 0}, {s = 0x0, len = 0}, {s = 0x0, len = 0}, {s = 0x0, len = 0}, {s = 0x0, len = 0}, {s = 0x0,
                len = 0}, {s = 0x0, len = 0}, {s = 0x0, len = 0}, {s = 0x0, len = 0}, {s = 0x0, len = 0}}, u_val = {{s = 0x0, len = 0}, {s = 0x0, len = 0}, {s = 0x0, len = 0}, {s = 0x0, len = 0}, {s = 0x0, len = 0}, {s = 0x0, len = 0}, {s = 0x0,
                len = 0}, {s = 0x0, len = 0}, {s = 0x0, len = 0}, {s = 0x0, len = 0}}, u_params_no = 0}, parsed_orig_ruri_ok = 0, parsed_orig_ruri = {user = {s = 0x0, len = 0}, passwd = {s = 0x0, len = 0}, host = {s = 0x0, len = 0}, port = {s = 0x0,
              len = 0}, params = {s = 0x0, len = 0}, headers = {s = 0x0, len = 0}, port_no = 0, proto = 0, type = ERROR_URI_T, transport = {s = 0x0, len = 0}, ttl = {s = 0x0, len = 0}, user_param = {s = 0x0, len = 0}, maddr = {s = 0x0, len = 0},
            method = {s = 0x0, len = 0}, lr = {s = 0x0, len = 0}, r2 = {s = 0x0, len = 0}, gr = {s = 0x0, len = 0}, transport_val = {s = 0x0, len = 0}, ttl_val = {s = 0x0, len = 0}, user_param_val = {s = 0x0, len = 0}, maddr_val = {s = 0x0, len = 0},
            method_val = {s = 0x0, len = 0}, lr_val = {s = 0x0, len = 0}, r2_val = {s = 0x0, len = 0}, gr_val = {s = 0x0, len = 0}, u_name = {{s = 0x0, len = 0}, {s = 0x0, len = 0}, {s = 0x0, len = 0}, {s = 0x0, len = 0}, {s = 0x0, len = 0}, {s = 0x0,
                len = 0}, {s = 0x0, len = 0}, {s = 0x0, len = 0}, {s = 0x0, len = 0}, {s = 0x0, len = 0}}, u_val = {{s = 0x0, len = 0}, {s = 0x0, len = 0}, {s = 0x0, len = 0}, {s = 0x0, len = 0}, {s = 0x0, len = 0}, {s = 0x0, len = 0}, {s = 0x0,
                len = 0}, {s = 0x0, len = 0}, {s = 0x0, len = 0}, {s = 0x0, len = 0}}, u_params_no = 0}, add_rm = 0x7fc9cfc7b940, body_lumps = 0x0, reply_lump = 0x0, add_to_branch_s = '\000' <repeats 57 times>, add_to_branch_len = 0,
          hash_index = 26212, flags = 16385, msg_flags = 360608, set_global_address = {s = 0x0, len = 0}, set_global_port = {s = 0x0, len = 0}, msg_cb = 0x0}
#2  t_should_relay_response (reply=<optimized out>, cancel_bitmap=<optimized out>, should_relay=<synthetic pointer>, should_store=<synthetic pointer>, branch=<optimized out>, new_code=477, Trans=0x7fc9d10aea40) at t_reply.c:972
        branch_cnt = <optimized out>
        inv_through = <optimized out>
        do_cancel = <optimized out>
#3  relay_reply (t=0x7fc9d10aea40, p_msg=<optimized out>, branch=<optimized out>, msg_status=477, cancel_bitmap=<optimized out>) at t_reply.c:1206
        relay = <optimized out>
        save_clone = <optimized out>
        buf = 0x0
        res_len = 0
        relayed_code = 0
        relayed_msg = 0x0
        bm = {to_tag_val = {s = 0x0, len = 0}}
        totag_retr = 0
        uas_rb = <optimized out>
        cb_s = {s = 0x7fca0b5c8608 "@\256\r", len = 4423818}
        text = {s = 0x0, len = 148029128}
        __FUNCTION__ = "relay_reply"
#4  0x00007fc9c367d04f in reply_received (p_msg=0x7fca0b5c8608) at t_reply.c:1598
        msg_status = 477
        last_uac_status = <optimized out>
        branch = 0
        __FUNCTION__ = "relay_reply"
        reply_status = <optimized out>
        timer = 9666280
        cancel_bitmap = 0
        uac = 0x7fc9d10aec18
        t = 0x7fc9d10aea40
        backup_list = <optimized out>
        has_reply_route = <optimized out>
        __FUNCTION__ = "reply_received"
#5  0x00000000004645a9 in forward_reply (msg=msg@entry=0x7fca0b5c8608) at forward.c:495
        new_buf = 0x0
        to = 0x0
        new_len = 0
        mod = 0x7fca08be7598
        proto = <optimized out>
        id = 0
        send_sock = <optimized out>
        s = <optimized out>
        len = <optimized out>
        __FUNCTION__ = "forward_reply"
#6  0x000000000043ecf6 in receive_msg (
    buf=0x937ec0 <buf> "SIP/2.0 477 Connection Failed/Rejected\r\nTo:  <sip:12287281018@testacdload18.byoc.inintca.com>;tag=8098-e15c244afe84f87df40fcdd8f630c2f4\r\nFrom:  <sip:+906028136945@testacdload18.byoc.inintca.com:5061>;"...,
    len=<optimized out>, rcv_info=rcv_info@entry=0x7ffc2a996120, existing_context=existing_context@entry=0x0, flags=flags@entry=0) at receive.c:258
        ctx = 0x7fca0b40df48
        msg = 0x7fca0b5c8608
        start = {tv_sec = 64, tv_usec = 140505754213112}
        rc = 3
        in_buff = {s = 0x937ec0 <buf> "SIP/2.0 477 Connection Failed/Rejected\r\nTo:  <sip:12287281018@testacdload18.byoc.inintca.com>;tag=8098-e15c244afe84f87df40fcdd8f630c2f4\r\nFrom:  <sip:+906028136945@testacdload18.byoc.inintca.com:5061>;"...,
          len = 363}
        __FUNCTION__ = "receive_msg"
#7  0x0000000000609856 in udp_read_req (si=<optimized out>, bytes_read=<optimized out>) at net/proto_udp/proto_udp.c:182
        ri = {src_ip = {af = 2, len = 4, u = {addrl = {4113047562, 18446744073709551615}, addr32 = {4113047562, 0, 4294967295, 4294967295}, addr16 = {8202, 62760, 0, 0, 65535, 65535, 65535, 65535},
              addr = "\n (\365\000\000\000\000\377\377\377\377\377\377\377\377"}}, dst_ip = {af = 2, len = 4, u = {addrl = {1761550346, 0}, addr32 = {1761550346, 0, 0, 0}, addr16 = {8202, 26879, 0, 0, 0, 0, 0, 0},
              addr = "\n \377h", '\000' <repeats 11 times>}}, src_port = 5060, dst_port = 6060, proto = 1, proto_reserved1 = 0, proto_reserved2 = 0, src_su = {s = {sa_family = 2, sa_data = "\023\304\n (\365\000\000\000\000\000\000\000"}, sin = {
              sin_family = 2, sin_port = 50195, sin_addr = {s_addr = 4113047562}, sin_zero = "\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 2, sin6_port = 50195, sin6_flowinfo = 4113047562, sin6_addr = {__in6_u = {
                  __u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 4350785}}, bind_address = 0x7fca08c333a0}
        len = <optimized out>
        buf = "SIP/2.0 477 Connection Failed/Rejected\r\nTo:  <sip:12287281018@testacdload18.byoc.inintca.com>;tag=8098-e15c244afe84f87df40fcdd8f630c2f4\r\nFrom:  <sip:+906028136945@testacdload18.byoc.inintca.com:5061>;"...
---Type <return> to continue, or q <return> to quit---
        fromlen = 16
        p = <optimized out>
        msg = {s = 0x937ec0 <buf> "SIP/2.0 477 Connection Failed/Rejected\r\nTo:  <sip:12287281018@testacdload18.byoc.inintca.com>;tag=8098-e15c244afe84f87df40fcdd8f630c2f4\r\nFrom:  <sip:+906028136945@testacdload18.byoc.inintca.com:5061>;"...,
          len = 363}
        __FUNCTION__ = "udp_read_req"
#8  0x00000000005ec6c9 in handle_io (idx=0, event_type=2, fm=<optimized out>) at net/net_udp.c:261
        n = 0
        read = 0
#9  io_wait_loop_epoll (h=<optimized out>, t=<optimized out>, repeat=<optimized out>) at net/../io_wait_loop.h:284
        ret = <optimized out>
        i = <optimized out>
        e = <optimized out>
        ep_event = {events = 1, data = {ptr = 0x4b3be600000000, fd = 0, u32 = 0, u64 = 21176482281816064}}
        fd = <optimized out>
        n = 1
        r = 0
#10 udp_start_processes (chd_rank=chd_rank@entry=0x924164 <chd_rank>, startup_done=startup_done@entry=0x7fc9cdecfba0) at net/net_udp.c:389
        si = <optimized out>
        pid = <optimized out>
        i = <optimized out>
        __FUNCTION__ = "udp_start_processes"
#11 0x000000000041c127 in main_loop () at main.c:782
        startup_done = 0x7fc9cdecfba0
        rc = <optimized out>
        chd_rank = 1
#12 main (argc=<optimized out>, argv=<optimized out>) at main.c:1439
        cfg_stream = <optimized out>
        c = <optimized out>
        r = 0
        tmp = 0x7ffc2a997ecf ""
        tmp_len = <optimized out>
        port = <optimized out>
        proto = <optimized out>
        protos_no = <optimized out>
        options = 0x653c68 "f:cCm:M:b:l:n:N:rRvdDFEVhw:t:u:g:P:G:W:o:"
        ret = -1
        seed = 736511634
        rfd = <optimized out>
        __FUNCTION__ = "main"

OS/environment information

razvancrainea commented 5 years ago

I believe I've narrowed down this issue. Will soon provide a fix for it.

adamoverbeeke commented 5 years ago

Thats great to hear. Please let me know if you need us to do anything.

razvancrainea commented 5 years ago

@adamoverbeeke did you manage to run with the commit a48aeca? Do you still experience the crash?

bcnewlin commented 5 years ago

@razvancrainea Adam is out on vacation. We are blocked from upgrading to any newer revisions of OpenSIPS due to #1779 and #1799 (via #1771). This has prevented us from testing a number of fixes that have been delivered recently.

stale[bot] commented 5 years ago

Any updates here? No progress has been made in the last 15 days, marking as stale. Will close this issue if no further updates are made in the next 30 days.

razvancrainea commented 5 years ago

Commit 17430901 seems to have done the trick. Closing this ticket now.