search

OpenSIPS / opensips

OpenSIPS is a GPL implementation of a multi-functionality SIP Server that targets to deliver a high-level technical solution (performance, security and quality) to be used in professional SIP server platforms.
https://opensips.org
Other
1.25k stars 575 forks source link

mid_registrar crash during contact cleanup #1094

Closed liviuchircu closed 7 years ago

liviuchircu commented 7 years ago

modparams

loadmodule "mid_registrar.so" modparam("mid_registrar", "mode", 1) modparam("mid_registrar", "insertion_mode", 0) modparam("mid_registrar", "received_avp", "$avp(rcv)") modparam("mid_registrar", "max_contacts", 4) modparam("mid_registrar", "tcp_persistent_flag", 10) modparam("mid_registrar", "contact_match_param", "regid")

backtrace

https://paste.fedoraproject.org/paste/GGfquSgHeN0ezjXHPqQzk15M1UNdIGYhyRLivL9gydE=

Reported by @volga629.

liviuchircu commented 7 years ago

@volga629 can you also provide a SIP trace and/or some OpenSIPS logs? (ideally with log_level = 4). Also, any nathelper pinging-related modparams could cause issues. If privacy is a concern, ship the info to liviu@opensips.org.

volga629 commented 7 years ago

Hello Liviu, NAT params

loadmodule "nathelper.so"
modparam("nathelper", "received_avp", "$avp(RECEIVED)")

#### Nat Traversal
loadmodule "nat_traversal.so
volga629 commented 7 years ago

Please check you inbox.

liviuchircu commented 7 years ago

Hi, @volga629 , so far I could only manage to find a restart persistence bug which could lead to this. Could it apply to your case, i.e. did you restart the mid-registrar with at least one contact persisted to DB?

volga629 commented 7 years ago

Yes, I think it was after restart when few contacts was present in DB. Right now if restart it just segfaulting.

The question. Is mid_registrar aware of transport=udp/tcp/tls parameter ? What expected behavior when on wan is tls/tcp and on lan udp only. Opensips contain 2 network interfaces wan and lan.

liviuchircu commented 7 years ago

Rephrasing your question: "should it be aware of URI parameters?", the answer is "no", as that should be handed by the transport / tm / dialog code (depending on how your script looks like), and not by either of the registrars. The expected behavior is that is should work, regardless of transport.

The fix involves persisting the mid_registrar attached contact data into the "location" table, so it will take a couple more days. Until then, it would be great if you could test the code with an usrloc db_mode = 0, and see if it's stable.

volga629 commented 7 years ago

I was testing in memory mode only and it more stable, but I think it have issue with constructing contact properly and filter all parameters. When request arrive on public wan side it sent to private side and then to fs that correct. Contact rewritten to lan ip address of opensips, but it not check which transport that incorrect, because on wan side is TCP on LAN udp as result contact looks like this. Expected that transport=tcp should be removed from contact, because opensips transport udp only on LAN side.

Try send to TCP on LAN side

Apr 17 22:56:06 casbc00 /usr/sbin/opensips[6814]: ERROR:core:tcpconn_async_connect: failed to retrieve SO_ERROR [server=10.18.130.27:5060] (111) Connection refused
Apr 17 22:56:06 casbc00 /usr/sbin/opensips[6814]: ERROR:core:proto_tcp_send: async TCP connect failed
Apr 17 22:56:06 casbc00 /usr/sbin/opensips[6814]: ERROR:tm:msg_send: send() for proto 2 failed
Apr 17 22:56:06 casbc00 /usr/sbin/opensips[6814]: ERROR:tm:t_forward_nonack: sending request failed
Apr 17 22:56:06 casbc00 /usr/sbin/opensips[6814]: ERROR:tm:w_t_relay: t_forward_nonack failed
Contact:        "volga629" <sip:4310@10.18.130.27:5060;rinstance=d0fdd5b90b922ce0;**transport=tcp**;regid=c2lwOjQzMTBAMTkyLjE2OC44OC4xNjQ6NDc2NjA7cmluc3RhbmNlPWQwZmRkNWI5MGI5MjJjZTA7dHJhbnNwb3J0PXRjcA--;fs_path=sip%3A10.18.130.27%3Br2%3Don%3Blr%2Csip%3A158.69.151.95%3A5082%3Btransport%3Dtcp%3Br2%3Don%3Blr>
volga629 commented 7 years ago

I am not sure if it related another crash when insert mode set by Path not Contact

Apr 18 15:52:14 casbc00 /usr/sbin/opensips[2531]: Forwarding REGISTER to main registrar (141351_rel51NmNjMDFmYmZhZTUxNTg4OTc1ZGZiNDQ5ZGMyZDI2NTI=141351_rel51NmNjMDFmYmZhZTUxNTg4OTc1ZGZiNDQ5ZGMyZDI2NTI)
Apr 18 15:52:14 casbc00 /usr/sbin/opensips[2531]: Forwarding REGISTER to main registrar (141351_rel51NmNjMDFmYmZhZTUxNTg4OTc1ZGZiNDQ5ZGMyZDI2NTI=141351_rel51NmNjMDFmYmZhZTUxNTg4OTc1ZGZiNDQ5ZGMyZDI2NTI)
Apr 18 15:52:14 casbc00 /usr/sbin/opensips[2527]: ERROR:core:qm_malloc: not enough free pkg memory (3169208 bytes left), please increase the "-M" command line parameter!
Apr 18 15:52:14 casbc00 /usr/sbin/opensips[2527]: ERROR:mid_registrar:decrypt_str: no more pkg memory
Apr 18 15:52:14 casbc00 /usr/sbin/opensips[2530]: INFO:core:probe_max_sock_buff: using snd buffer of 416 kb
Apr 18 15:52:14 casbc00 /usr/sbin/opensips[2530]: INFO:core:init_sock_keepalive: TCP keepalive enabled on socket 6
Apr 18 15:52:14 casbc00 /usr/sbin/opensips[2529]: INFO:core:probe_max_sock_buff: using snd buffer of 416 kb
Apr 18 15:52:14 casbc00 /usr/sbin/opensips[2529]: INFO:core:init_sock_keepalive: TCP keepalive enabled on socket 6
Apr 18 15:52:14 casbc00 /usr/sbin/opensips[2540]: CRITICAL:core:receive_fd: EOF on 20
Apr 18 15:52:14 casbc00 /usr/sbin/opensips[2522]: INFO:core:handle_sigs: child process 2527 exited by a signal 11
Apr 18 15:52:14 casbc00 /usr/sbin/opensips[2522]: INFO:core:handle_sigs: core was generated
Apr 18 15:52:14 casbc00 /usr/sbin/opensips[2522]: INFO:core:handle_sigs: terminating due to SIGCHLD
Apr 18 15:53:40 casbc00 kernel: opensips[2577]: segfault at 0 ip 00007fc3948b71a0 sp 00007fff15d20598 error 4 in libc-2.24.so[7fc39482a000+1bd000]
Apr 18 15:53:40 casbc00 abrt-hook-ccpp: Process 2577 (opensips) of user 992 killed by SIGSEGV - dumping core
volga629 commented 7 years ago

Here are coredump

root@casbc00 ~> [/var/spool/abrt/ccpp-2017-04-18-15:59:27-2836]# gdb coredump
GNU gdb (GDB) Fedora 7.12.1-47.fc25
Copyright (C) 2017 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
[New LWP 2836]
Reading symbols from /usr/sbin/opensips...Reading symbols from /usr/lib/debug/usr/sbin/opensips.debug...done.
done.

warning: Ignoring non-absolute filename: <linux-vdso.so.1>
Missing separate debuginfo for linux-vdso.so.1
Try: dnf --enablerepo='*debug*' install /usr/lib/debug/.build-id/d1/84b80698819d443af05b13eb01925efa18a73d
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/usr/sbin/opensips -P /var/run/opensips/opensips.pid -f /etc/opensips/opensips.'.
Program terminated with signal SIGABRT, Aborted.
#0  0x00007f013817a91f in raise () from /lib64/libc.so.6
Missing separate debuginfos, use: dnf debuginfo-install cyrus-sasl-lib-2.1.26-26.2.fc24.x86_64 glibc-2.24-4.fc25.x86_64 keyutils-libs-1.5.9-8.fc24.x86_64 krb5-libs-1.14.4-7.fc25.x86_64 libcom_err-1.43.3-1.fc25.x86_64 libcrypt-nss-2.24-4.fc25.x86_64 libselinux-2.5-13.fc25.x86_64 nspr-4.13.1-1.fc25.x86_64 nss-3.29.3-1.1.fc25.x86_64 nss-softokn-freebl-3.29.3-1.0.fc25.x86_64 nss-util-3.29.3-1.1.fc25.x86_64 openldap-2.4.44-10.fc25.x86_64 openssl-libs-1.0.2k-1.fc25.x86_64 pcre-8.40-6.fc25.x86_64 postgresql-libs-9.5.6-1.fc25.x86_64 zlib-1.2.8-10.fc24.x86_64
(gdb) bt
#0  0x00007f013817a91f in raise () from /lib64/libc.so.6
#1  0x00007f013817c51a in abort () from /lib64/libc.so.6
#2  0x00005618e7381373 in sig_alarm_abort (signo=<optimized out>) at main.c:414
#3  <signal handler called>
#4  0x00007f01382302c7 in sched_yield () from /lib64/libc.so.6
#5  0x00007f012e9ff0cd in get_lock (lock=0x7f013408640c) at ../../fastlock.h:221
#6  lock_ulslot (_d=_d@entry=0x7f0134094ce8, i=i@entry=463) at udomain.c:1233
#7  0x00007f012e9ff14a in free_udomain (_d=0x7f0134094ce8) at udomain.c:391
#8  0x00007f012ea1af7d in free_all_udomains () at dlist.c:742
#9  0x00007f012ea1c6e2 in destroy () at ul_mod.c:503
#10 0x00005618e7345176 in destroy_modules () at sr_module.c:527
#11 0x00005618e73815d0 in cleanup (show_status=1) at main.c:319
#12 0x00005618e738247d in handle_sigs () at main.c:522
#13 0x00005618e72edaf2 in main_loop () at main.c:722
#14 main (argc=<optimized out>, argv=<optimized out>) at main.c:1283
(gdb) bt full
#0  0x00007f013817a91f in raise () from /lib64/libc.so.6
No symbol table info available.
#1  0x00007f013817c51a in abort () from /lib64/libc.so.6
No symbol table info available.
#2  0x00005618e7381373 in sig_alarm_abort (signo=<optimized out>) at main.c:414
        __FUNCTION__ = "sig_alarm_abort"
#3  <signal handler called>
No symbol table info available.
#4  0x00007f01382302c7 in sched_yield () from /lib64/libc.so.6
No symbol table info available.
#5  0x00007f012e9ff0cd in get_lock (lock=0x7f013408640c) at ../../fastlock.h:221
        i = 0
#6  lock_ulslot (_d=_d@entry=0x7f0134094ce8, i=i@entry=463) at udomain.c:1233
No locals.
#7  0x00007f012e9ff14a in free_udomain (_d=0x7f0134094ce8) at udomain.c:391
        i = 463
        __FUNCTION__ = "free_udomain"
#8  0x00007f012ea1af7d in free_all_udomains () at dlist.c:742
        ptr = 0x7f0134094be8
        __FUNCTION__ = "free_all_udomains"
#9  0x00007f012ea1c6e2 in destroy () at ul_mod.c:503
        __FUNCTION__ = "destroy"
#10 0x00005618e7345176 in destroy_modules () at sr_module.c:527
        t = 0x7f0137d80a98
        foo = 0x7f0137d80748
        __FUNCTION__ = "destroy_modules"
#11 0x00005618e73815d0 in cleanup (show_status=1) at main.c:319
        __FUNCTION__ = "cleanup"
#12 0x00005618e738247d in handle_sigs () at main.c:522
        chld = 0
        chld_status = 139
        overall_status = 139
        i = <optimized out>
        do_exit = <optimized out>
        __FUNCTION__ = "handle_sigs"
#13 0x00005618e72edaf2 in main_loop () at main.c:722
        startup_done = <optimized out>
        chd_rank = 13
        chd_rank = 13
#14 main (argc=<optimized out>, argv=<optimized out>) at main.c:1283
        cfg_stream = <optimized out>
        c = <optimized out>
        r = <optimized out>
        tmp = 0x7fffc2fcdf09 ""
        tmp_len = <optimized out>
        port = <optimized out>
---Type <return> to continue, or q <return> to quit--- 
        proto = <optimized out>
        protos_no = <optimized out>
        options = 0x5618e7471618 "f:cCm:M:b:l:n:N:rRvdDFETSVhw:t:u:g:P:G:W:o:"
        seed = 2358207907
        rfd = <optimized out>
        __FUNCTION__ = "main"
(gdb)
liviuchircu commented 7 years ago

@volga629 that is an abort() backtrace from the main process, chances are the core file got overwritten. To prevent this, you can do echo 1 > /proc/sys/kernel/core_uses_pid before doing any future tests.

The error log has helped me catch a bug with "insert by Path" mode. Please let me know if issues persist on your side (except persistency related ones!), I've backported all recent registrar commits to 2.3 as well.

volga629 commented 7 years ago

@liviuchircu

Insert Path header works, with usrloc mod 0.
Is possible modify path header and add parameters like transport=udp ? 

modparam("usrloc", "db_url", "postgres://uri/opensips")
modparam("usrloc", "nat_bflag", "NAT")
modparam("usrloc", "db_mode",   0)

#### REGISTRAR module
loadmodule "mid_registrar.so"
modparam("mid_registrar", "mode", 1)
modparam("mid_registrar", "insertion_mode", 1)
modparam("mid_registrar", "received_avp", "$avp(rcv)")
modparam("mid_registrar", "max_contacts", 4)
modparam("mid_registrar", "tcp_persistent_flag", 10)
modparam("mid_registrar", "outgoing_expires", 7200)
modparam("mid_registrar", "contact_match_param", "regid")
liviuchircu commented 7 years ago

@volga629 I fully reproduced your scenario in my testing bed, and it worked fine. You just have to make sure to record_route() properly. You will notice OpenSIPS adds two "Record-Route" headers in the eventuality that you switch protocols, so it will know how to handle the sequential requests.

TL;DR: A plain Path: header should work just fine

volga629 commented 7 years ago

@liviuchircu I am testing on freeswitch side with originate and fs_path should inlcude transport=udp. If no transport=udp freeswitch refuse send traffic. Might be I need test inset mode contact and insert path with transport=udp.

Here are some output 

freeswitch@casip00.networklab.prod> sofia_contact 4310@sip.company.tld
sofia/internal-proxy/sip:4310@client_public_ip:49184;rinstance=7ec3bf5a828abf54;transport=tcp;fs_path=sip%3A4310%4010.18.130.27%3Blr
freeswitch@casip00.networklab.prod> originate sofia/internal-proxy/sip:4310@client_public_ip:49184;rinstance=7ec3bf5a828abf54;transport=tcp;fs_path=sip%3A4310%4010.18.130.27%3Blr inline &echo -ERR NORMAL_TEMPORARY_FAILURE
freeswitch@casip00.networklab.prod> originate sofia/internal-proxy/sip:4310@client_public_ip:49184;rinstance=7ec3bf5a828abf54;transport=tcp;fs_path=sip%3A4310%4010.18.130.27%3Btransport=udp%3Blr inline &echo
+OK 4c73f9bf-a9fd-4eec-865f-c2cb725e1dd0
volga629 commented 7 years ago

@liviuchircu I am trying test with insert mode contact and To header looks not right.

2017/04/24 14:08:48.488954 10.18.130.26:5160 -> 10.18.130.27:5060
INVITE sip:4300@10.18.130.27:5060;transport=tcp;regid=c2lwOjQzMDBAOTkuMjQ1LjE3Ni4xOTY6NDIxNzk7dHJhbnNwb3J0PXRjcA-- SIP/2.0
Via: SIP/2.0/UDP 10.18.130.26:5160;rport;branch=z9hG4bKjScFN5ZpKBt4H
Route: <sip:4300@10.18.130.27:5060>;lr;transport=udp
Max-Forwards: 68
From: "Test2" <sip:4310@sip.company.tld>;tag=jpj1jpNg2t3Ze
To: <sip:4300@10.18.130.27:5060;transport=tcp;regid=c2lwOjQzMDBAOTkuMjQ1LjE3Ni4xOTY6NDIxNzk7dHJhbnNwb3J0PXRjcA-->
Call-ID: e7cfe85d-a3bb-1235-f091-5254009e2382
CSeq: 106192600 INVITE
Contact: <sip:mod_sofia@10.18.130.26:5160;transport=tcp>
User-Agent: B2BUA01
Allow: INVITE, ACK, BYE, CANCEL, OPTIONS, MESSAGE, INFO, UPDATE, REGISTER, NOTIFY, PUBLISH, SUBSCRIBE
Supported: timer, path, replaces
Allow-Events: talk, hold, conference, presence, as-feature-event, dialog, line-seize, call-info, sla, include-session-description, presence.winfo, message-summary, refer
Privacy: none
Content-Type: application/sdp
Content-Disposition: session
Content-Length: 200
P-Asserted-Identity: "Test2" <sip:4310@sip.company.tld>

v=0
o=FreeSWITCH 3943987081 3943987082 IN IP4 10.18.130.26
s=FreeSWITCH
c=IN IP4 10.18.130.26
t=0 0
m=audio 25932 RTP/AVP 0 101
a=rtpmap:101 telephone-event/8000
a=fmtp:101 0-15
a=rtcp:11267
volga629 commented 7 years ago

Also I am trying mirroring mode 0 and do I need still use match regid ? Because I think not generate one

Apr 24 14:23:23 casbc00 /usr/sbin/opensips[8529]: ERROR:mid_registrar:get_match_token: a Contact from main registrar (sip:4300@client_public_ip:42199;transport=tcp) is missing the 'regid' hf parameter
Apr 24 14:23:23 casbc00 /usr/sbin/opensips[8529]: ERROR:mid_registrar:match_contact: failed to get match token
Apr 24 14:23:23 casbc00 /usr/sbin/opensips[8529]: ERROR:mid_registrar:insert_rpl_contacts: Contact 'sip:4300@client_public_ip:42199;transport=tcp' not found in reply from main registrar!
volga629 commented 7 years ago

@liviuchircu I did more testing with insert mode 2 AOR and usrloc db mode 2. Here are segfault trace


GNU gdb (GDB) Fedora 7.12.1-47.fc25
Copyright (C) 2017 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
[New LWP 9717]
Reading symbols from /usr/sbin/opensips...Reading symbols from /usr/lib/debug/usr/sbin/opensips.debug...done.
done.

warning: Ignoring non-absolute filename: <linux-vdso.so.1>
Missing separate debuginfo for linux-vdso.so.1
Try: dnf --enablerepo='*debug*' install /usr/lib/debug/.build-id/d1/84b80698819d443af05b13eb01925efa18a73d
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/usr/sbin/opensips -P /var/run/opensips/opensips.pid -f /etc/opensips/opensips.'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  process_contacts_by_aor (flags=0, urec=0x7f917e597a50, req=0x7f918e6e3248) at save.c:1827
1827        e_out = rinfo->expires_out;
Missing separate debuginfos, use: dnf debuginfo-install cyrus-sasl-lib-2.1.26-26.2.fc24.x86_64 glibc-2.24-4.fc25.x86_64 keyutils-libs-1.5.9-8.fc24.x86_64 krb5-libs-1.14.4-7.fc25.x86_64 libcom_err-1.43.3-1.fc25.x86_64 libcrypt-nss-2.24-4.fc25.x86_64 libselinux-2.5-13.fc25.x86_64 nspr-4.13.1-1.fc25.x86_64 nss-3.29.3-1.1.fc25.x86_64 nss-softokn-freebl-3.29.5-1.0.fc25.x86_64 nss-util-3.29.5-1.0.fc25.x86_64 openldap-2.4.44-10.fc25.x86_64 openssl-libs-1.0.2k-1.fc25.x86_64 pcre-8.40-6.fc25.x86_64 postgresql-libs-9.5.6-1.fc25.x86_64 zlib-1.2.8-10.fc24.x86_64
(gdb) bt
#0  process_contacts_by_aor (flags=0, urec=0x7f917e597a50, req=0x7f918e6e3248) at save.c:1827
#1  mid_reg_save (msg=<optimized out>, dom=<optimized out>, flags_gp=<optimized out>, to_uri_gp=<optimized out>, expires_gp=<optimized out>) at save.c:2052
#2  0x00005641ca5269ef in do_action (a=a@entry=0x7f918e254250, msg=msg@entry=0x7f918e6e3248) at action.c:1862
#3  0x00005641ca52c8b8 in run_action_list (a=<optimized out>, msg=msg@entry=0x7f918e6e3248) at action.c:172
#4  0x00005641ca52a32c in do_action (a=a@entry=0x7f918e254ab0, msg=msg@entry=0x7f918e6e3248) at action.c:1124
#5  0x00005641ca52c8b8 in run_action_list (a=<optimized out>, msg=msg@entry=0x7f918e6e3248) at action.c:172
#6  0x00005641ca52a32c in do_action (a=a@entry=0x7f918e254be0, msg=msg@entry=0x7f918e6e3248) at action.c:1124
#7  0x00005641ca52c8b8 in run_action_list (a=<optimized out>, msg=msg@entry=0x7f918e6e3248) at action.c:172
#8  0x00005641ca52a32c in do_action (a=a@entry=0x7f918e254d10, msg=msg@entry=0x7f918e6e3248) at action.c:1124
#9  0x00005641ca52c8b8 in run_action_list (a=a@entry=0x7f918e2504d0, msg=msg@entry=0x7f918e6e3248) at action.c:172
#10 0x00005641ca52cb98 in run_actions (msg=0x7f918e6e3248, a=0x7f918e2504d0) at action.c:137
#11 run_top_route (a=0x7f918e2504d0, msg=msg@entry=0x7f918e6e3248) at action.c:214
#12 0x00005641ca51bca4 in receive_msg (
    buf=0x5641ca941560 <tcp_current_req> "REGISTER sip:sip.company.tld SIP/2.0\r\nVia: SIP/2.0/TCP 192.168.88.164:47869;branch=z9hG4bK-524287-1---31745c71db99372c;rport\r\nMax-Forwards: 69\r\nContact: <sip:4310@99.245.176.196:44608;rinstance=987655"..., len=<optimized out>, len@entry=602, rcv_info=rcv_info@entry=0x7ffcaa7ff620, existing_context=existing_context@entry=0x0)
    at receive.c:208
#13 0x00005641ca6635ea in tcp_handle_req (_max_msg_chunks=<optimized out>, con=0x7f917e598290, req=0x5641ca941560 <tcp_current_req>) at net/proto_tcp/tcp_common.h:412
#14 tcp_read_req (con=0x7f917e598290, bytes_read=0x7ffcaa7ff6e0) at net/proto_tcp/proto_tcp.c:1150
#15 0x00005641ca644347 in handle_io (fm=<optimized out>, idx=idx@entry=0, event_type=event_type@entry=1) at net/net_tcp_proc.c:227
#16 0x00005641ca648630 in io_wait_loop_epoll (h=<optimized out>, t=<optimized out>, repeat=<optimized out>) at ./io_wait_loop.h:225
#17 tcp_worker_proc_loop () at net/net_tcp_proc.c:355
#18 0x00005641ca654368 in tcp_start_processes (chd_rank=0x5641ca91d9fc <chd_rank>, startup_done=0x0) at net/net_tcp.c:1836
#19 0x00005641ca5053bd in main_loop () at main.c:683
#20 main (argc=<optimized out>, argv=<optimized out>) at main.c:1283
(gdb) bt full
#0  process_contacts_by_aor (flags=0, urec=0x7f917e597a50, req=0x7f918e6e3248) at save.c:1827
        ci = <optimized out>
        e_out = <optimized out>
        e = 32657
        ret = <optimized out>
        rinfo = 0x0
        cflags = <optimized out>
        max_diff = -1
        cinfo = <optimized out>
        c = 0x0
        ct = <optimized out>
#1  mid_reg_save (msg=<optimized out>, dom=<optimized out>, flags_gp=<optimized out>, to_uri_gp=<optimized out>, expires_gp=<optimized out>) at save.c:2052
        ud = <optimized out>
        rec = 0x7f917e597a50
        flags_str = {s = 0x0, len = 0}
        to_uri = {
          s = 0x5641ca941659 <tcp_current_req+249> "sip:4310@sip.company.tld>\r\nFrom: \"volga629\"<sip:4310@sip.company.tld>;tag=2b129640\r\nCall-ID: 141351_rel51YjM3ODZmYTRmZGE5ZmViNDc5NDUzOGJjNzA3MzdmODg\r\nCSeq: 33 REGISTER\r\nExpires: 900\r\nAllow: INV"..., len = 24}
        sctx = {flags = 0, aor = {s = 0x7f9178a7f9c0 <aor_buf> "4310", len = 4}, max_contacts = 0, expires = 0, expires_out = 7200, star = 0, min_expires = 0, max_expires = 0}
        st = 0
        __FUNCTION__ = "mid_reg_save"
#2  0x00005641ca5269ef in do_action (a=a@entry=0x7f918e254250, msg=msg@entry=0x7f918e6e3248) at action.c:1862
        increment = <optimized out>
        decrement = <optimized out>
        val_number = <optimized out>
        j = <optimized out>
        val_s = {s = 0x1 <error: Cannot access memory at address 0x1>, len = 72}
        cdb_reply = 0x7ffcaa7fdead
        aux = {s = 0x1 <error: Cannot access memory at address 0x1>, len = 72}
        i = <optimized out>
        key_number = 32764
        it = <optimized out>
        avp_val = <optimized out>
        avp_name = {n = 313, s = {s = 0x139 <error: Cannot access memory at address 0x139>, len = -899903010}}
        avp_type = 43647
        ret = -5
        v = <optimized out>
        sec = <optimized out>
        usec = <optimized out>
        to = <optimized out>
        p = <optimized out>
        tmp = <optimized out>
        new_uri = <optimized out>
        end = <optimized out>
        crt = <optimized out>
        len = <optimized out>
---Type <return> to continue, or q <return> to quit---
        i = <optimized out>
        user = 0
        expires = 0
        vals = {{s = 0x5641ca954be0 <_ip_addr_A_buff> "99.245.176.196", len = -900041075}, {
            s = 0x5641ca9415de <tcp_current_req+126> "Max-Forwards: 69\r\nContact: <sip:4310@99.245.176.196:44608;rinstance=987655dc2a1aa8d2;transport=tcp>\r\nTo: \"volga629\"<sip:4310@sip.company.tld>\r\nFrom: \"volga629\"<sip:4310@sip.company.tld>;"..., len = 883760384}, {s = 0x7ffcaa7fda00 "", len = 883760384}, {s = 0x7ffcaa7fda10 "", len = 2119658616}, {
            s = 0x0, len = -1434461696}}
        result = {s = 0x7ffcaa7fdead "\335%$\340?%\216\221\177", len = 23}
        uri = {user = {s = 0x5641ca954be0 <_ip_addr_A_buff> "99.245.176.196", len = -900483410}, passwd = {s = 0x7ffcaa7feab6 "db\230Fn\216\221\177", len = 883760384}, host = {
            s = 0x5641ca951930 <int2str_buf+16> "46207", len = 5}, port = {s = 0x1c0000b47f <error: Cannot access memory at address 0x1c0000b47f>, len = 883760384}, params = {
            s = 0x7f918e6e3248 "\001", len = -1910024712}, headers = {s = 0x7f917e576c78 "8\240\356w\221\177", len = 2012118246}, port_no = 60064, proto = 43647, type = 4294967295, 
          transport = {s = 0x7f918e6e3248 "\001", len = 0}, ttl = {s = 0x4 <error: Cannot access memory at address 0x4>, len = 2119663480}, user_param = {
            s = 0x3f8 <error: Cannot access memory at address 0x3f8>, len = 2119658616}, maddr = {s = 0x7f917e576fb8 "\001", len = -1434461696}, method = {
            s = 0x7f917e578768 "\300\300\300\300\300\300\300\300\355\357ͫ\355\357ͫ", len = 16}, lr = {s = 0x7f9179721173 <w_process_maxfwd_header+243> "\205\300A\270\001", len = 0}, 
          r2 = {s = 0x0, len = 0}, gr = {s = 0x2425dd0c34ad1d00 <error: Cannot access memory at address 0x2425dd0c34ad1d00>, len = -896264724}, transport_val = {s = 0x7f918e257658 "\020", 
            len = -1905380792}, ttl_val = {s = 0x7f918e24a460 "\002", len = -1910024712}, user_param_val = {
            s = 0x2425dd0c34ad1d00 <error: Cannot access memory at address 0x2425dd0c34ad1d00>, len = -1910094088}, maddr_val = {s = 0x10 <error: Cannot access memory at address 0x10>, 
            len = -896185312}, method_val = {s = 0x5641ca5269ef <do_action+11839> "\017\266\023A\211\307XY\351\344\322\377\377M\213\275X\002", len = 0}, lr_val = {
            s = 0x7f917e576c78 "8\240\356w\221\177", len = 0}, r2_val = {s = 0x2425dd0c34ad1d00 <error: Cannot access memory at address 0x2425dd0c34ad1d00>, len = -1910147496}, gr_val = {
            s = 0x2425dd0c34ad1d00 <error: Cannot access memory at address 0x2425dd0c34ad1d00>, len = -1910104672}, u_name = {{s = 0x7f918e253e10 "\020", len = -1905380792}, {
              s = 0x5641ca954c20 <_oser_err_info> "", len = 0}, {s = 0x7f918e6e3248 "\001", len = -1905380792}, {s = 0x5641ca52c8b8 <run_action_list+72> "\205\300u\a\203\r=}=", 
              len = -1866375176}, {s = 0x5641cb7b0c60 ".", len = 0}, {s = 0xd <error: Cannot access memory at address 0xd>, len = -1910161600}, {
              s = 0x5641ca567165 <eval_elem+1941> "H\205\333t\n\307C\024\030", len = -1434457728}, {s = 0x7ffcaa7fea98 "", len = -1434457056}, {
              s = 0x7ffcaa7fdc50 "\220\334\177\252\374\177", len = -1434457832}, {s = 0x7ffcaa7fe920 "@\350\177\252\374\177", len = -1434457816}}, u_val = {{
              s = 0x100000016 <error: Cannot access memory at address 0x100000016>, len = 1986159621}, {s = 0x616c6b726f777465 <error: Cannot access memory at address 0x616c6b726f777465>, 
              len = 1919943778}, {s = 0x7f919021e068 "", len = -1434461040}, {s = 0x7f9190210d90 "", len = -1434461260}, {s = 0x7ffcaa7fdc80 "\377\377\377\377", len = -1866373784}, {
              s = 0x564100000000 <error: Cannot access memory at address 0x564100000000>, len = 5}, {s = 0x5641cb7a39e8 "M\031Aw\221\177", len = 1}, {s = 0x0, len = -1434460904}, {
              s = 0x5641cb7a2e08 "\340\223\301\220\221\177", len = -1434460896}, {s = 0x5641cb7a3a00 "p\031Aw\221\177", len = 1}}, u_params_no = 0}
        next_hop = {user = {s = 0x7ffcaa7fdd48 "0\337!\220\221\177", len = -1868575759}, passwd = {s = 0x3 <error: Cannot access memory at address 0x3>, len = -881182208}, host = {
            s = 0x1 <error: Cannot access memory at address 0x1>, len = 0}, port = {s = 0x1 <error: Cannot access memory at address 0x1>, len = -881186128}, params = {
            s = 0x5641cb7a2ab0 "", len = -1434461104}, headers = {s = 0x0, len = -881185272}, port_no = 56464, proto = 43647, type = 32764, transport = {
            s = 0x190a0049e <error: Cannot access memory at address 0x190a0049e>, len = -881186128}, ttl = {s = 0x7ffcaa7fdc80 "\377\377\377\377", len = 2000754635}, user_param = {
            s = 0x7c96f087 <error: Cannot access memory at address 0x7c96f087>, len = -1}, maddr = {
            s = 0x7f9190a0049e <_dl_map_object_deps+654> "H\213\215\200\373\377\377H\205\311\017\205~\004", len = -1876828368}, method = {s = 0x7f9190c15ff8 "", len = -1434461088}, lr = {
            s = 0x7ffcaa7fde00 " \347\177\252\374\177", len = 2002873744}, r2 = {s = 0x7f9190a0049e <_dl_map_object_deps+654> "H\213\215\200\373\377\377H\205\311\017\205~\004", len = 1}, 
          gr = {s = 0x7f91776170a0 "\020\306]\220\221\177", len = -881191008}, transport_val = {s = 0xffffffffffffff90 <error: Cannot access memory at address 0xffffffffffffff90>, 
            len = -1434457984}, ttl_val = {s = 0x7ffcaa7fe858 "\274\062)\220\221\177", len = -1434458336}, user_param_val = {s = 0x7f9177617020 "p2)\220\221\177", len = 1}, maddr_val = {
            s = 0x5641cb7b0c60 ".", len = -1434457984}, method_val = {s = 0x7ffcaa7fe858 "\274\062)\220\221\177", len = -1434458336}, lr_val = {
            s = 0x7f9190a01e24 <_dl_fixup+212> "I\211\300d\213\004%\030", len = 1}, r2_val = {s = 0x0, len = 774778414}, gr_val = {s = 0x7f919021df30 "\207\025", len = -1434460576}, 
          u_name = {{s = 0x7f9190a0a547 <_dl_runtime_resolve_sse_vex+167> "I\211\303f\017\032\234$\260", len = 0}, {
              s = 0xffffffffffffffff <error: Cannot access memory at address 0xffffffffffffffff>, len = -1}, {s = 0x0, len = 1869770798}, {
              s = 0x100000000 <error: Cannot access memory at address 0x100000000>, len = 0}, {s = 0xffff000000000000 <error: Cannot access memory at address 0xffff000000000000>, 
              len = 1077952576}, {s = 0x4040404040404040 <error: Cannot access memory at address 0x4040404040404040>, len = 1515870810}, {
              s = 0x5a5a5a5a5a5a5a5a <error: Cannot access memory at address 0x5a5a5a5a5a5a5a5a>, len = 538976288}, {
              s = 0x7f919030a71f <sbrk+95> "\205\300x\341I\213\034$\353\270\017\037\200", len = 199552}, {s = 0x7f91905cdb38 <main_arena+88> "0\250{\313AV", len = 65536}, {
              s = 0x7f91902961a9 <__default_morecore+9> "\272", len = -1434458432}}, u_val = {{s = 0x7f919028d36e <systrim.isra+142> "\220H\205\300t\247I)\306t\242H\213E", 
---Type <return> to continue, or q <return> to quit---
              len = -1434458336}, {s = 0x5641cb7ba480 "", len = -1872962848}, {s = 0x30b80 <error: Cannot access memory at address 0x30b80>, len = -881023856}, {s = 0x0, len = 134000}, {
              s = 0x2425dd0c34ad1d00 <error: Cannot access memory at address 0x2425dd0c34ad1d00>, len = -881128352}, {s = 0x2 <error: Cannot access memory at address 0x2>, 
              len = -1910161600}, {s = 0x7f918e6e3248 "\001", len = 0}, {s = 0x7f918e6e3248 "\001", len = 15}, {s = 0x5641ca56688d <eval_expr+525> "\213\035\215\345\071", 
              len = -2138708339}, {s = 0x7662640502000200 <error: Cannot access memory at address 0x7662640502000200>, len = 1846161712}}, u_params_no = 7424}
        u = <optimized out>
        port = <optimized out>
        cmatch = <optimized out>
        aitem = <optimized out>
        adefault = <optimized out>
        spec = <optimized out>
        model = <optimized out>
        val = {rs = {s = 0x139 <error: Cannot access memory at address 0x139>, len = -899903010}, ri = 86400, flags = 0}
        pve = <optimized out>
        name_s = {s = 0x5641cb7b0cbe "", len = 883760384}
        start = {tv_sec = 94840586701932, tv_usec = 140263136575240}
        end_time = <optimized out>
        aux_counter = 32764
        __FUNCTION__ = "do_action"
#3  0x00005641ca52c8b8 in run_action_list (a=<optimized out>, msg=msg@entry=0x7f918e6e3248) at action.c:172
        ret = -1
        t = 0x7f918e254250
#4  0x00005641ca52a32c in do_action (a=a@entry=0x7f918e254ab0, msg=msg@entry=0x7f918e6e3248) at action.c:1124
        increment = <optimized out>
        decrement = <optimized out>
        val_number = <optimized out>
        j = <optimized out>
        val_s = {s = 0x10043c00c8212 <error: Cannot access memory at address 0x10043c00c8212>, len = 67108865}
        cdb_reply = 0x70696163070a0080
        aux = {s = 0x10043c00c8212 <error: Cannot access memory at address 0x10043c00c8212>, len = 67108865}
        i = <optimized out>
        key_number = 1359020033
        it = <optimized out>
        avp_val = <optimized out>
        avp_name = {n = 4, s = {s = 0x4 <error: Cannot access memory at address 0x4>, len = -881186128}}
        avp_type = 2
        ret = 1
        v = 1
        sec = <optimized out>
        usec = <optimized out>
        to = <optimized out>
        p = <optimized out>
        tmp = <optimized out>
        new_uri = <optimized out>
        end = <optimized out>
        crt = <optimized out>
        len = <optimized out>
---Type <return> to continue, or q <return> to quit---
        i = <optimized out>
        user = 0
        expires = 0
        vals = {{s = 0x5641cb7a2ab0 "", len = -1434460112}, {s = 0x7f91774116ee "__cxa_finalize", len = 1842243024}, {s = 0xffffffff <error: Cannot access memory at address 0xffffffff>, 
            len = -1434459968}, {s = 0x7ffcaa7fe960 "\340\356\177\252\374\177", len = 1}, {s = 0x5641cb7b0c60 ".", len = -1434457408}}
        result = {s = 0x70696163070a0080 <error: Cannot access memory at address 0x70696163070a0080>, len = -1070518175}
        uri = {user = {s = 0x7ffcaa7fea98 "", len = -1876348228}, passwd = {s = 0x7ffcaa7fe960 "\340\356\177\252\374\177", len = 1}, host = {s = 0x5641cb7b0c60 ".", len = -1434457408}, 
          port = {s = 0x7ffcaa7fea98 "", len = -1434459968}, params = {s = 0x7ffcaa7fe960 "\340\356\177\252\374\177", len = 1}, headers = {s = 0x5641cb7b0c60 ".", len = 2000764223}, 
          port_no = 18269, proto = 32901, type = 16777472, transport = {s = 0x7662640502000200 <error: Cannot access memory at address 0x7662640502000200>, len = 1846161712}, ttl = {
            s = 0x72700462616c6b72 <error: Cannot access memory at address 0x72700462616c6b72>, len = 25711}, user_param = {
            s = 0x510100010001000c <error: Cannot access memory at address 0x510100010001000c>, len = 168034432}, maddr = {
            s = 0x5101000100020012 <error: Cannot access memory at address 0x5101000100020012>, len = 118096000}, method = {
            s = 0x12c012c0313061 <error: Cannot access memory at address 0x12c012c0313061>, len = 65538}, lr = {
            s = 0x306170696163070a <error: Cannot access memory at address 0x306170696163070a>, len = -1072512976}, r2 = {
            s = 0x2425dd0c34ad1d00 <error: Cannot access memory at address 0x2425dd0c34ad1d00>, len = -1072922094}, gr = {s = 0x7f918e2646f8 "\020", len = -1905380792}, transport_val = {
            s = 0x5641ca954c20 <_oser_err_info> "", len = 0}, ttl_val = {s = 0x7f918e6e3248 "\001", len = -1905380792}, user_param_val = {
            s = 0x2425dd0c34ad1d00 <error: Cannot access memory at address 0x2425dd0c34ad1d00>, len = 2000753288}, maddr_val = {s = 0x7f918e252a20 "\024", len = -1905380792}, method_val = {
            s = 0x5641ca954c20 <_oser_err_info> "", len = 0}, lr_val = {s = 0x7f918e6e3248 "\001", len = -1905380792}, r2_val = {
            s = 0x5641ca52c8b8 <run_action_list+72> "\205\300u\a\203\r=}=", len = -1905380792}, gr_val = {s = 0x5641ca52c8b8 <run_action_list+72> "\205\300u\a\203\r=}=", len = 0}, 
          u_name = {{s = 0xd <error: Cannot access memory at address 0xd>, len = -1910166704}, {s = 0x5641ca567165 <eval_elem+1941> "H\205\333t\n\307C\024\030", len = -1910104368}, {
              s = 0x5641ca567165 <eval_elem+1941> "H\205\333t\n\307C\024\030", len = -1434459456}, {s = 0x2425dd0c34ad1d00 <error: Cannot access memory at address 0x2425dd0c34ad1d00>, 
              len = -1910175256}, {s = 0x10 <error: Cannot access memory at address 0x10>, len = -896185312}, {
              s = 0x5641ca5269ef <do_action+11839> "\017\266\023A\211\307XY\351\344\322\377\377M\213\275X\002", len = 0}, {s = 0x7f9190a078c7 <dl_open_worker+391> "\220\366\005\262\003!", 
              len = -1872962840}, {s = 0x7ffcaa7fe230 "\350\332\\\220\221\177", len = 1}, {s = 0x7f9190c19100 <_r_debug> "\001", len = -1434459056}, {s = 0x5641cb7a2ab0 "", len = 0}}, 
          u_val = {{s = 0x7f9190210d90 "", len = -2147483647}, {s = 0x1 <error: Cannot access memory at address 0x1>, len = 4}, {
              s = 0x7f9100000000 <error: Cannot access memory at address 0x7f9100000000>, len = -1866375176}, {s = 0x0, len = -2147483647}, {
              s = 0x7f91909fbdd0 <check_match+112> "\205\300\017\205\257", len = -1875675104}, {s = 0x2b <error: Cannot access memory at address 0x2b>, len = -881186128}, {
              s = 0x7f9177411288 "", len = 2000754344}, {s = 0x7f91909fbea4 <check_match+324> "\205\300\017\205m\377\377\377\351z\377\377\377\017\037\200", len = 43}, {
              s = 0x6bc <error: Cannot access memory at address 0x6bc>, len = -1866375176}, {s = 0x7f9190210d90 "", len = -1876828056}}, u_params_no = 50923}
        next_hop = {user = {s = 0x6bc <error: Cannot access memory at address 0x6bc>, len = -1876828056}, passwd = {s = 0x7f9190c15ff8 "", len = -1434459192}, host = {
            s = 0x7ffcaa7fe3c4 "", len = -1868578703}, port = {s = 0x7ffc0000001a <error: Cannot access memory at address 0x7ffc0000001a>, len = 2000754609}, params = {
            s = 0x7f91774113c0 "\t\001", len = -1434459192}, headers = {s = 0x7f91905cdae8 <main_arena+8> "", len = -1434459217}, port_no = 56040, proto = 36956, type = 32657, transport = {
            s = 0x7ffcaa7fe3bf "", len = -1872962768}, ttl = {s = 0x7ffcaa7fe3c0 "\002", len = 1434459201}, user_param = {s = 0x7ffcaa7fe490 "", len = -1866373784}, maddr = {
            s = 0x564100000000 <error: Cannot access memory at address 0x564100000000>, len = 5}, method = {s = 0x0, len = 2}, lr = {s = 0x0, len = -1434458440}, r2 = {
            s = 0x7f91905cdae0 <main_arena> "", len = -1872962848}, gr = {s = 0x7f91905cdae0 <main_arena> "", len = -1872962848}, transport_val = {
            s = 0x1010 <error: Cannot access memory at address 0x1010>, len = 0}, ttl_val = {s = 0x3df <error: Cannot access memory at address 0x3df>, len = -881089392}, user_param_val = {
            s = 0x7f9190290eaf <_int_malloc+351> "L\211\340L\211\347L\215mXH\301\350\006H\301\357\tL\211\341H\211D$@\203\300\060H\211|$P\211D$L\203\307[D\211\340\301\350\004\211|$`H\215\274$\200", len = 99}, maddr_val = {s = 0x1000 <error: Cannot access memory at address 0x1000>, len = -1434458976}, method_val = {s = 0x7ffcaa7fe49f "", len = -881186128}, lr_val = {
            s = 0xffff800355801b71 <error: Cannot access memory at address 0xffff800355801b71>, len = 64}, r2_val = {
            s = 0xffff800355801b61 <error: Cannot access memory at address 0xffff800355801b61>, len = 64}, gr_val = {s = 0x7000000101 <error: Cannot access memory at address 0x7000000101>, 
            len = 8}, u_name = {{s = 0x1 <error: Cannot access memory at address 0x1>, len = 99}, {s = 0x0, len = 0}, {
              s = 0x7c00000077 <error: Cannot access memory at address 0x7c00000077>, len = -1876840656}, {s = 0x7f9190c15ff8 "", len = -881191856}, {s = 0x7f91905cdae0 <main_arena> "", 
              len = 4096}, {s = 0x2425dd0c34ad1d00 <error: Cannot access memory at address 0x2425dd0c34ad1d00>, len = 4096}, {s = 0x3 <error: Cannot access memory at address 0x3>, 
              len = -1910093784}, {s = 0x7f918e6e3248 "\001", len = 0}, {s = 0x7f918e6e3248 "\001", len = 15}, {
              s = 0x2425dd0c34ad1d00 <error: Cannot access memory at address 0x2425dd0c34ad1d00>, len = 4096}}, u_val = {{s = 0x3 <error: Cannot access memory at address 0x3>, 
              len = -1910166704}, {s = 0x7f918e6e3248 "\001", len = 0}, {s = 0x7f918e6e3248 "\001", len = 15}, {s = 0x5641ca56688d <eval_expr+525> "\213\035\215\345\071", len = 0}, {
---Type <return> to continue, or q <return> to quit---
              s = 0x5641ca5668df <eval_expr+607> "\203\370\001u\225H\213}(L\211\352L\211\346\350\215\375\377\377\213\035'\345\071", len = 8}, {
              s = 0x2425dd0c34ad1d00 <error: Cannot access memory at address 0x2425dd0c34ad1d00>, len = 566139219}, {s = 0x2 <error: Cannot access memory at address 0x2>, 
              len = -1910166544}, {s = 0x7f918e6e3248 "\001", len = 0}, {s = 0x5641ca5668a9 <eval_expr+553> "\205\300x\314\017\224\300\213\035j\345\071", len = 0}, {
              s = 0x5641ca566879 <eval_expr+505> "\213\035\241\345\071", len = -1905380792}}, u_params_no = 7424}
        u = <optimized out>
        port = <optimized out>
        cmatch = <optimized out>
        aitem = <optimized out>
        adefault = <optimized out>
        spec = <optimized out>
        model = <optimized out>
        val = {rs = {s = 0x4 <error: Cannot access memory at address 0x4>, len = -881186128}, ri = 18428, flags = 0}
        pve = <optimized out>
        name_s = {s = 0x80510100010002 <error: Cannot access memory at address 0x80510100010002>, len = 1633879818}
        start = {tv_sec = 281860451254320, tv_usec = 721702596267474945}
        end_time = <optimized out>
        aux_counter = 1359020033
        __FUNCTION__ = "do_action"
#5  0x00005641ca52c8b8 in run_action_list (a=<optimized out>, msg=msg@entry=0x7f918e6e3248) at action.c:172
        ret = -1
        t = 0x7f918e254ab0
#6  0x00005641ca52a32c in do_action (a=a@entry=0x7f918e254be0, msg=msg@entry=0x7f918e6e3248) at action.c:1124
        increment = <optimized out>
        decrement = <optimized out>
        val_number = <optimized out>
        j = <optimized out>
        val_s = {s = 0x7ffcaa7feb10 "\326\352\177\252\374\177", len = -1876348228}
        cdb_reply = 0x3e0
        aux = {s = 0x7ffcaa7feb10 "\326\352\177\252\374\177", len = -1876348228}
        i = <optimized out>
        key_number = 32657
        it = <optimized out>
        avp_val = <optimized out>
        avp_name = {n = 0, s = {s = 0x0, len = 0}}
        avp_type = 36904
        ret = 1
        v = 1
        sec = <optimized out>
        usec = <optimized out>
        to = <optimized out>
        p = <optimized out>
        tmp = <optimized out>
        new_uri = <optimized out>
        end = <optimized out>
        crt = <optimized out>
        len = <optimized out>
---Type <return> to continue, or q <return> to quit---
        i = <optimized out>
        user = 0
        expires = 0
        vals = {{s = 0x7ffcaa7feaa0 "dbv01.ne", len = 1434458241}, {s = 0x1 <error: Cannot access memory at address 0x1>, len = 5}, {s = 0x0, len = 0}, {
            s = 0x6e0000005b <error: Cannot access memory at address 0x6e0000005b>, len = 0}, {s = 0x0, len = 119}}
        result = {s = 0x3e0 <error: Cannot access memory at address 0x3e0>, len = -881089392}
        uri = {user = {s = 0x7ffcaa7fdf40 <incomplete sequence \361\205\200>, len = -881191904}, passwd = {s = 0x5641cb7ba490 "\210\334\\\220\221\177", len = -1434458112}, host = {
            s = 0x7ffcaa7feab6 "db\230Fn\216\221\177", len = -1872962848}, port = {s = 0x40 <error: Cannot access memory at address 0x40>, len = -1434458112}, params = {
            s = 0x7ffcaa7feef0 "", len = -1434458048}, headers = {s = 0x0, len = -1876349164}, port_no = 59792, proto = 43647, type = 32764, transport = {
            s = 0x7ffcaa7fea28 "\220\244{\313AV", len = -1434457712}, ttl = {s = 0x7f91902f9688 <gaih_inet.constprop+1288> "H\205\300H\211\003u\200M\211\376L\211\367\350%3\363\377\270\n", 
            len = 0}, user_param = {s = 0x7ffcaa7feaa0 "dbv01.ne", len = 2}, maddr = {s = 0x3000000028 <error: Cannot access memory at address 0x3000000028>, len = -1434456368}, method = {
            s = 0x3000000028 <error: Cannot access memory at address 0x3000000028>, len = -1434456352}, lr = {s = 0x7ffcaa7fee20 "\340\356\177\252\374\177", len = 0}, r2 = {
            s = 0x600000001 <error: Cannot access memory at address 0x600000001>, len = 29460}, gr = {s = 0x7f91902932bc <free+76> "H\203\304([]A\\A]\303f\017\037\204", len = -1434457840}, 
          transport_val = {s = 0x0, len = -1434457844}, ttl_val = {s = 0xaa7fe928 <error: Cannot access memory at address 0xaa7fe928>, len = -1434457828}, user_param_val = {
            s = 0x2425dd0c34ad1d00 <error: Cannot access memory at address 0x2425dd0c34ad1d00>, len = -1434457832}, maddr_val = {s = 0x7f918e251c40 "\020", len = -1905380792}, 
          method_val = {s = 0x5641ca954c20 <_oser_err_info> "", len = 0}, lr_val = {s = 0x7f918e6e3248 "\001", len = -1905380792}, r2_val = {
            s = 0x5641ca52c8b8 <run_action_list+72> "\205\300u\a\203\r=}=", len = 1024}, gr_val = {s = 0x2425dd0c34ad1d00 <error: Cannot access memory at address 0x2425dd0c34ad1d00>, 
            len = 0}, u_name = {{s = 0xd <error: Cannot access memory at address 0xd>, len = -1910170256}, {s = 0x5641ca567165 <eval_elem+1941> "H\205\333t\n\307C\024\030", len = 0}, {
              s = 0x7f918e6e3248 "\001", len = -1905380792}, {s = 0x5641ca52c8b8 <run_action_list+72> "\205\300u\a\203\r=}=", len = -1434458048}, {s = 0x7ffcaa7fe800 "", len = 0}, {
              s = 0xd <error: Cannot access memory at address 0xd>, len = -1910174952}, {s = 0x5641ca567165 <eval_elem+1941> "H\205\333t\n\307C\024\030", len = -1434456288}, {
              s = 0x57a1c00 <error: Cannot access memory at address 0x57a1c00>, len = -1434456352}, {s = 0x5641cb7aff00 "", len = -1434456336}, {s = 0x7ffcaa7feaa0 "dbv01.ne", 
              len = -881137712}}, u_val = {{s = 0x0, len = -1434456368}, {s = 0x7f91902fc78d <getaddrinfo+1597> "H\213\205X\373\377\377H\205\300\017\204\302\003", len = -881130728}, {
              s = 0x5641cb7ba490 "\210\334\\\220\221\177", len = -881130728}, {s = 0x5641cb7a7e10 "\240\004{\313AV", len = 0}, {
              s = 0x7f917dddb7dc <parseServiceInfo+572> "H\205\300H\211\303\017\205\377\375\377\377\061\300\351\321\376\377\377\220H\215=\361h\001", len = -1434456344}, {s = 0x0, 
              len = -898845988}, {s = 0x7f9100000001 <error: Cannot access memory at address 0x7f9100000001>, len = -1434456208}, {s = 0x7ffcaa7fea90 "\240\352\177\252\374\177", 
              len = -1434457432}, {s = 0x7ffcaa7fefa0 "@\361\177\252\374\177", len = -1434456176}, {s = 0x7f9100000000 <error: Cannot access memory at address 0x7f9100000000>, 
              len = 61720}}, u_params_no = 42128}
        next_hop = {user = {s = 0x0, len = 0}, passwd = {s = 0x7ffcaa7fef20 "\377\377\377\177\221\177", len = 5235}, host = {s = 0x7ffcaa7feaf8 "\260Hl\312AV", len = 10}, port = {
            s = 0x7ffcaa7feb20 "\320Fn\216\221\177", len = -1434455964}, params = {s = 0x0, len = 0}, headers = {s = 0x7ffc00000000 <error: Cannot access memory at address 0x7ffc00000000>, 
            len = 0}, port_no = 60064, proto = 43647, type = 32764, transport = {s = 0x400 <error: Cannot access memory at address 0x400>, len = 813064804}, ttl = {
            s = 0x2425dd0c34ad1d00 <error: Cannot access memory at address 0x2425dd0c34ad1d00>, len = 1869770798}, user_param = {s = 0x7f918e6e4698 "0", len = 1}, maddr = {
            s = 0x30 <error: Cannot access memory at address 0x30>, len = -1910456304}, method = {s = 0x5641ca6bb424 "parser/msg_parser.c", len = 132}, lr = {
            s = 0x5641ca5c91de <qm_malloc+1070> "I\203E(\001L\211\360H\213|$(dH3<%(", len = 1936681068}, r2 = {s = 0x5641ca6c48b0 <__FUNCTION__.8800> "get_hdr_field", len = 2059}, gr = {
            s = 0x2425dd0c34ad1d00 <error: Cannot access memory at address 0x2425dd0c34ad1d00>, len = -1434457386}, transport_val = {s = 0x5641ca9417ba <tcp_current_req+602> "", 
            len = -1905375536}, ttl_val = {s = 0x5641ca5f60e8 <parse_cseq+584> "H\205\300uU\351\"\002", len = -896264453}, user_param_val = {
            s = 0x2425dd0c34ad1d00 <error: Cannot access memory at address 0x2425dd0c34ad1d00>, len = -896264452}, maddr_val = {
            s = 0x5641ca9416fb <tcp_current_req+411> " 33 REGISTER\r\nExpires: 900\r\nAllow: INVITE, ACK, CANCEL, BYE, REFER, INFO, NOTIFY, OPTIONS, UPDATE, PRACK, SUBSCRIBE, MESSAGE\r\nUser-Agent: Bria Android 3.9.2 build 96033\r\nContent-Length: 0\r\n\r\n", len = -1905375712}, method_val = {s = 0x5641ca602ca3 <get_hdr_field+1379> "A\203>\377H\211\305\017\204~\a", 
            len = 313}, lr_val = {s = 0x5641ca5c91de <qm_malloc+1070> "I\203E(\001L\211\360H\213|$(dH3<%(", len = 0}, r2_val = {s = 0x5641ca6c48a0 <__FUNCTION__.8864> "parse_headers", 
            len = 2059}, gr_val = {s = 0x0, len = -1}, u_name = {{s = 0x2425dd0c34ad1d00 <error: Cannot access memory at address 0x2425dd0c34ad1d00>, len = -1434455472}, {
              s = 0x78 <error: Cannot access memory at address 0x78>, len = -896264458}, {s = 0x2425dd0c34ad1d00 <error: Cannot access memory at address 0x2425dd0c34ad1d00>, 
              len = -896264262}, {s = 0x7f918e20c010 "\356\354h\312AV", len = -1905374736}, {s = 0x7f918e6e4a28 "\022\027\224\312AV", len = -898873184}, {
              s = 0x5641ca6bb424 "parser/msg_parser.c", len = 28472760}, {s = 0x5641ca5c965c <qm_free+540> "H\213D$8dH3\004%(", len = 313}, {
              s = 0x149ca5c91de <error: Cannot access memory at address 0x149ca5c91de>, len = -1905374912}, {s = 0xb0 <error: Cannot access memory at address 0xb0>, len = 2059}, {
              s = 0x2425dd0c34ad1d00 <error: Cannot access memory at address 0x2425dd0c34ad1d00>, len = -1}}, u_val = {{s = 0x3 <error: Cannot access memory at address 0x3>, 
              len = -1910170256}, {s = 0x7f918e6e3248 "\001", len = 0}, {s = 0x7f918e6e3248 "\001", len = 15}, {s = 0x5641ca56688d <eval_expr+525> "\213\035\215\345\071", 
---Type <return> to continue, or q <return> to quit---
              len = -1905380792}, {s = 0x2425dd0c34ad1d00 <error: Cannot access memory at address 0x2425dd0c34ad1d00>, len = 0}, {s = 0x2 <error: Cannot access memory at address 0x2>, 
              len = -1910167864}, {s = 0x7f918e6e3248 "\001", len = 0}, {s = 0x7f918e6e3248 "\001", len = 15}, {s = 0x5641ca56688d <eval_expr+525> "\213\035\215\345\071", 
              len = -1905380792}, {s = 0x5641ca954c20 <_oser_err_info> "", len = -1905380792}}, u_params_no = 7424}
        u = <optimized out>
        port = <optimized out>
        cmatch = <optimized out>
        aitem = <optimized out>
        adefault = <optimized out>
        spec = <optimized out>
        model = <optimized out>
        val = {rs = {s = 0x0, len = 0}, ri = 5, flags = 22081}
        pve = <optimized out>
        name_s = {s = 0x0, len = -1872976960}
        start = {tv_sec = 94840586692560, tv_usec = 3}
        end_time = <optimized out>
        aux_counter = 32657
        __FUNCTION__ = "do_action"
#7  0x00005641ca52c8b8 in run_action_list (a=<optimized out>, msg=msg@entry=0x7f918e6e3248) at action.c:172
        ret = -1
        t = 0x7f918e254be0
#8  0x00005641ca52a32c in do_action (a=a@entry=0x7f918e254d10, msg=msg@entry=0x7f918e6e3248) at action.c:1124
        increment = <optimized out>
        decrement = <optimized out>
        val_number = <optimized out>
        j = <optimized out>
        val_s = {s = 0x7f919039851e "]", len = -1876577903}
        cdb_reply = 0x7ffcaa7fee80
        aux = {s = 0x7f919039851e "]", len = -1876577903}
        i = <optimized out>
        key_number = 0
        it = <optimized out>
        avp_val = <optimized out>
        avp_name = {n = -1434456392, s = {s = 0x7ffcaa7feeb8 "\221\261%\220\221\177", len = 10}}
        avp_type = 0
        ret = 1
        v = 1
        sec = <optimized out>
        usec = <optimized out>
        to = <optimized out>
        p = <optimized out>
        tmp = <optimized out>
        new_uri = <optimized out>
        end = <optimized out>
        crt = <optimized out>
        len = <optimized out>
        i = <optimized out>
---Type <return> to continue, or q <return> to quit---
        user = 0
        expires = 0
        vals = {{s = 0xffffffffffffffff <error: Cannot access memory at address 0xffffffffffffffff>, len = 0}, {s = 0x0, len = -1876382882}, {
            s = 0x7ffc00000001 <error: Cannot access memory at address 0x7ffc00000001>, len = 8}, {s = 0x7ffcaa7fef30 "", len = -881088448}, {s = 0x7ffcaa7ff410 "\020\300 \216\221\177", 
            len = 2007236248}}
        result = {s = 0x7ffcaa7fee80 "0\357\177\252\374\177", len = -881088448}
        uri = {user = {s = 0x7ffcaa7ff538 "+\033f\312AV", len = 0}, passwd = {s = 0x7f9177a3fed2 "enabled\n", len = -1876577903}, host = {
            s = 0xd68 <error: Cannot access memory at address 0xd68>, len = 1}, port = {s = 0x7ffcaa7fef78 "\332\376\243w\221\177", len = 0}, params = {
            s = 0x7ffcaa7fefa0 "@\361\177\252\374\177", len = 2007232507}, headers = {s = 0x0, len = 0}, port_no = 0, proto = 0, type = 32764, transport = {
            s = 0x7ffc00000000 <error: Cannot access memory at address 0x7ffc00000000>, len = -1}, ttl = {s = 0x7f910000004f <error: Cannot access memory at address 0x7f910000004f>, 
            len = 2147483647}, user_param = {s = 0x7f9100000000 <error: Cannot access memory at address 0x7f9100000000>, len = 0}, maddr = {
            s = 0x7ffc00000000 <error: Cannot access memory at address 0x7ffc00000000>, len = -1434456080}, method = {s = 0x0, len = 3432}, lr = {
            s = 0x7ffc0000004f <error: Cannot access memory at address 0x7ffc0000004f>, len = 2007236263}, r2 = {s = 0x0, len = -1434454352}, gr = {s = 0x7f9177a3feda "", len = 0}, 
          transport_val = {s = 0x7f9190292f14 <malloc+84> "H\205\300H\211\302\017\204", <incomplete sequence \310>, len = 1152}, ttl_val = {s = 0x7ffcaa7ff1d8 "\220\244{\313AV", 
            len = -1434455744}, user_param_val = {s = 0x7f91902f9688 <gaih_inet.constprop+1288> "H\205\300H\211\003u\200M\211\376L\211\367\350%3\363\377\270\n", len = 0}, maddr_val = {
            s = 0x3000000018 <error: Cannot access memory at address 0x3000000018>, len = -1434454512}, method_val = {s = 0x7ffcaa7ff550 "\340\363\243w\221\177", len = 2046601586}, 
          lr_val = {s = 0x7f9179fca930 <callid_buf+16> "-9717@10.18.130.27", len = 2046601586}, r2_val = {s = 0x0, len = 0}, gr_val = {
            s = 0x1100000002 <error: Cannot access memory at address 0x1100000002>, len = 44808}, u_name = {{s = 0x0, len = 1}, {s = 0x25ec <error: Cannot access memory at address 0x25ec>, 
              len = -1434517504}, {s = 0x0, len = -1434455824}, {s = 0x10 <error: Cannot access memory at address 0x10>, len = -1}, {s = 0x0, len = -1434454576}, {
              s = 0x5641ca6d0f21 "SIP receiver TCP", len = -1}, {s = 0x7ffcaa7ff0f0 "\f", len = 16}, {s = 0x0, len = 0}, {s = 0x7ffcaa7ff600 "\344\366\177\252\374\177", len = 1024}, {
              s = 0x7f918e6e3248 "\001", len = -1434455596}}, u_val = {{s = 0x2 <error: Cannot access memory at address 0x2>, len = -1434517502}, {s = 0x7f918e6e3248 "\001", 
              len = -898827280}, {
              s = 0x7f919025b191 <vfprintf+1937> "H\213\225h\373\377\377H\211\321L)\371H9\310\017\205h\371\377\377D\213\215H\373\377\377\271\377\377\377\177D)\311Hc\311H9\310\017\217\274\025", len = -1434456080}, {s = 0x7ffcaa7fefb0 "", len = 0}, {s = 0x7f918e250340 "10.18.130.27", len = 12}, {s = 0x7ffcaa7ff240 "\020\300 \216\221\177", len = -1905380782}, {
              s = 0x5324808 <error: Cannot access memory at address 0x5324808>, len = 3432}, {s = 0x7f918e6e3200 "\b", len = -1434454352}, {s = 0x7ffcaa7ff250 "@\b", len = -1905380792}, {
              s = 0x7f917e577878 "", len = -1434454400}}, u_params_no = 51085}
        next_hop = {user = {s = 0xd68 <error: Cannot access memory at address 0xd68>, len = 45}, passwd = {s = 0x5 <error: Cannot access memory at address 0x5>, len = -899150390}, host = {
            s = 0x0, len = -1434453872}, port = {s = 0x7ffcaa7ff6a0 "", len = -900356264}, params = {s = 0x7ffcaa7ff200 "[", len = 1}, headers = {
            s = 0x20 <error: Cannot access memory at address 0x20>, len = -1434455488}, port_no = 61984, proto = 43647, type = 32764, transport = {
            s = 0x7f917dde36f2 <pqSocketCheck.part.2+178> "\205\300\211\303y\027\350\203K\377\377\213\070\203\377\004u<E\205\344\017\205{\377\377\377\061\333H\213\214$(\001", len = 2}, 
          ttl = {s = 0x7ffc00000000 <error: Cannot access memory at address 0x7ffc00000000>, len = 40}, user_param = {s = 0x5641cb7ba490 "\210\334\\\220\221\177", len = 0}, maddr = {
            s = 0x0, len = -1905380782}, method = {s = 0x7f91902d1586 <__strftime_internal+6358> "L\211\361L\213\\$\020\351\306\356\377\377\017\037D", len = 91}, lr = {
            s = 0x1ff4 <error: Cannot access memory at address 0x1ff4>, len = 6}, r2 = {s = 0x2425dd0c34ad1d00 <error: Cannot access memory at address 0x2425dd0c34ad1d00>, 
            len = -1434454864}, gr = {s = 0x7f918e6e3db0 "@", len = 1}, transport_val = {s = 0x40 <error: Cannot access memory at address 0x40>, len = -1910456304}, ttl_val = {
            s = 0x5641ca698b23 "parser/parse_via.c", len = 2112}, user_param_val = {s = 0x5641ca5c91de <qm_malloc+1070> "I\203E(\001L\211\360H\213|$(dH3<%(", len = 124272}, maddr_val = {
            s = 0x5641ca6b0ad0 <__FUNCTION__.8968> "parse_via", len = 2059}, method_val = {s = 0x55800d41 <error: Cannot access memory at address 0x55800d41>, len = 0}, lr_val = {
            s = 0x2425dd0c34ad1d00 <error: Cannot access memory at address 0x2425dd0c34ad1d00>, len = 0}, r2_val = {
            s = 0x5641ca9415d7 <tcp_current_req+119> "rport\r\nMax-Forwards: 69\r\nContact: <sip:4310@99.245.176.196:44608;rinstance=987655dc2a1aa8d2;transport=tcp>\r\nTo: \"volga629\"<sip:4310@sip.company.tld>\r\nFrom: \"volga629\"<sip:4310@sip.compan"..., len = -896264745}, gr_val = {s = 0x0, len = -1905378328}, u_name = {{
              s = 0x5641ca9417ba <tcp_current_req+602> "", len = 6}, {s = 0x5641ca5e0375 <parse_via+5381> "H\205\300I\211\303\017\204\300", <incomplete sequence \321>, len = -1872962848}, {
              s = 0x5641ca94158b <tcp_current_req+43> "SIP/2.0/TCP 192.168.88.164:47869;branch=z9hG4bK-524287-1---31745c71db99372c;rport\r\nMax-Forwards: 69\r\nContact: <sip:4310@99.245.176.196:44608;rinstance=987655dc2a1aa8d2;transport=tcp>\r\nTo: \"volga629"..., len = 5}, {s = 0x7f918e6e3de8 <incomplete sequence \353>, len = -1905378328}, {s = 0x7f918e6e3be8 "\001", 
              len = 110}, {s = 0x5641ca5c9106 <qm_malloc+854> "m3", len = -1434455248}, {
              s = 0x5641ca9415d7 <tcp_current_req+119> "rport\r\nMax-Forwards: 69\r\nContact: <sip:4310@99.245.176.196:44608;rinstance=987655dc2a1aa8d2;transport=tcp>\r\nTo: \"volga629\"<sip:4310@sip.company.tld>\r\nFrom: \"volga629\"<sip:4310@sip.compan"..., len = 2059}, {s = 0x0, len = 132336}, {
              s = 0x2425dd0c34ad1d00 <error: Cannot access memory at address 0x2425dd0c34ad1d00>, len = -1434454728}, {
---Type <return> to continue, or q <return> to quit---
              s = 0x5641ca94158b <tcp_current_req+43> "SIP/2.0/TCP 192.168.88.164:47869;branch=z9hG4bK-524287-1---31745c71db99372c;rport\r\nMax-Forwards: 69\r\nContact: <sip:4310@99.245.176.196:44608;rinstance=987655dc2a1aa8d2;transport=tcp>\r\nTo: \"volga629"..., len = -896264822}}, u_val = {{s = 0x7f918e6e3b38 "\001", len = -896264262}, {s = 0x7f918e6e3be8 "\001", 
              len = 1}, {s = 0x5641ca602ff5 <get_hdr_field+2229> "A\203>\377H\211\305\017\204\234\003", len = 313}, {
              s = 0x5641ca5c91de <qm_malloc+1070> "I\203E(\001L\211\360H\213|$(dH3<%(", len = 9717}, {s = 0x5641ca6c48a0 <__FUNCTION__.8864> "parse_headers", len = 2059}, {
              s = 0x7f91905ca000 <_IO_mem_jumps> "", len = -881088448}, {s = 0x2425dd0c34ad1d00 <error: Cannot access memory at address 0x2425dd0c34ad1d00>, len = 1}, {
              s = 0x2 <error: Cannot access memory at address 0x2>, len = -896264826}, {
              s = 0x5641ca6bbb6c "T\177\364\377\224~\364\377\204}\364\377\374\206\364\377d\200\364\377$\200\364\377D\200\364\377\f\200\364\377܀\364\377\204\200\364\377\244\200\364\377\024\201\364\377\274\203\364\377|\203\364\377\234\203\364\377\004\202\364\377<\202\364\377\\\202\364\377\224\202\364\377l\201\364\377\244\201\364\377\364\203\364\377l\206\364\377\004\203\364\377\244\206\364\377d\204\364\377܆\364\377L\201\364\377\344\205\364\377,\204\364\377\004\206\364\377̂\364\377$\206\364\377\274\205\364\377D\206\364\377܁\364\377Ԅ\364\377\204\204\364\377\254\204\364\377\224~\364\377|\205\364\377\374\204\364\377<\205\364\377,\177\364\377<\203\364\377ƞ\364\377y\303\364\377\310\302\364\377\310\302\364\377y\303\364\377"..., len = -896264262}, {
              s = 0x2425dd0c34ad1d00 <error: Cannot access memory at address 0x2425dd0c34ad1d00>, len = -1905380792}}, u_params_no = 15960}
        u = <optimized out>
        port = <optimized out>
        cmatch = <optimized out>
        aitem = <optimized out>
        adefault = <optimized out>
        spec = <optimized out>
        model = <optimized out>
        val = {rs = {s = 0x7ffcaa7feeb8 "\221\261%\220\221\177", len = 10}, ri = -1434456352, flags = 32764}
        pve = <optimized out>
        name_s = {s = 0x7ffcaa7ff350 "\212\025\224\312AV", len = -1875278565}
        start = {tv_sec = 140723168998248, tv_usec = 0}
        end_time = <optimized out>
        aux_counter = 0
        __FUNCTION__ = "do_action"
#9  0x00005641ca52c8b8 in run_action_list (a=a@entry=0x7f918e2504d0, msg=msg@entry=0x7f918e6e3248) at action.c:172
        ret = -1
        t = 0x7f918e254d10
#10 0x00005641ca52cb98 in run_actions (msg=0x7f918e6e3248, a=0x7f918e2504d0) at action.c:137
        ret = <optimized out>
        msg = 0x7f918e6e3248
        a = 0x7f918e2504d0
        ret = <optimized out>
#11 run_top_route (a=0x7f918e2504d0, msg=msg@entry=0x7f918e6e3248) at action.c:214
        bk_action_flags = 0
        bk_rec_lev = 0
        ret = <optimized out>
        ctx = 0x0
        __FUNCTION__ = "run_top_route"
#12 0x00005641ca51bca4 in receive_msg (
    buf=0x5641ca941560 <tcp_current_req> "REGISTER sip:sip.company.tld SIP/2.0\r\nVia: SIP/2.0/TCP 192.168.88.164:47869;branch=z9hG4bK-524287-1---31745c71db99372c;rport\r\nMax-Forwards: 69\r\nContact: <sip:4310@99.245.176.196:44608;rinstance=987655"..., len=<optimized out>, len@entry=602, rcv_info=rcv_info@entry=0x7ffcaa7ff620, existing_context=existing_context@entry=0x0)
    at receive.c:208
        ctx = 0x7f918e6e3e90
        msg = 0x7f918e6e3248
        start = {tv_sec = 1, tv_usec = 140723168999072}
---Type <return> to continue, or q <return> to quit---
        rc = 3
        in_buff = {
          s = 0x5641ca941560 <tcp_current_req> "REGISTER sip:sip.company.tld SIP/2.0\r\nVia: SIP/2.0/TCP 192.168.88.164:47869;branch=z9hG4bK-524287-1---31745c71db99372c;rport\r\nMax-Forwards: 69\r\nContact: <sip:4310@99.245.176.196:44608;rinstance=987655"..., len = 602}
        __FUNCTION__ = "receive_msg"
#13 0x00005641ca6635ea in tcp_handle_req (_max_msg_chunks=<optimized out>, con=0x7f917e598290, req=0x5641ca941560 <tcp_current_req>) at net/proto_tcp/tcp_common.h:412
        local_rcv = {src_ip = {af = 2, len = 4, u = {addrl = {3299931491, 0}, addr32 = {3299931491, 0, 0, 0}, addr16 = {62819, 50352, 0, 0, 0, 0, 0, 0}, 
              addr = "c\365\260\304", '\000' <repeats 11 times>}}, dst_ip = {af = 2, len = 4, u = {addrl = {1603749278, 0}, addr32 = {1603749278, 0, 0, 0}, addr16 = {17822, 24471, 0, 0, 0, 
                0, 0, 0}, addr = "\236E\227_", '\000' <repeats 11 times>}}, src_port = 46207, dst_port = 5082, proto = 2, proto_reserved1 = 1353907697, proto_reserved2 = 0, src_su = {s = {
              sa_family = 2, sa_data = "\264\177c\365\260\304\000\000\000\000\000\000\000"}, sin = {sin_family = 2, sin_port = 32692, sin_addr = {s_addr = 3299931491}, 
              sin_zero = "\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 2, sin6_port = 32692, sin6_flowinfo = 3299931491, sin6_addr = {__in6_u = {
                  __u6_addr8 = "\000\000\000\000\000\000\000\000\377\377\377\377\377\377\377\377", __u6_addr16 = {0, 0, 0, 0, 65535, 65535, 65535, 65535}, __u6_addr32 = {0, 0, 4294967295, 
                    4294967295}}}, sin6_scope_id = 0}}, bind_address = 0x7f918e246080}
        msg_buf = <optimized out>
        msg_len = 602
        c = 0 '\000'
        size = 0
#14 tcp_read_req (con=0x7f917e598290, bytes_read=0x7ffcaa7ff6e0) at net/proto_tcp/proto_tcp.c:1150
        bytes = <optimized out>
        total_bytes = 602
        req = <optimized out>
        src_su = {s = {sa_family = 30840, sa_data = "W~\221\177\000\000\r=\243w\221\177\000"}, sin = {sin_family = 30840, sin_port = 32343, sin_addr = {s_addr = 32657}, 
            sin_zero = "\r=\243w\221\177\000"}, sin6 = {sin6_family = 30840, sin6_port = 32343, sin6_flowinfo = 32657, sin6_addr = {__in6_u = {
                __u6_addr8 = "\r=\243w\221\177\000\000xxW~\221\177\000", __u6_addr16 = {15629, 30627, 32657, 0, 30840, 32343, 32657, 0}, __u6_addr32 = {2007186701, 32657, 2119661688, 
                  32657}}}, sin6_scope_id = 3395643197}}
        dst_su = {s = {sa_family = 63204, sa_data = "\177\252\374\177\000\000=w\243w\221\177\000"}, sin = {sin_family = 63204, sin_port = 43647, sin_addr = {s_addr = 32764}, 
            sin_zero = "=w\243w\221\177\000"}, sin6 = {sin6_family = 63204, sin6_port = 43647, sin6_flowinfo = 32764, sin6_addr = {__in6_u = {
                __u6_addr8 = "=w\243w\221\177\000\000xxW~\221\177\000", __u6_addr16 = {30525, 30627, 32657, 0, 30840, 32343, 32657, 0}, __u6_addr32 = {2007201597, 32657, 2119661688, 
                  32657}}}, sin6_scope_id = 2418619068}}
        __FUNCTION__ = "tcp_read_req"
#15 0x00005641ca644347 in handle_io (fm=<optimized out>, idx=idx@entry=0, event_type=event_type@entry=1) at net/net_tcp_proc.c:227
        ret = 0
        n = <optimized out>
        con = 0x7f917e598290
        s = 6
        rw = <optimized out>
        resp = <optimized out>
        response = {140262866780816, 1}
        __FUNCTION__ = "handle_io"
#16 0x00005641ca648630 in io_wait_loop_epoll (h=<optimized out>, t=<optimized out>, repeat=<optimized out>) at ./io_wait_loop.h:225
        ret = <optimized out>
        e = <optimized out>
        n = 1
        r = 0
#17 tcp_worker_proc_loop () at net/net_tcp_proc.c:355
        __FUNCTION__ = "tcp_worker_proc_loop"
---Type <return> to continue, or q <return> to quit---
#18 0x00005641ca654368 in tcp_start_processes (chd_rank=0x5641ca91d9fc <chd_rank>, startup_done=0x0) at net/net_tcp.c:1836
        r = 0
        reader_fd = {32, 35}
        pid = <optimized out>
        load_p = 0x7f917e579ba8
        __FUNCTION__ = "tcp_start_processes"
#19 0x00005641ca5053bd in main_loop () at main.c:683
        startup_done = 0x0
        chd_rank = 5
        chd_rank = 5
#20 main (argc=<optimized out>, argv=<optimized out>) at main.c:1283
        cfg_stream = 0x5641cb78b010
        c = <optimized out>
        r = <optimized out>
        tmp = 0x7ffcaa800f13 ""
        tmp_len = <optimized out>
        port = <optimized out>
        proto = <optimized out>
        protos_no = <optimized out>
        options = 0x5641ca688778 "f:cCm:M:b:l:n:N:rRvdDFETSVhw:t:u:g:P:G:W:o:"
        seed = 2095616134
        rfd = <optimized out>
        __FUNCTION__ = "main"
volga629 commented 7 years ago

@liviuchircu What possible cause for this error insert mode 0 and mode 0. We apply fix_nat_register() then it not match contact any more. What possible solution here are ?

Code

        if(af==INET) {
                force_rport();
                if(!isflagset(FLAG_FROM_FS)) {
                        if(nat_uac_test("19")) {
                                if(is_method("REGISTER")) {
                                        fix_nated_register();
                                } else {
                                        fix_nated_contact();
                                }
                        }
                        if(is_method("REGISTER")) {
                                mid_registrar_save("location");
                                append_hf("Path: <sip:$fU@10.18.130.27:5060;transport=udp>\r\n");
                                setflag(10);
                                xlog("L_INFO", "Forwarding REGISTER to main registrar\n");
                        }
                }

        }

Error

Apr 26 21:11:44 casbc00 /usr/sbin/opensips[3758]: ERROR:mid_registrar:get_match_token: a Contact from main registrar (sip:4310@192.168.88.164:40368;rinstance=07bf843f000e41fe;transport=tcp) is missing the 'rid' hf parameter
Apr 26 21:11:44 casbc00 /usr/sbin/opensips[3758]: ERROR:mid_registrar:match_contact: failed to get match token
Apr 26 21:11:44 casbc00 /usr/sbin/opensips[3758]: ERROR:mid_registrar:insert_rpl_contacts: Contact 'sip:4310@client_public_ip:41494;rinstance=12b6c78cf8c6e6ff;transport=tcp' not found in reply from main registrar!
Apr 26 21:11:44 casbc00 /usr/sbin/opensips[3758]: ERROR:mid_registrar:mid_reg_resp_in: failed to process rpl contacts for AoR '4310'
liviuchircu commented 7 years ago

Your append_hf() workaround to make FreeSWITCH happy should work fine now, @volga629. I will open separate tickets regarding the ";transport" parameter for the appended Path headers, as well as adding restart persistency.

volga629 commented 7 years ago

Thank you, I will test it soon.

volga629 commented 7 years ago

@liviuchircu I posted in mailing list another crash I am not sure if related to mid_registrar

volga629 commented 7 years ago

Right now it set usr_loc mode 1

liviuchircu commented 7 years ago

@volga629 Please note that usrloc db_mode != 0 will crash the mid_registrar sooner or later, as you will inevitably restart the proxy. I will take care of adding persistency support after next week's Summit (maybe even during it!), but no sooner.

volga629 commented 7 years ago

or I see. Ok I will keep db mode 0

volga629 commented 7 years ago

Is should match contact only by regid ? Is this correct that regid will contain -- on the end ?

[sip:4300@10.18.130.27:5060;transport=tcp;regid=c2lwOjQzMDBAOTkuMjQ1LjE3Ni4xOTY6NTYwMDA7dHJhbnNwb3J0PXRjcA--]

volga629 commented 7 years ago

Hello Everyone, Is permanent storage ofr mid_registrar should work ?

liviuchircu commented 7 years ago

Not yet, @volga629, it's still on the TODO list