search

sipwise / mediaproxy-ng

[DEPRECATED] Kernel-based media relay for Kamailio
37 stars 12 forks source link

Segfault when cleaning up calls #4

Closed hughwaite closed 10 years ago

hughwaite commented 11 years ago

We are seeing segfaults on our mediaproxy-ng instance. From the backtrace they appear to occur on a timer, so I don't currently have a network trace of the call that caused it. This binary was compiled on August 22nd 2013 13:32 The command line is: /usr/sbin/mediaproxy-ng --pidfile /var/run/mediaproxy-ng.pid --table=0 --ip=10.yy.yy.yy --advertised-ip=50.xx.xx.xx --listen-ng=10.yy.yy.yy:2223 --port-min=30000 --port-max=40000

The backtrace from the coredump is: `(gdb) bt

0 0x00007f255c2d8686 in EVP_EncryptFinal_ex () from /usr/lib64/libcrypto.so.10

1 0x000000000041e26e in evp_session_key_cleanup (c=0x7f254800d328) at crypto.c:463

2 0x00000000004127b7 in crypto_cleanup (c=0x7f25480060d0) at crypto.h:114

3 kill_callstream (c=0x7f25480060d0) at call.c:1773

4 call_destroy (c=0x7f25480060d0) at call.c:1818

5 0x00000000004132e2 in kill_calls_timer (list=0x7f2540001490, m=) at call.c:1011

6 0x000000000041377b in callmaster_timer (ptr=0x2003e00) at call.c:1113

7 0x00000000004089cb in poller_timers_run (p=0x2001ed0, max=) at poller.c:282

8 poller_timers_wait_run (p=0x2001ed0, max=) at poller.c:505

9 0x00000000004058dd in timer_loop (d=0x2001ed0) at main.c:484

10 0x0000000000408c4f in thread_detach_func (d=) at aux.c:160

11 0x00007f255bbcf851 in start_thread () from /lib64/libpthread.so.0

12 0x00007f255b91d90d in clone () from /lib64/libc.so.6

(gdb) frame 1

1 0x000000000041e26e in evp_session_key_cleanup (c=0x7f254800d328) at crypto.c:463

463 EVP_EncryptFinal_ex(c->session_key_ctx[i], block, &len); (gdb) info locals block = "\000\320\000H%\177\000\000\320\000H%\177\000" len = 32549 i = 0 (gdb) p c->session_keyctx[i] $1 = (void ) 0x7f254c002cf0 (gdb) p *((EVP_CIPHERCTX)c->session_key_ctx[0]) $2 = {cipher = 0x0, engine = 0x0, encrypt = 0, buf_len = 0, oiv = '\000' <repeats 15 times>, iv = '\000' <repeats 15 times>, buf = '\000' <repeats 31 times>, num = 0, app_data = 0x0, key_len = 0, flags = 0, cipher_data = 0x0, final_used = 0, block_mask = 0, final = '\000' <repeats 31 times>} (gdb) ` Just ask if you need more info to investigate

pdunkley commented 11 years ago

The calls that were cleared up here were ones that failed during some testing of session-timer.

Some combination of rtpproxy_manage() had been called for the timer re-INVITEs (meaning that rtpproxy_manage() had been called multiple times with the +SP and -sp arguments for the same RTP streams).

rfuchs commented 11 years ago

Do you happen to have a log excerpt for the particular call which caused this? It would help to know the exact sequence of events which led up to this.

hughwaite commented 11 years ago

I don't have a log or network trace of the call that caused this and, so far, I am unable to recreate it.

From the coredump, at frame 3 (call.c:1773), the inbound stream has been freed successfully (on line 1772) and it is the outbound that causes a crash. I'm wondering if perhaps a reINVITE was started, but the response was not sent to rtpproxy-ng, leaving it in a half-way state when the timer fires. As I said, I can't recreate this at the moment.