PhantomVl / sccp_manager

SCCP Manager
35 stars 19 forks source link

Random Asterisk Crash after a Call is Hungup #75

Closed josh2893 closed 4 years ago

josh2893 commented 4 years ago

Since the last update to SCCP Manager our Asterisk Core now seems to randomly crash and restart after calls from extension 13 have hung-up.

I have no idea if this is an issue with SCCP manager or whether Asterisk is just broken on our VM now.

Can someone point me in the right direction?

[2020-05-14 11:51:28] VERBOSE[28722][C-00000003] pbx.c: Spawn extension (ext-local, h, 1) exited non-zero on 'SCCP/13-0000003D' [2020-05-14 11:51:28] ERROR[28722][C-00000003] astobj2.c: FRACK!, Failed assertion bad magic number 0x0 for object 0x321e310 (0) [2020-05-14 11:51:28] ERROR[28722][C-00000003] : Got 21 backtrace records

0: /usr/sbin/asterisk(__ast_assert_failed+0x84) [0x5feb50]

1: /usr/sbin/asterisk() [0x45ad75]

2: /usr/sbin/asterisk() [0x45b3ed]

3: /usr/sbin/asterisk(__ao2_ref+0x2e) [0x45b850]

4: /usr/sbin/asterisk(__ao2_cleanup+0x24) [0x45b8c0]

5: /usr/sbin/asterisk() [0x51cb96]

6: /usr/sbin/asterisk() [0x45b566]

7: /usr/sbin/asterisk(__ao2_ref+0x2e) [0x45b850]

8: /usr/sbin/asterisk() [0x51c882]

9: /usr/sbin/asterisk() [0x45b566]

10: /usr/sbin/asterisk(__ao2_ref+0x2e) [0x45b850]

11: /usr/sbin/asterisk(ast_channel_nativeformats_set+0x6b) [0x4cb0db]

12: /usr/sbin/asterisk() [0x4af2d0]

13: /usr/sbin/asterisk() [0x45b566]

14: /usr/sbin/asterisk(__ao2_ref+0x2e) [0x45b850]

15: /usr/sbin/asterisk(ast_hangup+0x435) [0x4b03cc]

16: /usr/sbin/asterisk() [0x57507a]

17: /usr/sbin/asterisk() [0x5753d6]

18: /usr/sbin/asterisk() [0x5fbb03]

19: /lib64/libpthread.so.0(+0x7dd5) [0x7f847fc4add5]

20: /lib64/libc.so.6(clone+0x6d) [0x7f847ef2802d]

[2020-05-14 11:51:38] ERROR[15050] astobj2.c: FRACK!, Failed assertion bad magic number 0x0 for object 0x321e310 (0) [2020-05-14 11:51:38] ERROR[15050] : Got 22 backtrace records

0: /usr/sbin/asterisk(__ast_assert_failed+0x84) [0x5feb50]

1: /usr/sbin/asterisk() [0x45ad75]

2: /usr/sbin/asterisk() [0x45b3ed]

3: /usr/sbin/asterisk(__ao2_ref+0x2e) [0x45b850]

4: /usr/sbin/asterisk(__ao2_cleanup+0x24) [0x45b8c0]

5: /usr/sbin/asterisk() [0x51cb96]

6: /usr/sbin/asterisk() [0x45b566]

7: /usr/sbin/asterisk(__ao2_ref+0x2e) [0x45b850]

8: /usr/sbin/asterisk() [0x51c882]

9: /usr/sbin/asterisk() [0x45b566]

10: /usr/sbin/asterisk(__ao2_ref+0x2e) [0x45b850]

11: /usr/sbin/asterisk(__ao2_cleanup+0x24) [0x45b8c0]

12: /usr/lib64/asterisk/modules/chan_sip.so(+0x27206) [0x7f843d598206]

13: /usr/sbin/asterisk() [0x45b566]

14: /usr/sbin/asterisk(__ao2_ref+0x2e) [0x45b850]

15: /usr/sbin/asterisk(__ao2_cleanup+0x24) [0x45b8c0]

16: /usr/lib64/asterisk/modules/chan_sip.so(+0x17c23) [0x7f843d588c23]

17: /usr/sbin/asterisk(ast_sched_runq+0x107) [0x5ba7de]

18: /usr/lib64/asterisk/modules/chan_sip.so(+0x94cf4) [0x7f843d605cf4]

19: /usr/sbin/asterisk() [0x5fbb03]

20: /lib64/libpthread.so.0(+0x7dd5) [0x7f847fc4add5]

21: /lib64/libc.so.6(clone+0x6d) [0x7f847ef2802d]

[2020-05-14 11:51:38] ERROR[15050] astobj2.c: FRACK!, Failed assertion bad magic number 0x0 for object 0x321e310 (0) [2020-05-14 11:51:38] ERROR[15050] : Got 22 backtrace records

0: /usr/sbin/asterisk(__ast_assert_failed+0x84) [0x5feb50]

1: /usr/sbin/asterisk() [0x45ad75]

2: /usr/sbin/asterisk() [0x45b3ed]

3: /usr/sbin/asterisk(__ao2_ref+0x2e) [0x45b850]

4: /usr/sbin/asterisk(__ao2_cleanup+0x24) [0x45b8c0]

5: /usr/sbin/asterisk() [0x51cb96]

6: /usr/sbin/asterisk() [0x45b566]

7: /usr/sbin/asterisk(__ao2_ref+0x2e) [0x45b850]

8: /usr/sbin/asterisk() [0x51c882]

9: /usr/sbin/asterisk() [0x45b566]

10: /usr/sbin/asterisk(__ao2_ref+0x2e) [0x45b850]

11: /usr/sbin/asterisk(__ao2_cleanup+0x24) [0x45b8c0]

12: /usr/lib64/asterisk/modules/chan_sip.so(+0x27206) [0x7f843d598206]

13: /usr/sbin/asterisk() [0x45b566]

14: /usr/sbin/asterisk(__ao2_ref+0x2e) [0x45b850]

15: /usr/sbin/asterisk(__ao2_cleanup+0x24) [0x45b8c0]

16: /usr/lib64/asterisk/modules/chan_sip.so(+0x17c23) [0x7f843d588c23]

17: /usr/sbin/asterisk(ast_sched_runq+0x107) [0x5ba7de]

18: /usr/lib64/asterisk/modules/chan_sip.so(+0x94cf4) [0x7f843d605cf4]

19: /usr/sbin/asterisk() [0x5fbb03]

20: /lib64/libpthread.so.0(+0x7dd5) [0x7f847fc4add5]

21: /lib64/libc.so.6(clone+0x6d) [0x7f847ef2802d]

[2020-05-14 11:51:57] ERROR[15050] astobj2.c: FRACK!, Failed assertion bad magic number 0x0 for object 0x321e310 (0) [2020-05-14 11:51:57] ERROR[15050] : Got 14 backtrace records

0: /usr/sbin/asterisk(__ast_assert_failed+0x84) [0x5feb50]

1: /usr/sbin/asterisk() [0x45ad75]

2: /usr/sbin/asterisk() [0x45b3ed]

3: /usr/sbin/asterisk(__ao2_ref+0x2e) [0x45b850]

4: /usr/sbin/asterisk(__ast_format_cap_append+0x77) [0x51d01d]

5: /usr/sbin/asterisk(ast_format_cap_append_from_cap+0x7d) [0x51d2f4]

6: /usr/lib64/asterisk/modules/chan_sip.so(+0x30897) [0x7f843d5a1897]

7: /usr/lib64/asterisk/modules/chan_sip.so(+0x96a3b) [0x7f843d607a3b]

8: /usr/lib64/asterisk/modules/chan_sip.so(+0x53e23) [0x7f843d5c4e23]

9: /usr/sbin/asterisk(ast_sched_runq+0x107) [0x5ba7de]

10: /usr/lib64/asterisk/modules/chan_sip.so(+0x94cf4) [0x7f843d605cf4]

11: /usr/sbin/asterisk() [0x5fbb03]

12: /lib64/libpthread.so.0(+0x7dd5) [0x7f847fc4add5]

13: /lib64/libc.so.6(clone+0x6d) [0x7f847ef2802d]

PhantomVl commented 4 years ago

You were mistaken, this question needs to be sent to https://github.com/chan-sccp/chan-sccp or https://gitter.im/chan-sccp/chan-sccp

josh2893 commented 4 years ago

Thankyou for the information!

dkgroot commented 4 years ago

@josh2893 Looking at the backtrace it looks like an issue caused by chan_sip. The reason for the crash is not fully clear, because it's only a partial/reduced backtrace. You would have to use addr2line to clear up the potential cause. You will have to translate ever line of the reduced backtrace by translating: /usr/sbin/asterisk() [0x45ad75] to addr2line -e asterisk -p -a -f -s 0x45ad75. Have a look at the addr2line manpage. Another option is to recompile asterisk with BETTER_BACKTRACES=1 set, but that might be tricky on a FreePBX system.

The ast_format_cap_append_from_cap string points to the cause being related to codecs. Do check if the codecs set for chan_sip are installed on your asterisk system (for example opus and silk and siren require the installation of external codec drivers). Maybe start by selecting only alaw/ulaw for the all sides of the call (ie: chan-sccp / chan_sip and asterisk defaults) and check if the issue remains, when not, slowly increase the number of codecs toward what you need/want.

@PhantomVl In the latest version of sccp_manager, we did remove some of the codecs which were not supported by skinny phones. Maybe there is still an issue there. But without a full/accurate backtrace it's hard to tell what the exact cause for the problem is. More debugging would be required.

josh2893 commented 4 years ago

Ill try using just codecs ulaw/alaw for SCCP and SIP and see how I go. First part you mentioned is a bit over my head.

Thanks!

dkgroot commented 4 years ago

@josh2893 Without the addr2line conversion i cannot exactly tell what is going on. I am not sure if chan_sccp is involved or not. I am mostly certain that it should not have anything to do with sccp_manager per se.

dkgroot commented 4 years ago

@josh2893 Can you copy the log file / backtrace lines from above to a file called asterisk.log and do

wget https://raw.githubusercontent.com/chan-sccp/chan-sccp/develop/contrib/backtrace_addr2line.py -O backtrace_addr2line.py
chmod a+x backtrace_addr2line.py
backtrace_addr2line -e /usr/sbin/asterisk -f asterisk.log -b4

That should convert the backtrace to something more usable. I cannot do this for you, because my binaries do not match yours.

josh2893 commented 4 years ago

This is what I got back from follow the above. Is this what you needed?

/usr/sbin/asterisk: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamic                                                                                                                                                             ally linked (uses shared libs), for GNU/Linux 2.6.32, BuildID[sha1]=20688afb183c                                                                                                                                                             de7a470cf61aa28f179c21e89f25, stripped

[2020-05-14 11:51:28] VERBOSE[28722][C-00000003] pbx.c: Spawn extension (ext-loc                                                                                                                                                             al, h, 1) exited non-zero on 'SCCP/13-0000003D'
[2020-05-14 11:51:28] ERROR[28722][C-00000003] astobj2.c: FRACK!, Failed asserti                                                                                                                                                             on bad magic number 0x0 for object 0x321e310 (0)
[2020-05-14 11:51:28] ERROR[28722][C-00000003] : Got 21 backtrace records

0: /usr/sbin/asterisk(__ast_assert_failed+0x84)
[Backtrace #0]
__ast_assert_failed at ??:?
ast_fd_init at ??:?
ao2_object_get_lockaddr at ??:?
__ao2_ref at ??:?
__ao2_cleanup at ??:?
ast_format_cap_set_framing at ??:?
ao2_object_get_lockaddr at ??:?
__ao2_ref at ??:?
ast_format_cache_is_slinear at ??:?
ao2_object_get_lockaddr at ??:?
__ao2_ref at ??:?
ast_channel_nativeformats_set at ??:?
ast_party_redirecting_free at ??:?
ao2_object_get_lockaddr at ??:?
__ao2_ref at ??:?
ast_hangup at ??:?
ast_pbx_h_exten_run at ??:?
ast_pbx_h_exten_run at ??:?
ast_inet_ntoa at ??:?
?? ??:0
?? ??:0

[2020-05-14 11:51:38] ERROR[15050] astobj2.c: FRACK!, Failed assertion bad magic                                                                                                                                                              number 0x0 for object 0x321e310 (0)
[2020-05-14 11:51:38] ERROR[15050] : Got 22 backtrace records

0: /usr/sbin/asterisk(__ast_assert_failed+0x84)
[Backtrace #1]
__ast_assert_failed at ??:?
ast_fd_init at ??:?
ao2_object_get_lockaddr at ??:?
__ao2_ref at ??:?
__ao2_cleanup at ??:?
ast_format_cap_set_framing at ??:?
ao2_object_get_lockaddr at ??:?
__ao2_ref at ??:?
ast_format_cache_is_slinear at ??:?
ao2_object_get_lockaddr at ??:?
__ao2_ref at ??:?
__ao2_cleanup at ??:?
?? ??:0
ao2_object_get_lockaddr at ??:?
__ao2_ref at ??:?
__ao2_cleanup at ??:?
?? ??:0
ast_sched_runq at ??:?
?? ??:0
ast_inet_ntoa at ??:?
?? ??:0
?? ??:0

[2020-05-14 11:51:38] ERROR[15050] astobj2.c: FRACK!, Failed assertion bad magic                                                                                                                                                              number 0x0 for object 0x321e310 (0)
[2020-05-14 11:51:38] ERROR[15050] : Got 22 backtrace records

0: /usr/sbin/asterisk(__ast_assert_failed+0x84)
[Backtrace #1] Already seen, not resolving again.

[2020-05-14 11:51:57] ERROR[15050] astobj2.c: FRACK!, Failed assertion bad magic                                                                                                                                                              number 0x0 for object 0x321e310 (0)
[2020-05-14 11:51:57] ERROR[15050] : Got 14 backtrace records

0: /usr/sbin/asterisk(__ast_assert_failed+0x84)
[Backtrace #2]
__ast_assert_failed at ??:?
ast_fd_init at ??:?
ao2_object_get_lockaddr at ??:?
__ao2_ref at ??:?
__ast_format_cap_append at ??:?
ast_format_cap_append_from_cap at ??:?
?? ??:0
?? ??:0
?? ??:0
ast_sched_runq at ??:?
?? ??:0
ast_inet_ntoa at ??:?
?? ??:0
?? ??:0
dkgroot commented 4 years ago

@josh2893 Ok i had hoped the script would do a better job filtering out the backtrace lines. I guess it needs a little more work (:-)). But it does confirm that chan_sccp does not seem to be involved in the error. Looks like chan_sip only (at the moment).

josh2893 commented 4 years ago

@dkgroot So far no crashes since un-checking additional codecs.

Thanks!

dkgroot commented 4 years ago

Can you tell us which codec you unchecked ? That might help test and fix sccp_manager/chan_sccp.

duartero84 commented 3 years ago

Hello all, I had similar issue too few days ago. I have read this issue and I just keep ulaw enable for all extension. So far, no issue at all. sccp_ulaw