Open tsawyer opened 12 months ago
Here's the log from the the day of the crash. There was binary in this. That's not normal.
Memory corruption and possible NULL pointer dereference:
The latter could likely be fixed by changing:
for (l = myrpt->links.next; l != &myrpt->links; l = l->next) {
to
for (l = myrpt->links.next; l && l != &myrpt->links; l = l->next) {
but there are other issues as well.
Thread 1 (Thread 0x7f91fe941700 (LWP 66456)):
#0 0x00007f921d31d03f in __mklinklist (myrpt=0x7f921d3541c0 <rpt_vars>, mylink=0x0, buf=0x7f91fe93ebe0 "", flag=0) at app_rpt/rpt_link.c:395
l = 0x0
mode = 0 '000'
i = 244
spos = 32657
#1 0x00007f921d31f3fb in rpt_manager_do_xstat (ses=0x7f91fe940cb0, m=0x7f91fe9407e0, str=0x7f92100142a0 'U' <repeats 200 times>...) at app_rpt/rpt_manager.c:194
rxchan = 0x0
rxchanname = "dahdi/pseudo", '000' <repeats 243 times>
pseudo = 0
i = 0
j = 489982912
ns = 1
lbuf = '000' <repeats 240 times>...
strs = {0x0 <repeats 3716 times>, 0x7f921407ddba "255336272355376", 0x0, 0x0, 0x0, 0x0, 0x0, 0x8057b1d970 <error: Cannot access memory at address 0x8057b1d970>, 0x0, 0x0, 0x0, 0x0, 0xffff00001fa0 <error: Cannot access memory at address 0xffff00001fa0>, 0xffffffffffffffff <error: Cannot access memory at address 0xffffffffffffffff>, 0x0, 0x0, 0x0, 0xffffffff <error: Cannot access memory at address 0xffffffff>, 0x0, 0x0, 0x7f91fe93c0f0 "", 0x303134ffffffffff <error: Cannot access memory at address 0x303134ffffffffff>, 0x7f92208fe230 <__strftime_internal+4256> "L213T$HL213D$0L001345L001303351367357377377I211366E1377A203374002270002", 0x55615820ac06 "255ޭޭޭޭ336376", 0xf5 <error: Cannot access memory at address 0xf5>, 0x7f91fe93c4f0 "a", 0x7f9220a0bc60 <_nl_C_LC_TIME> "Yk235 222177", 0x7f91fe93c2f7 "", 0x0, 0x1 <error: Cannot access memory at address 0x1>, 0x7f92208b2d10 <__vsprintf_internal+160> "H213T$(306002", 0x600000000 <error: Cannot access memory at address 0x600000000>, 0x7f9220a0f4a0 <_nl_global_locale> "240266240 222177", 0x7f9220a0f4a0 <_nl_global_locale> "240266240 222177", 0x55615820ac06 "255ޭޭޭޭ336376", 0x55615820ac06 "255ޭޭޭޭ336376", 0x55615820ac09 "ޭޭޭ336376", 0xffffffffffffffff <error: Cannot access memory at address 0xffffffffffffffff>, 0x55615820ac06 "255ޭޭޭޭ336376", 0xffffffffffffffff <error: Cannot access memory at address 0xffffffffffffffff>, 0x0, 0x0, 0x0, 0x0, 0x0, 0x80000000f0 <error: Cannot access memory at address 0x80000000f0>, 0x3 <error: Cannot access memory at address 0x3>, 0x373130320000 <error: Cannot access memory at address 0x373130320000>, 0x5b4bf03eeefa4700 <error: Cannot access memory at address 0x5b4bf03eeefa4700>, 0x7f921d2fd5c4 <rpt_exec> "UH211345AUATSH201354b021", 0x13 <error: Cannot access memory at address 0x13>, 0x7f91fe93c3eb "18:18:07.410", 0x7f91fe93c3eb "18:18:07.410", 0x0, 0x55615820ac04 "255ޭޭޭޭޭ336376", 0x7f92209d4f23 "%H:%M:%S", 0x7f92208fdc17 <__strftime_internal+2695> "Lc|$`^_D213\$XM001347L213T$HE205333017205372017", 0x7f9220a0f4a0 <_nl_global_locale> "240266240 222177", 0x7f9220a0b540 <_IO_str_jumps> "", 0x0, 0x100 <error: Cannot access memory at address 0x100>, 0x7f91fe93c4f0 "a", 0x7f9220a0bc60 <_nl_C_LC_TIME> "Yk235 222177", 0x7f91fe93c2f7 "", 0x0, 0xf5 <error: Cannot access memory at address 0xf5>, 0x556157c33b00 <enddata> "033[0m", 0x600000000 <error: Cannot access memory at address 0x600000000>, 0x7f9220a0f4a0 <_nl_global_locale> "240266240 222177", 0x8 <error: Cannot access memory at address 0x8>, 0x7f9200000000 " ", 0x7f921d2fd5c4 <rpt_exec> "UH211345AUATSH201354b021", 0x556157991780 <__ast_pthread_mutex_unlock+39> "211E374213E374311303UH211345H203354060H211}370211u364H211U350H211M340L211E330L211M320H213U320H213E330H211326H211307350006!360377311303UH211345H203354060H211}370211u364H211U350H211M340L211E330H213E330H211307350275'360377311303UH211345H203354060H211}370211u364H211U350H211M340L211E330H213E330H211307350d,360377311303UH211345H203354060H211}370211u364H211U350H211M340L211E330H213E330H211307350373035360377311303UH211345H203354@H211}350211u344H211"..., 0x7f921d2fd5c4 <rpt_exec> "UH211345AUATSH201354b021", 0x556157991780 <__ast_pthread_mutex_unlock+39> "211E374213E374311303UH211345H203354060H211}370211u364H211U350H211M340L211E330L211M320H213U320H213E330H211326H211307350006!360377311303UH211345H203354060H211}370211u364H211U350H211M340L211E330H213E330H211307350275'360377311303UH211345H203354060H211}370211u364H211U350H211M340L211E330H213E330H211307350d,360377311303UH211345H203354060H211}370211u364H211U350H211M340L211E330H213E330H211307350373035360377311303UH211345H203354@H211}350211u344H211"..., 0x556157b00b33 "®lock", 0x5b4bf03eeefa4700 <error: Cannot access memory at address 0x5b4bf03eeefa4700>, 0x556157b00b33 "®lock", 0x7f921d2fd5c4 <rpt_exec> "UH211345AUATSH201354b021", 0x7f91fe93c2c0 "", 0x556157c33b00 <enddata> "033[0m", 0x7f92140e3a93 'ޭ' <repeats 93 times>, <incomplete sequence 336>, 0x7f920000f1f2 "255", 'ޭ' <repeats 98 times>, <incomplete sequence 336>..., 0x556157c33b00 <enddata> "033[0m", 0x5561578c5dd8 <region_free+261> "220311303UH211345H203354060H211}330H307", <incomplete sequence 360>, 0x55615820ab70 "P233", 0x556157c2b080 <minnows> "p3", 0x0, 0x7f920c004cb0 "340314357WaU", 0x7f91fe93c300 "200Ó376221177", 0x5561578c61d4 <__ast_free+102> <incomplete sequence 351233>, 0x556157b43e50 <__PRETTY_FUNCTION__.0> "ast_strftime_locale", 0x9d8208ff728 <error: Cannot access memory at address 0x9d8208ff728>, 0x556157b43b63 "stdtime/localtime.c", 0x55615820ac00 "255ޭޭޭޭޭޭޭ336376", 0x0, 0x55615820ab70 "P233", 0x7f91fe93c380 "330t003024222177", 0x556157a45874 <ast_strftime_locale+648> "213E314311303UH211345H203354 H211}370H211u360H211U350H211M340H213M340H213U350H213u360H213E370A270", 0x64e8014f <error: Cannot access memory at address 0x64e8014f>, 0x0, 0x7f91fe93c4f0 "a", 0x556157c194e9 <dateformat+9> "", 0x556157b44a00 <__PRETTY_FUNCTION__.5> "__ast_string_field_alloc_space", 0x556157b44824 "stringfields.c", 0x556157b44850 "to_alloc % ast_alignof(ast_string_field_allocation) == 0", 0x1000000d4 <error: Cannot access memory at address 0x1000000d4>, 0x7f91fe93c3b0 "@Ɠ376221177", 0x556157a4ac87 <__ast_string_field_alloc_space+137> "H213E340H;E350017227300017266300H205300tOH213E320H213", 0x556157b63fd0 <__PRETTY_FUNCTION__.27> "format_log_message_ap", 0x7265820ac09 <error: Cannot access memory at address 0x7265820ac09>, 0x556157b62599 "logger.c", 0x9 <error: Cannot access memory at address 0x9>, 0x7f92140309d8 "255", 'ޭ' <repeats 98 times>, <incomplete sequence 336>..., 0x7f9214030a08 "255", 'ޭ' <repeats 98 times>, <incomplete sequence 336>..., 0x556157b44822 <__ast_string_field_empty_buffer+2> "", 0x556157a75766 <ast_get_tid+30> "211E374213E374311303UH211345H201354", <incomplete sequence 340>, 0x7f9214030ab8 "255", 'ޭ' <repeats 98 times>, <incomplete sequence 336>..., 0x1039700000009 <error: Cannot access memory at address 0x1039700000009>, 0x7f91fe93c640 "240Ɠ376202b", 0x556157acd2ba <format_log_message_ap+2758> "H213U200211B020H213E200311303UH211345H201354", <incomplete sequence 360>, 0x7f921d3440a0 <__FUNCTION__.7> "rpt_exec", 0x1a270000000f <error: Cannot access memory at address 0x1a270000000f>, 0x7f921d34104d "app_rpt.c", 0x3ffffffff <error: Cannot access memory at address 0x3ffffffff>, 0x2d38302d33323032 <error: Cannot access memory at address 0x2d38302d33323032>, 0x38313a3831203432 <error: Cannot access memory at address 0x38313a3831203432>, 0x3031342e37303a <error: Cannot access memory at address 0x3031342e37303a>, 0x556157bf5680 <reglock> "", 0x556157b00b33 "®lock", 0x7f9220a0b180 <_IO_strn_jumps> "", 0x0, 0x556157b00ac9 "astmm.c", 0x11 <error: Cannot access memory at address 0x11>, 0x57f96821 <error: Cannot access memory at address 0x57f96821>, 0x7f91fe93c490 "304325/035222177", 0x5561578c5a9a <__ast_alloc_region+596> "H213E370H005220", 0x10000005a <error: Cannot access memory at address 0x10000005a>, 0x38 <error: Cannot access memory at address 0x38>, 0x556157b1e6e0 <__PRETTY_FUNCTION__.1> "ast_dns_txt_get_strings", 0x556157b1e670 "dns_txt.c", 0x200000063 <error: Cannot access memory at address 0x200000063>, 0x5b4bf03eeefa4700 <error: Cannot access memory at address 0x5b4bf03eeefa4700>, 0x7f91fe93c4c0 "3v260WaU", 0x7f921d2fd5c4 <rpt_exec> "UH211345AUATSH201354b021", 0x7f91fe93c6d0 "h64035222177", 0x556157c33b00 <enddata> "033[0m", 0x7f921d2fd5c4 <rpt_exec> "UH211345AUATSH201354b021", 0x556157991780 <__ast_pthread_mutex_unlock+39> "211E374213E374311303UH211345H203354060H211}370211u364H211U350H211M340L211E330L211M320H213U320H213E330H211326H211307350006!360377311303UH211345H203354060H211}370211u364H211U350H211M340L211E330H213E330H211307350275'360377311303UH211345H203354060H211}370211u364H211U350H211M340L211E330H213E330H211307350d,360377311303UH211345H203354060H211}370211u364H211U350H211M340L211E330H213E330H211307350373035360377311303UH211345H203354@H211}350211u344H211"..., 0x7f921d2fd5c4 <rpt_exec> "UH211345AUATSH201354b021", 0x556157991780 <__ast_pthread_mutex_unlock+39> "211E374213E374311303UH211345H203354060H211}370211u364H211U350H211M340L211E330L211M320H213U320H213E330H211326H211307350006!360377311303UH211345H203354060H211}370211u364H211U350H211M340L211E330H213E330H211307350275'360377311303UH211345H203354060H211}370211u364H211U350H211M340L211E330H213E330H211307350d,360377311303UH211345H203354060H211}370211u364H211U350H211M340L211E330H213E330H211307350373035360377311303UH211345H203354@H211}350211u344H211"..., 0x556157b00b33 "®lock", 0x5b4bf03eeefa4700 <error: Cannot access memory at address 0x5b4bf03eeefa4700>, 0x556157b00b33 "®lock", 0x7f921d2fd5c4 <rpt_exec> "UH211345AUATSH201354b021", 0x7f91fe93c520 "", 0x556157c33b00 <enddata> "033[0m", 0x64e8014f <error: Cannot access memory at address 0x64e8014f>, 0x64314 <error: Cannot access memory at address 0x64314>, 0x1200000007 <error: Cannot access memory at address 0x1200000007>, 0x1800000012 <error: Cannot access memory at address 0x1800000012>, 0x7b00000007 <error: Cannot access memory at address 0x7b00000007>, 0xeb00000004 <error: Cannot access memory at address 0xeb00000004>, 0x0, 0xffffffffffff9d90 <error: Cannot access memory at address 0xffffffffffff9d90>, 0x0, 0x64314 <error: Cannot access memory at address 0x64314>, 0x7f92140c4640 "255", 'ޭ' <repeats 98 times>, <incomplete sequence 336>..., 0x9 <error: Cannot access memory at address 0x9>, 0x7f92140309f0 "255", 'ޭ' <repeats 98 times>, <incomplete sequence 336>..., 0x7f921d3440a0 <__FUNCTION__.7> "rpt_exec", 0xa <error: Cannot access memory at address 0xa>, 0x7f92140309e8 "255", 'ޭ' <repeats 98 times>, <incomplete sequence 336>..., 0x7f921d34104d "app_rpt.c", 0x8 <error: Cannot access memory at address 0x8>, 0x7f9214030a00 "255", 'ޭ' <repeats 98 times>, <incomplete sequence 336>..., 0x556157b625b5 "WARNING", 0x18 <error: Cannot access memory at address 0x18>, 0x7f92140309e0 "255", 'ޭ' <repeats 98 times>, <incomplete sequence 336>..., 0x7f91fe93c3e0 "2023-08-24 18:18:07.410", 0x2b <error: Cannot access memory at address 0x2b>, 0x7f92140309f8 "255", 'ޭ' <repeats 98 times>, <incomplete sequence 336>..., 0x7f92140c4658 "255", 'ޭ' <repeats 98 times>, <incomplete sequence 336>..., 0x200057b29810 <error: Cannot access memory at address 0x200057b29810>, 0x2a00002000 <error: Cannot access memory at address 0x2a00002000>, 0x1ba <error: Cannot access memory at address 0x1ba>, 0x114030ab8 <error: Cannot access memory at address 0x114030ab8>, 0x1 <error: Cannot access memory at address 0x1>, 0x556157c38648 <logcond+40> "", 0x7f9214030aac "255", 'ޭ' <repeats 98 times>, <incomplete sequence 336>..., 0x7f921d2fd5c4 <rpt_exec> "UH211345AUATSH201354b021"...}
myrpt = 0x7f921d3541c0 <rpt_vars>
newvariable = 0x2b71d348239
connstate = 0x7f921d31e997 <_ast_strlen_zero+67> "205300u$H213M350213U344H213E360L215005K230002"
l = 0x7f921d348bc0 <__PRETTY_FUNCTION__.1>
s = 0x0
t = 0x5561578c5a9a <__ast_alloc_region+596>
s_head = {next = 0x7f91fe9348e0, prev = 0x7f91fe9348e0, peer = '000' <repeats 30 times>, name = '000' <repeats 299 times>, mode = 0 '000', outbound = 0 '000', reconnects = 0, thisconnected = 0 '000', connecttime = 0, chan_stat = {{last = {tv_sec = 0, tv_usec = 0}, total = 0, count = 0, largest = 0, largest_time = {tv_sec = 0, tv_usec = 0}}, {last = {tv_sec = 0, tv_usec = 0}, total = 0, count = 0, largest = 0, largest_time = {tv_sec = 0, tv_usec = 0}}, {last = {tv_sec = 0, tv_usec = 0}, total = 0, count = 0, largest = 0, largest_time = {tv_sec = 0, tv_usec = 0}}, {last = {tv_sec = 0, tv_usec = 0}, total = 0, count = 0, largest = 0, largest_time = {tv_sec = 0, tv_usec = 0}}, {last = {tv_sec = 0, tv_usec = 0}, total = 0, count = 0, largest = 0, largest_time = {tv_sec = 0, tv_usec = 0}}, {last = {tv_sec = 0, tv_usec = 0}, total = 0, count = 0, largest = 0, largest_time = {tv_sec = 0, tv_usec = 0}}, {last = {tv_sec = 0, tv_usec = 0}, total = 0, count = 0, largest = 0, largest_time = {tv_sec = 0, tv_usec = 0}}}}
node = 0x5561581b4786 "28020"
nrpts = 3
parrot_ena = 0x7f921d348379 "0"
sys_ena = 0x7f921d348379 "0"
tot_ena = 0x7f921d348379 "0"
link_ena = 0x7f921d348379 "0"
patch_ena = 0x7f921d348379 "0"
patch_state = 0x7f921d34837f "4"
sch_ena = 0x7f921d348379 "0"
user_funs = 0x7f921d348379 "0"
tail_type = 0x7f921d348379 "0"
iconns = 0x7f921d348379 "0"
tot_state = 0x7f921d348379 "0"
ider_state = 0x7f921d34837b "2"
tel_mode = 0x7f921d348379 "0"
__PRETTY_FUNCTION__ = "rpt_manager_do_xstat"
__FUNCTION__ = "rpt_manager_do_xstat"
#2 0x00007f921d3218cc in manager_rpt_status (s=0x7f91fe940cb0, m=0x7f91fe9407e0) at app_rpt/rpt_manager.c:762
i = 2
res = 1474216920
len = 1024
index = 2
uptime = 1471562687
hours = 21857
minutes = 1474216944
now = 1692926287
cmd = 0x5561587a64a9 "XStat"
str = 0x7f92100142a0 'U' <repeats 200 times>...
MGRCMD_RPTSTAT = MGRCMD_RPTSTAT
MGRCMD_NODESTAT = MGRCMD_NODESTAT
MGRCMD_XSTAT = MGRCMD_XSTAT
MGRCMD_SAWSTAT = MGRCMD_SAWSTAT
mct = {{cmd = 0x7f921d348bd3 "RptStat", index = 0}, {cmd = 0x7f921d348bdb "NodeStat", index = 1}, {cmd = 0x7f921d348be4 "XStat", index = 2}, {cmd = 0x7f921d348bea "SawStat", index = 3}, {cmd = 0x0, index = 0}}
nrpts = 3
__PRETTY_FUNCTION__ = "manager_rpt_status"
#3 0x0000556157ae2ce7 in process_message (s=0x7f91fe940cb0, m=0x7f91fe9407e0) at manager.c:7098
mod_ref = 0x556158ab5a10
acted = 0
ret = -1
act_found = 0x556157dec3f0
user = 0x0
username = 0x1c8600000002 <error: Cannot access memory at address 0x1c8600000002>
action = 0x5561581dc178 "RptStatus"
__PRETTY_FUNCTION__ = "process_message"
__FUNCTION__ = "process_message"
#4 0x0000556157ae36ac in do_message (s=0x7f91fe940cb0) at manager.c:7311
m = {hdrcount = 4, headers = {0x5561581dc170 "ACTION: RptStatus", 0x5561587a64a0 "COMMAND: XStat", 0x5561581b4780 "NODE: 28020", 0x556158493610 " ActionID: 850f026c-fc65-4cfc-aa0e-4e8df42d04b5", 0x0 <repeats 124 times>}}
header_buf = "000ActionID: 850f026c-fc65-4cfc-aa0e-4e8df42d04b5", '000' <repeats 977 times>
res = 1
hdr_loss = 0
now = 1692926287
__FUNCTION__ = "do_message"
__PRETTY_FUNCTION__ = "do_message"
#5 0x0000556157ae3ae7 in session_do (data=0x7f921405c7d0) at manager.c:7416
ser = 0x7f921405c7d0
session = 0x7f9210010760
s = {session = 0x7f9210010760, stream = 0x7f9214088bf8, tcptls_session = 0x7f921405c7d0, parsing = MESSAGE_OKAY, write_error = 0, hook = 0x0, lock = {mutex = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 1, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '000' <repeats 16 times>, "001", '000' <repeats 22 times>, __align = 0}, _track = 0x0, _flags = {tracking = 0, setup = 0}}}
res = 0
arg = 1
ser_remote_address_tmp = {ss = {ss_family = 10, __ss_padding = "271320", '000' <repeats 14 times>, "377377- ^240000000000000222177000000220f224376221177000000343b214WaU000000240f224376221177000000220375264WaU000000000000000000273000000000240364264WaU000000004000000000000000000000001000000000000000000000004000000000000000000000200006001020222177000000340f224376221177000", __ss_align = 93876569017118}, len = 28}
__FUNCTION__ = "session_do"
__PRETTY_FUNCTION__ = "session_do"
#6 0x0000556157a53758 in handle_tcptls_connection (data=0x7f921405c7d0) at tcptls.c:274
tcptls_session = 0x7f921405c7d0
ssl = 0x5561578b86e7 <ast_register_thread+261>
__FUNCTION__ = "handle_tcptls_connection"
__PRETTY_FUNCTION__ = "handle_tcptls_connection"
#7 0x0000556157a7289c in dummy_start (data=0x7f9214000e70) at utils.c:1574
__cancel_buf = {__cancel_jmp_buf = {{__cancel_jmp_buf = {0, 6093457885168087157, 140265037757198, 140265037757199, 140265018101632, 140265176819088, 6093457885210030197, 103897042223444085}, __mask_was_saved = 0}}, __pad = {0x7f91fe940ed0, 0x0, 0x5b4bf03eeefa4700, 0x0}}
__cancel_routine = 0x5561578b86ec <ast_unregister_thread>
__cancel_arg = 0x7f91fe941700
__not_first_call = 0
ret = 0x7f920809e590
a = {start_routine = 0x556157a530ca <handle_tcptls_connection>, data = 0x7f921405c7d0, name = 0x7f9214047330 "handle_tcptls_connection started at [ 342] tcptls.c ast_tcptls_server_root()"}
__PRETTY_FUNCTION__ = "dummy_start"
#8 0x00007f9220d20ea7 in start_thread (arg=<optimized out>) at pthread_create.c:477
ret = <optimized out>
pd = <optimized out>
unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140265018103552, -6074315203554627467, 140265037757198, 140265037757199, 140265018101632, 140265176819088, 6093457885178572917, 6095607962968338549}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
not_first_call = 0
#9 0x00007f922093ba2f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
No locals.
That code exists is on 4 lines. Change all 4 occurrences?
apps/app_rpt/rpt_link.c:393: for (l = myrpt->links.next; l != &myrpt->links; l = l->next) {
apps/app_rpt/rpt_link.c:443: for (l = myrpt->links.next; l != &myrpt->links; l = l->next) {
apps/app_rpt.c:4289: for (l = myrpt->links.next; l != &myrpt->links; l = l->next) {
apps/app_rpt.c:4302: for (l = myrpt->links.next; l != &myrpt->links; l = l->next) {
What kind of queue is myrpt->links? At first look, it seems the code is treating myrpt->links as a circular queue. In total there are 59 accesses to links.next in the code, and not all of them are doing null checks. If this is a circular queue, then myrpt->links.next could (should?) be initialized to the value of &myrpt->links, which means the loops you mention are fine, and the null pointer has another root cause that needs to be sought out.
User complained there was "crosstalk" between DAHDI/pseudo nodes (28020 and 28030 are on the same server) and it was going "nuts". Found Asterisk was not running. Started with
asterisk -c
and found DAHDI would not load. Thinking we had a DAHDI/kernel issue again I recompiledphreaknet dahdi -f
.Later I found this core file which leads me to think maybe I didn't need to reinstall DAHDI. Is this core file any clew to what happened?
core-asterisk-2023-08-25T01-18-07Z-full.txt