Node 2509 Coredump

[tsawyer]

This one is a little old. 2509 is the node here at my house. Runs chan_echolink, chan_voter and chan_simpleusb.

From the log.

[2023-08-03 02:03:56.780] NOTICE[1739758] dnsmgr.c: dnssrv: host 'register.allstarlink.org' changed from 34.105.111.212:443 to 162.248.92.131:443
[2023-08-03 03:03:56.779] NOTICE[1739758] dnsmgr.c: dnssrv: host 'register.allstarlink.org' changed from 162.248.92.131:443 to 34.105.111.212:443
[2023-08-03 03:13:02.716] WARNING[1758486] app_rpt.c: statpost to URL 'http://stats.allstarlink.org/uhandler?node=25334&time=1691057582&seqno=2061&keyed=0&keytime=32944' failed with code 502
[2023-08-03 03:20:03.148] ERROR[1739802] res_rpt_http_registrations.c: Failed to retrieve URL 'https://register.allstarlink.org/': HTTP response code 502
[2023-08-03 03:26:02.430] Asterisk 20.1.0 built by root @ NewASL on a x86_64 running Linux on 2023-06-29 16:39:52 UTC
[2023-08-03 03:26:02.594] NOTICE[1758899] loader.c: 125 modules will be loaded.
[2023-08-03 03:26:02.754] WARNING[1758899] ccss.c: Could not find valid ccss.conf file. Using cc_max_requests default
[2023-08-03 03:26:02.754] WARNING[1758899] ccss.c: Could not find valid ccss.conf file. Using cc_[state]_devstate defaults
[2023-08-03 03:26:02.769] ERROR[1758899] config_options.c: Unable to load config file 'cdr.conf'
[2023-08-03 03:26:02.769] NOTICE[1758899] cdr.c: Failed to process CDR configuration; using defaults
[2023-08-03 03:26:02.769] NOTICE[1758899] cdr.c: CDR simple logging enabled.
[2023-08-03 03:26:02.775] ERROR[1758899] config_options.c: Unable to load config file 'cel.conf'
[2023-08-03 03:26:02.775] NOTICE[1758899] cel.c: Failed to process CEL configuration; using defaults
[2023-08-03 03:26:02.775] NOTICE[1758899] dnsmgr.c: Managed DNS entries will be refreshed every 1800 seconds.
[2023-08-03 03:26:02.808] WARNING[1758899] features_config.c: Parkinglots are no longer configurable in features.conf; parking is now handled by res_parking.conf
[2023-08-03 03:26:02.808] WARNING[1758899] features_config.c: The option 'parkext' is no longer configurable in features.conf.
[2023-08-03 03:26:02.808] WARNING[1758899] features_config.c: The option 'parkpos' is no longer configurable in features.conf.
[2023-08-03 03:26:02.808] WARNING[1758899] features_config.c: The option 'context' is no longer configurable in features.conf.

There were a number of restarts that day.

nodes=&apprptvers=3.0.0&apprptuptime=15477&totalkerchunks=1&totalkeyups=1&totaltxtime=7&timeouts=0&totalexecdcommands=0' failed with code 502
[2023-08-03 07:56:04.727] NOTICE[1758921] dnsmgr.c: dnssrv: host 'register.allstarlink.org' changed from 34.105.111.212:443 to 162.248.92.131:443
[2023-08-03 08:11:10.075] WARNING[1758959] app_rpt.c: rpt_thread restarted on node 2509
[2023-08-03 08:11:10.075] WARNING[1758959] app_rpt.c: rpt_thread restarted on node 25334
[2023-08-03 08:11:10.075] WARNING[1774193] channel.c: Channel allocation failed: Refusing due to active shutdown
[2023-08-03 08:11:10.075] WARNING[1774193] chan_simpleusb.c: Unable to create new usb channel
[2023-08-03 08:11:10.075] WARNING[1774193] app_rpt.c: Unable to obtain Rx channel
[2023-08-03 08:11:10.081] WARNING[1774192] channel.c: Channel allocation failed: Refusing due to active shutdown
[2023-08-03 08:11:10.081] ERROR[1774192] chan_voter.c: Cant alloc new asterisk channel
[2023-08-03 08:11:10.081] WARNING[1774192] app_rpt.c: Unable to obtain Rx channel
[2023-08-03 08:11:12.362] Asterisk 20.1.0 built by root @ NewASL on a x86_64 running Linux on 2023-06-29 16:39:52 UTC
[2023-08-03 08:11:12.514] NOTICE[1774195] loader.c: 125 modules will be loaded.
[2023-08-03 08:11:12.723] WARNING[1774195] ccss.c: Could not find valid ccss.conf file. Using cc_max_requests default
[2023-08-03 08:11:12.723] WARNING[1774195] ccss.c: Could not find valid ccss.conf file. Using cc_[state]_devstate defaults
[2023-08-03 08:11:12.738] ERROR[1774195] config_options.c: Unable to load config file 'cdr.conf'

core-asterisk-2023-08-03T10-25-55Z-full.txt

[tsawyer]

valgrind -s --leak-check=full --show-leak-kinds=all asterisk -g
==2028789== Memcheck, a memory error detector
==2028789== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==2028789== Using Valgrind-3.16.1 and LibVEX; rerun with -h for copyright info
==2028789== Command: asterisk -g
==2028789==
==2028789==
==2028789== HEAP SUMMARY:
==2028789==     in use at exit: 3,818 bytes in 18 blocks
==2028789==   total heap usage: 64 allocs, 46 frees, 85,741 bytes allocated
==2028789==
==2028789== 40 bytes in 1 blocks are possibly lost in loss record 1 of 3
==2028789==    at 0x483AB65: calloc (vg_replace_malloc.c:760)
==2028789==    by 0x52046E5: cap_init (in /usr/lib/x86_64-linux-gnu/libcap.so.2.44)
==2028789==    by 0x5206060: cap_from_text (in /usr/lib/x86_64-linux-gnu/libcap.so.2.44)
==2028789==    by 0x14E441: main (asterisk.c:4025)
==2028789==
==2028789== 93 bytes in 1 blocks are still reachable in loss record 2 of 3
==2028789==    at 0x483AB65: calloc (vg_replace_malloc.c:760)
==2028789==    by 0x1745E7: __ast_repl_calloc (astmm.c:1537)
==2028789==    by 0x1745E7: __ast_calloc (astmm.c:1607)
==2028789==    by 0x300310: cfmtime_new (config.c:1664)
==2028789==    by 0x300310: config_text_file_load (config.c:2236)
==2028789==    by 0x2FEAA5: ast_config_internal_load (config.c:3231)
==2028789==    by 0x3018AE: ast_config_load2 (config.c:3252)
==2028789==    by 0x231725: load_asterisk_conf (options.c:235)
==2028789==    by 0x14E1D4: main (asterisk.c:3610)
==2028789==
==2028789== 3,685 bytes in 16 blocks are still reachable in loss record 3 of 3
==2028789==    at 0x483AB65: calloc (vg_replace_malloc.c:760)
==2028789==    by 0x21B251: ast_module_register (loader.c:664)
==2028789==    by 0x3566C4: __libc_csu_init (in /usr/sbin/asterisk)
==2028789==    by 0x53E8C99: (below main) (libc-start.c:264)
==2028789==
==2028789== LEAK SUMMARY:
==2028789==    definitely lost: 0 bytes in 0 blocks
==2028789==    indirectly lost: 0 bytes in 0 blocks
==2028789==      possibly lost: 40 bytes in 1 blocks
==2028789==    still reachable: 3,778 bytes in 17 blocks
==2028789==         suppressed: 0 bytes in 0 blocks
==2028789==
==2028789== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)

[tsawyer]

When running valgrind with Asterisk in the foreground valgrind -s --leak-check=full --show-leak-kinds=all asterisk -g -f Asterisk doesn't crash until exit. Here's my ssh terminal session capture.

iTerm2 Session Aug 8, 2023 at 10:41:13 AM.txt

[InterLinked1]

I think the first thing is an issue with res_rpt_http_registrations, whenever there's a 5xx error. I've noticed that pattern in a few reports now. I'll try to see what's up with that.

valgrind often results in different behavior so not surprising that it didn't crash, though I'd have expected to see an error, I think the 1 error might be the "possible" leak.