FPP becomes unresponsive after a few days, light show stutters and hangs/flickers

[jkriegshauser]

NOTE: IF YOU DO NOT ATTACH YOUR LOG FILES, THEN THERE IS A HIGHER PROBABILITY THE ISSUE WON'T BE RESOLVED. (Steps at end)

Please answer the following:

FPP Version and Hardware: FPP v7.2 (latest) running on Kulp K16B

Describe the bug and Steps to reproduce: Reproduction steps unknown, seems to happen every 3-4 days running the show. Web interface becomes unresponsive, even the buttons and screen on the board take multiple seconds to respond. When this happened on 10/27 (Squid Game.fseq was playing at the time) I went to reboot the controller at ~9:10 pm and took seconds but managed to select Reboot from the menu on the K16B board, but it didn't reboot after minutes of waiting, so hard power reset was done. This morning, the web interface was also sluggish so we restarted. Both 10/27 and last night were playing our Halloween 2 playlist.

Expected behavior: FPP should not hang and become responsive.

Additional context: Add any other context about the problem here.

Additional Attachments Logs attached. FPP_Logs_20231031.1853.zip

If applicable, add screenshots to help explain your problem.

[darylc]

lots of errors around that time: 2023-10-27 21:12:04.770 (27378) [MediaOut] /opt/fpp/src/mediaoutput/SDLOut.cpp:273: Stalled audio, buffers still filling. 2023-10-27 21:12:05.731 (1952) [MediaOut] /opt/fpp/src/mediaoutput/SDLOut.cpp:1149: Audio has stalled 0 2023-10-27 21:12:06.617 (1952) [MediaOut] /opt/fpp/src/mediaoutput/SDLOut.cpp:1149: Audio has stalled 0 2023-10-27 21:12:10.283 (27378) [MediaOut] /opt/fpp/src/mediaoutput/SDLOut.cpp:273: Stalled audio, buffers still filling. 2023-10-27 21:12:10.940 (1952) [MediaOut] /opt/fpp/src/mediaoutput/SDLOut.cpp:1149: Audio has stalled 0 2023-10-27 21:12:11.529 (1952) [MediaOut] /opt/fpp/src/mediaoutput/SDLOut.cpp:1149: Audio has stalled 0 2023-10-27 21:12:12.299 (1952) [MediaOut] /opt/fpp/src/mediaoutput/SDLOut.cpp:1149: Audio has stalled 0 2023-10-27 21:12:12.627 (27378) [MediaOut] /opt/fpp/src/mediaoutput/SDLOut.cpp:273: Stalled audio, buffers still filling. 2023-10-27 21:12:14.916 (1952) [MediaOut] /opt/fpp/src/mediaoutput/SDLOut.cpp:1149: Audio has stalled 0 2023-10-27 21:12:16.470 (27378) [MediaOut] /opt/fpp/src/mediaoutput/SDLOut.cpp:273: Stalled audio, buffers still filling. 2023-10-27 21:12:17.086 (1952) [MediaOut] /opt/fpp/src/mediaoutput/SDLOut.cpp:1149: Audio has stalled 0 2023-10-27 21:12:17.750 (1952) [MediaOut] /opt/fpp/src/mediaoutput/SDLOut.cpp:1149: Audio has stalled 0 2023-10-27 21:12:21.149 (27378) [MediaOut] /opt/fpp/src/mediaoutput/SDLOut.cpp:273: Stalled audio, buffers still filling. 2023-10-27 21:12:27.371 (27378) [MediaOut] /opt/fpp/src/mediaoutput/SDLOut.cpp:273: Stalled audio, buffers still filling. 2023-10-27 21:12:27.963 (1952) [MediaOut] /opt/fpp/src/mediaoutput/SDLOut.cpp:1149: Audio has stalled 0

Often this happens because you are using a usb audio device that is not receiving enough power. Try adding a powered USB hub between your beagle and USB audio stick.

[jkriegshauser]

@darylc interesting. Though this controller usually functions fine, and worked all last year on FPP 6.x in the same configuration 🤔

[ebuechner]

I had the same experience with many of the same errors popping up in my FPPD file. I know of one other person who is having the same problem.

[darylc]

@ebuechner Can you please be more specific

[ebuechner]

These are some of the errors that I had before FPP restarted itself. I wasn't aware that it was even happening until the next night when someone said that they were watching the show and it was glitching so bad that they just left several days before this I ended up having to reinstall FPP because it became completely unresponsive it won't reboot. I'm currently running 7.2.1 on a pi4.

2023-10-22 00:00:00.060 (1699) [Schedule] /opt/fpp/src/Scheduler.cpp:724: + *G 2023-10-13 - 2028-10-31 Everyday 18:00:00 - 24:00:00 Holloween 2023-10-22 19:58:12.743 (1699) [MediaOut] /opt/fpp/src/mediaoutput/SDLOut.cpp:1149: Audio has stalled 0 2023-10-22 19:58:13.122 (18373) [MediaOut] /opt/fpp/src/mediaoutput/SDLOut.cpp:273: Stalled audio, buffers still filling. 2023-10-22 19:58:34.495 (1699) [MediaOut] /opt/fpp/src/mediaoutput/SDLOut.cpp:1149: Audio has stalled 0 2023-10-22 19:58:34.738 (18373) [MediaOut] /opt/fpp/src/mediaoutput/SDLOut.cpp:273: Stalled audio, buffers still filling. 2023-10-22 19:58:43.201 (16733) [ChannelOut] /opt/fpp/src/channeloutput/channeloutputthread.cpp:212: SLOW Output Thread: Loop: 444500us, Send: 65208us, Read: 13571us, Process: 19972us, FrameNum: 1669 2023-10-22 19:58:43.693 (16733) [ChannelOut] /opt/fpp/src/channeloutput/channeloutputthread.cpp:212: SLOW Output Thread: Loop: 25000us, Send: 103715us, Read: 11447us, Process: 31294us, FrameNum: 1670 2023-10-22 19:58:48.322 (1699) [MediaOut] /opt/fpp/src/mediaoutput/SDLOut.cpp:1149: Audio has stalled 0 2023-10-22 19:58:48.856 (16733) [ChannelOut] /opt/fpp/src/channeloutput/channeloutputthread.cpp:212: SLOW Output Thread: Loop: 25000us, Send: 66223us, Read: 13830us, Process: 89us, FrameNum: 1871 2023-10-22 19:58:51.808 (1699) [MediaOut] /opt/fpp/src/mediaoutput/SDLOut.cpp:1149: Audio has stalled 0 2023-10-22 19:58:52.234 (16733) [ChannelOut] /opt/fpp/src/channeloutput/channeloutputthread.cpp:212: SLOW Output Thread: Loop: 23500us, Send: 103095us, Read: 13136us, Process: 59us, FrameNum: 1925 2023-10-22 19:58:53.457 (1699) [MediaOut] /opt/fpp/src/mediaoutput/SDLOut.cpp:1149: Audio has stalled 0 2023-10-22 19:58:53.640 (18373) [MediaOut] /opt/fpp/src/mediaoutput/SDLOut.cpp:273: Stalled audio, buffers still filling. 2023-10-22 19:58:56.339 (16733) [ChannelOut] /opt/fpp/src/channeloutput/channeloutputthread.cpp:212: SLOW Output Thread: Loop: 41250us, Send: 54013us, Read: 33us, Process: 41us, FrameNum: 2064 2023-10-22 19:59:01.570 (1699) [MediaOut] /opt/fpp/src/mediaoutput/SDLOut.cpp:1149: Audio has stalled 0 2023-10-22 19:59:02.574 (1699) [MediaOut] /opt/fpp/src/mediaoutput/SDLOut.cpp:1149: Audio has stalled 0 2023-10-22 19:59:08.080 (1699) [MediaOut] /opt/fpp/src/mediaoutput/SDLOut.cpp:1149: Audio has stalled 0 2023-10-22 19:59:15.545 (16733) [ChannelOut] /opt/fpp/src/channeloutput/channeloutputthread.cpp:212: SLOW Output Thread: Loop: 172250us, Send: 65080us, Read: 15912us, Process: 56us, FrameNum: 2490 2023-10-22 19:59:30.111 (1699) [MediaOut] /opt/fpp/src/mediaoutput/SDLOut.cpp:1149: Audio has stalled 0 2023-10-22 19:59:32.589 (1699) [MediaOut] /opt/fpp/src/mediaoutput/SDLOut.cpp:1149: Audio has stalled 0 2023-10-22 19:59:32.796 (18373) [MediaOut] /opt/fpp/src/mediaoutput/SDLOut.cpp:273: Stalled audio, buffers still filling. 2023-10-22 19:59:33.090 (16733) [ChannelOut] /opt/fpp/src/channeloutput/channeloutputthread.cpp:212: SLOW Output Thread: Loop: 57000us, Send: 86126us, Read: 21610us, Process: 89us, FrameNum: 2863 2023-10-22 19:59:54.424 (16733) [ChannelOut] /opt/fpp/src/channeloutput/channeloutputthread.cpp:212: SLOW Output Thread: Loop: 24500us, Send: 78057us, Read: 91us, Process: 23269us, FrameNum: 3535 2023-10-22 20:00:02.907 (1699) [MediaOut] /opt/fpp/src/mediaoutput/SDLOut.cpp:1149: Audio has stalled 0 2023-10-22 20:00:04.414 (18373) [MediaOut] /opt/fpp/src/mediaoutput/SDLOut.cpp:273: Stalled audio, buffers still filling. 2023-10-22 20:00:05.598 (1699) [MediaOut] /opt/fpp/src/mediaoutput/SDLOut.cpp:1149: Audio has stalled 0 2023-10-22 20:00:07.445 (1699) [MediaOut] /opt/fpp/src/mediaoutput/SDLOut.cpp:1149: Audio has stalled 0 2023-10-22 20:00:08.200 (16733) [ChannelOut] /opt/fpp/src/channeloutput/channeloutputthread.cpp:212: SLOW Output Thread: Loop: 20000us, Send: 86850us, Read: 49us, Process: 74us, FrameNum: 3805 2023-10-22 20:00:11.686 (16733) [ChannelOut] /opt/fpp/src/channeloutput/channeloutputthread.cpp:212: SLOW Output Thread: Loop: 297750us, Send: 73783us, Read: 31451us, Process: 64us, FrameNum: 3860 2023-10-22 20:00:15.130 (16733) [ChannelOut] /opt/fpp/src/channeloutput/channeloutputthread.cpp:212: SLOW Output Thread: Loop: 338250us, Send: 66095us, Read: 28142us, Process: 9076us, FrameNum: 3867 2023-10-22 20:00:16.265 (16733) [ChannelOut] /opt/fpp/src/channeloutput/channeloutputthread.cpp:212: SLOW Output Thread: Loop: 375500us, Send: 43397us, Read: 17192us, Process: 16592us, FrameNum: 3869 2023-10-22 20:00:17.141 (16733) [ChannelOut] /opt/fpp/src/channeloutput/channeloutputthread.cpp:212: SLOW Output Thread: Loop: 405000us, Send: 101480us, Read: 49us, Process: 55us, FrameNum: 3871 2023-10-22 20:00:19.300 (16733) [ChannelOut] /opt/fpp/src/channeloutput/channeloutputthread.cpp:212: SLOW Output Thread: Loop: 461000us, Send: 98323us, Read: 7589us, Process: 14313us, FrameNum: 3876 2023-10-22 20:00:19.771 (16733) [ChannelOut] /opt/fpp/src/channeloutput/channeloutputthread.cpp:212: SLOW Output Thread: Loop: 470000us, Send: 104511us, Read: 7108us, Process: 8709us, FrameNum: 3877 2023-10-22 20:00:43.674 (1699) [MediaOut] /opt/fpp/src/mediaoutput/SDLOut.cpp:1149: Audio has stalled 0 2023-10-22 20:00:44.090 (18373) [MediaOut] /opt/fpp/src/mediaoutput/SDLOut.cpp:273: Stalled audio, buffers still filling. 2023-10-22 20:00:44.865 (1699) [MediaOut] /opt/fpp/src/mediaoutput/SDLOut.cpp:1149: Audio has stalled 0 2023-10-22 20:00:45.884 (18373) [MediaOut] /opt/fpp/src/mediaoutput/SDLOut.cpp:273: Stalled audio, buffers still filling. 2023-10-22 20:00:47.800 (16733) [ChannelOut] /opt/fpp/src/channeloutput/channeloutputthread.cpp:212: SLOW Output Thread: Loop: 37750us, Send: 43627us, Read: 34642us, Process: 43457us, FrameNum: 4844 2023-10-22 20:00:48.725 (1699) [MediaOut] /opt/fpp/src/mediaoutput/SDLOut.cpp:1149: Audio has stalled 0 2023-10-22 20:00:52.080 (16733) [ChannelOut] /opt/fpp/src/channeloutput/channeloutputthread.cpp:212: SLOW Output Thread: Loop: 122750us, Send: 112600us, Read: 38us, Process: 80us, FrameNum: 4908 2023-10-22 20:01:04.103 (16733) [ChannelOut] /opt/fpp/src/channeloutput/channeloutputthread.cpp:212: SLOW Output Thread: Loop: 25000us, Send: 77323us, Read: 16070us, Process: 109us, FrameNum: 5353 2023-10-22 20:01:25.518 (1699) [MediaOut] /opt/fpp/src/mediaoutput/SDLOut.cpp:1149: Audio has stalled 0 2023-10-22 20:01:26.261 (18373) [MediaOut] /opt/fpp/src/mediaoutput/SDLOut.cpp:273: Stalled audio, buffers still filling. 2023-10-22 20:01:27.405 (17680) [ChannelOut] /opt/fpp/src/channeloutput/channeloutputthread.cpp:212: SLOW Output Thread: Loop: 25000us, Send: 100436us, Read: 32us, Process: 41527us, FrameNum: 212 2023-10-22 20:01:36.094 (30441) [ChannelOut] /opt/fpp/src/channeloutput/UDPOutput.cpp:515: sendmmsg() failed for UDP output (key: 1705A8C0 output count: 19/19 time: 112 ms errCount: 1) with error: 0 Success 2023-10-22 20:01:36.094 (30440) [ChannelOut] /opt/fpp/src/channeloutput/UDPOutput.cpp:515: sendmmsg() failed for UDP output (key: 1405A8C0 output count: 40/40 time: 112 ms errCount: 1) with error: 0 Success 2023-10-22 20:01:36.094 (30443) [ChannelOut] /opt/fpp/src/channeloutput/UDPOutput.cpp:515: sendmmsg() failed for UDP output (key: 1505A8C0 output count: 48/48 time: 112 ms errCount: 1) with error: 0 Success 2023-10-22 20:01:43.127 (17680) [ChannelOut] /opt/fpp/src/channeloutput/channeloutputthread.cpp:212: SLOW Output Thread: Loop: 27000us, Send: 75098us, Read: 48us, Process: 88us, FrameNum: 232 2023-10-22 20:02:06.176 (30440) [ChannelOut] /opt/fpp/src/channeloutput/UDPOutput.cpp:515: sendmmsg() failed for UDP output (key: 1505A8C0 output count: 48/48 time: 112 ms errCount: 1) with error: 0 Success 2023-10-22 20:02:06.176 (30441) [ChannelOut] /opt/fpp/src/channeloutput/UDPOutput.cpp:515: sendmmsg() failed for UDP output (key: 1405A8C0 output count: 40/40 time: 111 ms errCount: 1) with error: 0 Success 2023-10-22 20:02:06.176 (30443) [ChannelOut] /opt/fpp/src/channeloutput/UDPOutput.cpp:515: sendmmsg() failed for UDP output (key: 1705A8C0 output count: 19/19 time: 112 ms errCount: 1) with error: 0 Success 2023-10-22 20:02:29.876 (17680) [ChannelOut] /opt/fpp/src/channeloutput/channeloutputthread.cpp:212: SLOW Output Thread: Loop: 40250us, Send: 120080us, Read: 56us, Process: 66us, FrameNum: 300 2023-10-22 20:02:45.009 (1699) [MediaOut] /opt/fpp/src/mediaoutput/SDLOut.cpp:1149: Audio has stalled 0 2023-10-22 20:04:02.733 (30440) [ChannelOut] /opt/fpp/src/channeloutput/UDPOutput.cpp:515: sendmmsg() failed for UDP output (key: 1405A8C0 output count: 40/40 time: 121 ms errCount: 1) with error: 0 Success 2023-10-22 20:04:02.733 (30443) [ChannelOut] /opt/fpp/src/channeloutput/UDPOutput.cpp:515: sendmmsg() failed for UDP output (key: 1705A8C0 output count: 19/19 time: 122 ms errCount: 1) with error: 0 Success 2023-10-22 20:04:02.733 (30441) [ChannelOut] /opt/fpp/src/channeloutput/UDPOutput.cpp:515: sendmmsg() failed for UDP output (key: 1505A8C0 output count: 48/48 time: 122 ms errCount: 1) with error: 0 Success 2023-10-22 20:04:15.551 (17680) [ChannelOut] /opt/fpp/src/channeloutput/channeloutputthread.cpp:212: SLOW Output Thread: Loop: 15500us, Send: 110389us, Read: 53us, Process: 12687us, FrameNum: 411 2023-10-22 20:04:46.880 (17719) [General] /opt/fpp/src/log.cpp:349: ========================================= 2023-10-22 20:04:46.894 (17719) [General] /opt/fpp/src/log.cpp:350: FPP 7.2-1-g1ca81bce 2023-10-22 20:04:46.895 (17719) [General] /opt/fpp/src/log.cpp:351: Branch: v7.2

[ebuechner]

After a few days of everything working OK I had this pop up.

2023-10-29 00:00:00.047 (17719) [Schedule] /opt/fpp/src/Scheduler.cpp:724: + *G 2023-10-13 - 2023-10-31 Everyday 18:00:00 - 24:00:00 Holloween 2023-10-29 22:06:49.320 (17765) [ChannelOut] /opt/fpp/src/channeloutput/UDPOutput.cpp:515: sendmmsg() failed for UDP output (key: 1505A8C0 output count: 48/48 time: 102 ms errCount: 1) with error: 0 Success 2023-10-29 22:06:49.320 (17764) [ChannelOut] /opt/fpp/src/channeloutput/UDPOutput.cpp:515: sendmmsg() failed for UDP output (key: 1405A8C0 output count: 40/40 time: 102 ms errCount: 1) with error: 0 Success 2023-10-29 22:06:49.320 (17763) [ChannelOut] /opt/fpp/src/channeloutput/UDPOutput.cpp:515: sendmmsg() failed for UDP output (key: 1705A8C0 output count: 19/19 time: 102 ms errCount: 1) with error: 0 Success 2023-10-29 22:07:06.339 (17764) [ChannelOut] /opt/fpp/src/channeloutput/UDPOutput.cpp:515: sendmmsg() failed for UDP output (key: 1705A8C0 output count: 19/19 time: 107 ms errCount: 1) with error: 0 Success 2023-10-29 22:07:06.339 (17765) [ChannelOut] /opt/fpp/src/channeloutput/UDPOutput.cpp:515: sendmmsg() failed for UDP output (key: 1505A8C0 output count: 48/48 time: 107 ms errCount: 1) with error: 0 Success 2023-10-29 22:07:06.339 (17763) [ChannelOut] /opt/fpp/src/channeloutput/UDPOutput.cpp:515: sendmmsg() failed for UDP output (key: 1405A8C0 output count: 40/40 time: 107 ms errCount: 1) with error: 0 Success 2023-10-29 22:07:41.283 (17765) [ChannelOut] /opt/fpp/src/channeloutput/UDPOutput.cpp:515: sendmmsg() failed for UDP output (key: 1405A8C0 output count: 40/40 time: 160 ms errCount: 1) with error: 0 Success 2023-10-29 22:07:41.283 (17763) [ChannelOut] /opt/fpp/src/channeloutput/UDPOutput.cpp:515: sendmmsg() failed for UDP output (key: 1705A8C0 output count: 19/19 time: 160 ms errCount: 1) with error: 0 Success 2023-10-29 22:07:41.283 (17764) [ChannelOut] /opt/fpp/src/channeloutput/UDPOutput.cpp:515: sendmmsg() failed for UDP output (key: 1505A8C0 output count: 48/48 time: 160 ms errCount: 1) with error: 0 Success 2023-10-29 22:07:54.260 (17765) [ChannelOut] /opt/fpp/src/channeloutput/UDPOutput.cpp:515: sendmmsg() failed for UDP output (key: 1405A8C0 output count: 40/40 time: 152 ms errCount: 1) with error: 0 Success 2023-10-29 22:07:54.260 (17763) [ChannelOut] /opt/fpp/src/channeloutput/UDPOutput.cpp:515: sendmmsg() failed for UDP output (key: 1705A8C0 output count: 19/19 time: 152 ms errCount: 1) with error: 0 Success 2023-10-29 22:07:54.260 (17764) [ChannelOut] /opt/fpp/src/channeloutput/UDPOutput.cpp:515: sendmmsg() failed for UDP output (key: 1505A8C0 output count: 48/48 time: 152 ms errCount: 1) with error: 0 Success 2023-10-29 22:08:02.426 (17764) [ChannelOut] /opt/fpp/src/channeloutput/UDPOutput.cpp:515: sendmmsg() failed for UDP output (key: 1405A8C0 output count: 40/40 time: 114 ms errCount: 1) with error: 0 Success 2023-10-29 22:08:02.426 (17765) [ChannelOut] /opt/fpp/src/channeloutput/UDPOutput.cpp:515: sendmmsg() failed for UDP output (key: 1505A8C0 output count: 48/48 time: 114 ms errCount: 1) with error: 0 Success 2023-10-29 22:08:02.426 (17763) [ChannelOut] /opt/fpp/src/channeloutput/UDPOutput.cpp:515: sendmmsg() failed for UDP output (key: 1705A8C0 output count: 19/19 time: 114 ms errCount: 1) with error: 0 Success 2023-10-29 22:08:22.013 (17764) [ChannelOut] /opt/fpp/src/channeloutput/UDPOutput.cpp:515: sendmmsg() failed for UDP output (key: 1505A8C0 output count: 48/48 time: 161 ms errCount: 1) with error: 0 Success 2023-10-29 22:08:22.013 (17765) [ChannelOut] /opt/fpp/src/channeloutput/UDPOutput.cpp:515: sendmmsg() failed for UDP output (key: 1405A8C0 output count: 40/40 time: 161 ms errCount: 1) with error: 0 Success 2023-10-29 22:08:22.013 (17763) [ChannelOut] /opt/fpp/src/channeloutput/UDPOutput.cpp:515: sendmmsg() failed for UDP output (key: 1705A8C0 output count: 19/19 time: 161 ms errCount: 1) with error: 0 Success 2023-10-29 22:09:37.856 (17763) [ChannelOut] /opt/fpp/src/channeloutput/UDPOutput.cpp:515: sendmmsg() failed for UDP output (key: 1405A8C0 output count: 40/40 time: 116 ms errCount: 1) with error: 0 Success 2023-10-29 22:09:37.856 (17764) [ChannelOut] /opt/fpp/src/channeloutput/UDPOutput.cpp:515: sendmmsg() failed for UDP output (key: 1705A8C0 output count: 19/19 time: 116 ms errCount: 1) with error: 0 Success 2023-10-29 22:09:37.856 (17765) [ChannelOut] /opt/fpp/src/channeloutput/UDPOutput.cpp:515: sendmmsg() failed for UDP output (key: 1505A8C0 output count: 48/48 time: 116 ms errCount: 1) with error: 0 Success 2023-10-29 22:09:40.878 (17765) [ChannelOut] /opt/fpp/src/channeloutput/UDPOutput.cpp:515: sendmmsg() failed for UDP output (key: 1405A8C0 output count: 40/40 time: 124 ms errCount: 1) with error: 0 Success

it's inconsistent and I can't figure out a way to force a failure to figure out what's going on.

[darylc]

@ebuechner @jkriegshauser Please confirm how you go with powered USB hubs for the audio stalling issue. We've seen this fix the issue before (you can find the examples in github if you like).

@ebuechner That looks like a network related problem, can you describe your network connections on the FPP device? Are there any USB network interfaces involved, if so please place them on the powered USB hub as well.

[dkulp]

When these things happen, can you ssh in an run "ps -aef"? I'm wondering if a network glitch is causing a ton of processes waiting on something, like possibly a git call or something.

[ebuechner]

I'm using a sound blaster Play 3 for audio. The same that I've been using for many years now. Not on a powered hub.

My network connection is also remained the same for many years a wired connection to a gigabit switch that's on a dedicated show network 192.168.5.xxx from the pi 4 built-in ethernet connection eth0.

As far as power goes, I have four raspberry pi 4 running in one box. One master three remotes all powered from one meanwell power supply. This configuration hasn't changed and when I look at the logs for the remotes I don't see errors. I'm only guessing but if I had power issues I think I would see something in the logs for the same time and day on everything that's connected?

[darylc]

@jkriegshauser Please see Dan's request above

[jkriegshauser]

I did see that, just haven't had it happen again and haven't had a chance to try to repro.

On Sat, Nov 18, 2023 at 5:50 PM darylc @.***> wrote:

@jkriegshauser https://github.com/jkriegshauser Please see Dan's request above

— Reply to this email directly, view it on GitHub https://github.com/FalconChristmas/fpp/issues/1625#issuecomment-1817715547, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAM6MXZ7NNKKW3UBOWX4SC3YFFQVDAVCNFSM6AAAAAA6YGPWJOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMJXG4YTKNJUG4 . You are receiving this because you were mentioned.Message ID: @.***>

[jkriegshauser]

Alright, I have this happening again right now. Pinging the iface it is responding in <1ms, but ssh through PuTTY is not connecting and the web interface is not responding. I'll hard power-cycle it. Logs attached: FPP_Logs_20231206.0350.zip

Looks like a bunch of this:

2023-12-05 18:10:03.723 (1963) [MediaOut] /opt/fpp/src/mediaoutput/SDLOut.cpp:1149: Audio has stalled   0
2023-12-05 18:10:04.388 (17378) [MediaOut] /opt/fpp/src/mediaoutput/SDLOut.cpp:273: Stalled audio, buffers still filling.
2023-12-05 18:10:04.983 (1963) [MediaOut] /opt/fpp/src/mediaoutput/SDLOut.cpp:1149: Audio has stalled   0
2023-12-05 18:10:05.471 (17378) [MediaOut] /opt/fpp/src/mediaoutput/SDLOut.cpp:273: Stalled audio, buffers still filling.

which is also what we were seeing with the Halloween show.

This device has been up since 11/25 which is when we started the show.

[cabrower]

@jkriegshauser i had the same thing happen tonight. My Kulp K32 which is my master player has been running for a few days no problem. The around 9:40 PM EST the lights all went black except my "Tune To" sign which is an eseq. The K32 was unresponsive via the web and via the controls on the board. I rebooted the K32 and the time it synced was for tomorrow at like 3PM EST, which was weird.

Upon inspecting the logs I noticed the same errors you are getting as well.

Only other things to note are that I am using the onboard sound from the K32 and also I noticed a bunch of remote falcon errors. I have remote falcon installed but I don't use it.

Dec 5 21:39:10 fpp-kulp-01 CRON[11716]: (root) CMD ( [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi) Dec 5 21:39:15 fpp-kulp-01 systemd[1]: Starting phpsessionclean.service - Clean php session files... Dec 5 21:39:32 fpp-kulp-01 kernel: [802895.679810] rtc-pcf8523 2-0068: low voltage detected, time is unreliable Dec 5 21:39:54 fpp-kulp-01 fppd[2341]: hanneloutput/channeloutputthread.cpp:212: SLOW Output Thread: Loop: 22750us, Send: 866us, Read: 5760us, Process: 70734us, FrameNum: 3877 Dec 5 21:40:00 fpp-kulp-01 fppd[2341]: 2023-12-05 21:39:15.976 (2341) [MediaOut] /opt/fpp/src/mediaoutput/SDLOut.cpp:1149: Audio has stalled 0 Dec 5 21:40:07 fpp-kulp-01 fppd[2341]: 2023-12-05 21:39:17.256 (20533) [MediaOut] /opt/fpp/src/mediaoutput/SDLOut.cpp:273: Stalled audio, buffers still filling. Dec 5 21:40:15 fpp-kulp-01 fppd[2341]: 2023-12-05 21:39:18.085 (2341) [MediaOut] /opt/fpp/src/mediaoutput/SDLOut.cpp:1149: Audio has stalled 0 Dec 5 21:40:21 fpp-kulp-01 fppd[2341]: 2023-12-05 21:39:18.882 (2341) [MediaOut] /opt/fpp/src/mediaoutput/SDLOut.cpp:1149: Audio has stalled 0 Dec 5 21:40:30 fpp-kulp-01 fppd[2341]: 2023-12-05 21:39:19.576 (2341) [MediaOut] /opt/fpp/src/mediaoutput/SDLOut.cpp:1149: Audio has stalled 0 Dec 5 21:40:41 fpp-kulp-01 fppd[2341]: 2023-12-05 21:39:20.114 (20533) [MediaOut] /opt/fpp/src/mediaoutput/SDLOut.cpp:273: Stalled audio, buffers still filling. Dec 5 21:40:49 fpp-kulp-01 fppd[2341]: 2023-12-05 21:39:20.751 (2341) [MediaOut] /opt/fpp/src/mediaoutput/SDLOut.cpp:1149: Audio has stalled 0 Dec 5 21:40:54 fpp-kulp-01 fppd[2341]: 2023-12-05 21:39:21.175 (20533) [MediaOut] /opt/fpp/src/mediaoutput/SDLOut.cpp:273: Stalled audio, buffers still filling. Dec 5 21:41:06 fpp-kulp-01 fppd[2341]: 2023-12-05 21:39:21.507 (2341) [MediaOut] /opt/fpp/src/mediaoutput/SDLOut.cpp:1149: Audio has stalled 0 Dec 5 21:41:13 fpp-kulp-01 fppd[2341]: 2023-12-05 21:39:22.239 (2341) [MediaOut] /opt/fpp/src/mediaoutput/SDLOut.cpp:1149: Audio has stalled 0 Dec 5 21:41:16 fpp-kulp-01 fppd[2341]: 2023-12-05 21:39:22.806 (20533) [MediaOut] /opt/fpp/src/mediaoutput/SDLOut.cpp:273: Stalled audio, buffers still filling. Dec 5 21:41:19 fpp-kulp-01 fppd[2341]: 2023-12-05 21:39:23.449 (2341) [MediaOut] /opt/fpp/src/mediaoutput/SDLOut.cpp:1149: Audio has stalled 0 Dec 5 21:41:25 fpp-kulp-01 fppd[2341]: 2023-12-05 21:39:25.126 (2341) [MediaOut] /opt/fpp/src/mediaoutput/SDLOut.cpp:1149: Audio has stalled 0 Dec 5 21:41:29 fpp-kulp-01 fppd[2341]: 2023-12-05 21:39:26.414 (2341) [MediaOut] /opt/fpp/src/mediaoutput/SDLOut.cpp:1149: Audio has stalled 0 Dec 5 21:41:32 fpp-kulp-01 fppd[2341]: 2023-12-05 21:39:26.863 (20533) [MediaOut] /opt/fpp/src/mediaoutput/SDLOut.cpp:273: Stalled audio, buffers still filling. Dec 5 21:41:36 fpp-kulp-01 fppd[2341]: 2023-12-05 21:39:27.776 (2341) [MediaOut] /opt/fpp/src/mediaoutput/SDLOut.cpp:1149: Audio has stalled 0 Dec 5 21:41:39 fpp-kulp-01 fppd[2341]: 2023-12-05 21:39:28.245 (20533) [MediaOut] /opt/fpp/src/mediaoutput/SDLOut.cpp:273: Stalled audio, buffers still filling. Dec 5 21:41:42 fpp-kulp-01 fppd[2341]: 2023-12-05 21:39:30.205 (2341) [MediaOut] /opt/fpp/src/mediaoutput/SDLOut.cpp:1149: Audio has stalled 0 Dec 5 21:41:44 fpp-kulp-01 fppd[2341]: 2023-12-05 21:39:31.673 (2341) [MediaOut] /opt/fpp/src/mediaoutput/SDLOut.cpp:1149: Audio has stalled 0 Dec 5 21:41:47 fpp-kulp-01 fppd[2341]: 2023-12-05 21:39:31.713 (20533) [MediaOut] /opt/fpp/src/mediaoutput/SDLOut.cpp:273: Stalled audio, buffers still filling. Dec 5 21:41:49 fpp-kulp-01 fppd[2341]: 2023-12-05 21:39:35.374 (2341) [MediaOut] /opt/fpp/src/mediaoutput/SDLOut.cpp:1149: Audio has stalled 0 Dec 5 21:41:51 fpp-kulp-01 fppd[2341]: 2023-12-05 21:39:36.357 (2341) [MediaOut] /opt/fpp/src/mediaoutput/SDLOut.cpp:1149: Audio has stalled 0 Dec 5 21:41:53 fpp-kulp-01 fppd[2341]: 2023-12-05 21:39:36.859 (20533) [MediaOut] /opt/fpp/src/mediaoutput/SDLOut.cpp:273: Stalled audio, buffers still filling. Dec 5 21:41:55 fpp-kulp-01 fppd[2341]: 2023-12-05 21:39:37.677 (2341) [MediaOut] /opt/fpp/src/mediaoutput/SDLOut.cpp:1149: Audio has stalled 0 Dec 5 21:41:57 fpp-kulp-01 fppd[2341]: 2023-12-05 21:39:40.392 (2341) [MediaOut] /opt/fpp/src/mediaoutput/SDLOut.cpp:1149: Audio has stalled 0 Dec 5 21:41:59 fpp-kulp-01 fppd[2341]: 2023-12-05 21:39:40.786 (20533) [MediaOut] /opt/fpp/src/mediaoutput/SDLOut.cpp:273: Stalled audio, buffers still filling. Dec 5 21:42:02 fpp-kulp-01 fppd[2341]: 2023-12-05 21:39:41.745 (2341) [MediaOut] /opt/fpp/src/mediaoutput/SDLOut.cpp:1149: Audio has stalled 0 Dec 5 21:42:03 fpp-kulp-01 fppd[2341]: 2023-12-05 21:39:42.475 (20533) [MediaOut] /opt/fpp/src/mediaoutput/SDLOut.cpp:273: Stalled audio, buffers still filling. Dec 5 21:42:06 fpp-kulp-01 fppd[2341]: 2023-12-05 21:39:46.396 (2341) [MediaOut] /opt/fpp/src/mediaoutput/SDLOut.cpp:1149: Audio has stalled 0 Dec 5 21:42:08 fpp-kulp-01 fppd[2341]: 2023-12-05 21:39:46.983 (20533) [MediaOut] /opt/fpp/src/mediaoutput/SDLOut.cpp:273: Stalled audio, buffers still filling. Dec 5 21:42:10 fpp-kulp-01 fppd[2341]: 2023-12-05 21:39:47.595 (2341) [MediaOut] /opt/fpp/src/mediaoutput/SDLOut.cpp:1149: Audio has stalled 0 Dec 5 21:42:12 fpp-kulp-01 fppd[2341]: 2023-12-05 21:39:49.021 (20533) [MediaOut] /opt/fpp/src/mediaoutput/SDLOut.cpp:273: Stalled audio, buffers still filling. Dec 5 21:42:12 fpp-kulp-01 fppd[2341]: 2023-12-05 21:39:50.101 (2341) [MediaOut] /opt/fpp/src/mediaoutput/SDLOut.cpp:1149: Audio has stalled 0 Dec 5 21:42:14 fpp-kulp-01 fppd[2341]: 2023-12-05 21:39:50.751 (2341) [MediaOut] /opt/fpp/src/mediaoutput/SDLOut.cpp:1149: Audio has stalled 0 Dec 5 21:42:15 fpp-kulp-01 fppd[2341]: 2023-12-05 21:39:51.566 (2341) [MediaOut] /opt/fpp/src/mediaoutput/SDLOut.cpp:1149: Audio has stalled 0 Dec 5 21:42:17 fpp-kulp-01 fppd[2341]: 2023-12-05 21:39:51.651 (20533) [MediaOut] /opt/fpp/src/mediaoutput/SDLOut.cpp:273: Stalled audio, buffers still filling. Dec 5 21:42:19 fpp-kulp-01 kernel: [803062.893597] davinci-mcasp 48038000.mcasp: Sample-rate is off by 400 PPM

[FPP_Logs_20231206.0251.zip] (https://github.com/FalconChristmas/fpp/files/13575885/FPP_Logs_20231206.0251.zip)

[cabrower]

Had it happen again tonight. Tonight it was stuck playing a sequence which was frozen. My Tune To sign which is an eseq never stopped playing.

I could ping the Kulp 32, but i couldn't ssh or bring up the page. This is what I got from the page:

After a power cycle it came up with this error:

After another power cycle the kulp would not respond on the network. I power cycled again and it came up with no problems.

Coincidentally both time this happened were towards the end of the night/show. Not to create any noise but here is my schedule:

FPP_Logs_20231212.0416.zip

[dkulp]

@cabrower At this point, there is no point in reporting crashes/hangs with anything other than 7.3.6 or 7.4.

[jkriegshauser]

@dkulp do the latest versions have any changes that could address this issue, or better logging/diagnostics when the issue occurs? It's a tall ask to update FPP in the middle of the season :)

Is there documentation around running a development branch from a fork? I might play around with some debugging stuff if I can find the time. I had a brief look but didn't find any.

[jkriegshauser]

Had a crash just now, may or may not be related. Unfortunately it looks like the gdb callstack is empty and everything restarted before I could ssh in and get gdb connected.

fpp-BBB-7.2-2-ge35c0df7-2023-12-14_20-42-40.zip

ps -ef reports:

UID        PID  PPID  C STIME TTY          TIME CMD
root         1     0  0 Dec05 ?        00:03:50 /sbin/init
root         2     0  0 Dec05 ?        00:00:00 [kthreadd]
root         3     2  0 Dec05 ?        00:00:00 [rcu_gp]
root         4     2  0 Dec05 ?        00:00:00 [rcu_par_gp]
root         8     2  0 Dec05 ?        00:00:00 [mm_percpu_wq]
root         9     2  0 Dec05 ?        00:00:00 [rcu_tasks_kthre]
root        10     2  0 Dec05 ?        00:00:00 [rcu_tasks_trace]
root        11     2  0 Dec05 ?        00:06:29 [ksoftirqd/0]
root        12     2  0 Dec05 ?        00:29:14 [rcu_preempt]
root        13     2  0 Dec05 ?        00:00:00 [kdevtmpfs]
root        14     2  0 Dec05 ?        00:00:00 [netns]
root        16     2  0 Dec05 ?        00:00:00 [kauditd]
root        17     2  0 Dec05 ?        00:00:01 [khungtaskd]
root        18     2  0 Dec05 ?        00:00:00 [oom_reaper]
root        19     2  0 Dec05 ?        00:00:00 [writeback]
root        20     2  0 Dec05 ?        00:01:08 [kcompactd0]
root        21     2  0 Dec05 ?        00:00:00 [ksmd]
root        61     2  0 Dec05 ?        00:00:00 [kintegrityd]
root        62     2  0 Dec05 ?        00:00:00 [kblockd]
root        63     2  0 Dec05 ?        00:00:00 [blkcg_punt_bio]
root        64     2  0 Dec05 ?        00:00:00 [tpm_dev_wq]
root        65     2  0 Dec05 ?        00:00:00 [ata_sff]
root        66     2  0 Dec05 ?        00:00:00 [devfreq_wq]
root        67     2  0 Dec05 ?        00:00:00 [watchdogd]
root        69     2  0 Dec05 ?        00:00:00 [rpciod]
root        70     2  0 Dec05 ?        00:00:00 [kworker/u3:0]
root        71     2  0 Dec05 ?        00:00:00 [xprtiod]
root        81     2  0 Dec05 ?        00:06:05 [kswapd0]
root        82     2  0 Dec05 ?        00:00:00 [nfsiod]
root        84     2  0 Dec05 ?        00:00:00 [kthrotld]
root        85     2  0 Dec05 ?        00:00:00 [irq/29-4802a000]
root        87     2  0 Dec05 ?        00:21:47 [irq/41-4819c000]
root        88     2  0 Dec05 ?        00:00:00 [hwrng]
root        90     2  0 Dec05 ?        00:00:00 [spi0]
root        91     2  0 Dec05 ?        00:00:00 [spi1]
root        92     2  0 Dec05 ?        00:00:00 [sdhci]
root        93     2  0 Dec05 ?        00:00:00 [53500000.aes-en]
root        94     2  0 Dec05 ?        00:00:00 [irq/50-mmc1]
root       100     2  0 Dec05 ?        00:00:00 [53100000.sham-e]
root       109     2  0 Dec05 ?        00:00:00 [irq/22-TI-am335]
root       111     2  0 Dec05 ?        00:00:00 [irq/32-48038000]
root       112     2  0 Dec05 ?        00:00:00 [irq/31-48038000]
root       114     2  0 Dec05 ?        00:00:00 [ipv6_addrconf]
root       126     2  0 Dec05 ?        00:00:00 [kstrp]
root       127     2  0 Dec05 ?        00:00:00 [zswap-shrink]
root       128     2  0 Dec05 ?        00:00:00 [irq/21-44e0b000]
root       129     2  0 Dec05 ?        00:00:00 [mmc_complete]
root       130     2  0 Dec05 ?        00:00:00 [irq/70-tps65217]
root       132     2  0 Dec05 ?        00:00:00 [sdhci]
root       133     2  0 Dec05 ?        00:00:00 [irq/39-mmc0]
root       135     2  0 Dec05 ?        00:00:00 [irq/144-4806000]
root       136     2  0 Dec05 ?        00:00:00 [mmc_complete]
root       138     2  0 Dec05 ?        00:00:06 [jbd2/mmcblk0p1-]
root       139     2  0 Dec05 ?        00:00:00 [ext4-rsv-conver]
root       163     1  0 Dec05 ?        00:01:45 /lib/systemd/systemd-journald
root       183     1  0 Dec05 ?        00:00:08 /lib/systemd/systemd-udevd
systemd+   191     1  0 Dec05 ?        00:00:53 /lib/systemd/systemd-networkd
root       198     1  0 Dec05 ?        00:00:57 /usr/sbin/haveged --Foreground --verbose=1
systemd+   200     1  0 Dec05 ?        00:00:42 /lib/systemd/systemd-resolved
root       213     2  0 Dec05 ?        00:00:00 [cfg80211]
avahi      256     1  1 Dec05 ?        02:36:12 avahi-daemon: running [FPP-Porch.local]
root       267     1  0 Dec05 ?        00:00:07 /usr/sbin/cron -f
message+   272     1  0 Dec05 ?        00:02:30 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
root       286     1  0 Dec05 ?        00:00:01 /usr/bin/python3 /usr/bin/networkd-dispatcher --run-startup-triggers
root       289     1  0 Dec05 ?        00:00:15 /usr/sbin/rsyslogd -n -iNONE
root       297     1  0 Dec05 ?        00:01:17 /lib/systemd/systemd-logind
root       299     1  0 Dec05 ?        00:00:08 /sbin/wpa_supplicant -u -s -O DIR=/run/wpa_supplicant GROUP=netdev
daemon     311     1  0 Dec05 ?        00:00:00 /usr/sbin/atd -f
avahi      317   256  0 Dec05 ?        00:00:00 avahi-daemon: chroot helper
root       382     1  0 Dec05 ?        00:39:36 /opt/fpp/src/fppoled
root       493     2  0 Dec05 ?        00:00:00 [irq/134-FPPOLED]
root       630     2  0 Dec05 ?        00:00:07 [ptp0]
root      1237     1  0 Dec05 ?        00:01:52 php-fpm: master process (/etc/php/7.4/fpm/php-fpm.conf)
ntp       1270     1  0 Dec05 ?        00:02:24 /usr/sbin/ntpd -p /var/run/ntpd.pid -g -u 108:118
root      1274     1  0 Dec05 ?        00:00:00 sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups
root      1313     1  0 Dec05 tty1     00:00:00 /sbin/agetty -o -p -- \u --noclear - linux
root      1320     1  0 Dec05 ttyGS0   00:00:00 /sbin/agetty -o -p -- \u --keep-baud 115200,57600,38400,9600 - vt220
root      1335     1  0 Dec05 ttyS0    00:00:00 /sbin/agetty -o -p -- \u --keep-baud 115200,57600,38400,9600 - vt220
fpp       1347  1237  0 Dec05 ?        00:21:39 php-fpm: pool www
root      1348     1  0 Dec05 ?        00:01:40 /usr/sbin/apache2 -k start
fpp       1353  1237  0 Dec05 ?        00:12:27 php-fpm: pool www
fpp       1356  1348  0 Dec05 ?        00:00:00 /usr/sbin/apache2 -k start
fpp       1367  1348  0 Dec05 ?        00:31:00 /usr/sbin/apache2 -k start
fpp       1371  1348  0 Dec05 ?        01:24:22 /usr/sbin/apache2 -k start
shellin+  1442     1  0 Dec05 ?        00:00:00 /usr/bin/shellinaboxd -q --background=/var/run/shellinaboxd.pid -c /var/tmp/ -p 4200 -u shellinabox -g shellinabox --user-css Black on White:+/etc/shellinabox/options-enabled/00+Black on White.css,White On Black:-/etc/shellinabox/opt
shellin+  1446  1442  0 Dec05 ?        00:00:00 /usr/bin/shellinaboxd -q --background=/var/run/shellinaboxd.pid -c /var/tmp/ -p 4200 -u shellinabox -g shellinabox --user-css Black on White:+/etc/shellinabox/options-enabled/00+Black on White.css,White On Black:-/etc/shellinabox/opt
Debian-+  1892     1  0 Dec05 ?        00:00:01 /usr/sbin/exim4 -bd -q30m
fpp       1991  1237  0 Dec05 ?        00:44:26 php-fpm: pool www
fpp       2037  1237  0 Dec05 ?        00:01:08 php-fpm: pool www
fpp       2045  1237  0 Dec05 ?        00:10:50 php-fpm: pool www
fpp       2160  1237  0 Dec05 ?        00:19:00 php-fpm: pool www
root      2529     2  0 20:22 ?        00:00:00 [kworker/u2:0-events_unbound]
fpp       2798  1237  0 Dec05 ?        00:21:53 php-fpm: pool www
fpp       3174  1237  0 Dec05 ?        00:32:40 php-fpm: pool www
fpp       3211  1237  0 Dec05 ?        00:23:12 php-fpm: pool www
root      3223     2  0 20:42 ?        00:00:00 [kworker/u2:1-ext4-rsv-conversion]
root      3224     2  0 20:42 ?        00:00:00 [kworker/u2:2]
root      3584 10510  0 20:42 ?        00:00:00 sh -c zip -r crashes/fpp-BBB-7.2-2-ge35c0df7-2023-12-14_20-42-40.zip /tmp/fppd_crash.log settings config tmp logs/fppd.log logs/apache2-error.log playlists
root      3585  3584 13 20:42 ?        00:00:08 zip -r crashes/fpp-BBB-7.2-2-ge35c0df7-2023-12-14_20-42-40.zip /tmp/fppd_crash.log settings config tmp logs/fppd.log logs/apache2-error.log playlists
root      4264  1274  7 20:43 ?        00:00:01 sshd: fpp [priv]
fpp       4276  1237  0 Dec07 ?        00:18:54 php-fpm: pool www
root      4320     2  0 20:43 ?        00:00:00 [kworker/0:0-events]
fpp       4468     1 18 20:43 ?        00:00:01 /lib/systemd/systemd --user
fpp       4472  4468  0 20:43 ?        00:00:00 (sd-pam)
fpp       4501  1347  0 20:43 ?        00:00:00 sh -c (git --git-dir=/opt/fpp/.git/ ls-remote -q -h origin v7.2 | awk '$1 > 0 { print substr($1,1,7)}')
fpp       4503  4501  0 20:43 ?        00:00:00 sh -c (git --git-dir=/opt/fpp/.git/ ls-remote -q -h origin v7.2 | awk '$1 > 0 { print substr($1,1,7)}')
fpp       4504  4503  0 20:43 ?        00:00:00 git --git-dir=/opt/fpp/.git/ ls-remote -q -h origin v7.2
fpp       4505  4503  0 20:43 ?        00:00:00 awk $1 > 0 { print substr($1,1,7)}
fpp       4508  4504  0 20:43 ?        00:00:00 /usr/lib/git-core/git remote-https origin https://github.com/FalconChristmas/fpp.git
fpp       4516  4508  6 20:43 ?        00:00:00 /usr/lib/git-core/git-remote-https origin https://github.com/FalconChristmas/fpp.git
fpp       4524 12546  0 20:43 ?        00:00:00 sh -c (git --git-dir=/opt/fpp/.git/ ls-remote -q -h origin v7.2 | awk '$1 > 0 { print substr($1,1,7)}')
fpp       4525  4524  0 20:43 ?        00:00:00 sh -c (git --git-dir=/opt/fpp/.git/ ls-remote -q -h origin v7.2 | awk '$1 > 0 { print substr($1,1,7)}')
fpp       4526  4525  0 20:43 ?        00:00:00 git --git-dir=/opt/fpp/.git/ ls-remote -q -h origin v7.2
fpp       4527  4525  0 20:43 ?        00:00:00 awk $1 > 0 { print substr($1,1,7)}
fpp       4531  4526  0 20:43 ?        00:00:00 /usr/lib/git-core/git remote-https origin https://github.com/FalconChristmas/fpp.git
fpp       4532  4531  7 20:43 ?        00:00:00 /usr/lib/git-core/git-remote-https origin https://github.com/FalconChristmas/fpp.git
fpp       4542  4264  3 20:43 ?        00:00:00 sshd: fpp@pts/0
fpp       4547  4542 10 20:43 pts/0    00:00:00 -bash
fpp       4581  3174  0 20:43 ?        00:00:00 sh -c (git --git-dir=/opt/fpp/.git/ ls-remote -q -h origin v7.2 | awk '$1 > 0 { print substr($1,1,7)}')
fpp       4582  4581  0 20:43 ?        00:00:00 sh -c (git --git-dir=/opt/fpp/.git/ ls-remote -q -h origin v7.2 | awk '$1 > 0 { print substr($1,1,7)}')
fpp       4583  4582  1 20:43 ?        00:00:00 git --git-dir=/opt/fpp/.git/ ls-remote -q -h origin v7.2
fpp       4584  4582  0 20:43 ?        00:00:00 awk $1 > 0 { print substr($1,1,7)}
fpp       4585  4583  0 20:43 ?        00:00:00 /usr/lib/git-core/git remote-https origin https://github.com/FalconChristmas/fpp.git
fpp       4586  4585  7 20:43 ?        00:00:00 /usr/lib/git-core/git-remote-https origin https://github.com/FalconChristmas/fpp.git
fpp       4587  1353  0 20:43 ?        00:00:00 sh -c /opt/fpp/scripts/get_uuid
fpp       4588  4587  1 20:43 ?        00:00:00 /bin/bash /opt/fpp/scripts/get_uuid
fpp       4593  4547 17 20:43 pts/0    00:00:00 ps -ef
root      6130     2  0 19:38 ?        00:00:00 [kworker/0:2-events]
root     10510     1 12 Dec11 ?        09:34:51 /opt/fpp/src/fppd -f --log-level info:General,ChannelOut,ChannelData,Command,E131Bridge,Effect,MediaOut,Playlist,Schedule,Sequence,Settings,Control,Sync,Plugin,GPIO,HTTP -r
root     10531     1  0 Dec11 ?        00:14:40 /usr/bin/php /home/fpp/media/plugins/remote-falcon/remote_falcon_listener.php
fpp      12101  1237  0 17:07 ?        00:00:09 php-fpm: pool www
fpp      12111  1237  0 17:07 ?        00:00:00 php-fpm: pool www
fpp      12300  1237  0 17:09 ?        00:00:03 php-fpm: pool www
fpp      12391  1237  0 17:09 ?        00:01:11 php-fpm: pool www
fpp      12425  1237  0 17:09 ?        00:00:08 php-fpm: pool www
fpp      12517  1237  0 17:09 ?        00:00:00 php-fpm: pool www
fpp      12533  1237  0 17:09 ?        00:00:00 php-fpm: pool www
fpp      12546  1237  0 17:09 ?        00:01:53 php-fpm: pool www
fpp      14218  1237  0 19:24 ?        00:00:19 php-fpm: pool www
fpp      14240  1237  0 19:24 ?        00:00:13 php-fpm: pool www
fpp      14257  1237  0 19:24 ?        00:00:12 php-fpm: pool www
root     20511     2  0 20:33 ?        00:00:02 [kworker/0:2H+kblockd]
fpp      23709  1237  0 Dec07 ?        00:00:00 php-fpm: pool www
root     25406     2  0 20:15 ?        00:00:00 [kworker/0:1H]
fpp      26196  1237  0 Dec06 ?        00:11:40 php-fpm: pool www
root     26463     2  0 19:09 ?        00:00:00 [kworker/u2:3+flush-179:768]
root     28238     2  0 20:37 ?        00:00:00 [kworker/0:1-events_power_efficient]
root     29995     2  0 20:39 ?        00:00:00 [kworker/0:0H]
fpp      30298  1237  0 Dec06 ?        00:12:35 php-fpm: pool www

[dkulp]

@jkriegshauser As i said, please don't bother sending logs for anything other than 7.3.6 or 7.4. No one is going to bother to look at them so it's just a waste of time. There are a ton of fixes that went into 7.4 which fix a bunch of crashes and memory leaks and other issues.

[jkriegshauser]

Updated to 7.4 today since we had another crash last night 🤞

[ghormann]

All the crashlogs for 7.x were reviewed looking for patterns. Fixes were applied. If problem can be reproduced in 7.5 please open another ticket.