gtxaspec
commented
4 months ago thanks for report, working on a resolution
Closed kohalmi666 closed 1 month ago
gtxaspec
commented
4 months ago thanks for report, working on a resolution
johndoewashere2
commented
3 months ago Noticed a similar issue as well on the Wyze Cam C3 builds as well, is there any workaround that we can try in the mean time or anything we can help?
gtxaspec
commented
3 months ago @johndoewashere2 what client are you using where you experience the issues?
johndoewashere2
commented
3 months ago @johndoewashere2 what client are you using where you experience the issues?
Having this issue when using the RTSP/TCP source directly in Frigate and re-stream using its built-in go2rtc implementation.
Frigate will report errors similar to something below:
2024-05-19 18:52:03.352206142 [h264 @ 0x7f38640f4480] error while decoding MB 30 29, bytestream -28
2024-05-19 18:52:33.364850242 [h264 @ 0x7f3864149080] error while decoding MB 49 31, bytestream -16
2024-05-19 18:52:43.388542996 [h264 @ 0x7f38640f5680] error while decoding MB 26 26, bytestream -34
2024-05-19 18:53:33.342862830 [h264 @ 0x7f38641841c0] error while decoding MB 82 38, bytestream -18... and eventually its ffmpeg process will crash and restart after these logs show up on the cameras (ignore the timestamp as I've been restarting Frigate to tune and test different input options, albeit to no avail):
May 19 18:34:16 thingino-t31x user.err prudynt[1111]: [ERROR:Encoder.cpp]: Sink IMPDeviceSource clogged! Discarding NAL.
May 19 18:34:16 thingino-t31x user.err prudynt[1111]: [ERROR:Encoder.cpp]: Sink IMPDeviceSource clogged! Discarding NAL.
May 19 18:34:16 thingino-t31x user.err prudynt[1111]: [ERROR:Encoder.cpp]: Sink IMPDeviceSource clogged! Discarding NAL.
May 19 18:34:16 thingino-t31x user.err prudynt[1111]: [ERROR:Encoder.cpp]: Sink IMPDeviceSource clogged! Discarding NAL.
May 19 18:34:16 thingino-t31x user.err prudynt[1111]: [ERROR:Encoder.cpp]: Sink IMPDeviceSource clogged! Discarding NAL.
May 19 18:34:16 thingino-t31x user.err prudynt[1111]: [ERROR:Encoder.cpp]: Sink IMPDeviceSource clogged! Discarding NAL.
May 19 18:34:16 thingino-t31x user.err prudynt[1111]: [ERROR:Encoder.cpp]: Sink IMPDeviceSource clogged! Discarding NAL.
May 19 18:34:16 thingino-t31x user.err prudynt[1111]: [ERROR:Encoder.cpp]: Sink IMPDeviceSource clogged! Discarding NAL.
May 19 18:34:16 thingino-t31x user.err prudynt[1111]: [ERROR:Encoder.cpp]: Sink IMPDeviceSource clogged! Discarding NAL.
May 19 18:34:16 thingino-t31x user.err prudynt[1111]: [ERROR:Encoder.cpp]: Sink IMPDeviceSource clogged! Discarding NAL.
May 19 18:34:16 thingino-t31x user.err prudynt[1111]: [ERROR:Encoder.cpp]: Sink IMPDeviceSource clogged! Discarding NAL.
May 19 18:34:16 thingino-t31x user.err prudynt[1111]: [ERROR:Encoder.cpp]: Sink IMPDeviceSource clogged! Discarding NAL.
May 19 18:34:17 thingino-t31x user.err prudynt[1111]: [ERROR:Encoder.cpp]: Sink IMPDeviceSource clogged! Discarding NAL.
May 19 18:34:17 thingino-t31x user.err prudynt[1111]: [ERROR:Encoder.cpp]: Sink IMPDeviceSource clogged! Discarding NAL.
May 19 18:34:17 thingino-t31x user.err prudynt[1111]: [ERROR:Encoder.cpp]: Sink IMPDeviceSource clogged! Discarding NAL.
May 19 18:34:17 thingino-t31x user.debug prudynt[1111]: [DEBUG:IMPDeviceSource.cpp]: Device source destruct
May 19 18:34:17 thingino-t31x user.debug prudynt[1111]: [DEBUG:Encoder.hpp]: Destroy Sink: 50
May 19 18:34:17 thingino-t31x user.debug prudynt[1111]: [DEBUG:IMPServerMediaSubsession.cpp]: Create Stream Source.
May 19 18:34:17 thingino-t31x user.debug prudynt[1111]: [DEBUG:IMPDeviceSource.cpp]: Device source construct
May 19 18:34:17 thingino-t31x user.debug prudynt[1111]: [DEBUG:Encoder.hpp]: Create Sink: 51
May 19 18:34:53 thingino-t31x user.debug prudynt[1111]: [DEBUG:IMPDeviceSource.cpp]: Device source destruct
May 19 18:34:53 thingino-t31x user.debug prudynt[1111]: [DEBUG:Encoder.hpp]: Destroy Sink: 51Switching the stream to RTSP/UDP in Frigate doesn't quite help either as it resulted smearing and corruption.
Tweaking the /etc/prudynt.cfg didn't help much either, it appears to behave the same with the default config.
Currently config is as per below to try and improve the stream quality/reduce smearing:
rtsp: {
# port: 554;
# name: "thingino prudynt";
est_bitrate: 5000;
out_buffer_size: 1000000;
send_buffer_size: 614400;
auth_required: true;
# username: "thingino";
# password: "thingino";
};
# Sensor Settings
# ---------------
sensor: {
# model: "gc2053";
# i2c_address: 0x37;
fps: 15;
width: 1920;
height: 1080;
};
# Stream0 Settings
# ----------------
stream0: {
rtsp_endpoint: "ch0";
format: "H264";
bitrate: 2000;
width: 1920;
height: 1080;
buffers: 4;
fps: 15;
gop: 6;
max_gop: 16;
# osd_pos_time_x: 10;
# osd_pos_time_y: 10;
# osd_pos_user_text_x: 900;
# osd_pos_user_text_y: 5;
# osd_pos_uptime_x: 1600;
# osd_pos_uptime_y: 5;
# osd_pos_logo_x: 1800;
# osd_pos_logo_y: 1030;
# osd_logo_transparency: 255;
# rotation: 0;
# scale_enabled: false;
# scale_width: 640;
# scale_height: 360;looking into this, my setup is similar with go2rtc and frigate. stay tuned!
johndoewashere2
commented
3 months ago looking into this, my setup is similar with go2rtc and frigate. stay tuned!
Not sure if this is helpful, but here's the logs that's time synced between Frigate and the camera:
Frigate:
2024-05-20 16:59:11.595286559 [h264 @ 0x7f479c015840] error while decoding MB 32 20, bytestream -28
2024-05-20 17:00:21.616495912 [h264 @ 0x7f479c015840] error while decoding MB 115 20, bytestream -30
2024-05-20 17:03:06.313373637 [2024-05-20 17:03:06] frigate.video ERROR : front_yard: Unable to read frames from ffmpeg process.
2024-05-20 17:03:06.313910919 [2024-05-20 17:03:06] frigate.video ERROR : front_yard: ffmpeg process is not running. exiting capture thread...
2024-05-20 17:03:06.520702076 [2024-05-20 17:03:06] watchdog.front_yard ERROR : Ffmpeg process crashed unexpectedly for front_yard.
2024-05-20 17:03:06.521102535 [2024-05-20 17:03:06] watchdog.front_yard ERROR : The following ffmpeg logs include the last 100 lines prior to exit.
2024-05-20 17:03:06.521497581 [2024-05-20 17:03:06] ffmpeg.front_yard.detect ERROR : rtsp://127.0.0.1:8554/cam_front_yard: Connection timed out
2024-05-20 17:03:06.521805334 [2024-05-20 17:03:06] ffmpeg.front_yard.detect ERROR : Last message repeated 1 times
2024-05-20 17:03:16.318904265 [2024-05-20 17:03:16] frigate.video ERROR : car_port: Unable to read frames from ffmpeg process.
2024-05-20 17:03:16.319341117 [2024-05-20 17:03:16] frigate.video ERROR : car_port: ffmpeg process is not running. exiting capture thread...
2024-05-20 17:03:16.565097453 [2024-05-20 17:03:16] watchdog.car_port ERROR : Ffmpeg process crashed unexpectedly for car_port.
2024-05-20 17:03:16.566262519 [2024-05-20 17:03:16] watchdog.car_port ERROR : The following ffmpeg logs include the last 100 lines prior to exit.
2024-05-20 17:03:16.566501355 [2024-05-20 17:03:16] ffmpeg.car_port.detect ERROR : rtsp://127.0.0.1:8554/cam_car_port: Connection timed out
2024-05-20 17:03:16.566749632 [2024-05-20 17:03:16] ffmpeg.car_port.detect ERROR : Last message repeated 1 times
2024-05-20 17:04:31.645534108 [h264 @ 0x7f479c159200] error while decoding MB 53 66, bytestream -16
2024-05-20 17:07:41.639286212 [h264 @ 0x7f479c015840] cabac decode of qscale diff failed at 25 37
2024-05-20 17:07:41.639291019 [h264 @ 0x7f479c015840] error while decoding MB 25 37, bytestream -10
2024-05-20 17:11:11.649841396 [h264 @ 0x7f479c105a40] error while decoding MB 25 33, bytestream -18
2024-05-20 17:12:51.656009128 [h264 @ 0x7f479c105a40] error while decoding MB 43 42, bytestream -20
2024-05-20 17:17:51.626158242 [h264 @ 0x7f479c105a40] error while decoding MB 77 20, bytestream -16
2024-05-20 17:21:16.975930071 [2024-05-20 17:21:16] frigate.video ERROR : front_yard: Unable to read frames from ffmpeg process.
2024-05-20 17:21:16.976251794 [2024-05-20 17:21:16] frigate.video ERROR : front_yard: ffmpeg process is not running. exiting capture thread...
2024-05-20 17:21:26.585385726 [2024-05-20 17:21:26] watchdog.front_yard ERROR : Ffmpeg process crashed unexpectedly for front_yard.
2024-05-20 17:21:26.586050775 [2024-05-20 17:21:26] watchdog.front_yard ERROR : The following ffmpeg logs include the last 100 lines prior to exit.
2024-05-20 17:21:26.586411218 [2024-05-20 17:21:26] ffmpeg.front_yard.detect ERROR : rtsp://127.0.0.1:8554/cam_front_yard: Connection timed out
2024-05-20 17:21:26.586720794 [2024-05-20 17:21:26] ffmpeg.front_yard.detect ERROR : Last message repeated 1 times
2024-05-20 17:21:26.807027529 [2024-05-20 17:21:26] frigate.video ERROR : car_port: Unable to read frames from ffmpeg process.
2024-05-20 17:21:26.808152779 [2024-05-20 17:21:26] frigate.video ERROR : car_port: ffmpeg process is not running. exiting capture thread...
2024-05-20 17:21:36.618564472 [2024-05-20 17:21:36] watchdog.car_port ERROR : Ffmpeg process crashed unexpectedly for car_port.
2024-05-20 17:21:36.619070497 [2024-05-20 17:21:36] watchdog.car_port ERROR : The following ffmpeg logs include the last 100 lines prior to exit.
2024-05-20 17:21:36.619358977 [2024-05-20 17:21:36] ffmpeg.car_port.detect ERROR : rtsp://127.0.0.1:8554/cam_car_port: Connection timed out
2024-05-20 17:21:36.619616722 [2024-05-20 17:21:36] ffmpeg.car_port.detect ERROR : Last message repeated 1 times
2024-05-20 17:24:41.683261288 [h264 @ 0x7f479c041f00] error while decoding MB 51 19, bytestream -12"front_yard" camera:
May 20 17:21:18 thingino-t31x user.err prudynt[1100]: [ERROR:Encoder.cpp]: Sink IMPDeviceSource clogged! Discarding NAL.
May 20 17:21:18 thingino-t31x user.err prudynt[1100]: [ERROR:Encoder.cpp]: Sink IMPDeviceSource clogged! Discarding NAL.
May 20 17:21:19 thingino-t31x user.err prudynt[1100]: [ERROR:Encoder.cpp]: Sink IMPDeviceSource clogged! Discarding NAL.
May 20 17:21:19 thingino-t31x user.err prudynt[1100]: [ERROR:Encoder.cpp]: Sink IMPDeviceSource clogged! Discarding NAL.
May 20 17:21:19 thingino-t31x user.err prudynt[1100]: [ERROR:Encoder.cpp]: Sink IMPDeviceSource clogged! Discarding NAL.
May 20 17:21:19 thingino-t31x user.err prudynt[1100]: [ERROR:Encoder.cpp]: Sink IMPDeviceSource clogged! Discarding NAL.
May 20 17:21:19 thingino-t31x user.err prudynt[1100]: [ERROR:Encoder.cpp]: Sink IMPDeviceSource clogged! Discarding NAL.
May 20 17:21:19 thingino-t31x user.err prudynt[1100]: [ERROR:Encoder.cpp]: Sink IMPDeviceSource clogged! Discarding NAL.
May 20 17:21:19 thingino-t31x user.err prudynt[1100]: [ERROR:Encoder.cpp]: Sink IMPDeviceSource clogged! Discarding NAL.
May 20 17:21:19 thingino-t31x user.err prudynt[1100]: [ERROR:Encoder.cpp]: Sink IMPDeviceSource clogged! Discarding NAL.
May 20 17:21:19 thingino-t31x user.err prudynt[1100]: [ERROR:Encoder.cpp]: Sink IMPDeviceSource clogged! Discarding NAL.
May 20 17:21:19 thingino-t31x user.err prudynt[1100]: [ERROR:Encoder.cpp]: Sink IMPDeviceSource clogged! Discarding NAL.
.
.
.
.
May 20 17:21:26 thingino-t31x user.err prudynt[1100]: [ERROR:Encoder.cpp]: Sink IMPDeviceSource clogged! Discarding NAL.
May 20 17:21:26 thingino-t31x user.err prudynt[1100]: [ERROR:Encoder.cpp]: Sink IMPDeviceSource clogged! Discarding NAL.
May 20 17:21:26 thingino-t31x user.debug prudynt[1100]: [DEBUG:IMPServerMediaSubsession.cpp]: Create Stream Source.
May 20 17:21:26 thingino-t31x user.debug prudynt[1100]: [DEBUG:IMPDeviceSource.cpp]: Device source construct
May 20 17:21:26 thingino-t31x user.debug prudynt[1100]: [DEBUG:Encoder.hpp]: Create Sink: 118
May 20 17:21:26 thingino-t31x user.err prudynt[1100]: [ERROR:Encoder.cpp]: Sink IMPDeviceSource clogged! Discarding NAL.
May 20 17:21:26 thingino-t31x user.err prudynt[1100]: [ERROR:Encoder.cpp]: Sink IMPDeviceSource clogged! Discarding NAL.
May 20 17:21:26 thingino-t31x user.err prudynt[1100]: [ERROR:Encoder.cpp]: Sink IMPDeviceSource clogged! Discarding NAL.
May 20 17:21:26 thingino-t31x user.err prudynt[1100]: [ERROR:Encoder.cpp]: Sink IMPDeviceSource clogged! Discarding NAL.
May 20 17:21:26 thingino-t31x user.err prudynt[1100]: [ERROR:Encoder.cpp]: Sink IMPDeviceSource clogged! Discarding NAL.
May 20 17:21:26 thingino-t31x user.err prudynt[1100]: [ERROR:Encoder.cpp]: Sink IMPDeviceSource clogged! Discarding NAL.
May 20 17:21:26 thingino-t31x user.err prudynt[1100]: [ERROR:Encoder.cpp]: Sink IMPDeviceSource clogged! Discarding NAL.
May 20 17:21:26 thingino-t31x user.err prudynt[1100]: [ERROR:Encoder.cpp]: Sink IMPDeviceSource clogged! Discarding NAL.
May 20 17:21:27 thingino-t31x user.err prudynt[1100]: [ERROR:Encoder.cpp]: Sink IMPDeviceSource clogged! Discarding NAL.
May 20 17:21:27 thingino-t31x user.err prudynt[1100]: [ERROR:Encoder.cpp]: Sink IMPDeviceSource clogged! Discarding NAL.
May 20 17:21:27 thingino-t31x user.err prudynt[1100]: [ERROR:Encoder.cpp]: Sink IMPDeviceSource clogged! Discarding NAL.
May 20 17:21:27 thingino-t31x user.err prudynt[1100]: [ERROR:Encoder.cpp]: Sink IMPDeviceSource clogged! Discarding NAL.
May 20 17:21:27 thingino-t31x user.err prudynt[1100]: [ERROR:Encoder.cpp]: Sink IMPDeviceSource clogged! Discarding NAL.
May 20 17:21:27 thingino-t31x user.err prudynt[1100]: [ERROR:Encoder.cpp]: Sink IMPDeviceSource clogged! Discarding NAL.
May 20 17:21:27 thingino-t31x user.err prudynt[1100]: [ERROR:Encoder.cpp]: Sink IMPDeviceSource clogged! Discarding NAL.
May 20 17:21:27 thingino-t31x user.err prudynt[1100]: [ERROR:Encoder.cpp]: Sink IMPDeviceSource clogged! Discarding NAL.
May 20 17:21:27 thingino-t31x user.err prudynt[1100]: [ERROR:Encoder.cpp]: Sink IMPDeviceSource clogged! Discarding NAL.
May 20 17:21:27 thingino-t31x user.err prudynt[1100]: [ERROR:Encoder.cpp]: Sink IMPDeviceSource clogged! Discarding NAL.
May 20 17:21:27 thingino-t31x user.err prudynt[1100]: [ERROR:Encoder.cpp]: Sink IMPDeviceSource clogged! Discarding NAL.
May 20 17:21:27 thingino-t31x user.err prudynt[1100]: [ERROR:Encoder.cpp]: Sink IMPDeviceSource clogged! Discarding NAL.
May 20 17:21:27 thingino-t31x user.err prudynt[1100]: [ERROR:Encoder.cpp]: Sink IMPDeviceSource clogged! Discarding NAL.
May 20 17:21:27 thingino-t31x user.err prudynt[1100]: [ERROR:Encoder.cpp]: Sink IMPDeviceSource clogged! Discarding NAL.
May 20 17:21:27 thingino-t31x user.err prudynt[1100]: [ERROR:Encoder.cpp]: Sink IMPDeviceSource clogged! Discarding NAL.
May 20 17:21:27 thingino-t31x user.err prudynt[1100]: [ERROR:Encoder.cpp]: Sink IMPDeviceSource clogged! Discarding NAL.
May 20 17:21:27 thingino-t31x user.err prudynt[1100]: [ERROR:Encoder.cpp]: Sink IMPDeviceSource clogged! Discarding NAL.
May 20 17:21:28 thingino-t31x user.err prudynt[1100]: [ERROR:Encoder.cpp]: Sink IMPDeviceSource clogged! Discarding NAL.
.
.
.
.
May 20 17:21:49 thingino-t31x user.err prudynt[1100]: [ERROR:Encoder.cpp]: Sink IMPDeviceSource clogged! Discarding NAL.
May 20 17:21:49 thingino-t31x user.err prudynt[1100]: [ERROR:Encoder.cpp]: Sink IMPDeviceSource clogged! Discarding NAL.
May 20 17:21:49 thingino-t31x user.err prudynt[1100]: [ERROR:Encoder.cpp]: Sink IMPDeviceSource clogged! Discarding NAL.
May 20 17:21:49 thingino-t31x user.err prudynt[1100]: [ERROR:Encoder.cpp]: Sink IMPDeviceSource clogged! Discarding NAL.
May 20 17:21:49 thingino-t31x user.err prudynt[1100]: [ERROR:Encoder.cpp]: Sink IMPDeviceSource clogged! Discarding NAL.
May 20 17:21:49 thingino-t31x user.err prudynt[1100]: [ERROR:Encoder.cpp]: Sink IMPDeviceSource clogged! Discarding NAL.
May 20 17:21:49 thingino-t31x user.err prudynt[1100]: [ERROR:Encoder.cpp]: Sink IMPDeviceSource clogged! Discarding NAL.
May 20 17:21:49 thingino-t31x user.err prudynt[1100]: [ERROR:Encoder.cpp]: Sink IMPDeviceSource clogged! Discarding NAL.
May 20 17:21:49 thingino-t31x user.err prudynt[1100]: [ERROR:Encoder.cpp]: Sink IMPDeviceSource clogged! Discarding NAL.
May 20 17:21:49 thingino-t31x user.err prudynt[1100]: [ERROR:Encoder.cpp]: Sink IMPDeviceSource clogged! Discarding NAL.
May 20 17:21:49 thingino-t31x user.err prudynt[1100]: [ERROR:Encoder.cpp]: Sink IMPDeviceSource clogged! Discarding NAL.
May 20 17:21:49 thingino-t31x user.err prudynt[1100]: [ERROR:Encoder.cpp]: Sink IMPDeviceSource clogged! Discarding NAL.
May 20 17:21:49 thingino-t31x user.err prudynt[1100]: [ERROR:Encoder.cpp]: Sink IMPDeviceSource clogged! Discarding NAL.
May 20 17:21:49 thingino-t31x user.err prudynt[1100]: [ERROR:Encoder.cpp]: Sink IMPDeviceSource clogged! Discarding NAL.
May 20 17:21:49 thingino-t31x user.err prudynt[1100]: [ERROR:Encoder.cpp]: Sink IMPDeviceSource clogged! Discarding NAL.
May 20 17:21:49 thingino-t31x user.err prudynt[1100]: [ERROR:Encoder.cpp]: Sink IMPDeviceSource clogged! Discarding NAL.
May 20 17:21:49 thingino-t31x user.debug prudynt[1100]: [DEBUG:IMPDeviceSource.cpp]: Device source destruct
May 20 17:21:49 thingino-t31x user.debug prudynt[1100]: [DEBUG:Encoder.hpp]: Destroy Sink: 116
May 20 17:22:00 thingino-t31x cron.info crond[1054]: USER root pid 5355 cmd /usr/sbin/daynightA few repeating lines has been omitted on the camera log.
gtxaspec
commented
2 months ago fixed in an upcoming prudynt release... stay tuned.
dparson55
commented
1 month ago @gtxaspec while the fix is being worked on, can you provide any guidance on what we could look for using a bash script to detect when streaming has stopped and trigger the camera to reboot?
I'm planning to write a script that looks at the camera's cpu usage to determine if streaming has stopped. When things are working the camera's cpu usage is usually above 5% but when streaming gets stuck, it seems to be consistently < 5%. I'm wondering if there is a more elegant method to determine streaming has stopped and if a surgical service restart could be used (rather than a camera reboot), so any guidance would be great.
I'm experiencing the same problem as OP with a wyze_c2_jxf22 and this particular thingino version info:
VERSION="1 (Ciao)"
BUILD_ID="master+6875626, 2024-07-29 06:36:42 -0400"
BUILDROOT_VERSION=-g568a8009
BUILDROOT_VERSION_ID=2024.05After I notice Frigate has stopped recording new files, I confirm the stream is stopped by checking the thingino web ui and it refreshes an old snapshot date time in the preview:
To fix it I ssh into the camera and reboot it or just use the web ui.
Edit: I finished a watchdog script and am giving this restart approach a shot for when the cpu usage of the prudynt process is < 5% (which seems to indicate that streaming has stopped). I'll post the entire script and installation details if it successfully recovers from the hung stream situation.
log "stopping stream-related services"
/etc/init.d/S96record stop
/etc/init.d/S96onvif stop
/etc/init.d/S96impconfig stop
/etc/init.d/S95prudynt stop
log "starting stream-related services"
/etc/init.d/S95prudynt start
/etc/init.d/S96impconfig start
/etc/init.d/S96onvif start
/etc/init.d/S96record start@kohalmi666 and @johndoewashere2 Here's an approach that seems to have solved the issue for my Wyze V2 camera...a script that runs on the camera and monitors the cpu usage of the prudynt process to indirectly determine if streaming has stopped, and restarts the streaming-related services when needed. It logs to a file and to the camera's main log. If you want to give it a try for yourself, here are the steps:
/etc/rc.local to add a command that runs the script /root/stream_watchdog.sh &/root/stream_watchdog.sh and make it executable chmod +x /root/stream_watchdog.sh#!/bin/sh
required_prudynt_cpu_usage=5 # if cpu usage isn't this or greater, restart the stream-related services
prudynt_check_seconds=15 # frigate has a 20 second watchdog for ffmpeg so we'll check more often than that
log_heartbeat_seconds=3600 # (3600=1 hour) we'll log how long the script has been running on this interval
logfile_max_number_of_lines=500 # limit the log file to this size
script_name="${0##*/}" # eg. stream_watchdog.sh
script_prefix=${script_name%.*} # eg. stream_watchdog
logfile="${0%/*}/$script_prefix.log" # logs go to the main system log and to this file
is_enabled=1 # allows the main loop to end if a SIGTERM is received (a user kills the process)
trap handle_sigterm SIGTERM
function handle_sigterm {
log "SIGTERM received"
is_enabled=0
}
function log {
echo "$(date): $1" >> $logfile
logger "$script_prefix: $1"
}
### MAIN ###
sleep 10 # wait for boot to complete
log "start"
script_start_time_seconds=$(date +%s)
last_heartbeat_message_seconds=$script_start_time_seconds
while [ $is_enabled -eq 1 ]; do
is_healthy=0
# check 3 different times to see if cpu usage is ok to hopefully avoid false positives
for wait_time in 0 2 3; do
sleep $wait_time
# grab cpu usage of the '/bin/prudynt' process
prudynt_cpu_usage_percent=$(top -n 1 | awk '/\/bin\/prudynt/ {print $7}')
# remove the trailing % character
prudynt_cpu_usage=${prudynt_cpu_usage_percent/%/}
if [ $prudynt_cpu_usage -ge $required_prudynt_cpu_usage ]; then
is_healthy=1
break # stop checking
fi
done
if [ $is_healthy -eq 0 ]; then
log "prudynt_cpu_usage=$prudynt_cpu_usage_percent, stream may have stopped"
log "stopping stream-related services"
/etc/init.d/S96record stop
/etc/init.d/S96onvif stop
/etc/init.d/S96impconfig stop
/etc/init.d/S95prudynt stop
log "starting stream-related services"
/etc/init.d/S95prudynt start
/etc/init.d/S96impconfig start
/etc/init.d/S96onvif start
/etc/init.d/S96record start
fi
# log a heartbeat showing how long the script has been running
current_time_seconds=$(date +%s)
seconds_since_last_heartbeat=$((current_time_seconds - last_heartbeat_message_seconds))
if [ $seconds_since_last_heartbeat -ge $log_heartbeat_seconds ]; then
last_heartbeat_message_seconds=$current_time_seconds
uptime_seconds=$((current_time_seconds - script_start_time_seconds))
uptime_days_hours_minutes=$(date -ud "@$uptime_seconds" +"$(( $uptime_seconds/3600/24 )) days %H hours %M minutes")
log "uptime $uptime_days_hours_minutes"
log "limiting log file size to $logfile_max_number_of_lines lines"
tail -n $logfile_max_number_of_lines $logfile > $logfile.tmp
mv $logfile.tmp $logfile
fi
# wait before checking again
sleep $prudynt_check_seconds
done
log "end"
exit 0/root/stream_watchdog.log)
I noticed the camera has the mosquitto mqtt client so in my own version of the script I added a call to publish logs to my node-red-based dashboard. And twice today the script has detected that streaming has stopped and restarted the necessary services. Here are the logs (in reverse order):
gtxaspec
commented
1 month ago we are working really hard to switch to the new prudynt nextgen, within the next few days hopefully. This solves the stream drops.
the stream watchdog was something that was planned, @dparson55 perhaps it can be included with the main repo?
dparson55
commented
1 month ago I'm really new to all this and just have been hacking my way through trying to get things working, so I'd just slow you all down with questions (I don't even know what the main repo is for example). I've never compiled anything in linux and am just learning with a raspberry pi 4.
BTW the stream_watchdog solution above still seems to be working well after 2 days of testing. It caught another stream stoppage and a totally separate frigate_watchdog didn't miss a beat (frigate itself is failing sometimes so I'm working on a watchdog for it as well).
gtxaspec
commented
1 month ago new streamer is live. you can sysupgrade or compile your own!
please test!
dparson55
commented
1 month ago Awesome, I did the sysupgrade and am giving it a shot, thanks!
dparson55
commented
1 month ago During the past 30 hours (since doing the upgrade) I haven't experienced any stream stoppages with my wyze_c2_jxf22 and build master+53a7bdc 2024-08-02 19:32:22 -0400. Previously I would have seen 2 or 3 stoppages during this time so it looks like the issue has been solved for me at least!
I had to turn off my stream watchdog since its indirect way of determining if streaming was working (by looking at prudynt's cpu usage) no longer worked. Too often prudynt was at 0% even though streaming was working perfectly. Frigate successfully collected the 30 hours of activity and spot checking its logs and recordings when prudynt was at 0% confirmed that streaming was not interrupted. There are still various Frigate errors that I attribute to using this wireless camera, but it handled them and the Frigate watchdog I made did not detect any missing recordings during the entire time.
So now I'm bummed that I'll eventually have to buy some more substantial hardware than my raspberry pi 4 to run Frigate object detection and add another camera since this is working so well :) Actually thank you though, it's working great!
gtxaspec
commented
1 month ago for frigate i would suggest anything you can plug a coral TPU into. a cheap used slim business desktop PC like the lenovo thinkcenter's are great and can be had for good pricing. couple that with a pci-e coral tpu (less than 30$) you get really fast object detection in frigate.
or you can get a USB coral tpu accellerator and use it with your RPI, expect to pay around 75-90$ USD for those.
closing this issue =D
(time stamp is not real because there is no internet connection.) logread:
Apr 17 18:09:32 thingino-t31x user.err prudynt[938]: [ERROR:Encoder.cpp]: Sink IMPDeviceSource clogged! Discarding NAL. Please alert the developers =D Apr 17 18:09:32 thingino-t31x user.err prudynt[938]: [ERROR:Encoder.cpp]: Sink IMPDeviceSource clogged! Discarding NAL. Please alert the developers =D Apr 17 18:09:32 thingino-t31x user.err prudynt[938]: [ERROR:Encoder.cpp]: Sink IMPDeviceSource clogged! Discarding NAL. Please alert the developers =D Apr 17 18:09:32 thingino-t31x user.err prudynt[938]: [ERROR:Encoder.cpp]: Sink IMPDeviceSource clogged! Discarding NAL. Please alert the developers =D Apr 17 18:09:32 thingino-t31x user.err prudynt[938]: [ERROR:Encoder.cpp]: Sink IMPDeviceSource clogged! Discarding NAL. Please alert the developers =D Apr 17 18:09:32 thingino-t31x user.err prudynt[938]: [ERROR:Encoder.cpp]: Sink IMPDeviceSource clogged! Discarding NAL. Please alert the developers =D Apr 17 18:09:32 thingino-t31x user.err prudynt[938]: [ERROR:Encoder.cpp]: Sink IMPDeviceSource clogged! Discarding NAL. Please alert the developers =D Apr 17 18:09:32 thingino-t31x user.err prudynt[938]: [ERROR:Encoder.cpp]: Sink IMPDeviceSource clogged! Discarding NAL. Please alert the developers =D Apr 17 18:09:32 thingino-t31x user.err prudynt[938]: [ERROR:Encoder.cpp]: Sink IMPDeviceSource clogged! Discarding NAL. Please alert the developers =D Apr 17 18:09:32 thingino-t31x user.err prudynt[938]: [ERROR:Encoder.cpp]: Sink IMPDeviceSource clogged! Discarding NAL. Please alert the developers =D Apr 17 18:09:32 thingino-t31x user.err prudynt[938]: [ERROR:Encoder.cpp]: Sink IMPDeviceSource clogged! Discarding NAL. Please alert the developers =D Apr 17 18:09:32 thingino-t31x user.err prudynt[938]: [ERROR:Encoder.cpp]: Sink IMPDeviceSource clogged! Discarding NAL. Please alert the developers =D Apr 17 18:09:32 thingino-t31x user.err prudynt[938]: [ERROR:Encoder.cpp]: Sink IMPDeviceSource clogged! Discarding NAL. Please alert the developers =D Apr 17 18:09:32 thingino-t31x user.err prudynt[938]: [ERROR:Encoder.cpp]: Sink IMPDeviceSource clogged! Discarding NAL. Please alert the developers =D Apr 17 18:09:32 thingino-t31x user.err prudynt[938]: [ERROR:Encoder.cpp]: Sink IMPDeviceSource clogged! Discarding NAL. Please alert the developers =D