blakeblackshear / frigate

NVR with realtime local object detection for IP cameras
https://frigate.video
MIT License
19k stars 1.73k forks source link

[Support]: Ffmpeg continuously crashing #2858

Closed tonitonae closed 2 years ago

tonitonae commented 2 years ago

Describe the problem you are having

Hi.

I am struggling to have frigate set up for 24/7 recording from two cameras on a RPi 4. I have been stucked as everybody else with the hwaccel bug for some time. With the latest release, now that is working (no more green screens, I can see the images from both cameras). Thanks!

My problem now is that I have a lot of errors on my logs and many frames dropped. Clips are not stored regularly every 1 minute. Instead, I have some clips of 10 seconds, other longer, sometimes no clip for 2-3 minutes. If I go to the recordings page in the UI, the largest aggregated length of a recording is of 8-9 minutes for an hour, which means that a lot of information is missing.

This is due, as the logs say, to the fact that ffmpeg crashes a lot (see below the errors). I do not know if this is due to my cameras or network setup. Which is weird is that this setup (network and cameras) has been working flawlessly most of the time with Synology Surveillance Station. I can also watch the streams with VLC and they seem quite fluent. I did not watch them for a long time, but I do not see those drops in a 3-5 minutes interval in which ffmpeg in the frigate container crashes several times.

As per the logs there is one thing that I do not understand. Whenever an error happens, the logs refer to a detect process which I, supposedly, have disabled in my config of frigate. I do not know if the name of the process does not actually mean that detection is being conducted or if the detection process is used for something else. I also see a warning at the beginning of the log about cpu detection not being recommended though I think that I am explicitly disabling it.

Sorry for the long message. I just wanted to provide enough context.

Thanks in advance for your help.

PS: both cameras are running unofficial yi-hack firmware, just in case someone has experience with those FWs.

PS2: maybe I should tweak some config of fmmpeg to make everything work properly, but I do not have any experience with that, so I do not know what to touch.

Version

0.11.0-3de1948

Frigate config file

mqtt:
  host: 192.168.1.XXX
  port: 1883
  user: XXXXX
  password: XXXXXXX
  topic_prefix: frigate
  client_id: frigate
  stats_interval: 60

#detectors:
#  coral:
#    type: edgetpu
#    device: usb

record:
  enabled: True
  expire_interval: 60
  retain:
    days: 21
    mode: all

#detect:
#  enabled: False

cameras:
  yi_outdoor:
    ffmpeg:
      inputs:
        - path: rtsp://192.168.1.XXX/ch0_0.h264
          roles:
#            - detect
            - rtmp
            - record
      hwaccel_args: -c:v h264_v4l2m2m
    detect:
      enabled: False
#      width:  1920
#      height: 1080
  yi_dome:
    ffmpeg:
      inputs:
        - path: rtsp://192.168.1.XXX/ch0_0.h264
          roles:
#            - detect
            - rtmp
            - record
      hwaccel_args: -c:v h264_v4l2m2m
    detect:
      enabled: False
#      width:  1920
#      height: 1080

Relevant log output

[s6-init] making user provided files available at /var/run/s6/etc...exited 0.
[s6-init] ensuring user provided files have correct perms...exited 0.
[fix-attrs.d] applying ownership & permissions fixes...
[fix-attrs.d] done.
[cont-init.d] executing container initialization scripts...
[cont-init.d] done.
[services.d] starting services
[services.d] done.
[2022-02-22 09:29:28] frigate.app                    INFO    : Starting Frigate (0.11.0-3de1948)
Starting migrations
[2022-02-22 09:29:28] peewee_migrate                 INFO    : Starting migrations
There is nothing to migrate
[2022-02-22 09:29:29] peewee_migrate                 INFO    : There is nothing to migrate
[2022-02-22 09:29:29] frigate.app                    INFO    : Output process started: 217
[2022-02-22 09:29:29] detector.cpu                   INFO    : Starting detection process: 216
[2022-02-22 09:29:29] frigate.edgetpu                WARNING : CPU detectors are not recommended and should only be used for testing or for trial purposes.
[2022-02-22 09:29:29] frigate.app                    INFO    : Camera processor started for yi_outdoor: 220
[2022-02-22 09:29:29] ws4py                          INFO    : Using epoll
[2022-02-22 09:29:29] frigate.app                    INFO    : Camera processor started for yi_dome: 223
[2022-02-22 09:29:29] frigate.app                    INFO    : Capture process started for yi_outdoor: 225
[2022-02-22 09:29:29] frigate.app                    INFO    : Capture process started for yi_dome: 226
[2022-02-22 09:29:29] ws4py                          INFO    : Using epoll
[2022-02-22 09:29:59] watchdog.yi_dome               INFO    : No frames received from yi_dome in 20 seconds. Exiting ffmpeg...
[2022-02-22 09:29:59] watchdog.yi_dome               INFO    : Waiting for ffmpeg to exit gracefully...
[2022-02-22 09:30:09] ws4py                          INFO    : Managing websocket [Local => 127.0.0.1:8082 | Remote => 127.0.0.1:41974]
[2022-02-22 09:30:11] frigate.video                  ERROR   : yi_dome: Unable to read frames from ffmpeg process.
[2022-02-22 09:30:11] frigate.video                  ERROR   : yi_dome: Unable to read frames from ffmpeg process.
[2022-02-22 09:30:11] frigate.video                  ERROR   : yi_dome: ffmpeg process is not running. exiting capture thread...
[2022-02-22 09:30:21] watchdog.yi_dome               ERROR   : Ffmpeg process crashed unexpectedly for yi_dome.
[2022-02-22 09:30:21] watchdog.yi_dome               ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
[2022-02-22 09:30:21] ffmpeg.yi_dome.detect          ERROR   : [flv @ 0x55a3e53be0] Failed to update header with correct duration.
[2022-02-22 09:30:21] ffmpeg.yi_dome.detect          ERROR   : [flv @ 0x55a3e53be0] Failed to update header with correct filesize.
[2022-02-22 09:31:54] frigate.video                  ERROR   : yi_outdoor: Unable to read frames from ffmpeg process.
[2022-02-22 09:31:54] frigate.video                  ERROR   : yi_outdoor: ffmpeg process is not running. exiting capture thread...
[2022-02-22 09:31:59] watchdog.yi_outdoor            ERROR   : Ffmpeg process crashed unexpectedly for yi_outdoor.
[2022-02-22 09:31:59] watchdog.yi_outdoor            ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
[2022-02-22 09:31:59] ffmpeg.yi_outdoor.detect       ERROR   : [rtsp @ 0x5578060390] CSeq 4 expected, 0 received.
[2022-02-22 09:31:59] ffmpeg.yi_outdoor.detect       ERROR   : [segment @ 0x5578081ea0] Timestamps are unset in a packet for stream 0. This is deprecated and will stop working in the future. Fix your code to set the timestamps properly
[2022-02-22 09:31:59] ffmpeg.yi_outdoor.detect       ERROR   : [flv @ 0x5578082af0] Timestamps are unset in a packet for stream 0. This is deprecated and will stop working in the future. Fix your code to set the timestamps properly
[2022-02-22 09:31:59] ffmpeg.yi_outdoor.detect       ERROR   : [segment @ 0x5578081ea0] Non-monotonous DTS in output stream 0:0; previous: 0, current: 0; changing to 1. This may result in incorrect timestamps in the output file.
[2022-02-22 09:31:59] ffmpeg.yi_outdoor.detect       ERROR   : [flv @ 0x5578082af0] Failed to update header with correct duration.
[2022-02-22 09:31:59] ffmpeg.yi_outdoor.detect       ERROR   : [flv @ 0x5578082af0] Failed to update header with correct filesize.
[2022-02-22 09:32:31] frigate.video                  ERROR   : yi_dome: Unable to read frames from ffmpeg process.
[2022-02-22 09:32:31] frigate.video                  ERROR   : yi_dome: ffmpeg process is not running. exiting capture thread...
[2022-02-22 09:32:31] watchdog.yi_dome               ERROR   : Ffmpeg process crashed unexpectedly for yi_dome.
[2022-02-22 09:32:31] watchdog.yi_dome               ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
[2022-02-22 09:32:31] ffmpeg.yi_dome.detect          ERROR   : [flv @ 0x55b41d0db0] Failed to update header with correct duration.
[2022-02-22 09:32:31] ffmpeg.yi_dome.detect          ERROR   : [flv @ 0x55b41d0db0] Failed to update header with correct filesize.
[2022-02-22 09:34:25] frigate.video                  ERROR   : yi_outdoor: Unable to read frames from ffmpeg process.
[2022-02-22 09:34:25] frigate.video                  ERROR   : yi_outdoor: Unable to read frames from ffmpeg process.
[2022-02-22 09:34:25] frigate.video                  ERROR   : yi_outdoor: Unable to read frames from ffmpeg process.
[2022-02-22 09:34:25] frigate.video                  ERROR   : yi_outdoor: ffmpeg process is not running. exiting capture thread...
[2022-02-22 09:34:29] watchdog.yi_outdoor            ERROR   : Ffmpeg process crashed unexpectedly for yi_outdoor.
[2022-02-22 09:34:29] watchdog.yi_outdoor            ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
[2022-02-22 09:34:29] ffmpeg.yi_outdoor.detect       ERROR   : [rtsp @ 0x5588a86390] CSeq 4 expected, 0 received.
[2022-02-22 09:34:29] ffmpeg.yi_outdoor.detect       ERROR   : [segment @ 0x5588aa6570] Timestamps are unset in a packet for stream 0. This is deprecated and will stop working in the future. Fix your code to set the timestamps properly
[2022-02-22 09:34:29] ffmpeg.yi_outdoor.detect       ERROR   : [flv @ 0x5588aa9680] Timestamps are unset in a packet for stream 0. This is deprecated and will stop working in the future. Fix your code to set the timestamps properly
[2022-02-22 09:34:29] ffmpeg.yi_outdoor.detect       ERROR   : [segment @ 0x5588aa6570] Non-monotonous DTS in output stream 0:0; previous: 0, current: 0; changing to 1. This may result in incorrect timestamps in the output file.
[2022-02-22 09:34:29] ffmpeg.yi_outdoor.detect       ERROR   : [flv @ 0x5588aa9680] Failed to update header with correct duration.
[2022-02-22 09:34:29] ffmpeg.yi_outdoor.detect       ERROR   : [flv @ 0x5588aa9680] Failed to update header with correct filesize.
[2022-02-22 09:35:02] frigate.video                  ERROR   : yi_dome: Unable to read frames from ffmpeg process.
[2022-02-22 09:35:02] frigate.video                  ERROR   : yi_dome: ffmpeg process is not running. exiting capture thread...
[2022-02-22 09:35:11] watchdog.yi_dome               ERROR   : Ffmpeg process crashed unexpectedly for yi_dome.
[2022-02-22 09:35:11] watchdog.yi_dome               ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
[2022-02-22 09:35:11] ffmpeg.yi_dome.detect          ERROR   : [rtsp @ 0x558c7b1390] CSeq 5 expected, 0 received.
[2022-02-22 09:35:11] ffmpeg.yi_dome.detect          ERROR   : [flv @ 0x558c8c06b0] Failed to update header with correct duration.
[2022-02-22 09:35:11] ffmpeg.yi_dome.detect          ERROR   : [flv @ 0x558c8c06b0] Failed to update header with correct filesize.
[2022-02-22 09:35:39] watchdog.yi_outdoor            INFO    : No frames received from yi_outdoor in 20 seconds. Exiting ffmpeg...
[2022-02-22 09:35:39] watchdog.yi_outdoor            INFO    : Waiting for ffmpeg to exit gracefully...
[2022-02-22 09:35:44] frigate.video                  ERROR   : yi_outdoor: Unable to read frames from ffmpeg process.
[2022-02-22 09:35:44] frigate.video                  ERROR   : yi_outdoor: ffmpeg process is not running. exiting capture thread...
[2022-02-22 09:35:54] watchdog.yi_outdoor            ERROR   : Ffmpeg process crashed unexpectedly for yi_outdoor.
[2022-02-22 09:35:54] watchdog.yi_outdoor            ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
[2022-02-22 09:35:54] ffmpeg.yi_outdoor.detect       ERROR   : [rtsp @ 0x558b3bf390] CSeq 4 expected, 0 received.
[2022-02-22 09:35:54] ffmpeg.yi_outdoor.detect       ERROR   : [segment @ 0x558b3c3950] Timestamps are unset in a packet for stream 0. This is deprecated and will stop working in the future. Fix your code to set the timestamps properly
[2022-02-22 09:35:54] ffmpeg.yi_outdoor.detect       ERROR   : [flv @ 0x558b3e1980] Timestamps are unset in a packet for stream 0. This is deprecated and will stop working in the future. Fix your code to set the timestamps properly
[2022-02-22 09:35:54] ffmpeg.yi_outdoor.detect       ERROR   : [segment @ 0x558b3c3950] Non-monotonous DTS in output stream 0:0; previous: 0, current: 0; changing to 1. This may result in incorrect timestamps in the output file.
[2022-02-22 09:35:54] ffmpeg.yi_outdoor.detect       ERROR   : [flv @ 0x558b3e1980] Failed to update header with correct duration.
[2022-02-22 09:35:54] ffmpeg.yi_outdoor.detect       ERROR   : [flv @ 0x558b3e1980] Failed to update header with correct filesize.
[2022-02-22 09:36:24] watchdog.yi_outdoor            INFO    : No frames received from yi_outdoor in 20 seconds. Exiting ffmpeg...
[2022-02-22 09:36:24] watchdog.yi_outdoor            INFO    : Waiting for ffmpeg to exit gracefully...

...

[2022-02-22 10:18:55] frigate.video                  ERROR   : yi_dome: Unable to read frames from ffmpeg process.
[2022-02-22 10:18:55] frigate.video                  ERROR   : yi_dome: ffmpeg process is not running. exiting capture thread...
[2022-02-22 10:19:01] frigate.video                  ERROR   : yi_outdoor: Unable to read frames from ffmpeg process.
[2022-02-22 10:19:01] frigate.video                  ERROR   : yi_outdoor: ffmpeg process is not running. exiting capture thread...
[2022-02-22 10:19:04] watchdog.yi_dome               ERROR   : Ffmpeg process crashed unexpectedly for yi_dome.
[2022-02-22 10:19:04] watchdog.yi_dome               ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
[2022-02-22 10:19:04] ffmpeg.yi_dome.detect          ERROR   : [h264 @ 0x55a2d66260] error while decoding MB 59 40, bytestream -7
[2022-02-22 10:19:04] ffmpeg.yi_dome.detect          ERROR   : [h264_v4l2m2m @ 0x55a2d678e0] === poll unexpected TIMEOUT: events=0x43, cap buffers=20
[2022-02-22 10:19:04] ffmpeg.yi_dome.detect          ERROR   : [flv @ 0x55a2d85de0] Failed to update header with correct duration.
[2022-02-22 10:19:04] ffmpeg.yi_dome.detect          ERROR   : [flv @ 0x55a2d85de0] Failed to update header with correct filesize.
[2022-02-22 10:19:11] watchdog.yi_outdoor            ERROR   : Ffmpeg process crashed unexpectedly for yi_outdoor.
[2022-02-22 10:19:11] watchdog.yi_outdoor            ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
[2022-02-22 10:19:11] ffmpeg.yi_outdoor.detect       ERROR   : [rtsp @ 0x55ab05a390] CSeq 4 expected, 0 received.
[2022-02-22 10:19:11] ffmpeg.yi_outdoor.detect       ERROR   : [segment @ 0x55ab07bea0] Timestamps are unset in a packet for stream 0. This is deprecated and will stop working in the future. Fix your code to set the timestamps properly
[2022-02-22 10:19:11] ffmpeg.yi_outdoor.detect       ERROR   : [flv @ 0x55ab07cf30] Timestamps are unset in a packet for stream 0. This is deprecated and will stop working in the future. Fix your code to set the timestamps properly
[2022-02-22 10:19:11] ffmpeg.yi_outdoor.detect       ERROR   : [segment @ 0x55ab07bea0] Non-monotonous DTS in output stream 0:0; previous: 0, current: 0; changing to 1. This may result in incorrect timestamps in the output file.
[2022-02-22 10:19:11] ffmpeg.yi_outdoor.detect       ERROR   : [flv @ 0x55ab07cf30] Failed to update header with correct duration.
[2022-02-22 10:19:11] ffmpeg.yi_outdoor.detect       ERROR   : [flv @ 0x55ab07cf30] Failed to update header with correct filesize.
[2022-02-22 10:19:46] ws4py                          INFO    : Terminating websocket [Local => 127.0.0.1:5002 | Remote => 127.0.0.1:60538]
[2022-02-22 10:19:54] ws4py                          INFO    : Terminating websocket [Local => 127.0.0.1:8082 | Remote => 127.0.0.1:44998]
[2022-02-22 10:20:04] watchdog.yi_dome               INFO    : No frames received from yi_dome in 20 seconds. Exiting ffmpeg...
[2022-02-22 10:20:04] watchdog.yi_dome               INFO    : Waiting for ffmpeg to exit gracefully...
[2022-02-22 10:20:12] frigate.video                  ERROR   : yi_dome: Unable to read frames from ffmpeg process.
[2022-02-22 10:20:12] frigate.video                  ERROR   : yi_dome: ffmpeg process is not running. exiting capture thread...
[2022-02-22 10:20:22] watchdog.yi_dome               ERROR   : Ffmpeg process crashed unexpectedly for yi_dome.
[2022-02-22 10:20:22] watchdog.yi_dome               ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
[2022-02-22 10:20:22] ffmpeg.yi_dome.detect          ERROR   : [NULL @ 0x5574c9f260] missing picture in access unit with size 52
[2022-02-22 10:20:22] ffmpeg.yi_dome.detect          ERROR   : [flv @ 0x5574daf4b0] Failed to update header with correct duration.
[2022-02-22 10:20:22] ffmpeg.yi_dome.detect          ERROR   : [flv @ 0x5574daf4b0] Failed to update header with correct filesize.
[2022-02-22 10:20:31] frigate.video                  ERROR   : yi_outdoor: Unable to read frames from ffmpeg process.
[2022-02-22 10:20:31] frigate.video                  ERROR   : yi_outdoor: Unable to read frames from ffmpeg process.
[2022-02-22 10:20:31] frigate.video                  ERROR   : yi_outdoor: ffmpeg process is not running. exiting capture thread...
[2022-02-22 10:20:31] watchdog.yi_outdoor            ERROR   : Ffmpeg process crashed unexpectedly for yi_outdoor.
[2022-02-22 10:20:31] watchdog.yi_outdoor            ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
[2022-02-22 10:20:31] ffmpeg.yi_outdoor.detect       ERROR   : [rtsp @ 0x559baff390] CSeq 4 expected, 0 received.
[2022-02-22 10:20:31] ffmpeg.yi_outdoor.detect       ERROR   : [h264_v4l2m2m @ 0x559bb03b10] === poll unexpected TIMEOUT: events=0x43, cap buffers=20
[2022-02-22 10:20:31] ffmpeg.yi_outdoor.detect       ERROR   : [flv @ 0x559bb21680] Failed to update header with correct duration.
[2022-02-22 10:20:31] ffmpeg.yi_outdoor.detect       ERROR   : [flv @ 0x559bb21680] Failed to update header with correct filesize.
[2022-02-22 10:21:12] watchdog.yi_dome               INFO    : No frames received from yi_dome in 20 seconds. Exiting ffmpeg...
[2022-02-22 10:21:12] watchdog.yi_dome               INFO    : Waiting for ffmpeg to exit gracefully...
[2022-02-22 10:21:21] frigate.video                  ERROR   : yi_dome: Unable to read frames from ffmpeg process.
[2022-02-22 10:21:21] frigate.video                  ERROR   : yi_dome: ffmpeg process is not running. exiting capture thread...
[2022-02-22 10:21:25] frigate.video                  ERROR   : yi_outdoor: Unable to read frames from ffmpeg process.
[2022-02-22 10:21:25] frigate.video                  ERROR   : yi_outdoor: ffmpeg process is not running. exiting capture thread...
[2022-02-22 10:21:31] watchdog.yi_outdoor            ERROR   : Ffmpeg process crashed unexpectedly for yi_outdoor.
[2022-02-22 10:21:31] watchdog.yi_outdoor            ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
[2022-02-22 10:21:31] ffmpeg.yi_outdoor.detect       ERROR   : [rtsp @ 0x5591f6c390] CSeq 4 expected, 0 received.
[2022-02-22 10:21:31] ffmpeg.yi_outdoor.detect       ERROR   : [segment @ 0x55920cb5f0] Timestamps are unset in a packet for stream 0. This is deprecated and will stop working in the future. Fix your code to set the timestamps properly
[2022-02-22 10:21:31] ffmpeg.yi_outdoor.detect       ERROR   : [flv @ 0x5591f8e980] Timestamps are unset in a packet for stream 0. This is deprecated and will stop working in the future. Fix your code to set the timestamps properly
[2022-02-22 10:21:31] ffmpeg.yi_outdoor.detect       ERROR   : [segment @ 0x55920cb5f0] Non-monotonous DTS in output stream 0:0; previous: 0, current: 0; changing to 1. This may result in incorrect timestamps in the output file.
[2022-02-22 10:21:31] ffmpeg.yi_outdoor.detect       ERROR   : [flv @ 0x5591f8e980] Failed to update header with correct duration.
[2022-02-22 10:21:31] ffmpeg.yi_outdoor.detect       ERROR   : [flv @ 0x5591f8e980] Failed to update header with correct filesize.
[2022-02-22 10:21:31] watchdog.yi_dome               ERROR   : Ffmpeg process crashed unexpectedly for yi_dome.
[2022-02-22 10:21:31] watchdog.yi_dome               ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
[2022-02-22 10:21:31] ffmpeg.yi_dome.detect          ERROR   : [flv @ 0x55bbead520] Failed to update header with correct duration.
[2022-02-22 10:21:31] ffmpeg.yi_dome.detect          ERROR   : [flv @ 0x55bbead520] Failed to update header with correct filesize.

FFprobe output from your camera

Camera #1

❯ ffprobe rtsp://192.168.1.XXX/ch0_0.h264
ffprobe version 4.4.1 Copyright (c) 2007-2021 the FFmpeg developers
  built with Apple clang version 12.0.0 (clang-1200.0.32.29)
  configuration: --prefix=/usr/local/Cellar/ffmpeg/4.4.1_3 --enable-shared --enable-pthreads --enable-version3 --cc=clang --host-cflags= --host-ldflags= --enable-ffplay --enable-gnutls --enable-gpl --enable-libaom --enable-libbluray --enable-libdav1d --enable-libmp3lame --enable-libopus --enable-librav1e --enable-librist --enable-librubberband --enable-libsnappy --enable-libsrt --enable-libtesseract --enable-libtheora --enable-libvidstab --enable-libvmaf --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libxvid --enable-lzma --enable-libfontconfig --enable-libfreetype --enable-frei0r --enable-libass --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libspeex --enable-libsoxr --enable-libzmq --enable-libzimg --disable-libjack --disable-indev=jack --enable-avresample --enable-videotoolbox
  libavutil      56. 70.100 / 56. 70.100
  libavcodec     58.134.100 / 58.134.100
  libavformat    58. 76.100 / 58. 76.100
  libavdevice    58. 13.100 / 58. 13.100
  libavfilter     7.110.100 /  7.110.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  9.100 /  5.  9.100
  libswresample   3.  9.100 /  3.  9.100
  libpostproc    55.  9.100 / 55.  9.100
[rtsp @ 0x7fd264009000] CSeq 4 expected, 0 received.
Input #0, rtsp, from 'rtsp://192.168.1.XXX/ch0_0.h264':
  Metadata:
    title           : Session streamed by "rtspv4"
    comment         : ch0_0.h264
  Duration: N/A, start: 0.050000, bitrate: N/A
  Stream #0:0: Video: h264 (High), yuvj420p(pc, bt709, progressive), 1920x1080, 20 fps, 20 tbr, 90k tbn, 180k tbc

Camera #2

❯ ffprobe rtsp://192.168.1.XXX/ch0_0.h264
ffprobe version 4.4.1 Copyright (c) 2007-2021 the FFmpeg developers
  built with Apple clang version 12.0.0 (clang-1200.0.32.29)
  configuration: --prefix=/usr/local/Cellar/ffmpeg/4.4.1_3 --enable-shared --enable-pthreads --enable-version3 --cc=clang --host-cflags= --host-ldflags= --enable-ffplay --enable-gnutls --enable-gpl --enable-libaom --enable-libbluray --enable-libdav1d --enable-libmp3lame --enable-libopus --enable-librav1e --enable-librist --enable-librubberband --enable-libsnappy --enable-libsrt --enable-libtesseract --enable-libtheora --enable-libvidstab --enable-libvmaf --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libxvid --enable-lzma --enable-libfontconfig --enable-libfreetype --enable-frei0r --enable-libass --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libspeex --enable-libsoxr --enable-libzmq --enable-libzimg --disable-libjack --disable-indev=jack --enable-avresample --enable-videotoolbox
  libavutil      56. 70.100 / 56. 70.100
  libavcodec     58.134.100 / 58.134.100
  libavformat    58. 76.100 / 58. 76.100
  libavdevice    58. 13.100 / 58. 13.100
  libavfilter     7.110.100 /  7.110.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  9.100 /  5.  9.100
  libswresample   3.  9.100 /  3.  9.100
  libpostproc    55.  9.100 / 55.  9.100
Input #0, rtsp, from 'rtsp://192.168.1.XXX/ch0_0.h264':
  Metadata:
    title           : Session streamed by "rRTSPServer"
    comment         : ch0_0.h264
  Duration: N/A, start: 0.050000, bitrate: N/A
  Stream #0:0: Video: h264 (Main), yuvj420p(pc, progressive), 1920x1088, 20 fps, 20 tbr, 90k tbn, 180k tbc

Frigate stats

{"detection_fps":0.0,"detectors":{"cpu":{"detection_start":0.0,"inference_speed":10.0,"pid":216}},"service":{"storage":{"/dev/shm":{"free":64.0,"mount_type":"tmpfs","total":67.1,"used":3.2},"/media/frigate/clips":{"free":877255.6,"mount_type":"ext4","total":984280.7,"used":67006.4},"/media/frigate/recordings":{"free":877255.6,"mount_type":"ext4","total":984280.7,"used":67006.4},"/tmp/cache":{"free":999.7,"mount_type":"tmpfs","total":1000.0,"used":0.3}},"temperatures":{},"uptime":3263,"version":"0.11.0-3de1948"},"yi_dome":{"camera_fps":2.76,"capture_pid":226,"detection_fps":0.0,"pid":223,"process_fps":1.6,"skipped_fps":0.0},"yi_outdoor":{"camera_fps":5.0,"capture_pid":225,"detection_fps":0.0,"pid":220,"process_fps":5.0,"skipped_fps":0.0}}

Operating system

Other Linux

Install method

Docker Compose

Coral version

CPU (no coral)

Network connection

Wireless

Camera make and model

Yi Outdoor and Yo Dome 1080p

Any other information that may be helpful

I do have a coral device (USB) but I can not connect it to my RPi as it hangs for some reason :/

I use this box:

https://www.amazon.es/gp/product/B095WD7BTW

Which has a sort of USB bridge to connect the 2.5 HDD to one of the USB 3 ports. If I connect the Coral to the other USB 3 port, then it complains about power during boot. I have a USB hub:

https://www.amazon.es/gp/product/B00TPMEOYM

But, if I connect it to the second USB port, then the RPi does not boot. Or, If I connect it when the Rpi is running, then if hangs at some point. It seems like some kind of interference with the "USB bridge". I need more testing (maybe connecting the HDD to the hub directly with a USB A - USB A cable) but using coral is not an option right now.

Regarding the OS, I use DietPi 64bit.

Thanks for your help.

blakeblackshear commented 2 years ago

It looks like instability in the stream is causing the RTMP relay to fail. It is very sensitive to bad data. I would try disabling RTMP with:

rtmp:
  enabled: False

The detect role cannot be removed and Frigate is adding it back in automatically. This is the role that decodes the video stream, does motion detection, and provides images for the frontend. Disabling detect just prevents the object detection from being run on motion.

If you are still seeing failures that cause the recording segments to go missing, you can try running a separate process for record and detect like this:

cameras:
  yi_outdoor:
    ffmpeg:
      inputs:
        - path: rtsp://192.168.1.XXX/ch0_0.h264
          hwaccel_args: -c:v h264_v4l2m2m
          roles:
            - detect
        - path: rtsp://192.168.1.XXX/ch0_0.h264
          roles:
            - record
    detect:
      enabled: False
#      width:  1920
#      height: 1080

Lastly, your issues with the Coral sound like the normal power related issues. The Coral uses a lot of power for a USB device, so it is recommended to use a USB hub with an external power supply with a Pi.

tonitonae commented 2 years ago

Hi @blakeblackshear.

Thanks for the prompt reply.

I have made the suggested changes but I still get crashes (although log messages are partly different):

[2022-02-22 13:59:46] frigate.video                  ERROR   : yi_outdoor: Unable to read frames from ffmpeg process.
[2022-02-22 13:59:46] frigate.video                  ERROR   : yi_outdoor: ffmpeg process is not running. exiting capture thread...
[2022-02-22 13:59:47] ffmpeg.yi_dome.record          ERROR   : [NULL @ 0x5585284030] missing picture in access unit with size 52
[2022-02-22 13:59:47] watchdog.yi_dome               INFO    : Terminating the existing ffmpeg process...
[2022-02-22 13:59:47] watchdog.yi_dome               INFO    : Waiting for ffmpeg to exit gracefully...
[2022-02-22 13:59:48] watchdog.yi_outdoor            ERROR   : Ffmpeg process crashed unexpectedly for yi_outdoor.
[2022-02-22 13:59:48] watchdog.yi_outdoor            ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
[2022-02-22 13:59:48] ffmpeg.yi_outdoor.detect       ERROR   : [rtsp @ 0x55bff34130] CSeq 4 expected, 0 received.
[2022-02-22 13:59:48] ffmpeg.yi_outdoor.detect       ERROR   : [h264_v4l2m2m @ 0x55c001b820] === poll unexpected TIMEOUT: events=0x43, cap buffers=20
[2022-02-22 13:59:57] ffmpeg.yi_dome.record          ERROR   : rtsp://192.168.1.164/ch0_0.h264: Invalid data found when processing input
[2022-02-22 13:59:57] watchdog.yi_dome               INFO    : Terminating the existing ffmpeg process...
[2022-02-22 13:59:57] watchdog.yi_dome               INFO    : Waiting for ffmpeg to exit gracefully...
[2022-02-22 14:00:07] ffmpeg.yi_dome.record          ERROR   : rtsp://192.168.1.164/ch0_0.h264: Invalid data found when processing input
[2022-02-22 14:00:07] watchdog.yi_dome               INFO    : Terminating the existing ffmpeg process...
[2022-02-22 14:00:07] watchdog.yi_dome               INFO    : Waiting for ffmpeg to exit gracefully...
[2022-02-22 14:00:17] ffmpeg.yi_dome.record          ERROR   : rtsp://192.168.1.164/ch0_0.h264: Invalid data found when processing input
[2022-02-22 14:00:17] watchdog.yi_dome               INFO    : Terminating the existing ffmpeg process...
[2022-02-22 14:00:17] watchdog.yi_dome               INFO    : Waiting for ffmpeg to exit gracefully...
[2022-02-22 14:00:18] watchdog.yi_outdoor            INFO    : No frames received from yi_outdoor in 20 seconds. Exiting ffmpeg...
[2022-02-22 14:00:18] watchdog.yi_outdoor            INFO    : Waiting for ffmpeg to exit gracefully...
[2022-02-22 14:00:26] frigate.video                  ERROR   : yi_outdoor: Unable to read frames from ffmpeg process.
[2022-02-22 14:00:26] frigate.video                  ERROR   : yi_outdoor: Unable to read frames from ffmpeg process.
[2022-02-22 14:00:26] frigate.video                  ERROR   : yi_outdoor: Unable to read frames from ffmpeg process.
[2022-02-22 14:00:26] frigate.video                  ERROR   : yi_outdoor: ffmpeg process is not running. exiting capture thread...
[2022-02-22 14:00:27] ffmpeg.yi_dome.record          ERROR   : rtsp://192.168.1.164/ch0_0.h264: Invalid data found when processing input
[2022-02-22 14:00:27] watchdog.yi_dome               INFO    : Terminating the existing ffmpeg process...
[2022-02-22 14:00:27] watchdog.yi_dome               INFO    : Waiting for ffmpeg to exit gracefully...
[2022-02-22 14:00:36] watchdog.yi_outdoor            ERROR   : Ffmpeg process crashed unexpectedly for yi_outdoor.
[2022-02-22 14:00:36] watchdog.yi_outdoor            ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
[2022-02-22 14:00:36] ffmpeg.yi_outdoor.detect       ERROR   : [rtsp @ 0x557c85f130] CSeq 4 expected, 0 received.
[2022-02-22 14:00:37] ffmpeg.yi_dome.record          ERROR   : rtsp://192.168.1.164/ch0_0.h264: Invalid data found when processing input
[2022-02-22 14:00:37] watchdog.yi_dome               INFO    : Terminating the existing ffmpeg process...
[2022-02-22 14:00:37] watchdog.yi_dome               INFO    : Waiting for ffmpeg to exit gracefully...
[2022-02-22 14:00:47] ffmpeg.yi_dome.record          ERROR   : rtsp://192.168.1.164/ch0_0.h264: Invalid data found when processing input
[2022-02-22 14:00:47] watchdog.yi_dome               INFO    : Terminating the existing ffmpeg process...
[2022-02-22 14:00:47] watchdog.yi_dome               INFO    : Waiting for ffmpeg to exit gracefully...
[2022-02-22 14:00:57] ffmpeg.yi_dome.record          ERROR   : rtsp://192.168.1.164/ch0_0.h264: Invalid data found when processing input
[2022-02-22 14:00:57] watchdog.yi_dome               INFO    : Terminating the existing ffmpeg process...
[2022-02-22 14:00:57] watchdog.yi_dome               INFO    : Waiting for ffmpeg to exit gracefully...
[2022-02-22 14:01:07] ffmpeg.yi_dome.record          ERROR   : rtsp://192.168.1.164/ch0_0.h264: Invalid data found when processing input
[2022-02-22 14:01:07] watchdog.yi_dome               INFO    : Terminating the existing ffmpeg process...
[2022-02-22 14:01:07] watchdog.yi_dome               INFO    : Waiting for ffmpeg to exit gracefully...
[2022-02-22 14:01:17] ffmpeg.yi_dome.record          ERROR   : rtsp://192.168.1.164/ch0_0.h264: Invalid data found when processing input
[2022-02-22 14:01:17] watchdog.yi_dome               INFO    : Terminating the existing ffmpeg process...
[2022-02-22 14:01:17] watchdog.yi_dome               INFO    : Waiting for ffmpeg to exit gracefully...
[2022-02-22 14:01:27] ffmpeg.yi_dome.record          ERROR   : rtsp://192.168.1.164/ch0_0.h264: Invalid data found when processing input
[2022-02-22 14:01:27] watchdog.yi_dome               INFO    : Terminating the existing ffmpeg process...
[2022-02-22 14:01:27] watchdog.yi_dome               INFO    : Waiting for ffmpeg to exit gracefully...
[2022-02-22 14:01:31] ws4py                          INFO    : Terminating websocket [Local => 127.0.0.1:5002 | Remote => 127.0.0.1:47024]
[2022-02-22 14:01:37] ffmpeg.yi_dome.record          ERROR   : rtsp://192.168.1.164/ch0_0.h264: Invalid data found when processing input
[2022-02-22 14:01:37] watchdog.yi_dome               INFO    : Terminating the existing ffmpeg process...
[2022-02-22 14:01:37] watchdog.yi_dome               INFO    : Waiting for ffmpeg to exit gracefully...
[2022-02-22 14:01:41] frigate.video                  ERROR   : yi_dome: Unable to read frames from ffmpeg process.
[2022-02-22 14:01:41] frigate.video                  ERROR   : yi_dome: ffmpeg process is not running. exiting capture thread...
[2022-02-22 14:01:47] watchdog.yi_dome               ERROR   : Ffmpeg process crashed unexpectedly for yi_dome.
[2022-02-22 14:01:47] watchdog.yi_dome               ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
[2022-02-22 14:01:47] ffmpeg.yi_dome.detect          ERROR   : [rtsp @ 0x557c812130] CSeq 5 expected, 0 received.
[2022-02-22 14:01:47] ffmpeg.yi_dome.detect          ERROR   : [h264_v4l2m2m @ 0x557c816840] === poll unexpected TIMEOUT: events=0x43, cap buffers=20

Clips are still missing on both cameras (especially on the outdoor one, with a long crash every 2-3 minutes). Any other suggestions?

I also observe (actually hear the fans) that the ffmpeg process gets stuck frequently with 100% usage of the cpu. I do not think that happened before (but I am not sure).

Finally, regarding the coral, I forgot to say that the USB hub is powered, so that should not be the problem (I guess...).

Thanks again for your help!

tonitonae commented 2 years ago

Hi again.

I have been playing around and it results that it was RSTP server's fault. I was using yi-hack-mstar v0.3.8 on one of the cameras and after updating to v0.4.4 and switching to an alternate server provided in this version everything seems to run much better. 15 minutes of uptime an no errors on the logs. Recordings also seem to be working fine.

I will try to upgrade the version of the firmware of the other camera tomorrow. It is installed outside and now is dark here.

I have however a couple of questions.

  1. I thought recordings where stored in 1 minute long files. However, in my case at least a new file is created every 10 seconds. Is this the expected behavior?
  2. When I watch the live stream through the UI I observe some glitches (the bottom of the image is distorted, like the same horizontal line being copied over and over til the half of the image, sometimes more, sometimes less). It is not that important as clips are recorded fine. Maybe it is a problem with my browser (I use firefox). I will try Chrome and see if that also happens there.

Thanks for everything. I will report back with the second camera issue, just in case somebody else is facing similar problems.

tonitonae commented 2 years ago

Well... it crashed after 30 minutes. More work needed server-side... :/

NickM-27 commented 2 years ago

@tonitonae Yeah, 30 minutes of no crashing is good. My Wyze cam also has crashes here and there just due to "meh" firmware unfortunately.

As far as your questions go:

  1. Frigate, by default, records in 10 second segments
  2. I see this behavior rarely with one of my amcrest cameras where as my dog runs across the yard he leaves a trail of smeared pixels behind him. From what I can tell, in my case at least, it seems to be a compression issue on the camera itself.
tonitonae commented 2 years ago

Hi @NickM-27. Thanks for the prompt reply.

I am trying to figure out if something can be done server side to avoid these problems.

Regarding 1, understood.

As per 2: I checked the recorded clips and did not see anything on them, so it has to be something on the client (maybe specific to firefox, maybe not, I have to check).

Thanks!

tonitonae commented 2 years ago

Hi @blakeblackshear.

I am experimenting a little bit more with the RTSP server. I do not know if I should say this, but now if has been running without issues for more than 80 minutes (compared to a maximum of half an hour yesterday it seems that something I have touched server-side has significantly improved its reliability).

Anyway, what I have done now is monitoring frigate. I see no errors on the logs (good) but I see that the ffmpeg process in charge of the detection, even with the HW acceleration flags, is consuming 100% of one of the CPUs of the RPi 4. Is this the expected behavior?

I still need to grab a USB A - USB A cable to see if I can use my HDD and the coral simultaneously, both connected to the USB hub (powered) but I wanted first to check if I can make the cameras and frigate work together.

Thanks for everything!

blakeblackshear commented 2 years ago

It simply takes a lot of resources to decode a 1080p video stream at 20fps. This is why it is recommended to use cameras with configurable frame rates and resolutions. The Pi can handle many more cameras if the streams are 720p at 5fps.

In the config you posted, the detect resolution is commented out. This will cause ffmpeg to resize the stream to 720p before frigate processes it. This will reduce the CPU used by frigate, but increase the CPU used by ffmpeg. If you specify the resolution as 1080p in the detect config, it should lower the CPU used by ffmpeg and increase it for other processes.

tonitonae commented 2 years ago

Thanks @blakeblackshear for the suggestion. I uncommented the resolution lines and now ffmpeg uses a lot less resources (around 15% of CPU, more or less).

Another thing that I seem not to have understood completely is the detect role. You told me in a previous comment that it is mandatory and it is re-enabled automatically by frigate. In my config I had it with enabled: False and I had the warning in the log file about CPU detectors, so I thought that frigate was ignoring my False and using a True for that option.

However, if I explicitly put True in the config file, then a frigate.detector (or similar, I did note it, sorry) process spawns taking 180-200% of CPU. Reverting it back to False that process does not appear.

Obviously, it seems to be a difference between using True and False even if frigate is reenabling it. Does this mean that the detector process is always running but it does different things whether the config option is True or False?

Thanks!

blakeblackshear commented 2 years ago

The role is required, and frigate will add it to the list of roles under your input. Setting enabled to False tells frigate not to run object detection. It still needs the role assigned to an input so it knows which stream to use for motion detection and to display images and the live view in the UI.

tonitonae commented 2 years ago

Ok, that makes sense.

One last question (sorry for making too many of them :/): which is the purpose of motion detection? I do not see anything in the UI related to motion detection (I see the object detection and I even picked my wife while I was testing with the config XDDD) but I do not see how motion detection can be used. Is it described in the docs? I did not seem to find it, just about motion masks, but not motion detection itself.

Thanks one more time.

NickM-27 commented 2 years ago

@tonitonae Motion detection has a handful of different uses.

Frigate uses motion detection to decide when to run object detection. It isn't run on every frame just the frames where motion is detected.

In 0.10.0 there is also the ability to set retain modes on 24/7 recording and events so that it only saves the recording segments that have motion.

Currently this is only documented in the config file: https://docs.frigate.video/configuration/index

tonitonae commented 2 years ago

Thanks @NickM-27. If it is only in the config that is why I was unable to find it.

You did not mention and I did not see it so I imagine it is not possible, but I will just ask. In principle, I like to have 24/7 recordings, but it would be very nice if frigate could label the moments in which motion was detected similarly to what is done with object detection. Would that be possible? Maybe also creating a snapshot with a timestamp to easily identify where to look would be interesting.

I am aware that there are many things to do, but I think that some kind of tagging for motion events in 24/7 recordings would be interesting.

Thanks again for both the cool project and the help given here.

Cheers!

NickM-27 commented 2 years ago

It is currently not supported, however it is something that has been requested somewhat often. I am not 100% sure, but believe all the pieces besides the UI are there. Not sure where it fits in to the roadmap, that would be a question for blakeblackshear.

tonitonae commented 2 years ago

Ok, thanks for the feedback. Hope it is not very low in the todo list ;-)

stale[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

outofsight commented 2 years ago

Hello @tonitonae,

I'm running into same problems with Yi Hacked cameras (AllWinner and AllWinner2 hacks).

Did you managed to solve issues and avoid ffmpeg crash?

What are your best settings/configurations?

Thanks

tonitonae commented 2 years ago

Hello @tonitonae,

I'm running into same problems with Yi Hacked cameras (AllWinner and AllWinner2 hacks).

Did you managed to solve issues and avoid ffmpeg crash?

What are your best settings/configurations?

Thanks

Hi @outofsight.

I do not have any AllWinner camera. I am using yi-hack v5 for my outdoor camera and yi-hack-mstar for my Dome camera. The first works more or less flawlessly (some frame drops here and there, but no crashes for weeks) but the MStar one crashes frequently. I think I have narrowed it down and crashes seem to be related to the switch of the camera to night vision when light from the outside gets too low. I have downloaded the source code of the RTSP server, but I did not have the time to work on it (too busy at the moment). If I find something interesting, I will let you know.

Cheers!

cdk222 commented 2 years ago

Hi @tonitonae

Would you be able to post the config.yml file that you finally arrived at. Im having same issue

thanks

husnainali09 commented 2 years ago

Anyone have good working ffmped args for the nyihack5 that works reliably.

r00HuNTeR commented 2 years ago

I got rid of the errors with the following line input_args: -rtsp_transport tcp mqtt: host: 192.168.1... port: 1883 user: user password: pass cameras: yi-hack-v4: ffmpeg: input_args: -rtsp_transport tcp inputs:

max-mayorov commented 2 years ago

input_args: -rtsp_transport tcp

Saved my night! 🦄

harambe88 commented 1 year ago

Saved my night too!

I had the problem that frigate didn't record clips and snapshots anymore while live view was still working. There were lots of errors in the frigate log like "'Unable to open RTSP for listening'" and "ffmpeg process crashed unexpectedly" and the like.

Adding this option in /config/frigate.yml

input_args: -rtsp_transport tcp

solved this. THANK YOU!!

ale-saglia commented 4 months ago

I'm in the same situation with a Yi camera with custom firmware.

Mister-A commented 4 weeks ago

Commenting only in case useful to future Google searchers like myself.

I've gone from all FFMPEG processess crashing every 10 minutes to 1.5 hours so far with no issues at all by adding:

ffmpeg:
    input_args: -rtsp_transport tcp

In my case all Reolink and Veezoom cameras.