search

blakeblackshear / frigate

NVR with realtime local object detection for IP cameras
https://frigate.video
MIT License
19.57k stars 1.8k forks source link

[Support]: frigate not resuming detection stream even if its available and working #8624

Closed ristoavila closed 1 year ago

ristoavila commented 1 year ago

Describe the problem you are having

I have a problem where the sub stream just stops and frigate doesn't restart the ffmpeg process for the sub_stream. When accessing the substream it works perfectly fine. How ever it is not able to recover from the failure it received. I also dont see anything wrong in go2rtc logs. To me it seems like watchdog waits for ever for something that never happens.

I was thinking that could/should there bere a mqtt API to show possible error if camera detection stream doesn't receive frames for x period of time and after this one could force restart of the detection stream (also through mqtt).

Version

0.12

Frigate config file

mqtt:
  host: 192.168.0.13
  port: 1883
  topic_prefix: frigate
  client_id: frigate
  user: frigate
  password: frigate123
  stats_interval: 300

go2rtc:
  api:
    listen: "0.0.0.0:1984"
  rtsp:
    listen: "0.0.0.0:8554"
  webrtc:
    candidates:
      - 192.168.0.1:8555
      - stun:8555
    listen: "0.0.0.0:8555"
  streams:
    talli_etu: rtsp://192.168.0.36:554/0/h264major
    talli_etu_sub: rtsp://192.168.0.36:554/0/h264minor
    #640x480 5fps i-frame 5
    talli_taka: rtsp://192.168.0.30/user=_password=_channel=1_stream=0.sdp
    talli_taka_sub: rtsp://192.168.0.30/user=_password=_channel=1_stream=1.sdp
    #704x576 5fps i-frame 5
    talli_sivu: rtsp://192.168.0.33/user=_password=_channel=1_stream=0.sdp
    talli_sivu_sub: rtsp://192.168.0.33/user=_password=_channel=1_stream=1.sdp
    #704x576 5fps i-frame 5
    takaovi: rtsp://u:p@192.168.0.32:554/cam/realmonitor?channel=1&subtype=0
    takaovi_sub: rtsp://u:p@192.168.0.32:554/cam/realmonitor?channel=1&subtype=1
    #704x576 5 fps i-frame 5
    etuovi: rtsp://192.168.0.34:554/user=_password=_channel=1_stream=0.sdp?real_stream
    etuovi_sub: rtsp://192.168.0.34:554/user=_password=_channel=1_stream=1.sdp?real_stream
    #704x576 5fps i-frame 5
    katos: rtsp://u:p@192.168.0.35:554/cam/realmonitor?channel=1&subtype=0
    katos_sub: rtsp://u:p@192.168.0.35:554/cam/realmonitor?channel=1&subtype=1
    #704x576 5fps i-frame 5
    katostie: rtsp://192.168.0.31/user=_password=_channel=1_stream=0.sdp
    katostie_sub: rtsp://192.168.0.31/user=_password=_channel=1_stream=1.sdp
    #704x576 5fps i-frame 5

detectors:
  coral:
    type: edgetpu
    device: usb

record:
  enabled: true
  expire_interval: 10
  retain:
    days: 7
    mode: all
  events:
    retain:
      default: 14
      mode: active_objects

detect:
  # Optional: width of the frame for the input with the detect role (default: shown below)
  width: 704
  # Optional: height of the frame for the input with the detect role (default: shown below)
  height: 576
  # Optional: desired fps for your camera for the input with the detect role (default: shown below)
  # NOTE: Recommended value of 5. Ideally, try and reduce your FPS on the camera.
  fps: 5
#  stationary:
#    interval: 10
#    threshold: 50
#    max_frames:
      # Optional: Default for all object types (default: not set, track forever)
#      default: 3000
      # Optional: Object specific values
#      objects:
#        person: 1000
#        car: 1000
#        dog: 1000
#        cat: 1000

model:
  width: 320 #default 320
  height: 320 #default 320

snapshots:
  # Optional: Enable writing jpg snapshot to /media/frigate/clips (default: shown below)
  enabled: true
  # Optional: save a clean PNG copy of the snapshot image (default: shown below)
  clean_copy: true
  # Optional: print a timestamp on the snapshots (default: shown below)
  timestamp: true
  # Optional: draw bounding box on the snapshots (default: shown below)
  bounding_box: true
  retain:
    # Required: Default retention days (default: shown below)
    default: 3
    # Optional: Per object retention days
    objects:
      person: 5

birdseye:
  # Optional: Enable birdseye view (default: shown below)
  enabled: true
  # NOTE: Enabling this will set birdseye to run 24/7 which may increase CPU usage somewhat
  restream: false
  # Optional: Mode of the view. Available options are: objects, motion, and continuous
  #   objects - cameras are included if they have had a tracked object within the last 30 seconds
  #   motion - cameras are included if motion was detected in the last 30 seconds
  #   continuous - all cameras are included always
  mode: continuous

objects:
  # Optional: list of objects to track from labelmap.txt (default: shown below)
  track:
    - person
    - dog
    - car
    - bus
  filters:
    cat:
      min_score: 0.4
      threshold: 0.5
    dog:
      min_score: 0.4
      threshold: 0.5
    person:
      min_score: 0.5
      threshold: 0.6
    car:
      min_score: 0.3
      threshold: 0.5

cameras:
#  test:
#    ffmpeg:
#      inputs:
#        - path: /media/frigate/10.17.mp4
#          input_args: -re -stream_loop -1 -fflags +genpts
#          roles:
#            - detect
#    detect:
#      width: 704
#      height: 576
#      fps: 5

  talli_etu:
    ffmpeg:
      hwaccel_args: preset-intel-qsv-h264
      inputs:
        - path: rtsp://127.0.0.1:8554/talli_etu_sub
          input_args: preset-rtsp-restream
          roles:
            - detect
        - path: rtsp://127.0.0.1:8554/talli_etu
          input_args: preset-rtsp-restream
          roles:
            - record
    live:
       stream_name: talli_etu_sub
    detect:
      width: 640
      height: 360
      fps: 5
    zones:
      piha:
        coordinates: 640,480,640,208,332,123,0,184,0,480
      tie:
        coordinates: 183,57,176,75,640,198,640,149
      metsa:
        coordinates: 155,48,153,0,640,0,640,0,640,143
      parkki:
        coordinates: 0,0,125,0,179,146,0,169
    record:
      events:
        required_zones:
          - piha
          - tie
          - metsa
    snapshots:
      required_zones:
        - piha
        - tie
        - metsa

  talli_taka:
    ffmpeg:
      hwaccel_args: preset-intel-qsv-h264
      inputs:
        - path: rtsp://127.0.0.1:8554/talli_taka_sub
          input_args: preset-rtsp-restream
          roles:
            - detect
        - path: rtsp://127.0.0.1:8554/talli_taka
          input_args: preset-rtsp-restream
          roles:
            - record
    objects:
      track:
        - person
        - dog
        - cat
    live:
       stream_name: talli_taka_sub

  talli_sivu:
    ffmpeg:
      hwaccel_args: preset-intel-qsv-h264
      inputs:
        - path: rtsp://127.0.0.1:8554/talli_sivu_sub
          input_args: preset-rtsp-restream
          roles:
            - detect
        - path: rtsp://127.0.0.1:8554/talli_sivu
          input_args: preset-rtsp-restream
          roles:
            - record
    objects:
      track:
        - person
        - dog
        - cat
    live:
       stream_name: talli_sivu_sub

  takaovi:
    ffmpeg:
      hwaccel_args: preset-intel-qsv-h264
      inputs:
        - path: rtsp://127.0.0.1:8554/takaovi_sub
          input_args: preset-rtsp-restream
          roles:
            - detect
        - path: rtsp://127.0.0.1:8554/takaovi
          input_args: preset-rtsp-restream
          roles:
            - record
    objects:
      track:
        - person
        - dog
        - cat
    live:
       stream_name: takaovi_sub

  etuovi:
    ffmpeg:
      hwaccel_args: preset-intel-qsv-h264
      inputs:
        - path: rtsp://127.0.0.1:8554/etuovi_sub
          input_args: preset-rtsp-restream
          roles:
            - detect
        - path: rtsp://127.0.0.1:8554/etuovi
          input_args: preset-rtsp-restream
          roles:
            - record
    objects:
      track:
        - person
        - dog
        - cat
    live:
       stream_name: etuovi_sub

  katos:
    ffmpeg:
      hwaccel_args: preset-intel-qsv-h264
      inputs:
        - path: rtsp://127.0.0.1:8554/katos_sub
          input_args: preset-rtsp-restream
          roles:
            - detect
        - path: rtsp://127.0.0.1:8554/katos
          input_args: preset-rtsp-restream
          roles:
            - record
    zones:
      parkki:
        coordinates: 704,576,704,48,657,93,363,81,391,576
      piha:
        coordinates: 0,0,298,0,704,0,704,0,704,38,661,72,356,67,365,576,0,576
    objects:
      filters:
        car:
          mask:
            - 646,89,704,48,704,576,440,576,364,93
    record:
      events:
        required_zones:
          - piha
    snapshots:
      required_zones:
        - piha
    live:
       stream_name: katos_sub

  katostie:
    ffmpeg:
      hwaccel_args: preset-intel-qsv-h264
      inputs:
        - path: rtsp://127.0.0.1:8554/katostie_sub
          input_args: preset-rtsp-restream
          roles:
            - detect
        - path: rtsp://127.0.0.1:8554/katostie
          input_args: preset-rtsp-restream
          roles:
            - record
    zones:
      tie:
        coordinates: 704,229,704,352,399,182,399,88
      metsa:
        coordinates: 704,0,409,0,404,103,704,237
      piha:
        coordinates: 704,355,400,193,411,0,89,0,85,77,0,102,0,576,704,576,704,354
      talli:
        coordinates: 0,92,73,77,86,0,0,0
    record:
      events:
        required_zones:
          - tie
          - piha
          - metsa
    snapshots:
      required_zones:
        - tie
        - piha
        - metsa
    live:
       stream_name: katostie_sub

Relevant log output

2023-11-14 20:18:37.066294925  [INFO] Starting Frigate...
2023-11-14 20:18:37.819409119  [2023-11-14 20:18:37] frigate.app                    INFO    : Starting Frigate (0.12.1-367d724)
2023-11-14 20:18:37.848228213  [2023-11-14 20:18:37] peewee_migrate                 INFO    : Starting migrations
2023-11-14 20:18:37.850952294  [2023-11-14 20:18:37] peewee_migrate                 INFO    : There is nothing to migrate
2023-11-14 20:18:37.861324986  [2023-11-14 20:18:37] detector.coral                 INFO    : Starting detection process: 579
2023-11-14 20:18:40.506627037  [2023-11-14 20:18:37] frigate.detectors.plugins.edgetpu_tfl INFO    : Attempting to load TPU as usb
2023-11-14 20:18:40.506686197  [2023-11-14 20:18:37] frigate.app                    INFO    : Output process started: 581
2023-11-14 20:18:40.506722271  [2023-11-14 20:18:37] frigate.app                    INFO    : Camera processor started for talli_etu: 587
2023-11-14 20:18:40.506754971  [2023-11-14 20:18:37] frigate.app                    INFO    : Camera processor started for talli_taka: 589
2023-11-14 20:18:40.506786013  [2023-11-14 20:18:37] frigate.app                    INFO    : Camera processor started for talli_sivu: 591
2023-11-14 20:18:40.506812767  [2023-11-14 20:18:37] frigate.app                    INFO    : Camera processor started for takaovi: 592
2023-11-14 20:18:40.506844828  [2023-11-14 20:18:37] frigate.app                    INFO    : Camera processor started for etuovi: 594
2023-11-14 20:18:40.506871613  [2023-11-14 20:18:37] frigate.app                    INFO    : Camera processor started for katos: 595
2023-11-14 20:18:40.506901565  [2023-11-14 20:18:37] frigate.app                    INFO    : Camera processor started for katostie: 597
2023-11-14 20:18:40.506930695  [2023-11-14 20:18:37] frigate.app                    INFO    : Capture process started for talli_etu: 599
2023-11-14 20:18:40.506958355  [2023-11-14 20:18:37] frigate.app                    INFO    : Capture process started for talli_taka: 602
2023-11-14 20:18:40.506986280  [2023-11-14 20:18:37] frigate.app                    INFO    : Capture process started for talli_sivu: 607
2023-11-14 20:18:40.507014346  [2023-11-14 20:18:37] frigate.app                    INFO    : Capture process started for takaovi: 614
2023-11-14 20:18:40.507042764  [2023-11-14 20:18:37] frigate.app                    INFO    : Capture process started for etuovi: 621
2023-11-14 20:18:40.507072066  [2023-11-14 20:18:37] frigate.app                    INFO    : Capture process started for katos: 630
2023-11-14 20:18:40.507100627  [2023-11-14 20:18:37] frigate.app                    INFO    : Capture process started for katostie: 659
2023-11-14 20:18:40.514368861  [2023-11-14 20:18:40] frigate.detectors.plugins.edgetpu_tfl INFO    : TPU found
2023-11-15 06:56:09.405972689  [2023-11-15 06:56:09] watchdog.katos                 INFO    : katos exceeded fps limit. Exiting ffmpeg...
2023-11-15 06:56:09.406067195  [2023-11-15 06:56:09] watchdog.katos                 INFO    : Waiting for ffmpeg to exit gracefully...
2023-11-15 06:56:22.964826494  [2023-11-15 06:56:22] frigate.video                  ERROR   : katos: Unable to read frames from ffmpeg process.
2023-11-15 06:56:22.973920700  [2023-11-15 06:56:22] frigate.video                  ERROR   : katos: ffmpeg process is not running. exiting capture thread...

go2rtc log
2023-11-14 20:18:37.066308309  [INFO] Preparing go2rtc config...
2023-11-14 20:18:37.284260994  [INFO] Not injecting WebRTC candidates into go2rtc config as it has been set manually
2023-11-14 20:18:37.309467646  [WARN] Using go2rtc binary from '/config/go2rtc' instead of the embedded one
2023-11-14 20:18:37.309471490  [INFO] Starting go2rtc...
2023-11-14 20:18:37.375044223  20:18:37.374 INF go2rtc version 1.8.1 linux/amd64
2023-11-14 20:18:37.375262846  20:18:37.375 INF [api] listen addr=0.0.0.0:1984
2023-11-14 20:18:37.375579158  20:18:37.375 INF [rtsp] listen addr=0.0.0.0:8554
2023-11-14 20:18:37.376307391  20:18:37.376 INF [webrtc] listen addr=0.0.0.0:8555
2023-11-14 20:18:47.066880086  [INFO] Starting go2rtc healthcheck service...

FFprobe output from your camera

ffprobe rtsp://192.168.0.1:8554/katos_sub
ffprobe version 4.3.6-0+deb11u1 Copyright (c) 2007-2023 the FFmpeg developers
  built with gcc 10 (Debian 10.2.1-6)
  configuration: --prefix=/usr --extra-version=0+deb11u1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --arch=amd64 --enable-gpl --disable-stripping --enable-avresample --disable-filter=resample --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libdav1d --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librabbitmq --enable-librsvg --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --enable-pocketsphinx --enable-libmfx --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared
  libavutil      56. 51.100 / 56. 51.100
  libavcodec     58. 91.100 / 58. 91.100
  libavformat    58. 45.100 / 58. 45.100
  libavdevice    58. 10.100 / 58. 10.100
  libavfilter     7. 85.100 /  7. 85.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  7.100 /  5.  7.100
  libswresample   3.  7.100 /  3.  7.100
  libpostproc    55.  7.100 / 55.  7.100
[rtsp @ 0x555ee5bf8a00] method SETUP failed: 461 Unsupported transport
Input #0, rtsp, from 'rtsp://192.168.0.1:8554/katos_sub':
  Metadata:
    title           : go2rtc/1.8.1
  Duration: N/A, start: 1.280000, bitrate: N/A
    Stream #0:0: Video: h264 (High), yuvj420p(pc, bt470bg, progressive), 704x576 [SAR 27:22 DAR 3:2], 5 fps, 5 tbr, 90k tbn, 10 tbc

with forced tcp
ffprobe rtsp://192.168.0.1:8554/katos_sub -rtsp_transport tcp
ffprobe version 4.3.6-0+deb11u1 Copyright (c) 2007-2023 the FFmpeg developers
  built with gcc 10 (Debian 10.2.1-6)
  configuration: --prefix=/usr --extra-version=0+deb11u1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --arch=amd64 --enable-gpl --disable-stripping --enable-avresample --disable-filter=resample --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libdav1d --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librabbitmq --enable-librsvg --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --enable-pocketsphinx --enable-libmfx --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared
  libavutil      56. 51.100 / 56. 51.100
  libavcodec     58. 91.100 / 58. 91.100
  libavformat    58. 45.100 / 58. 45.100
  libavdevice    58. 10.100 / 58. 10.100
  libavfilter     7. 85.100 /  7. 85.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  7.100 /  5.  7.100
  libswresample   3.  7.100 /  3.  7.100
  libpostproc    55.  7.100 / 55.  7.100
Input #0, rtsp, from 'rtsp://192.168.0.1:8554/katos_sub':
  Metadata:
    title           : go2rtc/1.8.1
  Duration: N/A, start: 1.280000, bitrate: N/A
    Stream #0:0: Video: h264 (High), yuvj420p(pc, bt470bg, progressive), 704x576 [SAR 27:22 DAR 3:2], 5 fps, 5 tbr, 90k tbn, 10 tbc

Frigate stats

No response

Operating system

Debian

Install method

Docker Compose

Coral version

USB

Network connection

Wired

Camera make and model

Dahua

Any other information that may be helpful

Everything is else than the detect stream is working perfect. I can access streams and recordings inside frigate.

ristoavila commented 1 year ago

for test purposes I've upgraded go2rtc to 1.8.3

NickM-27 commented 1 year ago

more logs are needed, please let frigate run for 1 minute then provide logs

ristoavila commented 1 year ago

frigate was running 24hours and this happened in the morning at 06:56 at 14:00 same day there were no more logs. This has now happened few times and maybe there is some async problem here. E.g. when checking with mqtt it was showing some pid number for detect stream but when doing ps -aux|grep ffmpeg I could not find the ffmpeg line for the katos_sub which is used for detection.

Frigate was still recording just fine the katos main stream but the sub was not visible to frigate but still accessible with web,vlc and ffmpegprobe.

Is there anyway to enable more logs?

NickM-27 commented 1 year ago

if frigate does not receive frames then it already restarts the ffmpeg process (which can be seen in the logs). it is also worth noting that the camera_fps field in mqtt can be used to indicate whether the detect stream is currently running as it will be 0 when the stream is closed

it has been seen before where go2rtc live view would work (mse / webrtc) but it fails to output an rtsp stream that works for frigate. However, this was the case on older go2rtc versions not recent ones.

ristoavila commented 1 year ago

Ok, next time I will also check the camera_fps from mqtt. Is there also way to enforce restart of the detetction stream? To me it seemed like frigate is waiting for ffmpeg to exit where in reality it had already exited.

I also tested the rtsp stream directly from go2rtc and the detection stream was working fine, I was able to record it with ffmpeg get the probe and also view with vlc, only thing that was not happening was frigate to restart the detection stream.

NickM-27 commented 1 year ago

the restart is already fully enforced. if the ffmpeg process doesn't exit gracefully after 30 seconds frigate will forcibly exit the process

https://github.com/blakeblackshear/frigate/blob/37b82c0d605d2530a15809ab078356fc35764600/frigate/video.py#L272-L278

you may find better results running the go2rtc streams as ffmpeg: instead of directly through go2rtc

ristoavila commented 1 year ago

hmm in my case I never see the timeout. What happens if exit is ok and next restart you get no frames and you do end up exiting like in the log? Should it loop forever and try to restart ffmepg?

NickM-27 commented 1 year ago

yes, it will continue to restart if no frames are received. This logic has not changed for many frigate versions

ristoavila commented 1 year ago

Some more logs from last night

2023-11-15 11:47:58.960146116  [INFO] Starting Frigate...
2023-11-15 11:47:59.704740293  [2023-11-15 11:47:59] frigate.app                    INFO    : Starting Frigate (0.12.1-367d724)
2023-11-15 11:47:59.738729099  [2023-11-15 11:47:59] peewee_migrate                 INFO    : Starting migrations
2023-11-15 11:47:59.741316440  [2023-11-15 11:47:59] peewee_migrate                 INFO    : There is nothing to migrate
2023-11-15 11:47:59.753280411  [2023-11-15 11:47:59] detector.coral                 INFO    : Starting detection process: 577
2023-11-15 11:48:02.399108362  [2023-11-15 11:47:59] frigate.app                    INFO    : Output process started: 579
2023-11-15 11:48:02.406697223  [2023-11-15 11:47:59] frigate.detectors.plugins.edgetpu_tfl INFO    : Attempting to load TPU as usb
2023-11-15 11:48:02.406812316  [2023-11-15 11:48:02] frigate.detectors.plugins.edgetpu_tfl INFO    : TPU found
2023-11-15 11:48:02.406867543  [2023-11-15 11:47:59] frigate.app                    INFO    : Camera processor started for talli_etu: 586
2023-11-15 11:48:02.406912138  [2023-11-15 11:47:59] frigate.app                    INFO    : Camera processor started for talli_taka: 587
2023-11-15 11:48:02.406954665  [2023-11-15 11:47:59] frigate.app                    INFO    : Camera processor started for talli_sivu: 589
2023-11-15 11:48:02.406997172  [2023-11-15 11:47:59] frigate.app                    INFO    : Camera processor started for takaovi: 590
2023-11-15 11:48:02.407038682  [2023-11-15 11:47:59] frigate.app                    INFO    : Camera processor started for etuovi: 592
2023-11-15 11:48:02.407074882  [2023-11-15 11:47:59] frigate.app                    INFO    : Camera processor started for katos: 594
2023-11-15 11:48:02.407112060  [2023-11-15 11:47:59] frigate.app                    INFO    : Camera processor started for katostie: 595
2023-11-15 11:48:02.407147015  [2023-11-15 11:47:59] frigate.app                    INFO    : Capture process started for talli_etu: 597
2023-11-15 11:48:02.407187057  [2023-11-15 11:47:59] frigate.app                    INFO    : Capture process started for talli_taka: 601
2023-11-15 11:48:02.407225096  [2023-11-15 11:47:59] frigate.app                    INFO    : Capture process started for talli_sivu: 605
2023-11-15 11:48:02.407256908  [2023-11-15 11:47:59] frigate.app                    INFO    : Capture process started for takaovi: 612
2023-11-15 11:48:02.407280940  [2023-11-15 11:47:59] frigate.app                    INFO    : Capture process started for etuovi: 621
2023-11-15 11:48:02.407304715  [2023-11-15 11:47:59] frigate.app                    INFO    : Capture process started for katos: 627
2023-11-15 11:48:02.407326735  [2023-11-15 11:47:59] frigate.app                    INFO    : Capture process started for katostie: 681
2023-11-15 16:00:40.252470046  [2023-11-15 16:00:40] watchdog.katos                 INFO    : No frames received from katos in 20 seconds. Exiting ffmpeg...
2023-11-15 16:00:40.252519701  [2023-11-15 16:00:40] watchdog.katos                 INFO    : Waiting for ffmpeg to exit gracefully...
2023-11-15 16:00:40.445926091  [2023-11-15 16:00:40] frigate.video                  ERROR   : katos: Unable to read frames from ffmpeg process.
2023-11-15 16:00:40.445972836  [2023-11-15 16:00:40] frigate.video                  ERROR   : katos: ffmpeg process is not running. exiting capture thread...
2023-11-15 16:00:50.447519403  [2023-11-15 16:00:50] watchdog.katos                 ERROR   : Ffmpeg process crashed unexpectedly for katos.
2023-11-15 16:00:50.447571088  [2023-11-15 16:00:50] watchdog.katos                 ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2023-11-15 16:00:50.447616277  [2023-11-15 16:00:50] ffmpeg.katos.detect            ERROR   : [rtsp @ 0x562c8cc7db40] RTP: PT=60: bad cseq d41b expected=eb26
2023-11-15 16:43:50.562157224  [2023-11-15 16:43:50] watchdog.katos                 INFO    : No frames received from katos in 20 seconds. Exiting ffmpeg...
2023-11-15 16:43:50.562168892  [2023-11-15 16:43:50] watchdog.katos                 INFO    : Waiting for ffmpeg to exit gracefully...
2023-11-15 16:43:50.620302380  [2023-11-15 16:43:50] frigate.video                  ERROR   : katos: Unable to read frames from ffmpeg process.
2023-11-15 16:43:50.620360838  [2023-11-15 16:43:50] frigate.video                  ERROR   : katos: ffmpeg process is not running. exiting capture thread...
2023-11-15 16:44:00.621919966  [2023-11-15 16:44:00] watchdog.katos                 ERROR   : Ffmpeg process crashed unexpectedly for katos.
2023-11-15 16:44:00.621931611  [2023-11-15 16:44:00] watchdog.katos                 ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2023-11-15 16:44:00.621968072  [2023-11-15 16:44:00] ffmpeg.katos.detect            ERROR   : [rtsp @ 0x555dda7c8b40] RTP: PT=60: bad cseq b5de expected=6315
2023-11-15 23:49:11.900268744  [2023-11-15 23:49:11] watchdog.katos                 INFO    : katos exceeded fps limit. Exiting ffmpeg...
2023-11-15 23:49:11.900372795  [2023-11-15 23:49:11] watchdog.katos                 INFO    : Waiting for ffmpeg to exit gracefully...
2023-11-15 23:53:57.814418614  [2023-11-15 23:53:51] watchdog.katostie              INFO    : No frames received from katostie in 20 seconds. Exiting ffmpeg...
2023-11-15 23:54:00.745106118  [2023-11-15 23:53:51] watchdog.talli_sivu            INFO    : No frames received from talli_sivu in 20 seconds. Exiting ffmpeg...
2023-11-15 23:54:02.817027904  [2023-11-15 23:53:51] watchdog.takaovi               INFO    : No frames received from takaovi in 20 seconds. Exiting ffmpeg...
2023-11-15 23:54:09.051259960  [2023-11-15 23:53:51] watchdog.talli_taka            INFO    : No frames received from talli_taka in 20 seconds. Exiting ffmpeg...
2023-11-15 23:54:09.737436133  [2023-11-15 23:53:51] watchdog.talli_etu             INFO    : No frames received from talli_etu in 20 seconds. Exiting ffmpeg...
2023-11-15 23:54:11.603059414  [2023-11-15 23:53:55] watchdog.takaovi               INFO    : Waiting for ffmpeg to exit gracefully...
2023-11-15 23:54:22.215919633  [2023-11-15 23:53:55] watchdog.talli_taka            INFO    : Waiting for ffmpeg to exit gracefully...
2023-11-15 23:54:23.644007459  [2023-11-15 23:53:55] watchdog.talli_sivu            INFO    : Waiting for ffmpeg to exit gracefully...
2023-11-15 23:54:25.615518936  [2023-11-15 23:53:55] watchdog.katostie              INFO    : Waiting for ffmpeg to exit gracefully...
2023-11-15 23:54:26.536913601  [2023-11-15 23:53:56] watchdog.talli_etu             INFO    : Waiting for ffmpeg to exit gracefully...
2023-11-15 23:54:27.254924651  [2023-11-15 23:54:10] watchdog.etuovi                ERROR   : No new recording segments were created for etuovi in the last 120s. restarting the ffmpeg record process...
2023-11-15 23:54:27.490031415  [2023-11-15 23:54:23] watchdog.etuovi                INFO    : Terminating the existing ffmpeg process...
2023-11-15 23:54:27.490241774  [2023-11-15 23:54:26] watchdog.etuovi                INFO    : Waiting for ffmpeg to exit gracefully...
2023-11-15 23:54:27.490359441  [2023-11-15 23:54:26] watchdog.takaovi               INFO    : FFmpeg did not exit. Force killing...
2023-11-15 23:54:27.490429587  [2023-11-15 23:54:26] watchdog.talli_taka            INFO    : FFmpeg did not exit. Force killing...
2023-11-15 23:54:27.554172743  [2023-11-15 23:54:27] frigate.video                  ERROR   : takaovi: Unable to read frames from ffmpeg process.
2023-11-15 23:54:27.554440233  [2023-11-15 23:54:27] watchdog.talli_etu             INFO    : FFmpeg did not exit. Force killing...
2023-11-15 23:54:27.554580317  [2023-11-15 23:54:27] watchdog.katostie              INFO    : FFmpeg did not exit. Force killing...
2023-11-15 23:54:27.555259520  [2023-11-15 23:54:27] frigate.video                  ERROR   : takaovi: ffmpeg process is not running. exiting capture thread...
2023-11-15 23:54:27.559513741  [2023-11-15 23:54:27] watchdog.talli_sivu            INFO    : FFmpeg did not exit. Force killing...
2023-11-15 23:54:27.559808284  [2023-11-15 23:54:27] watchdog.takaovi               ERROR   : No new recording segments were created for takaovi in the last 120s. restarting the ffmpeg record process...
2023-11-15 23:54:27.571663067  [2023-11-15 23:54:27] frigate.video                  ERROR   : talli_taka: Unable to read frames from ffmpeg process.
2023-11-15 23:54:27.571700042  [2023-11-15 23:54:27] watchdog.takaovi               INFO    : Terminating the existing ffmpeg process...
2023-11-15 23:54:27.581426000  [2023-11-15 23:54:27] frigate.video                  ERROR   : talli_etu: Unable to read frames from ffmpeg process.
2023-11-15 23:54:27.591512654  [2023-11-15 23:54:27] frigate.video                  ERROR   : talli_taka: ffmpeg process is not running. exiting capture thread...
2023-11-15 23:54:27.591544243  [2023-11-15 23:54:27] watchdog.takaovi               INFO    : Waiting for ffmpeg to exit gracefully...
2023-11-15 23:54:27.591568709  [2023-11-15 23:54:27] frigate.video                  ERROR   : talli_etu: Unable to read frames from ffmpeg process.
2023-11-15 23:54:27.596254229  [2023-11-15 23:54:27] frigate.video                  ERROR   : katostie: Unable to read frames from ffmpeg process.
2023-11-15 23:54:27.596311871  [2023-11-15 23:54:27] frigate.video                  ERROR   : talli_etu: ffmpeg process is not running. exiting capture thread...
2023-11-15 23:54:27.608019380  [2023-11-15 23:54:27] frigate.video                  ERROR   : talli_sivu: Unable to read frames from ffmpeg process.
2023-11-15 23:54:27.623801322  [2023-11-15 23:54:27] frigate.video                  ERROR   : katostie: ffmpeg process is not running. exiting capture thread...
2023-11-15 23:54:27.623849866  [2023-11-15 23:54:27] watchdog.talli_etu             ERROR   : No new recording segments were created for talli_etu in the last 120s. restarting the ffmpeg record process...
2023-11-15 23:54:27.922429162  [2023-11-15 23:54:27] watchdog.talli_sivu            ERROR   : No new recording segments were created for talli_sivu in the last 120s. restarting the ffmpeg record process...
2023-11-15 23:54:27.922493316  [2023-11-15 23:54:27] watchdog.katostie              ERROR   : No new recording segments were created for katostie in the last 120s. restarting the ffmpeg record process...
2023-11-15 23:54:27.922522411  [2023-11-15 23:54:27] watchdog.talli_etu             INFO    : Terminating the existing ffmpeg process...
2023-11-15 23:54:27.922547271  [2023-11-15 23:54:27] watchdog.katostie              INFO    : Terminating the existing ffmpeg process...
2023-11-15 23:54:27.922570566  [2023-11-15 23:54:27] watchdog.talli_etu             INFO    : Waiting for ffmpeg to exit gracefully...
2023-11-15 23:54:27.922593948  [2023-11-15 23:54:27] watchdog.talli_sivu            INFO    : Terminating the existing ffmpeg process...
2023-11-15 23:54:27.922617120  [2023-11-15 23:54:27] watchdog.katostie              INFO    : Waiting for ffmpeg to exit gracefully...
2023-11-15 23:54:27.922638395  [2023-11-15 23:54:27] watchdog.talli_sivu            INFO    : Waiting for ffmpeg to exit gracefully...
2023-11-15 23:54:27.922661531  [2023-11-15 23:54:27] frigate.video                  ERROR   : talli_sivu: Unable to read frames from ffmpeg process.
2023-11-15 23:54:27.922684478  [2023-11-15 23:54:27] frigate.video                  ERROR   : talli_sivu: ffmpeg process is not running. exiting capture thread...
2023-11-15 23:54:29.130816552  [2023-11-15 23:54:29] frigate.video                  ERROR   : etuovi: Unable to read frames from ffmpeg process.
2023-11-15 23:54:29.130821444  [2023-11-15 23:54:29] frigate.video                  ERROR   : etuovi: ffmpeg process is not running. exiting capture thread...
2023-11-15 23:55:12.670811422  [2023-11-15 23:54:40] watchdog.talli_taka            ERROR   : Ffmpeg process crashed unexpectedly for talli_taka.
2023-11-15 23:55:12.677017103  [2023-11-15 23:54:47] watchdog.talli_etu             ERROR   : Ffmpeg process crashed unexpectedly for talli_etu.
2023-11-15 23:55:12.677019698  [2023-11-15 23:54:47] watchdog.katostie              ERROR   : Ffmpeg process crashed unexpectedly for katostie.
2023-11-15 23:55:12.677024763  [2023-11-15 23:54:48] watchdog.talli_sivu            ERROR   : Ffmpeg process crashed unexpectedly for talli_sivu.
2023-11-15 23:55:12.677026364  [2023-11-15 23:54:48] watchdog.takaovi               ERROR   : Ffmpeg process crashed unexpectedly for takaovi.
2023-11-15 23:55:12.677250528  [2023-11-15 23:55:12] watchdog.talli_taka            ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2023-11-15 23:55:12.684077596  [2023-11-15 23:54:47] watchdog.etuovi                ERROR   : Ffmpeg process crashed unexpectedly for etuovi.
2023-11-15 23:55:12.713509919  [2023-11-15 23:55:12] watchdog.talli_etu             ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2023-11-15 23:55:12.722243803  [2023-11-15 23:55:12] watchdog.katostie              ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2023-11-15 23:55:12.722300876  [2023-11-15 23:55:12] watchdog.talli_sivu            ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2023-11-15 23:55:12.722329646  [2023-11-15 23:55:12] watchdog.takaovi               ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2023-11-15 23:55:12.722372063  [2023-11-15 23:55:12] watchdog.etuovi                ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2023-11-15 23:55:12.736430406  [2023-11-15 23:55:12] ffmpeg.talli_etu.detect        ERROR   : More than 1000 frames duplicated
2023-11-15 23:55:12.736545718  [2023-11-15 23:55:12] ffmpeg.katostie.detect         ERROR   : More than 1000 frames duplicated
2023-11-15 23:55:12.736587386  [2023-11-15 23:55:12] ffmpeg.talli_sivu.detect       ERROR   : More than 1000 frames duplicated
2023-11-15 23:55:12.736619159  [2023-11-15 23:55:12] ffmpeg.etuovi.detect           ERROR   : [h264_qsv @ 0x556892406900] A decode call did not consume any data: expect more data at input (-10)
2023-11-15 23:55:12.736646774  [2023-11-15 23:55:12] ffmpeg.katostie.detect         ERROR   : More than 10000 frames duplicated
2023-11-15 23:55:12.736670987  [2023-11-15 23:55:12] ffmpeg.talli_etu.detect        ERROR   : More than 10000 frames duplicated
2023-11-15 23:55:12.736694529  [2023-11-15 23:55:12] ffmpeg.talli_sivu.detect       ERROR   : More than 10000 frames duplicated
2023-11-15 23:55:12.736716827  [2023-11-15 23:55:12] ffmpeg.etuovi.detect           ERROR   :     Last message repeated 2 times
2023-11-15 23:55:13.141500307  [2023-11-15 23:55:13] ffmpeg.talli_taka.record       ERROR   : [segment @ 0x563ddd65c7c0] Timestamps are unset in a packet for stream 0. This is deprecated and will stop working in the future. Fix your code to set the timestamps properly
2023-11-15 23:55:13.142139023  [2023-11-15 23:55:13] ffmpeg.talli_taka.record       ERROR   : [segment @ 0x563ddd65c7c0] Non-monotonous DTS in output stream 0:0; previous: 0, current: 0; changing to 1. This may result in incorrect timestamps in the output file.
2023-11-15 23:55:13.142262508  [2023-11-15 23:55:13] watchdog.talli_taka            INFO    : Terminating the existing ffmpeg process...
2023-11-15 23:55:13.142338024  [2023-11-15 23:55:13] watchdog.talli_taka            INFO    : Waiting for ffmpeg to exit gracefully...

All the other cameras recoved but "katos" camera is just not coming back online. Now also the capture ffmpeg session didn't recover. From the katos camera the last seen logging is always waiting for ffmpeg to exit. 2023-11-15 23:49:11.900372795 [2023-11-15 23:49:11] watchdog.katos INFO : Waiting for ffmpeg to exit gracefully...

There is nothing in go2rtc logs and ffprobe doesn't give errors when accessing both detect and record streams.

Is there anyway to enable more logs on this? To me it seems that it's just stuck waiting.

NickM-27 commented 1 year ago

for starts I would suggest using preset-vaapi not qsv, qsv has often caused weird bugs and issues

ristoavila commented 1 year ago

no 1 day solid running with preset-vaapi. After another day I close this as resolved. @NickM-27 thank you for the support!

ristoavila commented 1 year ago

Working solid with vaapi