search

blakeblackshear / frigate

NVR with realtime local object detection for IP cameras
https://frigate.video
MIT License
18.88k stars 1.72k forks source link

Frigate 0.13 RC: No frames have been received, check error logs #9406

Closed SuperMaximus1984 closed 3 months ago

SuperMaximus1984 commented 9 months ago

Describe the problem you are having

No frames have been received, check error logs appears periodically in Frigate's Web Interface Restart of Frigate may temporarily help with some cameras, but soon the problem reappears. After a while camera image may show up again normally. image

There were no problems like this with the previously used Frigate 0.12.1 Release.

I tried to use the embedded go2rtc 1.8.4, then after no lock included a go2rtc 1.8.5 in docker volume, but nothing's changed. Docker-compose.yml:

version: "3.9"
services:
  frigate:
    container_name: frigate
    runtime: nvidia
    deploy:    # <------------- Add this section
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1 # number of GPUs
              capabilities: [gpu]
    privileged: true # this may not be necessary for all setups
    restart: unless-stopped
    image: ghcr.io/blakeblackshear/frigate:0.13.0-rc1-tensorrt
    shm_size: "512mb" # update for your cameras based on calculation above
    devices:
      - /dev/bus/usb:/dev/bus/usb # passes the USB Coral, needs to be modified for other versions
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /usr/share/hassio/homeassistant/:/config/
      - /home/max/VMS/frigate-media:/media/frigate
      - /usr/share/hassio/media/trt-models:/trt-models
      - /usr/share/hassio/homeassistant/frigate13-docker/go2rtc:/config/go2rtc
      - type: tmpfs # Optional: 1GB of memory, reduces SSD/SD Card wear
        target: /tmp/cache
        tmpfs:
          size: 1000000000
    ports:
      - "5000:5000"
      - "8554:8554" # RTSP feeds
      - "8555:8555/tcp" # WebRTC over tcp
      - "8555:8555/udp" # WebRTC over udp
    environment:
      FRIGATE_RTSP_PASSWORD: "password"

Version

0.13 RC1

Frigate config file

mqtt:
  host: 192.168.0.3
  user: user
  password: pass
detectors:
  coral:
    type: edgetpu
    device: usb
model:
  path: /trt-models/efficientdet_lite3_512_ptq_edgetpu.tflite
  width: 512
  height: 512

go2rtc:
  streams:
    ptz:
      - rtsp://login:pass@192.168.0.8:554/axis-media/media.amp
    front:
      - rtsp://login:pass@192.168.0.4:554/axis-media/media.amp
    back:
      - rtsp://login:pass@192.168.0.5:554/axis-media/media.amp
    sosed:
      - rtsp://login:pass@192.168.0.6:554/axis-media/media.amp
    trucks:
      - rtsp://login:pass@192.168.0.7:554/axis-media/media.amp
    trucks_back:
      - rtsp://admin:pass@192.168.0.13:554/trackID=1
  webrtc:
    candidates:
      - 192.168.0.3:8555
      - stun:8555

cameras:
  ptz:
    ffmpeg:
      hwaccel_args: preset-nvidia-h264
      inputs:
        - path: rtsp://login:pass@192.168.0.8:554/axis-media/media.amp
          roles:
            - detect
    detect:
      width: 1920
      height: 1080
      fps: 5
    objects:
      track:
        - person
        - car
        - truck
        - bicycle
        - motorcycle
    record:
      enabled: true
      retain:
        days: 10
        mode: motion

  front:
    ffmpeg:
      hwaccel_args: preset-nvidia-h264
      inputs:
        - path: rtsp://login:pass@192.168.0.4:554/axis-media/media.amp
          roles:
            - detect
    detect:
      width: 3840
      height: 2160
      fps: 5
      track:
        - person
#        - car
        - truck
        - bicycle
        - motorcycle

    record:
      enabled: True
      events:
        post_capture: 14
    zones:
      front_yard:
        coordinates: 3840,2160,3840,1313,3733,1351,3381,954,3072,942,2991,755,2169,702,1201,758,446,870,240,1257,63,1818,0,2160

  back:
    ffmpeg:
      hwaccel_args: preset-nvidia-h264
      inputs:
        - path: rtsp://login:pass@192.168.0.5:554/axis-media/media.amp
          roles:
            - detect
    detect:
      width: 3840
      height: 2160
      fps: 5
    zones:
      back_yard:
        coordinates: 3768,2160,3596,1915,3471,1850,3531,1678,3393,1425,3128,979,2798,536,2579,334,2547,159,1803,75,1125,216,1155,312,0,930,0,2160
        objects:
          - person
    objects:
      track:
        - person

  sosed:
    ffmpeg:
      hwaccel_args: preset-nvidia-h264
      inputs:
        - path: rtsp://login:pass@192.168.0.6:554/axis-media/media.amp
          roles:
            - detect
    detect:
      width: 2688
      height: 1520
      fps: 5
    objects:
      track:
        - person
#        - car
        - truck
        - bicycle
        - motorcycle
    zones:
      sosed_yard:
        coordinates: 2688,1520,2688,941,2094,880,941,769,926,880,825,858,812,749,0,670,0,1520
        objects:
          - person
        filters:
          person:
            min_area: 5000

  trucks:
    ffmpeg:
      hwaccel_args: preset-nvidia-h264
      inputs:
        - path: rtsp://login:pass@192.168.0.7:554/axis-media/media.amp
          roles:
            - detect
    detect:
      width: 2688
      height: 1520
      fps: 5
    objects:
      track:
        - person
#        - car
#        - bus
        - truck
    zones:
      trucks_yard:
        coordinates: 2688,1520,2688,905,2370,612,1982,443,1661,424,753,424,0,612,0,1520
        objects:
          - person
        filters:
          person:
            min_area: 5000

  trucks_back:
    ffmpeg:
      hwaccel_args: preset-nvidia-h264
        # - -c:v
        # - h264_cuvid
      inputs:
        - path: rtsp://admin:pass@192.168.0.13:554/trackID=1
          roles:
            - detect
    detect:
      width: 2592
      height: 1944
      fps: 5
    objects:
      track:
        - person
#        - car
        - truck
        - bicycle
        - motorcycle
    zones:
      trucks_back_yard:
        coordinates: 2592,1944,2592,474,2103,446,0,1829,0,1944
        objects:
          - person
        filters:
          person:
            min_area: 10000 
snapshots:
  enabled: True
record:
  enabled: True
  events:
    pre_capture: 10
    retain:
      default: 10
ui:
  time_format: 24hour
  date_style: medium

Relevant log output

2024-01-21 19:26:11.638685836  [INFO] Preparing Frigate...
2024-01-21 19:26:11.654147899  [INFO] Starting Frigate...
2024-01-21 19:26:12.819158254  [2024-01-21 19:26:12] frigate.app                    INFO    : Starting Frigate (0.13.0-49814b3)
2024-01-21 19:26:12.900367356  [2024-01-21 19:26:12] peewee_migrate.logs            INFO    : Starting migrations
2024-01-21 19:26:12.903403550  [2024-01-21 19:26:12] peewee_migrate.logs            INFO    : There is nothing to migrate
2024-01-21 19:26:12.908102417  [2024-01-21 19:26:12] frigate.app                    INFO    : Recording process started: 450
2024-01-21 19:26:12.909924195  [2024-01-21 19:26:12] frigate.app                    INFO    : go2rtc process pid: 98
2024-01-21 19:26:12.935610345  [2024-01-21 19:26:12] detector.coral                 INFO    : Starting detection process: 460
2024-01-21 19:26:12.939019317  [2024-01-21 19:26:12] frigate.app                    INFO    : Output process started: 462
2024-01-21 19:26:12.969405598  [2024-01-21 19:26:12] frigate.app                    INFO    : Camera processor started for ptz: 472
2024-01-21 19:26:12.974536063  [2024-01-21 19:26:12] frigate.app                    INFO    : Camera processor started for front: 473
2024-01-21 19:26:12.979871679  [2024-01-21 19:26:12] frigate.app                    INFO    : Camera processor started for back: 475
2024-01-21 19:26:12.985551614  [2024-01-21 19:26:12] frigate.app                    INFO    : Camera processor started for sosed: 477
2024-01-21 19:26:12.993970975  [2024-01-21 19:26:12] frigate.app                    INFO    : Camera processor started for trucks: 479
2024-01-21 19:26:13.002088009  [2024-01-21 19:26:13] frigate.app                    INFO    : Camera processor started for trucks_back: 483
2024-01-21 19:26:13.010532870  [2024-01-21 19:26:13] frigate.app                    INFO    : Capture process started for ptz: 486
2024-01-21 19:26:13.018459909  [2024-01-21 19:26:13] frigate.app                    INFO    : Capture process started for front: 490
2024-01-21 19:26:13.025124759  [2024-01-21 19:26:13] frigate.app                    INFO    : Capture process started for back: 495
2024-01-21 19:26:13.032454413  [2024-01-21 19:26:13] frigate.app                    INFO    : Capture process started for sosed: 515
2024-01-21 19:26:13.039736511  [2024-01-21 19:26:13] frigate.app                    INFO    : Capture process started for trucks: 520
2024-01-21 19:26:13.046619286  [2024-01-21 19:26:13] frigate.app                    INFO    : Capture process started for trucks_back: 525
2024-01-21 19:26:15.583059964  [2024-01-21 19:26:12] frigate.detectors.plugins.edgetpu_tfl INFO    : Attempting to load TPU as usb
2024-01-21 19:26:15.591805902  [2024-01-21 19:26:15] frigate.detectors.plugins.edgetpu_tfl INFO    : TPU found

Frigate stats

{"cameras":{"back":{"audio_dBFS":0.0,"audio_rms":0.0,"camera_fps":5.1,"capture_pid":495,"detection_enabled":1,"detection_fps":3.1,"ffmpeg_pid":516,"pid":475,"process_fps":3.1,"skipped_fps":2.1},"front":{"audio_dBFS":0.0,"audio_rms":0.0,"camera_fps":5.1,"capture_pid":490,"detection_enabled":1,"detection_fps":3.0,"ffmpeg_pid":512,"pid":473,"process_fps":0.5,"skipped_fps":4.6},"ptz":{"audio_dBFS":0.0,"audio_rms":0.0,"camera_fps":5.1,"capture_pid":486,"detection_enabled":1,"detection_fps":3.0,"ffmpeg_pid":491,"pid":472,"process_fps":1.5,"skipped_fps":3.5},"sosed":{"audio_dBFS":0.0,"audio_rms":0.0,"camera_fps":5.1,"capture_pid":515,"detection_enabled":1,"detection_fps":0.0,"ffmpeg_pid":521,"pid":477,"process_fps":5.1,"skipped_fps":0.0},"trucks":{"audio_dBFS":0.0,"audio_rms":0.0,"camera_fps":5.0,"capture_pid":520,"detection_enabled":1,"detection_fps":0.0,"ffmpeg_pid":528,"pid":479,"process_fps":5.0,"skipped_fps":0.0},"trucks_back":{"audio_dBFS":0.0,"audio_rms":0.0,"camera_fps":5.0,"capture_pid":525,"detection_enabled":1,"detection_fps":0.0,"ffmpeg_pid":531,"pid":483,"process_fps":5.0,"skipped_fps":0.0}},"cpu_usages":{"1":{"cmdline":"/package/admin/s6/command/s6-svscan -d4 -- /run/service","cpu":"0.0","cpu_average":"0","mem":"0.0"},"15":{"cmdline":"s6-supervise s6-linux-init-shutdownd","cpu":"0.0","cpu_average":"0","mem":"0.0"},"16":{"cmdline":"/package/admin/s6-linux-init/command/s6-linux-init-shutdownd -c /run/s6/basedir -g 3000 -C -B","cpu":"0.0","cpu_average":"0","mem":"0.0"},"24":{"cmdline":"s6-supervise s6rc-fdholder","cpu":"0.0","cpu_average":"0","mem":"0.0"},"25":{"cmdline":"s6-supervise s6rc-oneshot-runner","cpu":"0.0","cpu_average":"0","mem":"0.0"},"26":{"cmdline":"s6-supervise nginx","cpu":"0.0","cpu_average":"0","mem":"0.0"},"27":{"cmdline":"s6-supervise go2rtc-log","cpu":"0.0","cpu_average":"0","mem":"0.0"},"28":{"cmdline":"s6-supervise go2rtc","cpu":"0.0","cpu_average":"0","mem":"0.0"},"29":{"cmdline":"s6-supervise frigate-log","cpu":"0.0","cpu_average":"0","mem":"0.0"},"30":{"cmdline":"s6-supervise nginx-log","cpu":"0.0","cpu_average":"0","mem":"0.0"},"31":{"cmdline":"s6-supervise frigate","cpu":"0.0","cpu_average":"0","mem":"0.0"},"32":{"cmdline":"s6-supervise go2rtc-healthcheck","cpu":"0.0","cpu_average":"0","mem":"0.0"},"43":{"cmdline":"/package/admin/s6-2.11.3.2/command/s6-fdholderd -1 -i data/rules","cpu":"0.0","cpu_average":"0","mem":"0.0"},"44":{"cmdline":"/package/admin/s6/command/s6-ipcserverd -1 -- /package/admin/s6/command/s6-ipcserver-access -v0 -E -l0 -i data/rules -- /package/admin/s6/command/s6-sudod -t 30000 -- /package/admin/s6-rc/command/s6-rc-oneshot-run -l ../.. --","cpu":"0.0","cpu_average":"0","mem":"0.0"},"89":{"cmdline":"s6-log -b -- T 1 n0 s10000000 T /dev/shm/logs/nginx","cpu":"0.0","cpu_average":"0","mem":"0.0"},"90":{"cmdline":"s6-log -b -- T 1 n0 s10000000 T /dev/shm/logs/go2rtc","cpu":"0.0","cpu_average":"0","mem":"0.0"},"91":{"cmdline":"s6-log -b -- T 1 n0 s10000000 T /dev/shm/logs/frigate","cpu":"0.0","cpu_average":"0","mem":"0.0"},"98":{"cmdline":"/config/go2rtc -config=/dev/shm/go2rtc.yaml","cpu":"0.0","cpu_average":"0","mem":"0.0"},"106":{"cmdline":"bash ./run.user go2rtc-healthcheck","cpu":"0.0","cpu_average":"0","mem":"0.0"},"107":{"cmdline":"python3 -u -m frigate","cpu":"4.4","cpu_average":"3","mem":"1.4"},"121":{"cmdline":"nginx: master process nginx","cpu":"0.0","cpu_average":"0","mem":"0.0"},"154":{"cmdline":"nginx: worker process","cpu":"0.0","cpu_average":"0","mem":"0.0"},"155":{"cmdline":"nginx: worker process","cpu":"0.0","cpu_average":"0","mem":"0.0"},"156":{"cmdline":"nginx: worker process","cpu":"0.0","cpu_average":"0","mem":"0.0"},"157":{"cmdline":"nginx: worker process","cpu":"0.0","cpu_average":"0","mem":"0.0"},"159":{"cmdline":"nginx: worker process","cpu":"0.0","cpu_average":"0","mem":"0.0"},"172":{"cmdline":"nginx: worker process","cpu":"0.0","cpu_average":"0","mem":"0.0"},"192":{"cmdline":"nginx: worker process","cpu":"0.0","cpu_average":"0","mem":"0.0"},"225":{"cmdline":"nginx: worker process","cpu":"0.0","cpu_average":"0","mem":"0.0"},"266":{"cmdline":"nginx: cache manager process","cpu":"0.0","cpu_average":"0","mem":"0.0"},"448":{"cmdline":"frigate.logger       ","cpu":"0.0","cpu_average":"0","mem":"0.3"},"450":{"cmdline":"frigate.recording_manager","cpu":"0.0","cpu_average":"2","mem":"0.4"},"459":{"cmdline":"/usr/bin/python3 -c from multiprocessing.resource_tracker import main;main(61)","cpu":"0.5","cpu_average":"0","mem":"0.0"},"460":{"cmdline":"frigate.detector.coral","cpu":"76.8","cpu_average":"69","mem":"0.5"},"462":{"cmdline":"frigate.output       ","cpu":"2.4","cpu_average":"0","mem":"0.3"},"468":{"cmdline":"ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 1920x1080 -i pipe: -f mpegts -s 1280x720 -codec:v mpeg1video -q 8 -bf 0 pipe:","cpu":"0.0","cpu_average":"0","mem":"0.0"},"469":{"cmdline":"ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 3840x2160 -i pipe: -f mpegts -s 1280x720 -codec:v mpeg1video -q 8 -bf 0 pipe:","cpu":"0.0","cpu_average":"0","mem":"0.0"},"471":{"cmdline":"ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 3840x2160 -i pipe: -f mpegts -s 1280x720 -codec:v mpeg1video -q 8 -bf 0 pipe:","cpu":"0.0","cpu_average":"0","mem":"0.0"},"472":{"cmdline":"frigate.process:ptz  ","cpu":"3.9","cpu_average":"2","mem":"0.4"},"473":{"cmdline":"frigate.process:front","cpu":"4.4","cpu_average":"6","mem":"0.4"},"475":{"cmdline":"frigate.process:back ","cpu":"4.4","cpu_average":"3","mem":"0.4"},"476":{"cmdline":"ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 2688x1520 -i pipe: -f mpegts -s 1273x720 -codec:v mpeg1video -q 8 -bf 0 pipe:","cpu":"0.0","cpu_average":"0","mem":"0.0"},"477":{"cmdline":"frigate.process:sosed","cpu":"1.0","cpu_average":"2","mem":"0.4"},"478":{"cmdline":"ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 2688x1520 -i pipe: -f mpegts -s 1273x720 -codec:v mpeg1video -q 8 -bf 0 pipe:","cpu":"0.0","cpu_average":"0","mem":"0.0"},"479":{"cmdline":"frigate.process:trucks","cpu":"1.5","cpu_average":"3","mem":"0.4"},"483":{"cmdline":"frigate.process:trucks_back","cpu":"1.0","cpu_average":"3","mem":"0.4"},"484":{"cmdline":"ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 2592x1944 -i pipe: -f mpegts -s 960x720 -codec:v mpeg1video -q 8 -bf 0 pipe:","cpu":"0.0","cpu_average":"0","mem":"0.0"},"486":{"cmdline":"frigate.capture:ptz  ","cpu":"3.9","cpu_average":"2","mem":"0.3"},"489":{"cmdline":"ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 1280x720 -i pipe: -f mpegts -s 1280x720 -codec:v mpeg1video -q 8 -bf 0 pipe:","cpu":"0.0","cpu_average":"0","mem":"0.0"},"490":{"cmdline":"frigate.capture:front","cpu":"8.8","cpu_average":"8","mem":"0.4"},"491":{"cmdline":"ffmpeg -hide_banner -loglevel warning -threads 2 -user_agent FFmpeg Frigate/0.13.0-49814b3 -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -timeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://admin:pass@192.168.0.8:554/axis-media/media.amp -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy -an /tmp/cache/ptz@%Y%m%d%H%M%S%z.mp4 -r 5 -vf fps=5,scale=1920:1080 -threads 2 -f rawvideo -pix_fmt yuv420p pipe:","cpu":"31.6","cpu_average":"33","mem":"0.1"},"495":{"cmdline":"frigate.capture:back ","cpu":"10.2","cpu_average":"8","mem":"0.4"},"512":{"cmdline":"ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel cuda -hwaccel_output_format cuda -user_agent FFmpeg Frigate/0.13.0-49814b3 -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -timeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://admin:pass@192.168.0.4:554/axis-media/media.amp -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy -an /tmp/cache/front@%Y%m%d%H%M%S%z.mp4 -r 5 -vf fps=5,scale_cuda=w=3840:h=2160:format=nv12,hwdownload,format=nv12,format=yuv420p -threads 2 -f rawvideo -pix_fmt yuv420p pipe:","cpu":"19.0","cpu_average":"19","mem":"0.5"},"515":{"cmdline":"frigate.capture:sosed","cpu":"3.9","cpu_average":"4","mem":"0.4"},"516":{"cmdline":"ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel cuda -hwaccel_output_format cuda -user_agent FFmpeg Frigate/0.13.0-49814b3 -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -timeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://admin:pass@192.168.0.5:554/axis-media/media.amp -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy -an /tmp/cache/back@%Y%m%d%H%M%S%z.mp4 -r 5 -vf fps=5,scale_cuda=w=3840:h=2160:format=nv12,hwdownload,format=nv12,format=yuv420p -threads 2 -f rawvideo -pix_fmt yuv420p pipe:","cpu":"18.5","cpu_average":"17","mem":"0.4"},"520":{"cmdline":"frigate.capture:trucks","cpu":"5.3","cpu_average":"4","mem":"0.4"},"521":{"cmdline":"ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel cuda -hwaccel_output_format cuda -user_agent FFmpeg Frigate/0.13.0-49814b3 -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -timeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://admin:pass@192.168.0.6:554/axis-media/media.amp -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy -an /tmp/cache/sosed@%Y%m%d%H%M%S%z.mp4 -r 5 -vf fps=5,scale_cuda=w=2688:h=1520:format=nv12,hwdownload,format=nv12,format=yuv420p -threads 2 -f rawvideo -pix_fmt yuv420p pipe:","cpu":"11.2","cpu_average":"10","mem":"0.3"},"525":{"cmdline":"frigate.capture:trucks_back","cpu":"6.3","cpu_average":"5","mem":"0.4"},"528":{"cmdline":"ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel cuda -hwaccel_output_format cuda -user_agent FFmpeg Frigate/0.13.0-49814b3 -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -timeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://admin:pass@192.168.0.7:554/axis-media/media.amp -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy -an /tmp/cache/trucks@%Y%m%d%H%M%S%z.mp4 -r 5 -vf fps=5,scale_cuda=w=2688:h=1520:format=nv12,hwdownload,format=nv12,format=yuv420p -threads 2 -f rawvideo -pix_fmt yuv420p pipe:","cpu":"11.7","cpu_average":"9","mem":"0.3"},"531":{"cmdline":"ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel cuda -hwaccel_output_format cuda -user_agent FFmpeg Frigate/0.13.0-49814b3 -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -timeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://admin:pass@192.168.0.13:554/trackID=1 -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy -an /tmp/cache/trucks_back@%Y%m%d%H%M%S%z.mp4 -r 5 -vf fps=5,scale_cuda=w=2592:h=1944:format=nv12,hwdownload,format=nv12,format=yuv420p -threads 2 -f rawvideo -pix_fmt yuv420p pipe:","cpu":"13.6","cpu_average":"12","mem":"0.4"},"3228":{"cmdline":"sleep 30s","cpu":"0.0","cpu_average":"0","mem":"0.0"}},"detection_fps":9.1,"detectors":{"coral":{"detection_start":1705858405.229372,"inference_speed":114.51,"pid":460}},"gpu_usages":{"NVIDIA GeForce GTX 1650":{"dec":"45.0%","enc":"0.0%","gpu":"4.0%","mem":"33.61%"}},"processes":{"go2rtc":{"pid":98},"logger":{"pid":448},"recording":{"pid":450}},"service":{"last_updated":1705858405,"latest_version":"0.12.1","storage":{"/dev/shm":{"free":376.8,"mount_type":"tmpfs","total":512.0,"used":135.2},"/media/frigate/clips":{"free":1617886.5,"mount_type":"fuseblk","total":3815430.0,"used":2197543.4},"/media/frigate/recordings":{"free":1617886.5,"mount_type":"fuseblk","total":3815430.0,"used":2197543.4},"/tmp/cache":{"free":912.1,"mount_type":"tmpfs","total":953.7,"used":41.6}},"temperatures":{},"uptime":432,"version":"0.13.0-49814b3"}}

Operating system

Debian

Install method

Docker Compose

Coral version

USB

Any other information that may be helpful

Go2RTC logs contain no errors:

2024-01-21 19:26:11.644011786  [INFO] Preparing new go2rtc config...
2024-01-21 19:26:11.893152752  [INFO] Not injecting WebRTC candidates into go2rtc config as it has been set manually
2024-01-21 19:26:11.930066460  [WARN] Using go2rtc binary from '/config/go2rtc' instead of the embedded one
2024-01-21 19:26:11.930069280  [INFO] Starting go2rtc...
2024-01-21 19:26:11.993388698  19:26:11.993 INF go2rtc version 1.8.5 linux/amd64
2024-01-21 19:26:11.993804381  19:26:11.993 INF [api] listen addr=:1984
2024-01-21 19:26:11.993882725  19:26:11.993 INF [rtsp] listen addr=:8554
2024-01-21 19:26:11.994160339  19:26:11.994 INF [webrtc] listen addr=:8555
2024-01-21 19:26:21.639203783  [INFO] Starting go2rtc healthcheck service...
NickM-27 commented 9 months ago

Have you recompiled the trt models to use the new version? Like the release notes said models generated previously are not supported.

SuperMaximus1984 commented 9 months ago 
/trt-models/efficientdet_lite3_512_ptq_edgetpu.tflite

No, I'm not using TRT Models here, it's just the homonymous name of folder where I placed TRT & TFLITE models. Here I use Coral USD & EfficientDet model. By the way, did Tensorflow become more efficient in 0.13 compared to 0.12. It's kind of an off-topic, but according to my tests TRT YoloV7 was much inferior compared to EfficientDet model in terms of false positives detection. As soon as I get Coral detector running smoothly, I'll start experimenting with TRT.

NickM-27 commented 9 months ago

The main change in 0.13 is that models are generated automatically for trt on first startup.

What are the frigate logs after running for 5 minutes?

SuperMaximus1984 commented 9 months ago

The Firgate logs are unchanged after running for 5 minutes:

2024-01-21 22:14:43.468230607  [INFO] Preparing Frigate...
2024-01-21 22:14:43.477142601  [INFO] Starting Frigate...
2024-01-21 22:14:44.534857817  [2024-01-21 22:14:44] frigate.app                    INFO    : Starting Frigate (0.13.0-49814b3)
2024-01-21 22:14:44.615900839  [2024-01-21 22:14:44] peewee_migrate.logs            INFO    : Starting migrations
2024-01-21 22:14:44.619030073  [2024-01-21 22:14:44] peewee_migrate.logs            INFO    : There is nothing to migrate
2024-01-21 22:14:44.623817329  [2024-01-21 22:14:44] frigate.app                    INFO    : Recording process started: 450
2024-01-21 22:14:44.625873411  [2024-01-21 22:14:44] frigate.app                    INFO    : go2rtc process pid: 98
2024-01-21 22:14:44.649159622  [2024-01-21 22:14:44] detector.coral                 INFO    : Starting detection process: 460
2024-01-21 22:14:47.296180979  [2024-01-21 22:14:44] frigate.app                    INFO    : Output process started: 462
2024-01-21 22:14:47.296364168  [2024-01-21 22:14:44] frigate.detectors.plugins.edgetpu_tfl INFO    : Attempting to load TPU as usb
2024-01-21 22:14:47.296763798  [2024-01-21 22:14:44] frigate.app                    INFO    : Camera processor started for ptz: 471
2024-01-21 22:14:47.296844796  [2024-01-21 22:14:44] frigate.app                    INFO    : Camera processor started for front: 472
2024-01-21 22:14:47.296921032  [2024-01-21 22:14:44] frigate.app                    INFO    : Camera processor started for back: 475
2024-01-21 22:14:47.296993620  [2024-01-21 22:14:44] frigate.app                    INFO    : Camera processor started for sosed: 476
2024-01-21 22:14:47.297069255  [2024-01-21 22:14:44] frigate.app                    INFO    : Camera processor started for trucks: 478
2024-01-21 22:14:47.297174027  [2024-01-21 22:14:44] frigate.app                    INFO    : Camera processor started for trucks_back: 482
2024-01-21 22:14:47.297246923  [2024-01-21 22:14:44] frigate.app                    INFO    : Capture process started for ptz: 485
2024-01-21 22:14:47.297318148  [2024-01-21 22:14:44] frigate.app                    INFO    : Capture process started for front: 489
2024-01-21 22:14:47.297389394  [2024-01-21 22:14:44] frigate.app                    INFO    : Capture process started for back: 494
2024-01-21 22:14:47.297464049  [2024-01-21 22:14:44] frigate.app                    INFO    : Capture process started for sosed: 500
2024-01-21 22:14:47.297536440  [2024-01-21 22:14:44] frigate.app                    INFO    : Capture process started for trucks: 520
2024-01-21 22:14:47.297610643  [2024-01-21 22:14:44] frigate.app                    INFO    : Capture process started for trucks_back: 525
2024-01-21 22:14:47.304767666  [2024-01-21 22:14:47] frigate.detectors.plugins.edgetpu_tfl INFO    : TPU found

Was something changed in FFMPEG interaction in 0.13? I see frames skipped in Detect here, which was not the issue in 0.12 (0 skipped there all the time): image

The weird thing is that camera screenshots on the main Frigate dashboard are not getting updated after Frigate restart. And the more I refresh the page browser the less screenshots I see (I see "No frames have been received, check error logs" messages on all camera preview windows). In 0.12 I saw regular camera image updates (once per 5 seconds or so).

NickM-27 commented 9 months ago

No nothing changed with ffmpeg. Also if I understand what you're saying that is not correct, there is no such functionality to refresh the images on the main camera dashboard. Can you send a full screenshot of the entire system page please

SuperMaximus1984 commented 9 months ago

@NickM-27 Yes, you're right, I misinterpreted it a bit: after I refresh the page in 0.12 I see changes in camera images, while in 0.13 I see either a static (previous) image or the message about "No frames.." on a black screen. In the meantime I need an operable Frigate by tomorrow, thus I had to revert to 0.12. If this is what you need, it's same for 0.12 and 0.13 (unless the latter starts to skip the frames): image

NickM-27 commented 9 months ago

100 milliseconds of inference time is crazy high, so that is most likely the problem. I'd suggest using the built in model and see if that fixes the issue

SuperMaximus1984 commented 9 months ago

100 milliseconds of inference time is crazy high, so that is most likely the problem. I'd suggest using the built in model and see if that fixes the issue

Same inference time in 0.12, and no problems at all, not a single frame skipped. Built-in model, albeit gives lesser inference time, quality-wise is much more inferior to EfficientDet.

There's something wrong/different in 0.13 for sure.

NickM-27 commented 9 months ago

The skipped frames counter was broken in 0.12, so no skipped frames doesn't mean it wasn't actually skipping frames

NickM-27 commented 9 months ago

100ms is really slow, it means you can only run 10 inferences per second, which is not nearly enough for 5 fps across 6 cameras

NickM-27 commented 9 months ago

In any case, we've not had any other reports of this issue so it will likely be something fairly specific

SuperMaximus1984 commented 9 months ago

@NickM-27 OK, I can try reconfiguring Frigate for TensorRT despite its model's inferiority in terms of accuracy, but I don't think the inference time has something to do with constantly disappearing streams in 0.13. When "No frames.." appears, the system doesn't work, logically. And in 0.12 it works. Would you please elaborate on your earlier message regarding TensorRT models generated at first run in 0.13? Does it mean I should not compile them on my own? Here's my previous TensorRT 0.12 config part:

detectors:
 tensorrt:
   type: tensorrt
   device: 0
model:
 path: /trt-models/yolov7x-640.trt
 input_tensor: nchw
 input_pixel_format: rgb
 width: 640
 height: 640
SuperMaximus1984 commented 9 months ago

I've tried running with TRT, now the inference time is much better, but it doesn't change the "No frames.." issue appearing periodically (very frequently). image image

NickM-27 commented 9 months ago

the inference time has a direct affect on this issue, because when frigate is sitting waiting for the object detection inference to complete it is unable to process other frames and that means the frame times fall behind causing frigate to return the no frames received image.

we still see loads of skipped frames which means something is still very wrong here. I wouldn't expect that to occur with 47 ms inference speed but it seems something is quite wrong here. I think the best thing for testing is to use the built in coral model and that should give an inference time of 10 ms and see if it still occurs at that point

NickM-27 commented 9 months ago

another thing you could try is just turning off detect in the dashboard for all cameras and see if the issue improves

SuperMaximus1984 commented 9 months ago

@NickM-27 Yeah, I managed to get 27-33 ms inference time with TRT Yolov7-640 model. Now it doesn't skip anything and "No frames" message doesn't appear.

However I regret that the EfficientDet model takes so much time for inference. That model is much more false-proof than the standard one.

NickM-27 commented 9 months ago

Because the models aren't trained on security camera images they definitely vary highly from camera to camera in terms of false positives.

SuperMaximus1984 commented 9 months ago

@NickM-27 One more question - I have a PTZ camera with parameters like this:

cameras:
  ptz:
    ffmpeg:
      hwaccel_args: preset-nvidia-h264
      inputs:
        - path: rtsp://login:pass@192.168.0.8:554/axis-media/media.amp
          roles:
            - detect
    detect:
      width: 1920
      height: 1080
      fps: 5
    objects:
      track:
        - person
        - car
        - truck
        - bicycle
        - motorcycle
    record:
      enabled: true
      retain:
        days: 10
        mode: motion

In Frigate 0.12 there was a clear recording from motion start till motion end (PTZ moves quickly, I assume the motion detection was leveraged immediately and recording started as well). Now in Frigate 0.13 I see only a part of my target motion recorded, it's abrupt (sometimes records only the end of clip). I assume it happens due to a novel Norfair motion detection. Is it somehow configurable so that I could get the needed result?

NickM-27 commented 9 months ago

there are some things to clear up here:

  1. norfair does not do motion detection, it handles tracking objects as they move across the camera frame
  2. what you are likely seeing is the motion recalibration logic (calling lightning_detection in the docs) which will stop running object detection when a large motion change occurs like lightning, a camera switching from IR to color, etc. you can increase the lightning_threhsold to 1.0 and that should fix the issue
ithesk commented 8 months ago

I have the same problem, and in version 0.12 it doesn't happen, only in version 13

Screenshot (5) Screenshot (6) Screenshot (7) Screenshot (8)

If anyone has any help, I have Reolink cameras. but in version 12 it works

jarru66 commented 8 months ago

I have Reolink RLC-523WA actually two separate HA setups. With Frigate 0.12.1 both setups are working, but upgrading Frigate to 0.13.2 those are not working. The following errors are coming: frigate.video ERROR : sipoo-piha: Unable to read frames from ffmpeg process. rigate.video ERROR : sipoo-piha: ffmpeg process is not running. exiting capture thread... watchdog.sipoo-piha ERROR : Ffmpeg process crashed unexpectedly for sipoo-piha.

By restoring back to Frigate 0.12.1 without any Config changes, the setups are working.

PeterPalenik commented 7 months ago

Very similar here. I get the 'No frames have been received, check error logs' very often on all cameras simultaneously. This wasn't happening on previous versions.

jarru66 commented 7 months ago

Very similar here. I get the 'No frames have been received, check error logs' very often on all cameras simultaneously. This wasn't happening on previous versions.

I have this nowadays working. It seems that it takes time to stabilize after upgrade from 0.12.1 to 0.13.2. It was taking even about 24 Hours.

ksanislo commented 7 months ago

I also have this same issue with a variety of Amcrest cameras after upgrading to 0.13.2. Nearly all of my cameras are giving the "no frames have been received" message, with a few intermittently successful frames being shown after multiple reloads of the camera page. Everything worked flawlessly with 0.12.1, so I've had to downgrade back to that version to have a functional NVR system.

ksanislo commented 7 months ago

So, after some fiddling, I found that I was able to get 0.13.2 working with a combination of using the ffmpeg:rtsp:// source in go2rtc along with the secondary lower VGA resolution camera streams at 640x480, then scaled up to 854x480 for detection to fix the aspect ratio. Previously with 0.12.1 I had been using the native camera resolution video scaled down to 1280x720 for detection without problem.

I'm still somewhat disappointed in this workaround, as JSMPEG quality suffers quite dramatically, but at least my home assistant integration is working again with the latest version.

Beeez commented 7 months ago

Also getting this after moving to v0.13. Though I am guessing it is due to performance and high inference.

tobernguyen commented 7 months ago

I'm getting this too after upgrading to v0.13. I'm having reolink camera with the recommended setup (using HTTP stream with lower resolution). I haven't found a solution yet. Part of the config (worked really well with v0.12)

go2rtc:
  ffmpeg:
    bin: ffmpeg
    homekit: "-map -an -tune zerolatency -pix_fmt yuv420p -payload_type 99"
    reolink: "-hwaccel vaapi -hwaccel_device /dev/dri/renderD128"
  webrtc:
    candidates:
      - 192.168.5.7:8555
  streams:
    front_door:
      - http://192.168.40.50/flv?port=1935&app=bcs&stream=channel2_sub.bcs&user=frigate&password={FRIGATE_RTSP_PASSWORD}
      - "ffmpeg:front_door_sub#hardware"

ffmpeg:
  hwaccel_args: preset-vaapi
  input_args: preset-rtsp-restream
  output_args:
    record: preset-record-generic-audio-aac

cameras:
  # Camera section...

Logs

2024-03-27 05:44:45.290634564  [INFO] Preparing Frigate...
2024-03-27 05:44:45.321618001  [INFO] Starting Frigate...
2024-03-27 05:44:46.431619316  [2024-03-27 05:44:46] frigate.app                    INFO    : Starting Frigate (0.13.2-6476f8a)
2024-03-27 05:44:46.459680918  [2024-03-27 05:44:46] frigate.config                 WARNING : Customizing more than a detector model path is unsupported.
2024-03-27 05:44:46.469615587  [2024-03-27 05:44:46] peewee_migrate.logs            INFO    : Starting migrations
2024-03-27 05:44:46.472715796  [2024-03-27 05:44:46] peewee_migrate.logs            INFO    : There is nothing to migrate
2024-03-27 05:44:46.472830402  [2024-03-27 05:44:46] frigate.app                    INFO    : Running database vacuum
2024-03-27 05:44:46.784920534  [2024-03-27 05:44:46] frigate.app                    INFO    : Recording process started: 728
2024-03-27 05:44:46.785851580  [2024-03-27 05:44:46] frigate.app                    INFO    : go2rtc process pid: 89
2024-03-27 05:44:46.800199393  [2024-03-27 05:44:46] detector.ov                    INFO    : Starting detection process: 737
2024-03-27 05:44:46.804000899  [2024-03-27 05:44:46] frigate.app                    INFO    : Output process started: 739
2024-03-27 05:44:46.842126066  [2024-03-27 05:44:46] frigate.app                    INFO    : Camera processor started for front_door: 751
2024-03-27 05:44:46.845837706  [2024-03-27 05:44:46] frigate.app                    INFO    : Camera processor started for garage: 754
2024-03-27 05:44:46.848592965  [2024-03-27 05:44:46] frigate.app                    INFO    : Camera processor started for side_door: 755
2024-03-27 05:44:46.851794630  [2024-03-27 05:44:46] frigate.app                    INFO    : Camera processor started for backyard_door: 775
2024-03-27 05:44:46.855017456  [2024-03-27 05:44:46] frigate.app                    INFO    : Camera processor started for backyard: 776
2024-03-27 05:44:46.857874371  [2024-03-27 05:44:46] frigate.app                    INFO    : Camera processor started for right_side: 778
2024-03-27 05:44:46.860685213  [2024-03-27 05:44:46] frigate.app                    INFO    : Camera processor started for left_side: 781
2024-03-27 05:44:46.863694533  [2024-03-27 05:44:46] frigate.app                    INFO    : Capture process started for front_door: 783
2024-03-27 05:44:46.867245543  [2024-03-27 05:44:46] frigate.app                    INFO    : Capture process started for garage: 786
2024-03-27 05:44:46.870647347  [2024-03-27 05:44:46] frigate.app                    INFO    : Capture process started for side_door: 790
2024-03-27 05:44:46.875149290  [2024-03-27 05:44:46] frigate.app                    INFO    : Capture process started for backyard_door: 795
2024-03-27 05:44:46.878962989  [2024-03-27 05:44:46] frigate.app                    INFO    : Capture process started for backyard: 800
2024-03-27 05:44:46.882721651  [2024-03-27 05:44:46] frigate.app                    INFO    : Capture process started for right_side: 804
2024-03-27 05:44:46.886981312  [2024-03-27 05:44:46] frigate.app                    INFO    : Capture process started for left_side: 808
2024-03-27 05:44:46.891577112  [2024-03-27 05:44:46] frigate.app                    INFO    : Audio process started: 812
2024-03-27 05:44:47.172987107  E: [xLinkUsb] [    287172] [frigate.detecto] usb_find_device_with_bcd:266   Library has not been initialized when loaded
2024-03-27 05:44:47.173264509  E: [xLinkUsb] [    287173] [frigate.detecto] usb_find_device_with_bcd:266   Library has not been initialized when loaded
2024-03-27 05:44:47.279115579  [2024-03-27 05:44:47] frigate.detectors.plugins.openvino INFO    : Model Input Shape: [1,300,300,3]
2024-03-27 05:44:47.279173734  [2024-03-27 05:44:47] frigate.detectors.plugins.openvino INFO    : Model Output-0 Shape: [1,1,100,7]
2024-03-27 05:44:47.279194609  [2024-03-27 05:44:47] frigate.detectors.plugins.openvino INFO    : Model has 1 Output Tensors
2024-03-27 05:44:48.365313260  [2024-03-27 05:44:48] frigate.util.services          ERROR   : Unable to poll intel GPU stats: Failed to initialize PMU! (Operation not permitted)
2024-03-27 05:44:48.365316903  
2024-03-27 05:44:51.998861727  [2024-03-27 05:44:51] frigate.video                  ERROR   : right_side: Unable to read frames from ffmpeg process.
2024-03-27 05:44:51.998891621  [2024-03-27 05:44:51] frigate.video                  ERROR   : right_side: ffmpeg process is not running. exiting capture thread...
2024-03-27 05:45:06.898587933  [2024-03-27 05:45:06] watchdog.right_side            ERROR   : Ffmpeg process crashed unexpectedly for right_side.
2024-03-27 05:45:06.898661620  [2024-03-27 05:45:06] watchdog.right_side            ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2024-03-27 05:45:06.898731034  [2024-03-27 05:45:06] ffmpeg.right_side.detect       ERROR   : rtsp://127.0.0.1:8554/right_side: Invalid data found when processing input
2024-03-27 05:45:12.013006916  [2024-03-27 05:45:12] frigate.video                  ERROR   : right_side: Unable to read frames from ffmpeg process.
2024-03-27 05:45:12.013071426  [2024-03-27 05:45:12] frigate.video                  ERROR   : right_side: ffmpeg process is not running. exiting capture thread...
2024-03-27 05:45:16.904831977  [2024-03-27 05:45:16] watchdog.right_side            ERROR   : Ffmpeg process crashed unexpectedly for right_side.
2024-03-27 05:45:16.904889109  [2024-03-27 05:45:16] watchdog.right_side            ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2024-03-27 05:45:16.904934025  [2024-03-27 05:45:16] ffmpeg.right_side.detect       ERROR   : rtsp://127.0.0.1:8554/right_side: Invalid data found when processing input
2024-03-27 05:45:22.018222863  [2024-03-27 05:45:22] frigate.video                  ERROR   : right_side: Unable to read frames from ffmpeg process.
2024-03-27 05:45:22.018286996  [2024-03-27 05:45:22] frigate.video                  ERROR   : right_side: ffmpeg process is not running. exiting capture thread...
2024-03-27 05:45:26.910221430  [2024-03-27 05:45:26] watchdog.right_side            ERROR   : Ffmpeg process crashed unexpectedly for right_side.
2024-03-27 05:45:26.910284924  [2024-03-27 05:45:26] watchdog.right_side            ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2024-03-27 05:45:26.910335333  [2024-03-27 05:45:26] ffmpeg.right_side.detect       ERROR   : rtsp://127.0.0.1:8554/right_side: Invalid data found when processing input
2024-03-27 05:45:32.024912364  [2024-03-27 05:45:32] frigate.video                  ERROR   : right_side: Unable to read frames from ffmpeg process.
2024-03-27 05:45:32.024926891  [2024-03-27 05:45:32] frigate.video                  ERROR   : right_side: ffmpeg process is not running. exiting capture thread...
2024-03-27 05:45:36.916651862  [2024-03-27 05:45:36] watchdog.right_side            ERROR   : Ffmpeg process crashed unexpectedly for right_side.
2024-03-27 05:45:36.916689981  [2024-03-27 05:45:36] watchdog.right_side            ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2024-03-27 05:45:36.916728912  [2024-03-27 05:45:36] ffmpeg.right_side.detect       ERROR   : rtsp://127.0.0.1:8554/right_side: Invalid data found when processing input
2024-03-27 05:45:42.030827381  [2024-03-27 05:45:42] frigate.video                  ERROR   : right_side: Unable to read frames from ffmpeg process.
2024-03-27 05:45:42.030868491  [2024-03-27 05:45:42] frigate.video                  ERROR   : right_side: ffmpeg process is not running. exiting capture thread...
2024-03-27 05:45:46.922121526  [2024-03-27 05:45:46] watchdog.right_side            ERROR   : Ffmpeg process crashed unexpectedly for right_side.
2024-03-27 05:45:46.922178375  [2024-03-27 05:45:46] watchdog.right_side            ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2024-03-27 05:45:46.922223299  [2024-03-27 05:45:46] ffmpeg.right_side.detect       ERROR   : rtsp://127.0.0.1:8554/right_side: Invalid data found when processing input
2024-03-27 05:45:52.036210046  [2024-03-27 05:45:52] frigate.video                  ERROR   : right_side: Unable to read frames from ffmpeg process.
2024-03-27 05:45:52.036234118  [2024-03-27 05:45:52] frigate.video                  ERROR   : right_side: ffmpeg process is not running. exiting capture thread...
2024-03-27 05:45:56.879248418  [2024-03-27 05:45:56] watchdog.front_door            INFO    : No frames received from front_door in 20 seconds. Exiting ffmpeg...
2024-03-27 05:45:56.879384732  [2024-03-27 05:45:56] watchdog.front_door            INFO    : Waiting for ffmpeg to exit gracefully...
2024-03-27 05:45:56.883268665  [2024-03-27 05:45:56] watchdog.garage                INFO    : No frames received from garage in 20 seconds. Exiting ffmpeg...
2024-03-27 05:45:56.883437595  [2024-03-27 05:45:56] watchdog.garage                INFO    : Waiting for ffmpeg to exit gracefully...
2024-03-27 05:45:56.887184508  [2024-03-27 05:45:56] watchdog.side_door             INFO    : No frames received from side_door in 20 seconds. Exiting ffmpeg...
2024-03-27 05:45:56.887337595  [2024-03-27 05:45:56] watchdog.side_door             INFO    : Waiting for ffmpeg to exit gracefully...
2024-03-27 05:45:56.902977193  [2024-03-27 05:45:56] watchdog.left_side             INFO    : No frames received from left_side in 20 seconds. Exiting ffmpeg...
2024-03-27 05:45:56.903092188  [2024-03-27 05:45:56] watchdog.left_side             INFO    : Waiting for ffmpeg to exit gracefully...
2024-03-27 05:45:56.911088904  [2024-03-27 05:45:56] frigate.video                  ERROR   : front_door: Unable to read frames from ffmpeg process.
2024-03-27 05:45:56.911544439  [2024-03-27 05:45:56] frigate.video                  ERROR   : front_door: ffmpeg process is not running. exiting capture thread...
2024-03-27 05:45:56.913336503  [2024-03-27 05:45:56] frigate.video                  ERROR   : garage: Unable to read frames from ffmpeg process.
2024-03-27 05:45:56.913475487  [2024-03-27 05:45:56] frigate.video                  ERROR   : garage: ffmpeg process is not running. exiting capture thread...
2024-03-27 05:45:56.927993926  [2024-03-27 05:45:56] watchdog.right_side            ERROR   : Ffmpeg process crashed unexpectedly for right_side.
2024-03-27 05:45:56.928056725  [2024-03-27 05:45:56] watchdog.right_side            ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2024-03-27 05:45:56.928106936  [2024-03-27 05:45:56] ffmpeg.right_side.detect       ERROR   : rtsp://127.0.0.1:8554/right_side: Invalid data found when processing input
2024-03-27 05:45:56.941687075  [2024-03-27 05:45:56] frigate.video                  ERROR   : left_side: Unable to read frames from ffmpeg process.
2024-03-27 05:45:56.942156469  [2024-03-27 05:45:56] frigate.video                  ERROR   : left_side: Unable to read frames from ffmpeg process.
2024-03-27 05:45:56.942243611  [2024-03-27 05:45:56] frigate.video                  ERROR   : left_side: Unable to read frames from ffmpeg process.
2024-03-27 05:45:56.942300725  [2024-03-27 05:45:56] frigate.video                  ERROR   : left_side: ffmpeg process is not running. exiting capture thread...
2024-03-27 05:45:56.942976070  [2024-03-27 05:45:56] watchdog.backyard              INFO    : No frames received from backyard in 20 seconds. Exiting ffmpeg...
2024-03-27 05:45:56.943228824  [2024-03-27 05:45:56] watchdog.backyard              INFO    : Waiting for ffmpeg to exit gracefully...
2024-03-27 05:45:56.943305827  [2024-03-27 05:45:56] watchdog.backyard_door         INFO    : No frames received from backyard_door in 20 seconds. Exiting ffmpeg...
2024-03-27 05:45:56.943590887  [2024-03-27 05:45:56] watchdog.backyard_door         INFO    : Waiting for ffmpeg to exit gracefully...
2024-03-27 05:45:56.944348248  [2024-03-27 05:45:56] frigate.video                  ERROR   : side_door: Unable to read frames from ffmpeg process.
2024-03-27 05:45:56.944754717  [2024-03-27 05:45:56] frigate.video                  ERROR   : side_door: ffmpeg process is not running. exiting capture thread...
2024-03-27 05:45:56.988484979  [2024-03-27 05:45:56] frigate.video                  ERROR   : backyard: Unable to read frames from ffmpeg process.
2024-03-27 05:45:56.988615490  [2024-03-27 05:45:56] frigate.video                  ERROR   : backyard: ffmpeg process is not running. exiting capture thread...
2024-03-27 05:45:57.016508291  [2024-03-27 05:45:57] frigate.video                  ERROR   : backyard_door: Unable to read frames from ffmpeg process.
2024-03-27 05:45:57.016549728  [2024-03-27 05:45:57] frigate.video                  ERROR   : backyard_door: ffmpeg process is not running. exiting capture thread...
2024-03-27 05:45:57.116533673  [2024-03-27 05:45:57] frigate.util.services          ERROR   : Unable to poll intel GPU stats: Failed to initialize PMU! (Operation not permitted)
2024-03-27 05:45:57.116536010  
2024-03-27 05:46:02.046230722  [2024-03-27 05:46:02] frigate.video                  ERROR   : right_side: Unable to read frames from ffmpeg process.
2024-03-27 05:46:02.046295837  [2024-03-27 05:46:02] frigate.video                  ERROR   : right_side: ffmpeg process is not running. exiting capture thread...
2024-03-27 05:46:06.912752970  [2024-03-27 05:46:06] watchdog.front_door            ERROR   : Ffmpeg process crashed unexpectedly for front_door.
2024-03-27 05:46:06.912791258  [2024-03-27 05:46:06] watchdog.front_door            ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2024-03-27 05:46:06.912845149  [2024-03-27 05:46:06] ffmpeg.front_door.detect       ERROR   : [segment @ 0x564aadb65b00] Non-monotonous DTS in output stream 0:0; previous: 4285470, current: 1079832; changing to 4285471. This may result in incorrect timestamps in the output file.
2024-03-27 05:46:06.912849275  [2024-03-27 05:46:06] ffmpeg.front_door.detect       ERROR   : [segment @ 0x564aadb65b00] Non-monotonous DTS in output stream 0:0; previous: 4285471, current: 1090632; changing to 4285472. This may result in incorrect timestamps in the output file.
2024-03-27 05:46:06.912855690  [2024-03-27 05:46:06] ffmpeg.front_door.detect       ERROR   : [segment @ 0x564aadb65b00] Non-monotonous DTS in output stream 0:0; previous: 4285472, current: 1097832; changing to 4285473. This may result in incorrect timestamps in the output file.
2024-03-27 05:46:06.912870973  [2024-03-27 05:46:06] ffmpeg.front_door.detect       ERROR   : [segment @ 0x564aadb65b00] Non-monotonous DTS in output stream 0:0; previous: 4285473, current: 1108632; changing to 4285474. This may result in incorrect timestamps in the output file.
tobernguyen commented 7 months ago

Update: Downgrading go2rtc to 1.6.2 seems to fix the issue. I tested 1.7.0 and got the same issue.

NickM-27 commented 7 months ago

@tobernguyen the config you have there is not the reolink recommended config, the difference being the recommended config uses ffmpeg:http

ithesk commented 7 months ago

I couldn't solve the problem, I had to downgrade to 12. I would have to try changing the go2rtc version. Someone who confirmed if that would be the problem?

tobernguyen commented 7 months ago

@tobernguyen the config you have there is not the reolink recommended config, the difference being the recommended config uses ffmpeg:http

@NickM-27 Thanks for pointing that out. I will consider trying that with the latest go2rtc. That said, I have used this configuration in v0.12 and have zero issues. Since downgrading go2rtc to 1.6.2, I have never seen the "No frames have been received, check error logs" anymore and the recordings are no longer broken midway.

@ithesk At least in my setup, I can confirm go2rtc was the culprit ^

NickM-27 commented 7 months ago

@tobernguyen yes, the later go2rtc has issues with http flv hence the recommendation to use ffmpeg with it

tobernguyen commented 7 months ago

@tobernguyen yes, the later go2rtc has issues with http flv hence the recommendation to use ffmpeg with it

I just tested using ffmpeg:http with the embedded go2rtc (1.8.4) and it seems to work well! Thank you very much @NickM-27 !

PeterPalenik commented 7 months ago

I changed it to ffmpeg:http but it did not help in my case. But I have one camera that I power on only during the night and I see in the logs that frigate is constantly trying to connect to it when it's. Wondering if that might be the problem. I'll try to check if the problem also happens when the camera is on.

jaaneo commented 6 months ago

Update: Downgrading go2rtc to 1.6.2 seems to fix the issue. I tested 1.7.0 and got the same issue.

how to download the version of go2rtc?
I currently have frigate in a docker container in proxmox.

sebdoan commented 5 months ago

@jaaneo try following the tips on this thread https://github.com/blakeblackshear/frigate/issues/6502#issuecomment-1556894356. Hopefully it steers you in the right path as all installations are slightly different.

Goeste commented 5 months ago

Hi, I also just wanted to add that I am having issues with the OP's topic.

Tried to change go2rtc version from 1.8.4 to 1.8.5 no change, to 1.6.0 no change. to 1.9.0 no change.

I can see how the streamy are being picked up by the GPU but immediately disapear from nvidia-smi.

Only one out of six camera (all of my six cams are of the same model [Annke c800; HVEC only]) is working without any issues.

Storage or GPU Memory (tried 4GB to 24GB on a Tesla P40) cannot be an issue... since streams are aborting after only (max) 5 seconds.

If needed I can attache some logs.

Best, goeste

Goeste commented 5 months ago

Update: Downgrading go2rtc to 1.6.2 seems to fix the issue. I tested 1.7.0 and got the same issue.

how to download the version of go2rtc? I currently have frigate in a docker container in proxmox.

just go here and download what you need: https://github.com/AlexxIT/go2rtc/releases make the file executable (chmod +x) and mv to go2rtc in /config of the frigate folder

sebdoan commented 5 months ago

@Goeste I tried all those versions you did as well and none of them worked for me. Go2rtc version 1.5.0 however has been perfect, super stable no issues across all my POE and WiFi based cameras.

Give 1.5.0 a go and good luck 🤞

Goeste commented 5 months ago

@sebdoan TY for that ;)

guess what.. i rebootet all cams that were not working and now even with 1.9.0 everything works again... let's see for how long if it will fail again, i'll switch to 1.5.0

Goeste commented 5 months ago

Ok, with 1.9.0. it failed again. I replade go2rtc with version 1.5.0 still failing... only one webcam is detecting objects, non of the other 5 is doing so...

Goeste commented 5 months ago

I also see it's only failing when using the mainstream instead of the substream for detection (except for one camera....) weird...

Goeste commented 5 months ago

OK, issue found.

all webcams were set to 6FPS where as the one that is working was on 12,5FPS. So... I set all others to 12,5 FPS and automagically streams reappear... no hanging since with go2rtc 1.9.1 :) My webcams are all Annke c800 (h265 streams)

stintov commented 4 months ago

I've also get the No frames have been received, check error logs issue

sebdoan commented 4 months ago

My experience is that eventually this will happen no matter what you do... My current fix is to automatically reboot frigate daily... The way I do this is running a nightly Proxmox lxc backup (which is the container in which my Proxmox instance is running)

Goeste commented 4 months ago

In the end i found it was due to too little gpu memory. i expanded the memory for the vm and and reduced the fps on the cams ow everything is working fine even with the latest go2rtc version

stintov commented 4 months ago

Rebooting makes no difference to me, it like this right after a reboot. I'm running a NUC and have loads of RAM. What I have found to work is remming out these lines in my config:

ffmpeg: hwaccel_args: preset-vaapi retry_interval: 10

Hope this helps someone.

kankadev commented 4 months ago

Hi, have the same problem since several days. I didn't change anything in my hardware, only tried some masks and birdseye options.

I have 6 cameras. Frigate runs for hours then I have this "No frames have been received" for always the same 1-2 cameras (Tapo), Reolink always works.

logs.txt

I can see Fatal Python error: Fatal Python error: Bus error which seems to be a hardware stuck / performance problem? Frigate runs in Proxmox. Had no problems for months until now.

Here's my config:

mqtt:
  host: 192.168.123.71
  user: frigate
  password: xxx
  port: 1883
  topic_prefix: frigate
  client_id: frigate

birdseye:
  enabled: true
  mode: objects
  #width: 2560
  #height: 1440
  inactivity_threshold: 10

ffmpeg:
  hwaccel_args: preset-vaapi
  output_args:
    record: preset-record-generic-audio-aac

detectors:
  coral:
    type: edgetpu
    device: usb

objects:
  track:
  - person
  - car
  - cat
  - dog
  - bicycle
  - motorcycle
  filters:
    person:
      min_area: 400 #5000
      max_area: 100000

record:
  enabled: true
  retain:
    days: 3
    mode: motion
  events:
    pre_capture: 5
    post_capture: 5
    retain:
      default: 30
      mode: active_objects

cameras:
  # Tapo C320WS - outdoor ueber Eingangstuer
  outside_door:
    ffmpeg:
      inputs:
        # High Resolution Stream
      - path: rtsp://super:secret@192.168.123.150/stream1
        roles:
        - record
        # Low Resolution Stream
      - path: rtsp://super:secret@192.168.123.150/stream2
        roles:
        - detect
    detect:
      width: 640
      height: 360
      fps: 15
    motion:
      mask:
      - 232,21,0,26,0,0,228,0

  # Tapo C310 - outdoor corner
  outside_corner:
    ffmpeg:
      inputs:
        # High Resolution Stream
      - path: rtsp://super:secret@192.168.123.151/stream1
        roles:
        - record
        # Low Resolution Stream
      - path: rtsp://super:secret@192.168.123.151/stream2
        roles:
        - detect
    detect:
      width: 2304
      height: 1296
      fps: 15
    motion:
      mask:
      - 0,273,1343,200,2304,665,2304,0,0,0
      - 2304,0,2304,571,851,82,851,0

  # Tapo C310 - outdoor Klima
  outside_klima:
    ffmpeg:
      inputs:
        # High Resolution Stream
      - path: rtsp://super:secret@192.168.123.152/stream1
        roles:
        - record
        # Low Resolution Stream
      - path: rtsp://super:secret@192.168.123.152/stream2
        roles:
        - detect
    detect:
      width: 640
      height: 360
      fps: 15
    motion:
      mask:
      - 640,0,640,82,0,31,0,0

  # Reolink E1 Pro - inside watching door
  inside_door:
    ffmpeg:
      inputs:
        # High Resolution Stream
      - path: rtsp://super:secret@192.168.123.160:554/h264Preview_01_main
        roles:
        - record
        # Low Resolution Stream
      - path: rtsp://super:secret@192.168.123.160:554/h264Preview_01_sub
        roles:
        - detect
    detect:
      width: 640
      height: 360
      fps: 7
    motion:
      mask:
      - 226,26,413,31,410,0,410,0,218,0

  # Reolink E1 Pro - above kitchen door
  inside_room_overview:
    ffmpeg:
      inputs:
        # High Resolution Stream
      - path: rtsp://super:secret@192.168.123.161:554/h264Preview_01_main
        roles:
        - record
        # Low Resolution Stream
      - path: rtsp://super:secret@192.168.123.161:554/h264Preview_01_sub
        roles:
        - detect
    detect:
      width: 640
      height: 360
      fps: 7
    motion:
      mask:
      - 412,0,411,23,218,24,222,0

  # Reolink E1 Pro - inside watching door
  inside_kitchen:
    ffmpeg:
      inputs:
        # High Resolution Stream
      - path: rtsp://super:secret@192.168.123.162:554/h264Preview_01_main
        roles:
        - record
        # Low Resolution Stream
      - path: rtsp://super:secret@192.168.123.162:554/h264Preview_01_sub
        roles:
        - detect
    detect:
      width: 640
      height: 360
      fps: 7
    motion:
      mask:
      - 223,0,222,28,415,27,410,0
NickM-27 commented 4 months ago

Usually means that you just need to increase SHM, otherwise we need to see logs

kankadev commented 4 months ago

Don't know how to increase SHM actually. I googled. Is this the equivalent to swap? I changed the CPU cores from 3 to 4 and the swap from 512 to 1024MB for the Proxmox LXC container now. Restarted Frigate and I immediatly see the same problem again for the same two cameras. 

2024-06-20 17:53:34.575692309  [INFO] Preparing Frigate...
2024-06-20 17:53:34.587122831  [INFO] Starting Frigate...
2024-06-20 17:53:35.760851113  [2024-06-20 17:53:35] frigate.app                    INFO    : Starting Frigate (0.13.2-6476f8a)
2024-06-20 17:53:35.816935451  [2024-06-20 17:53:35] peewee_migrate.logs            INFO    : Starting migrations
2024-06-20 17:53:35.820492565  [2024-06-20 17:53:35] peewee_migrate.logs            INFO    : There is nothing to migrate
2024-06-20 17:53:35.825324071  [2024-06-20 17:53:35] frigate.app                    INFO    : Recording process started: 296
2024-06-20 17:53:35.827674998  [2024-06-20 17:53:35] frigate.app                    INFO    : go2rtc process pid: 89
2024-06-20 17:53:35.851057949  [2024-06-20 17:53:35] detector.coral                 INFO    : Starting detection process: 306
2024-06-20 17:53:38.473631417  [2024-06-20 17:53:35] frigate.app                    INFO    : Output process started: 308
2024-06-20 17:53:38.473635564  [2024-06-20 17:53:35] frigate.app                    INFO    : Camera processor started for outside_door: 337
2024-06-20 17:53:38.473640371  [2024-06-20 17:53:35] frigate.app                    INFO    : Camera processor started for outside_corner: 338
2024-06-20 17:53:38.473642442  [2024-06-20 17:53:35] frigate.app                    INFO    : Camera processor started for outside_klima: 340
2024-06-20 17:53:38.473669026  [2024-06-20 17:53:35] frigate.app                    INFO    : Camera processor started for inside_door: 341
2024-06-20 17:53:38.473671409  [2024-06-20 17:53:35] frigate.app                    INFO    : Camera processor started for inside_room_overview: 344
2024-06-20 17:53:38.473672839  [2024-06-20 17:53:35] frigate.app                    INFO    : Camera processor started for inside_kitchen: 346
2024-06-20 17:53:38.473692454  [2024-06-20 17:53:35] frigate.app                    INFO    : Capture process started for outside_door: 348
2024-06-20 17:53:38.473693934  [2024-06-20 17:53:35] frigate.app                    INFO    : Capture process started for outside_corner: 352
2024-06-20 17:53:38.473695165  [2024-06-20 17:53:35] frigate.app                    INFO    : Capture process started for outside_klima: 356
2024-06-20 17:53:38.473709520  [2024-06-20 17:53:35] frigate.app                    INFO    : Capture process started for inside_door: 361
2024-06-20 17:53:38.473711070  [2024-06-20 17:53:35] frigate.app                    INFO    : Capture process started for inside_room_overview: 365
2024-06-20 17:53:38.473712455  [2024-06-20 17:53:35] frigate.app                    INFO    : Capture process started for inside_kitchen: 372
2024-06-20 17:53:38.478237829  [2024-06-20 17:53:36] frigate.video                  ERROR   : outside_klima: Unable to read frames from ffmpeg process.
2024-06-20 17:53:38.478242345  [2024-06-20 17:53:36] frigate.video                  ERROR   : outside_klima: ffmpeg process is not running. exiting capture thread...
2024-06-20 17:53:38.478243631  [2024-06-20 17:53:35] frigate.detectors.plugins.edgetpu_tfl INFO    : Attempting to load TPU as usb
2024-06-20 17:53:38.480700171  [2024-06-20 17:53:38] frigate.detectors.plugins.edgetpu_tfl INFO    : TPU found
2024-06-20 17:53:51.724712120  Fatal Python error: Bus error
2024-06-20 17:53:51.724715399  
2024-06-20 17:53:51.724722243  Thread 0x000073354cc006c0 (most recent call first):
2024-06-20 17:53:51.724723558    File "/usr/lib/python3.9/threading.py", line 312 in wait
2024-06-20 17:53:51.724724657    File "/usr/lib/python3.9/multiprocessing/queues.py", line 233 in _feed
2024-06-20 17:53:51.724725665    File "/usr/lib/python3.9/threading.py", line 892 in run
2024-06-20 17:53:51.724726714    File "/usr/lib/python3.9/threading.py", line 954 in _bootstrap_inner
2024-06-20 17:53:51.724727696    File "/usr/lib/python3.9/threading.py", line 912 in _bootstrap
2024-06-20 17:53:51.724728416  
2024-06-20 17:53:51.724729325  Thread 0x0000733543a006c0 (most recent call first):
2024-06-20 17:53:51.724730279    File "/opt/frigate/frigate/log.py", line 95 in run
2024-06-20 17:53:51.724731363    File "/usr/lib/python3.9/threading.py", line 954 in _bootstrap_inner
2024-06-20 17:53:51.724732384    File "/usr/lib/python3.9/threading.py", line 912 in _bootstrap
2024-06-20 17:53:51.724733147  
2024-06-20 17:53:51.724734025  Current thread 0x00007335430006c0 (most recent call first):
2024-06-20 17:53:51.724734996    File "/opt/frigate/frigate/video.py", line 117 in capture_frames
2024-06-20 17:53:51.724735858    File "/opt/frigate/frigate/video.py", line 344 in run
2024-06-20 17:53:51.724736866    File "/usr/lib/python3.9/threading.py", line 954 in _bootstrap_inner
2024-06-20 17:53:51.724737869    File "/usr/lib/python3.9/threading.py", line 912 in _bootstrap
2024-06-20 17:53:51.724738650  
2024-06-20 17:53:51.724739507  Thread 0x00007335426006c0 (most recent call first):
2024-06-20 17:53:53.726859891    File "/usr/lib/python3.9/multiprocessing/synchronize.py", line 261 in wait
2024-06-20 17:53:53.726864162    File "/usr/lib/python3.9/multiprocessing/synchronize.py", line 349 in wait
2024-06-20 17:53:53.726865455    File "/opt/frigate/frigate/video.py", line 193 in run
2024-06-20 17:53:53.726866653    File "/usr/lib/python3.9/threading.py", line 954 in _bootstrap_inner
2024-06-20 17:53:53.726867732    File "/usr/lib/python3.9/threading.py", line 912 in _bootstrap
2024-06-20 17:53:53.726868573  
2024-06-20 17:53:53.726869596  Thread 0x00007335416006c0 (most recent call first):
2024-06-20 17:53:53.726870638    File "/opt/frigate/frigate/log.py", line 95 in run
2024-06-20 17:53:53.726871643    File "/usr/lib/python3.9/threading.py", line 954 in _bootstrap_inner
2024-06-20 17:53:53.726872662    File "/usr/lib/python3.9/threading.py", line 912 in _bootstrap
2024-06-20 17:53:53.726873365  
2024-06-20 17:53:53.726874245  Thread 0x0000733569a9a740 (most recent call first):
2024-06-20 17:53:53.726875332    File "/usr/lib/python3.9/threading.py", line 1049 in _wait_for_tstate_lock
2024-06-20 17:53:53.726876282    File "/usr/lib/python3.9/threading.py", line 1033 in join
2024-06-20 17:53:53.726877307    File "/opt/frigate/frigate/video.py", line 380 in capture_camera
2024-06-20 17:53:53.726878381    File "/usr/lib/python3.9/multiprocessing/process.py", line 108 in run
2024-06-20 17:53:53.726879449    File "/usr/lib/python3.9/multiprocessing/process.py", line 315 in _bootstrap
2024-06-20 17:53:53.726880488    File "/usr/lib/python3.9/multiprocessing/popen_fork.py", line 71 in _launch
2024-06-20 17:53:53.726881566    File "/usr/lib/python3.9/multiprocessing/popen_fork.py", line 19 in __init__
2024-06-20 17:53:53.726882534    File "/usr/lib/python3.9/multiprocessing/context.py", line 277 in _Popen
2024-06-20 17:53:53.726883540    File "/usr/lib/python3.9/multiprocessing/context.py", line 224 in _Popen
2024-06-20 17:53:53.726909488    File "/usr/lib/python3.9/multiprocessing/process.py", line 121 in start
2024-06-20 17:53:53.726911242    File "/opt/frigate/frigate/app.py", line 553 in start_camera_capture_processes
2024-06-20 17:53:53.726912204    File "/opt/frigate/frigate/app.py", line 689 in start
2024-06-20 17:53:53.726913335    File "/opt/frigate/frigate/__main__.py", line 17 in <module>
2024-06-20 17:53:53.726914377    File "/usr/lib/python3.9/runpy.py", line 87 in _run_code
2024-06-20 17:53:53.726915398    File "/usr/lib/python3.9/runpy.py", line 197 in _run_module_as_main
2024-06-20 17:53:56.040363775  [2024-06-20 17:53:56] watchdog.outside_klima         ERROR   : Ffmpeg process crashed unexpectedly for outside_klima.
2024-06-20 17:53:56.040937215  [2024-06-20 17:53:56] watchdog.outside_klima         ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2024-06-20 17:53:56.040944533  [2024-06-20 17:53:56] ffmpeg.outside_klima.detect    ERROR   : [tcp @ 0x6112e18342c0] Connection to tcp://192.168.123.152:554?timeout=5000000 failed: Connection refused
2024-06-20 17:53:56.040946294  [2024-06-20 17:53:56] ffmpeg.outside_klima.detect    ERROR   : rtsp://*:*@192.168.123.152/stream2: Connection refused
2024-06-20 17:53:56.054521572  [2024-06-20 17:53:56] ffmpeg.outside_klima.record    ERROR   : [tcp @ 0x5f166e8cd400] Connection to tcp://192.168.123.152:554?timeout=5000000 failed: Connection refused
2024-06-20 17:53:56.054525689  [2024-06-20 17:53:56] ffmpeg.outside_klima.record    ERROR   : rtsp://*:*@192.168.123.152/stream1: Connection refused
2024-06-20 17:53:56.054527291  [2024-06-20 17:53:56] watchdog.outside_klima         INFO    : Terminating the existing ffmpeg process...
2024-06-20 17:53:56.054528718  [2024-06-20 17:53:56] watchdog.outside_klima         INFO    : Waiting for ffmpeg to exit gracefully...
2024-06-20 17:53:56.056477878  [2024-06-20 17:53:56] frigate.video                  ERROR   : outside_klima: Unable to read frames from ffmpeg process.
2024-06-20 17:53:56.056693549  [2024-06-20 17:53:56] frigate.video                  ERROR   : outside_klima: ffmpeg process is not running. exiting capture thread...
2024-06-20 17:54:06.060254823  [2024-06-20 17:54:06] watchdog.outside_klima         ERROR   : Ffmpeg process crashed unexpectedly for outside_klima.
2024-06-20 17:54:06.060414623  [2024-06-20 17:54:06] watchdog.outside_klima         ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2024-06-20 17:54:06.060417599  [2024-06-20 17:54:06] ffmpeg.outside_klima.detect    ERROR   : [tcp @ 0x629718e022c0] Connection to tcp://192.168.123.152:554?timeout=5000000 failed: Connection refused
2024-06-20 17:54:06.060476389  [2024-06-20 17:54:06] ffmpeg.outside_klima.detect    ERROR   : rtsp://*:*@192.168.123.152/stream2: Connection refused
2024-06-20 17:54:06.072509959  [2024-06-20 17:54:06] ffmpeg.outside_klima.record    ERROR   : [tcp @ 0x6124896f8400] Connection to tcp://192.168.123.152:554?timeout=5000000 failed: Connection refused
2024-06-20 17:54:06.072899638  [2024-06-20 17:54:06] ffmpeg.outside_klima.record    ERROR   : rtsp://*:*@192.168.123.152/stream1: Connection refused
2024-06-20 17:54:06.072902622  [2024-06-20 17:54:06] watchdog.outside_klima         INFO    : Terminating the existing ffmpeg process...
2024-06-20 17:54:06.072904070  [2024-06-20 17:54:06] watchdog.outside_klima         INFO    : Waiting for ffmpeg to exit gracefully...
2024-06-20 17:54:06.078971007  [2024-06-20 17:54:06] frigate.video                  ERROR   : outside_klima: Unable to read frames from ffmpeg process.
2024-06-20 17:54:06.079076376  [2024-06-20 17:54:06] frigate.video                  ERROR   : outside_klima: ffmpeg process is not running. exiting capture thread...
2024-06-20 17:54:16.088513219  [2024-06-20 17:54:16] watchdog.outside_klima         ERROR   : Ffmpeg process crashed unexpectedly for outside_klima.
2024-06-20 17:54:16.088635380  [2024-06-20 17:54:16] watchdog.outside_klima         ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2024-06-20 17:54:16.088648859  [2024-06-20 17:54:16] ffmpeg.outside_klima.detect    ERROR   : [tcp @ 0x5fb41b1b22c0] Connection to tcp://192.168.123.152:554?timeout=5000000 failed: Connection refused
2024-06-20 17:54:16.088694107  [2024-06-20 17:54:16] ffmpeg.outside_klima.detect    ERROR   : rtsp://*:*@192.168.123.152/stream2: Connection refused
2024-06-20 17:54:16.094562752  [2024-06-20 17:54:16] ffmpeg.outside_klima.record    ERROR   : [tcp @ 0x5c575737a400] Connection to tcp://192.168.123.152:554?timeout=5000000 failed: Connection refused
2024-06-20 17:54:16.094733780  [2024-06-20 17:54:16] ffmpeg.outside_klima.record    ERROR   : rtsp://*:*@192.168.123.152/stream1: Connection refused
2024-06-20 17:54:16.094737053  [2024-06-20 17:54:16] watchdog.outside_klima         INFO    : Terminating the existing ffmpeg process...
2024-06-20 17:54:16.094823303  [2024-06-20 17:54:16] watchdog.outside_klima         INFO    : Waiting for ffmpeg to exit gracefully...
2024-06-20 17:54:16.101102715  [2024-06-20 17:54:16] frigate.video                  ERROR   : outside_klima: Unable to read frames from ffmpeg process.
2024-06-20 17:54:16.101584449  [2024-06-20 17:54:16] frigate.video                  ERROR   : outside_klima: ffmpeg process is not running. exiting capture thread...
2024-06-20 17:54:26.105365641  [2024-06-20 17:54:26] watchdog.outside_klima         ERROR   : Ffmpeg process crashed unexpectedly for outside_klima.
2024-06-20 17:54:26.105523851  [2024-06-20 17:54:26] watchdog.outside_klima         ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2024-06-20 17:54:26.105527336  [2024-06-20 17:54:26] ffmpeg.outside_klima.detect    ERROR   : [tcp @ 0x5cc9d11742c0] Connection to tcp://192.168.123.152:554?timeout=5000000 failed: Connection refused
2024-06-20 17:54:26.105563426  [2024-06-20 17:54:26] ffmpeg.outside_klima.detect    ERROR   : rtsp://*:*@192.168.123.152/stream2: Connection refused
2024-06-20 17:54:26.117004865  [2024-06-20 17:54:26] ffmpeg.outside_klima.record    ERROR   : [tcp @ 0x5b12e92f5400] Connection to tcp://192.168.123.152:554?timeout=5000000 failed: Connection refused
2024-06-20 17:54:26.117226934  [2024-06-20 17:54:26] ffmpeg.outside_klima.record    ERROR   : rtsp://*:*@192.168.123.152/stream1: Connection refused
2024-06-20 17:54:26.117230354  [2024-06-20 17:54:26] watchdog.outside_klima         INFO    : Terminating the existing ffmpeg process...
2024-06-20 17:54:26.117231940  [2024-06-20 17:54:26] watchdog.outside_klima         INFO    : Waiting for ffmpeg to exit gracefully...
2024-06-20 17:54:26.123611632  [2024-06-20 17:54:26] frigate.video                  ERROR   : outside_klima: Unable to read frames from ffmpeg process.
2024-06-20 17:54:26.123738613  [2024-06-20 17:54:26] frigate.video                  ERROR   : outside_klima: ffmpeg process is not running. exiting capture thread...
2024-06-20 17:54:36.127895616  [2024-06-20 17:54:36] watchdog.outside_klima         ERROR   : Ffmpeg process crashed unexpectedly for outside_klima.
2024-06-20 17:54:36.127985094  [2024-06-20 17:54:36] watchdog.outside_klima         ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2024-06-20 17:54:36.128095179  [2024-06-20 17:54:36] ffmpeg.outside_klima.detect    ERROR   : [tcp @ 0x5ee627c882c0] Connection to tcp://192.168.123.152:554?timeout=5000000 failed: Connection refused
2024-06-20 17:54:36.128153559  [2024-06-20 17:54:36] ffmpeg.outside_klima.detect    ERROR   : rtsp://*:*@192.168.123.152/stream2: Connection refused
2024-06-20 17:54:36.139819512  [2024-06-20 17:54:36] ffmpeg.outside_klima.record    ERROR   : [tcp @ 0x5a527195b400] Connection to tcp://192.168.123.152:554?timeout=5000000 failed: Connection refused
2024-06-20 17:54:36.140047442  [2024-06-20 17:54:36] ffmpeg.outside_klima.record    ERROR   : rtsp://*:*@192.168.123.152/stream1: Connection refused
2024-06-20 17:54:36.140050783  [2024-06-20 17:54:36] watchdog.outside_klima         INFO    : Terminating the existing ffmpeg process...
2024-06-20 17:54:36.140052265  [2024-06-20 17:54:36] watchdog.outside_klima         INFO    : Waiting for ffmpeg to exit gracefully...
2024-06-20 17:54:36.148955148  [2024-06-20 17:54:36] frigate.video                  ERROR   : outside_klima: Unable to read frames from ffmpeg process.
2024-06-20 17:54:36.148959148  [2024-06-20 17:54:36] frigate.video                  ERROR   : outside_klima: ffmpeg process is not running. exiting capture thread...
2024-06-20 17:54:46.152511750  [2024-06-20 17:54:46] watchdog.outside_klima         ERROR   : Ffmpeg process crashed unexpectedly for outside_klima.
2024-06-20 17:54:46.152585453  [2024-06-20 17:54:46] watchdog.outside_klima         ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2024-06-20 17:54:46.152646656  [2024-06-20 17:54:46] ffmpeg.outside_klima.detect    ERROR   : [tcp @ 0x5f8175ffd2c0] Connection to tcp://192.168.123.152:554?timeout=5000000 failed: Connection refused
2024-06-20 17:54:46.152733264  [2024-06-20 17:54:46] ffmpeg.outside_klima.detect    ERROR   : rtsp://*:*@192.168.123.152/stream2: Connection refused
2024-06-20 17:54:46.164278850  [2024-06-20 17:54:46] ffmpeg.outside_klima.record    ERROR   : [tcp @ 0x63ba1becd400] Connection to tcp://192.168.123.152:554?timeout=5000000 failed: Connection refused
2024-06-20 17:54:46.164474007  [2024-06-20 17:54:46] ffmpeg.outside_klima.record    ERROR   : rtsp://*:*@192.168.123.152/stream1: Connection refused
2024-06-20 17:54:46.164477158  [2024-06-20 17:54:46] watchdog.outside_klima         INFO    : Terminating the existing ffmpeg process...
2024-06-20 17:54:46.165578294  [2024-06-20 17:54:46] watchdog.outside_klima         INFO    : Waiting for ffmpeg to exit gracefully...
2024-06-20 17:54:46.171694990  [2024-06-20 17:54:46] frigate.video                  ERROR   : outside_klima: Unable to read frames from ffmpeg process.
2024-06-20 17:54:46.172164585  [2024-06-20 17:54:46] frigate.video                  ERROR   : outside_klima: ffmpeg process is not running. exiting capture thread...
2024-06-20 17:54:56.175925974  [2024-06-20 17:54:56] watchdog.outside_klima         ERROR   : Ffmpeg process crashed unexpectedly for outside_klima.
2024-06-20 17:54:56.181152875  [2024-06-20 17:54:56] watchdog.outside_klima         ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2024-06-20 17:54:56.181230390  [2024-06-20 17:54:56] ffmpeg.outside_klima.detect    ERROR   : [tcp @ 0x5a94cbd7a2c0] Connection to tcp://192.168.123.152:554?timeout=5000000 failed: Connection refused
2024-06-20 17:54:56.181297784  [2024-06-20 17:54:56] ffmpeg.outside_klima.detect    ERROR   : rtsp://*:*@192.168.123.152/stream2: Connection refused
2024-06-20 17:54:56.196026120  [2024-06-20 17:54:56] ffmpeg.outside_klima.record    ERROR   : [tcp @ 0x62b6f06ca400] Connection to tcp://192.168.123.152:554?timeout=5000000 failed: Connection refused
2024-06-20 17:54:56.196116088  [2024-06-20 17:54:56] ffmpeg.outside_klima.record    ERROR   : rtsp://*:*@192.168.123.152/stream1: Connection refused
2024-06-20 17:54:56.196226493  [2024-06-20 17:54:56] watchdog.outside_klima         INFO    : Terminating the existing ffmpeg process...
2024-06-20 17:54:56.196343332  [2024-06-20 17:54:56] watchdog.outside_klima         INFO    : Waiting for ffmpeg to exit gracefully...
2024-06-20 17:55:00.839168594  [2024-06-20 17:55:00] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for outside_corner. Keeping the 6 most recent segments out of 7 and discarding the rest...
2024-06-20 17:55:01.197892636  [2024-06-20 17:55:01] frigate.video                  ERROR   : outside_klima: Unable to read frames from ffmpeg process.
2024-06-20 17:55:01.198002567  [2024-06-20 17:55:01] frigate.video                  ERROR   : outside_klima: ffmpeg process is not running. exiting capture thread...
2024-06-20 17:55:06.202616582  [2024-06-20 17:55:06] watchdog.outside_klima         ERROR   : Ffmpeg process crashed unexpectedly for outside_klima.
2024-06-20 17:55:06.202770544  [2024-06-20 17:55:06] watchdog.outside_klima         ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2024-06-20 17:55:06.202773946  [2024-06-20 17:55:06] ffmpeg.outside_klima.detect    ERROR   : [tcp @ 0x63bf5580f2c0] Connection to tcp://192.168.123.152:554?timeout=5000000 failed: Connection timed out
2024-06-20 17:55:06.203473567  [2024-06-20 17:55:06] ffmpeg.outside_klima.detect    ERROR   : rtsp://*:*@192.168.123.152/stream2: Connection timed out
2024-06-20 17:55:06.217898295  [2024-06-20 17:55:06] ffmpeg.outside_klima.record    ERROR   : [tcp @ 0x6552ae48f400] Connection to tcp://192.168.123.152:554?timeout=5000000 failed: Connection timed out
2024-06-20 17:55:06.218237537  [2024-06-20 17:55:06] ffmpeg.outside_klima.record    ERROR   : rtsp://*:*@192.168.123.152/stream1: Connection timed out
2024-06-20 17:55:06.218247241  [2024-06-20 17:55:06] watchdog.outside_klima         INFO    : Terminating the existing ffmpeg process...
2024-06-20 17:55:06.218248736  [2024-06-20 17:55:06] watchdog.outside_klima         INFO    : Waiting for ffmpeg to exit gracefully...
2024-06-20 17:55:09.332414524  [2024-06-20 17:55:09] frigate.video                  ERROR   : outside_klima: Unable to read frames from ffmpeg process.
2024-06-20 17:55:09.332520929  [2024-06-20 17:55:09] frigate.video                  ERROR   : outside_klima: ffmpeg process is not running. exiting capture thread...
2024-06-20 17:55:10.840229014  [2024-06-20 17:55:10] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for outside_corner. Keeping the 6 most recent segments out of 7 and discarding the rest...
2024-06-20 17:55:16.229051063  [2024-06-20 17:55:16] watchdog.outside_klima         ERROR   : Ffmpeg process crashed unexpectedly for outside_klima.
2024-06-20 17:55:16.229168177  [2024-06-20 17:55:16] watchdog.outside_klima         ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2024-06-20 17:55:16.229265130  [2024-06-20 17:55:16] ffmpeg.outside_klima.detect    ERROR   : [tcp @ 0x62967d56f2c0] Connection to tcp://192.168.123.152:554?timeout=5000000 failed: No route to host
2024-06-20 17:55:16.229294529  [2024-06-20 17:55:16] ffmpeg.outside_klima.detect    ERROR   : rtsp://*:*@192.168.123.152/stream2: No route to host
2024-06-20 17:55:16.240217435  [2024-06-20 17:55:16] ffmpeg.outside_klima.record    ERROR   : [tcp @ 0x61ef52593400] Connection to tcp://192.168.123.152:554?timeout=5000000 failed: No route to host
2024-06-20 17:55:16.245689344  [2024-06-20 17:55:16] ffmpeg.outside_klima.record    ERROR   : rtsp://*:*@192.168.123.152/stream1: No route to host
2024-06-20 17:55:16.246010340  [2024-06-20 17:55:16] watchdog.outside_klima         INFO    : Terminating the existing ffmpeg process...
2024-06-20 17:55:16.246561919  [2024-06-20 17:55:16] watchdog.outside_klima         INFO    : Waiting for ffmpeg to exit gracefully...
2024-06-20 17:55:16.246672119  [2024-06-20 17:55:16] frigate.video                  ERROR   : outside_klima: Unable to read frames from ffmpeg process.
2024-06-20 17:55:16.247138792  [2024-06-20 17:55:16] frigate.video                  ERROR   : outside_klima: ffmpeg process is not running. exiting capture thread...
2024-06-20 17:55:20.842537658  [2024-06-20 17:55:20] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for outside_corner. Keeping the 6 most recent segments out of 7 and discarding the rest...
2024-06-20 17:55:26.251011789  [2024-06-20 17:55:26] watchdog.outside_klima         ERROR   : Ffmpeg process crashed unexpectedly for outside_klima.
2024-06-20 17:55:26.251086512  [2024-06-20 17:55:26] watchdog.outside_klima         ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2024-06-20 17:55:26.251219174  [2024-06-20 17:55:26] ffmpeg.outside_klima.detect    ERROR   : [tcp @ 0x61b4e56f82c0] Connection to tcp://192.168.123.152:554?timeout=5000000 failed: Connection refused
2024-06-20 17:55:26.251222090  [2024-06-20 17:55:26] ffmpeg.outside_klima.detect    ERROR   : rtsp://*:*@192.168.123.152/stream2: Connection refused
2024-06-20 17:55:26.262843842  [2024-06-20 17:55:26] ffmpeg.outside_klima.record    ERROR   : [tcp @ 0x5947e58f5400] Connection to tcp://192.168.123.152:554?timeout=5000000 failed: Connection refused
2024-06-20 17:55:26.268292518  [2024-06-20 17:55:26] ffmpeg.outside_klima.record    ERROR   : rtsp://*:*@192.168.123.152/stream1: Connection refused
2024-06-20 17:55:26.268442389  [2024-06-20 17:55:26] watchdog.outside_klima         INFO    : Terminating the existing ffmpeg process...
2024-06-20 17:55:26.268546915  [2024-06-20 17:55:26] watchdog.outside_klima         INFO    : Waiting for ffmpeg to exit gracefully...
2024-06-20 17:55:26.269042479  [2024-06-20 17:55:26] frigate.video                  ERROR   : outside_klima: Unable to read frames from ffmpeg process.
2024-06-20 17:55:26.269269459  [2024-06-20 17:55:26] frigate.video                  ERROR   : outside_klima: ffmpeg process is not running. exiting capture thread...
2024-06-20 17:55:30.839936890  [2024-06-20 17:55:30] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for outside_corner. Keeping the 6 most recent segments out of 7 and discarding the rest...
2024-06-20 17:55:36.273334116  [2024-06-20 17:55:36] watchdog.outside_klima         ERROR   : Ffmpeg process crashed unexpectedly for outside_klima.
2024-06-20 17:55:36.273400161  [2024-06-20 17:55:36] watchdog.outside_klima         ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2024-06-20 17:55:36.273532638  [2024-06-20 17:55:36] ffmpeg.outside_klima.detect    ERROR   : [tcp @ 0x63d200e772c0] Connection to tcp://192.168.123.152:554?timeout=5000000 failed: Connection refused
2024-06-20 17:55:36.273535507  [2024-06-20 17:55:36] ffmpeg.outside_klima.detect    ERROR   : rtsp://*:*@192.168.123.152/stream2: Connection refused
2024-06-20 17:55:36.284850783  [2024-06-20 17:55:36] ffmpeg.outside_klima.record    ERROR   : [tcp @ 0x5da8ac6c0400] Connection to tcp://192.168.123.152:554?timeout=5000000 failed: Connection refused
2024-06-20 17:55:36.285075538  [2024-06-20 17:55:36] ffmpeg.outside_klima.record    ERROR   : rtsp://*:*@192.168.123.152/stream1: Connection refused
2024-06-20 17:55:36.285079310  [2024-06-20 17:55:36] watchdog.outside_klima         INFO    : Terminating the existing ffmpeg process...
2024-06-20 17:55:36.285080839  [2024-06-20 17:55:36] watchdog.outside_klima         INFO    : Waiting for ffmpeg to exit gracefully...
2024-06-20 17:55:36.291739249  [2024-06-20 17:55:36] frigate.video                  ERROR   : outside_klima: Unable to read frames from ffmpeg process.
2024-06-20 17:55:36.291744011  [2024-06-20 17:55:36] frigate.video                  ERROR   : outside_klima: ffmpeg process is not running. exiting capture thread...
2024-06-20 17:55:40.840544485  [2024-06-20 17:55:40] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for outside_corner. Keeping the 6 most recent segments out of 7 and discarding the rest...
2024-06-20 17:55:46.296315749  [2024-06-20 17:55:46] watchdog.outside_klima         ERROR   : Ffmpeg process crashed unexpectedly for outside_klima.
2024-06-20 17:55:46.301769476  [2024-06-20 17:55:46] watchdog.outside_klima         ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2024-06-20 17:55:46.301908571  [2024-06-20 17:55:46] ffmpeg.outside_klima.detect    ERROR   : [tcp @ 0x6395449782c0] Connection to tcp://192.168.123.152:554?timeout=5000000 failed: Connection refused
2024-06-20 17:55:46.301911990  [2024-06-20 17:55:46] ffmpeg.outside_klima.detect    ERROR   : rtsp://*:*@192.168.123.152/stream2: Connection refused
2024-06-20 17:55:46.307910510  [2024-06-20 17:55:46] ffmpeg.outside_klima.record    ERROR   : [tcp @ 0x5653f82e8400] Connection to tcp://192.168.123.152:554?timeout=5000000 failed: Connection refused
2024-06-20 17:55:46.308147333  [2024-06-20 17:55:46] ffmpeg.outside_klima.record    ERROR   : rtsp://*:*@192.168.123.152/stream1: Connection refused
2024-06-20 17:55:46.308150702  [2024-06-20 17:55:46] watchdog.outside_klima         INFO    : Terminating the existing ffmpeg process...
2024-06-20 17:55:46.308152200  [2024-06-20 17:55:46] watchdog.outside_klima         INFO    : Waiting for ffmpeg to exit gracefully...