search

blakeblackshear / frigate

NVR with realtime local object detection for IP cameras
https://frigate.video
MIT License
19.2k stars 1.76k forks source link

[Support]: Camera feeds keep crashing #7642

Closed Maxxell closed 1 year ago

Maxxell commented 1 year ago

Describe the problem you are having

I have had frigate for a while. It started intermittently losing connection to my cameras. I upgraded to latest version, which included upgrading all my HomeAssistant Notifications. Now that I am on the latest version, I am still having intermittent issues where one (or more) of my cameras start showing "no frames have been received, check error logs"

Rightnow its affecting only 1 of my cameras. But it does sometimes affect others. Logs below reflect current state, where only the "Frontdoor" camera is offline.

Version

0.12.1-367D724

Frigate config file

mqtt:
  host: 192.168.4.81
  port: 1883
  user: 
  password: 
detectors:
  coral:
    type: edgetpu
    device: usb
birdseye: 
  enabled: True
  mode: continuous
cameras:
  REDACTED:
    ffmpeg:
      hwaccel_args: -c:v h264_cuvid
      inputs:
        - path: rtsp://admin:REDACTED@192.168.4.252:554/cam/realmonitor?channel=1&subtype=0
    record: 
      enabled: True
      retain: 
        days: 2
      events:
        retain:
          default: 3
          mode: motion
    snapshots:
      enabled: True
    rtmp: 
      enabled: False
    detect:
      width: 640
      height: 480
  Backyard:
    ffmpeg:
      hwaccel_args: -c:v h264_cuvid
      inputs:
        - path: rtsp://admin:REDACTED!@192.168.7.180:554/cam/realmonitor?channel=1&subtype=0
    record: 
      enabled: True
      retain: 
        days: 2
      events:
        retain:
          default: 4
          mode: motion
    snapshots:
      enabled: True
    rtmp: 
      enabled: False
    detect:
      width: 640
      height: 480
    motion:
      mask:
        - 0,321,21,293,32,242,85,221,124,197,128,156,144,140,165,137,157,93,120,63,137,38,149,0,0,0
        - 225,0,358,0,350,53,325,62,307,83,267,72,241,66,231,47,229,0
    objects:
      track:
        - person
        - dog      
  REDACTED:
    ffmpeg:
      hwaccel_args: -c:v h264_cuvid
      inputs:
        - path: rtsp://admin:REDACTED@192.168.5.5:554/cam/realmonitor?channel=1&subtype=0
    record: 
      enabled: True
      retain: 
        days: 2
      events:
        retain:
          default: 3
          mode: motion
    snapshots:
      enabled: True
    rtmp: 
      enabled: False
    motion:
      mask:
        - 0,0,0,379,63,176,74,99,122,0
    detect:
      width: 640
      height: 480
  Frontdoor:
    ffmpeg:
      hwaccel_args: -c:v h264_cuvid
      inputs:
        - path: rtsp://admin:REDACTED@192.168.7.166:554/cam/realmonitor?channel=1&subtype=0
    record: 
      enabled: True
      retain: 
        days: 2
      events:
        retain:
          default: 4
          mode: motion
    snapshots:
      enabled: True
    rtmp: 
      enabled: False
    detect:
      width: 720
      height: 576
    objects:
      track:
        - person
        - dog
  Garage:
    ffmpeg:
      hwaccel_args: -c:v h264_cuvid
      inputs:
        - path: rtsp://admin:REDACTED@192.168.4.202:554/cam/realmonitor?channel=1&subtype=0
    record: 
      enabled: True
      retain: 
        days: 2
      events:
        retain:
          default: 4
          mode: motion
    snapshots:
      enabled: True
    rtmp: 
      enabled: False
    detect:
      width: 1280
      height: 720

Relevant log output

Here's what looks relevant to me.

2023-08-30 16:51:04.742161520  [2023-08-30 16:51:04] frigate.video                  ERROR   : Frontdoor: Unable to read frames from ffmpeg process.
2023-08-30 16:51:04.742491216  [2023-08-30 16:51:04] frigate.video                  ERROR   : Frontdoor: ffmpeg process is not running. exiting capture thread...
2023-08-30 16:51:09.726873377  [2023-08-30 16:51:09] watchdog.Frontdoor             ERROR   : Ffmpeg process crashed unexpectedly for Frontdoor.
2023-08-30 16:51:09.727454843  [2023-08-30 16:51:09] watchdog.Frontdoor             ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2023-08-30 16:51:09.727714029  [2023-08-30 16:51:09] ffmpeg.Frontdoor.detect        ERROR   : [tcp @ 0x5597d305d540] Connection to tcp://192.168.7.166:554?timeout=5000000 failed: Connection timed out
2023-08-30 16:51:09.728317532  [2023-08-30 16:51:09] ffmpeg.Frontdoor.detect        ERROR   : rtsp://*:*@192.168.7.166:554/cam/realmonitor?channel=1&subtype=0: Connection timed out
2023-08-30 16:51:12.809836361  [2023-08-30 16:51:12] frigate.video                  ERROR   : Frontdoor: Unable to read frames from ffmpeg process.
2023-08-30 16:51:12.810162091  [2023-08-30 16:51:12] frigate.video                  ERROR   : Frontdoor: ffmpeg process is not running. exiting capture thread...
2023-08-30 16:51:19.755044528  [2023-08-30 16:51:19] watchdog.Frontdoor             ERROR   : Ffmpeg process crashed unexpectedly for Frontdoor.
2023-08-30 16:51:19.755337704  [2023-08-30 16:51:19] watchdog.Frontdoor             ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2023-08-30 16:51:19.755588413  [2023-08-30 16:51:19] ffmpeg.Frontdoor.detect        ERROR   : [tcp @ 0x563a1d090540] Connection to tcp://192.168.7.166:554?timeout=5000000 failed: No route to host
2023-08-30 16:51:19.755815259  [2023-08-30 16:51:19] ffmpeg.Frontdoor.detect        ERROR   : rtsp://*:*@192.168.7.166:554/cam/realmonitor?channel=1&subtype=0: No route to host
2023-08-30 16:51:24.783033304  [2023-08-30 16:51:24] frigate.video                  ERROR   : Frontdoor: Unable to read frames from ffmpeg process.
2023-08-30 16:51:24.783146871  [2023-08-30 16:51:24] frigate.video                  ERROR   : Frontdoor: ffmpeg process is not running. exiting capture thread...
2023-08-30 16:51:29.765519222  [2023-08-30 16:51:29] watchdog.Frontdoor             ERROR   : Ffmpeg process crashed unexpectedly for Frontdoor.
2023-08-30 16:51:29.765769357  [2023-08-30 16:51:29] watchdog.Frontdoor             ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2023-08-30 16:51:29.765955917  [2023-08-30 16:51:29] ffmpeg.Frontdoor.detect        ERROR   : [tcp @ 0x55c5ad61a540] Connection to tcp://192.168.7.166:554?timeout=5000000 failed: Connection timed out
2023-08-30 16:51:29.766172144  [2023-08-30 16:51:29] ffmpeg.Frontdoor.detect        ERROR   : rtsp://*:*@192.168.7.166:554/cam/realmonitor?channel=1&subtype=0: Connection timed out
2023-08-30 16:51:31.671919661  [2023-08-30 16:51:31] frigate.video                  ERROR   : Frontdoor: Unable to read frames from ffmpeg process.
2023-08-30 16:51:31.672140085  [2023-08-30 16:51:31] frigate.video                  ERROR   : Frontdoor: ffmpeg process is not running. exiting capture thread...
2023-08-30 16:51:39.774300671  [2023-08-30 16:51:39] watchdog.Frontdoor             ERROR   : Ffmpeg process crashed unexpectedly for Frontdoor.
2023-08-30 16:51:39.774324526  [2023-08-30 16:51:39] watchdog.Frontdoor             ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2023-08-30 16:51:39.774390231  [2023-08-30 16:51:39] ffmpeg.Frontdoor.detect        ERROR   : [AVHWDeviceContext @ 0x556aa3253e80] cu->cuInit(0) failed -> CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected
2023-08-30 16:51:39.774403269  [2023-08-30 16:51:39] ffmpeg.Frontdoor.detect        ERROR   : Error while opening decoder for input stream #0:0 : Generic error in an external library
2023-08-30 16:51:41.487462987  [2023-08-30 16:51:41] frigate.video                  ERROR   : Frontdoor: Unable to read frames from ffmpeg process.
2023-08-30 16:51:41.487813125  [2023-08-30 16:51:41] frigate.video                  ERROR   : Frontdoor: ffmpeg process is not running. exiting capture thread...
2023-08-30 16:51:49.782399265  [2023-08-30 16:51:49] watchdog.Frontdoor             ERROR   : Ffmpeg process crashed unexpectedly for Frontdoor.

FFprobe output from your camera

For the camera that's currently offline:

[{"return_code":0,"stderr":"","stdout":{"programs":[],"streams":[{"avg_frame_rate":"15/1","codec_long_name":"H.264/AVC/MPEG-4AVC/MPEG-4part10","height":1920,"width":2560},{"avg_frame_rate":"0/0","bit_rate":"128000","codec_long_name":"PCMmu-law/G.711mu-law"}]}}]

Frigate stats

{
    "Backyard": {
        "camera_fps": 5,
        "capture_pid": 739,
        "detection_enabled": 1,
        "detection_fps": 0.5,
        "ffmpeg_pid": 378677,
        "pid": 729,
        "process_fps": 5.1,
        "skipped_fps": 0
    },
    "Frontdoor": {
        "camera_fps": 0,
        "capture_pid": 747,
        "detection_enabled": 1,
        "detection_fps": 0,
        "ffmpeg_pid": 690852,
        "pid": 732,
        "process_fps": 6.1,
        "skipped_fps": 0
    },
    "Garage": {
        "camera_fps": 5,
        "capture_pid": 755,
        "detection_enabled": 1,
        "detection_fps": 0,
        "ffmpeg_pid": 378639,
        "pid": 734,
        "process_fps": 5,
        "skipped_fps": 0
    },
    "REDACTED": {
        "camera_fps": 5.1,
        "capture_pid": 735,
        "detection_enabled": 1,
        "detection_fps": 0,
        "ffmpeg_pid": 378640,
        "pid": 727,
        "process_fps": 5.1,
        "skipped_fps": 0
    },
    "REDACTED": {
        "camera_fps": 5.1,
        "capture_pid": 744,
        "detection_enabled": 1,
        "detection_fps": 0,
        "ffmpeg_pid": 378641,
        "pid": 731,
        "process_fps": 5.1,
        "skipped_fps": 0
    },
    "cpu_usages": {
        "1": {
            "cpu": "0.0",
            "mem": "0.0"
        },
        "15": {
            "cpu": "0.0",
            "mem": "0.0"
        },
        "17": {
            "cpu": "0.0",
            "mem": "0.0"
        },
        "24": {
            "cpu": "0.0",
            "mem": "0.0"
        },
        "25": {
            "cpu": "0.0",
            "mem": "0.0"
        },
        "26": {
            "cpu": "0.0",
            "mem": "0.0"
        },
        "27": {
            "cpu": "0.0",
            "mem": "0.0"
        },
        "28": {
            "cpu": "0.0",
            "mem": "0.0"
        },
        "29": {
            "cpu": "0.0",
            "mem": "0.0"
        },
        "30": {
            "cpu": "0.0",
            "mem": "0.0"
        },
        "31": {
            "cpu": "0.0",
            "mem": "0.0"
        },
        "32": {
            "cpu": "0.0",
            "mem": "0.0"
        },
        "41": {
            "cpu": "0.0",
            "mem": "0.0"
        },
        "42": {
            "cpu": "0.0",
            "mem": "0.0"
        },
        "79": {
            "cpu": "0.0",
            "mem": "0.0"
        },
        "80": {
            "cpu": "0.0",
            "mem": "0.0"
        },
        "82": {
            "cpu": "0.0",
            "mem": "0.0"
        },
        "89": {
            "cpu": "0.0",
            "mem": "0.1"
        },
        "97": {
            "cpu": "0.0",
            "mem": "0.0"
        },
        "98": {
            "cpu": "2.0",
            "mem": "1.2"
        },
        "112": {
            "cpu": "0.0",
            "mem": "0.0"
        },
        "134": {
            "cpu": "0.0",
            "mem": "0.0"
        },
        "135": {
            "cpu": "0.0",
            "mem": "0.0"
        },
        "136": {
            "cpu": "0.0",
            "mem": "0.0"
        },
        "137": {
            "cpu": "0.0",
            "mem": "0.0"
        },
        "146": {
            "cpu": "0.0",
            "mem": "0.0"
        },
        "160": {
            "cpu": "0.0",
            "mem": "0.0"
        },
        "191": {
            "cpu": "0.0",
            "mem": "0.0"
        },
        "239": {
            "cpu": "0.0",
            "mem": "0.0"
        },
        "281": {
            "cpu": "0.0",
            "mem": "0.0"
        },
        "320": {
            "cpu": "0.0",
            "mem": "0.0"
        },
        "356": {
            "cpu": "0.0",
            "mem": "0.0"
        },
        "397": {
            "cpu": "0.0",
            "mem": "0.0"
        },
        "430": {
            "cpu": "0.0",
            "mem": "0.0"
        },
        "468": {
            "cpu": "0.0",
            "mem": "0.0"
        },
        "502": {
            "cpu": "0.0",
            "mem": "0.0"
        },
        "532": {
            "cpu": "0.0",
            "mem": "0.0"
        },
        "715": {
            "cpu": "0.0",
            "mem": "0.2"
        },
        "721": {
            "cpu": "0.7",
            "mem": "0.0"
        },
        "724": {
            "cpu": "1.0",
            "mem": "0.2"
        },
        "727": {
            "cpu": "0.7",
            "mem": "0.2"
        },
        "728": {
            "cpu": "0.0",
            "mem": "0.1"
        },
        "729": {
            "cpu": "0.7",
            "mem": "0.2"
        },
        "730": {
            "cpu": "0.0",
            "mem": "0.1"
        },
        "731": {
            "cpu": "0.7",
            "mem": "0.2"
        },
        "732": {
            "cpu": "0.0",
            "mem": "0.2"
        },
        "733": {
            "cpu": "0.0",
            "mem": "0.1"
        },
        "734": {
            "cpu": "1.0",
            "mem": "0.3"
        },
        "735": {
            "cpu": "1.0",
            "mem": "0.2"
        },
        "736": {
            "cpu": "0.0",
            "mem": "0.1"
        },
        "739": {
            "cpu": "0.7",
            "mem": "0.2"
        },
        "742": {
            "cpu": "0.0",
            "mem": "0.1"
        },
        "744": {
            "cpu": "1.0",
            "mem": "0.2"
        },
        "747": {
            "cpu": "0.7",
            "mem": "0.2"
        },
        "750": {
            "cpu": "0.0",
            "mem": "0.1"
        },
        "755": {
            "cpu": "1.7",
            "mem": "0.2"
        },
        "378639": {
            "cpu": "22.0",
            "mem": "0.5"
        },
        "378640": {
            "cpu": "18.3",
            "mem": "0.4"
        },
        "378641": {
            "cpu": "18.7",
            "mem": "0.4"
        },
        "378677": {
            "cpu": "24.7",
            "mem": "0.4"
        },
        "560792": {
            "cpu": "0.0",
            "mem": "0.8"
        },
        "690832": {
            "cpu": "0.0",
            "mem": "0.0"
        },
        "690870": {
            "cpu": "0.0",
            "mem": "0.0"
        },
        "%Cpu(s):": {
            "cpu": "id,",
            "mem": "0.0"
        },
        "MiB": {
            "cpu": "28995.2",
            "mem": "avail"
        },
        "PID": {
            "cpu": "%CPU",
            "mem": "%MEM"
        },
        "Tasks:": {
            "cpu": "stopped,",
            "mem": "0"
        },
        "top": {
            "cpu": "users,",
            "mem": "load"
        }
    },
    "detection_fps": 0.5,
    "detectors": {
        "coral": {
            "detection_start": 0,
            "inference_speed": 9.75,
            "pid": 560792
        }
    },
    "gpu_usages": {
        "error-gpu": {
            "gpu": -1,
            "mem": -1
        }
    },
    "service": {
        "last_updated": 1693511264,
        "latest_version": "0.12.1",
        "storage": {
            "/dev/shm": {
                "free": 112.5,
                "mount_type": "tmpfs",
                "total": 134.2,
                "used": 21.7
            },
            "/media/frigate/clips": {
                "free": 337449.5,
                "mount_type": "ext4",
                "total": 982820.9,
                "used": 595371.3
            },
            "/media/frigate/recordings": {
                "free": 337449.5,
                "mount_type": "ext4",
                "total": 982820.9,
                "used": 595371.3
            },
            "/tmp/cache": {
                "free": 983.5,
                "mount_type": "tmpfs",
                "total": 1000,
                "used": 16.5
            }
        },
        "temperatures": {},
        "uptime": 523519,
        "version": "0.12.1-367d724"
    }
}

Operating system

Other Linux

Install method

Docker Compose

Coral version

USB

Network connection

Wired

Camera make and model

Amcrest AD110

Any other information that may be helpful

No response

NickM-27 commented 1 year ago

tcp://192.168.7.166:554?timeout=5000000 failed: No route to host means the camera is not on the network at that ip address

Maxxell commented 1 year ago

I am still able to access that camera through Synology DS Cam, which I set up as a backup. So the camera is still online.

Why would frigate be unable to reach it?

If it is helpful/diagnostic, when I reboot frigate, all the cameras work again.

On Thu, Aug 31, 2023 at 3:51 PM Nicolas Mowen @.***> wrote:

tcp://192.168.7.166:554?timeout=5000000 failed: No route to host means the camera is not on the network at that ip address

— Reply to this email directly, view it on GitHub https://github.com/blakeblackshear/frigate/issues/7642#issuecomment-1701692567, or unsubscribe https://github.com/notifications/unsubscribe-auth/AOX6NXHMPNOQZAK6U65B7OLXYDTKXANCNFSM6AAAAAA4GSA2QY . You are receiving this because you authored the thread.Message ID: @.***>

NickM-27 commented 1 year ago

potentially because the camera is struggling to provide multiple streams at once. Ffmpeg makes no mistake on that, it is not seeing the device on the network or the stream is timing out and stops sending data

Maxxell commented 1 year ago

So that error about a CUDA-compatible device is irrelevant?

I only spun up the Synology system after these problems started happening with Frigate. The whole point of using frigate was that I had minimal connections to the camera itself.

NickM-27 commented 1 year ago

So that error about a CUDA-compatible device is irrelevant?

it is its own problem, but the logs are clear about the camera

The whole point of using frigate was that I had minimal connections to the camera itself.

you're not using go2rtc, so frigate is not restreaming

in any case, that is irrelevant, I was only offering an example, not saying that was definitely the reason

Maxxell commented 1 year ago

Entirely possible I have multiple issues here.

  1. I dont understand how there is an internet connection issue when other devices can still find the camera at the same location. Is there a way to have frigate 'try again' without manually restarting frigate? That seems the only reliable way to restore these connections.
  2. I dont re-stream through frigate. I have a HomeAssistant page which is a live view through frigate. Perhaps that is not re-streaming, but I thought that would still be going through frigate instead of connecting directly to camera.
  3. How would I investigate the CUDA-related error?
NickM-27 commented 1 year ago

I dont understand how there is an internet connection issue when other devices can still find the camera at the same location. Is there a way to have frigate 'try again' without manually restarting frigate? That seems the only reliable way to restore these connections.

frigate is "trying again" hence the repeating logs

2023-08-30 16:51:09.727454843  [2023-08-30 16:51:09] watchdog.Frontdoor             ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2023-08-30 16:51:09.727714029  [2023-08-30 16:51:09] ffmpeg.Frontdoor.detect        ERROR   : [tcp @ 0x5597d305d540] Connection to tcp://192.168.7.166:554?timeout=5000000 failed: Connection timed out
2023-08-30 16:51:09.728317532  [2023-08-30 16:51:09] ffmpeg.Frontdoor.detect        ERROR   : rtsp://*:*@192.168.7.166:554/cam/realmonitor?channel=1&subtype=0: Connection timed out
2023-08-30 16:51:12.809836361  [2023-08-30 16:51:12] frigate.video                  ERROR   : Frontdoor: Unable to read frames from ffmpeg process.
2023-08-30 16:51:12.810162091  [2023-08-30 16:51:12] frigate.video                  ERROR   : Frontdoor: ffmpeg process is not running. exiting capture thread...
2023-08-30 16:51:19.755044528  [2023-08-30 16:51:19] watchdog.Frontdoor             ERROR   : Ffmpeg process crashed unexpectedly for Frontdoor.
2023-08-30 16:51:19.755337704  [2023-08-30 16:51:19] watchdog.Frontdoor             ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2023-08-30 16:51:19.755588413  [2023-08-30 16:51:19] ffmpeg.Frontdoor.detect        ERROR   : [tcp @ 0x563a1d090540] Connection to tcp://192.168.7.166:554?timeout=5000000 failed: No route to host
2023-08-30 16:51:19.755815259  [2023-08-30 16:51:19] ffmpeg.Frontdoor.detect        ERROR   : rtsp://*:*@192.168.7.166:554/cam/realmonitor?channel=1&subtype=0: No route to host
2023-08-30 16:51:24.783033304  [2023-08-30 16:51:24] frigate.video                  ERROR   : Frontdoor: Unable to read frames from ffmpeg process.
2023-08-30 16:51:24.783146871  [2023-08-30 16:51:24] frigate.video                  ERROR   : Frontdoor: ffmpeg process is not running. exiting capture thread...
2023-08-30 16:51:29.765519222  [2023-08-30 16:51:29] watchdog.Frontdoor             ERROR   : Ffmpeg process crashed unexpectedly for Frontdoor.
2023-08-30 16:51:29.765769357  [2023-08-30 16:51:29] watchdog.Frontdoor             ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2023-08-30 16:51:29.765955917  [2023-08-30 16:51:29] ffmpeg.Frontdoor.detect        ERROR   : [tcp @ 0x55c5ad61a540] Connection to tcp://192.168.7.166:554?timeout=5000000 failed: Connection timed out
2023-08-30 16:51:29.766172144  [2023-08-30 16:51:29] ffmpeg.Frontdoor.detect        ERROR   : rtsp://*:*@192.168.7.166:554/cam/realmonitor?channel=1&subtype=0: Connection timed out
2023-08-30 16:51:31.671919661  [2023-08-30 16:51:31] frigate.video                  ERROR   : Frontdoor: Unable to read frames from ffmpeg process.

I dont re-stream through frigate. I have a HomeAssistant page which is a live view through frigate. Perhaps that is not re-streaming, but I thought that would still be going through frigate instead of connecting directly to camera.

if you don't have go2rtc setup then the home assistant entity is just a snapshot that updates once per second

How would I investigate the CUDA-related error?

for one I would start by using the recommended ffmpeg preset instead of manual args. Other then that, need to look up the error and see what others say on it

Maxxell commented 1 year ago

the recommended ffmpeg preset

It seems like my setup is pretty out-dated. Can you point me to a setup page where I can learn more about this? I remember battling for a while to find the proper manual arguments to make these cameras work, but like I said earlier that was several versions of frigate ago.

NickM-27 commented 1 year ago

https://docs.frigate.video/configuration/hardware_acceleration#nvidia-gpus

Maxxell commented 1 year ago

So I should

  1. change my config file: hwaccel_args: -c:v h264_cuvid

to this: hwaccel_args: preset-nvidia-h264

  1. [And add the whole deploy section about the NVidia stuff to my dockercompose.] nevermind, i realized I already have this part.

OK I'll make those changes and try again. Thank you!

Maxxell commented 1 year ago

if you don't have go2rtc setup then the home assistant entity is just a snapshot that updates once per second

After making your recommended changes, I went back to HomeAssistant and I'm definitely getting more than one frame per second on my dashboard through frigate. Is there some other way (perhaps an old, abandon system) that was able to make that happen without go2rtc?

NickM-27 commented 1 year ago

Is there some other way (perhaps an old, abandon system) that was able to make that happen without go2rtc?

yes, it used to use rtmp, that has been deprecated, as is explained in the 0.12 release notes

Maxxell commented 1 year ago

Is there some other way (perhaps an old, abandon system) that was able to make that happen without go2rtc?

yes, it used to use rtmp, that has been deprecated, as is explained in the 0.12 release notes

Thank you very much for pointing out those Release Notes to me, I had not seen those before. That was the first I saw your involvement. Let me take a second to say THANK YOU for all of these changes. It looks like I was not paying attention for too long and you've accomplished much of what I was disappointed by not having previously. I have some tinkering to do.

I'm not sure I understand how my frontdoor feed is still working if rtmp is deprecated and i have rtmp enabled: false in my current config. Is there something more I would need to investigate/do to ensure that is truly turned off as I look into enabling the built-in go2rtc?

NickM-27 commented 1 year ago

I'm not sure I understand how my frontdoor feed is still working if rtmp is deprecated and i have rtmp enabled: false in my current config. Is there something more I would need to investigate/do to ensure that is truly turned off as I look into enabling the built-in go2rtc?

Still working where? as I said previously, the HA feed is just a snapshot that updates once per second if go2rtc is not setup

Maxxell commented 1 year ago

I'm not sure I understand how my frontdoor feed is still working if rtmp is deprecated and i have rtmp enabled: false in my current config. Is there something more I would need to investigate/do to ensure that is truly turned off as I look into enabling the built-in go2rtc?

Still working where? as I said previously, the HA feed is just a snapshot that updates once per second if go2rtc is not setup

I wish you were sitting next to me. Here I have the pro telling me there's no way I'm getting more than 1fps, but I'm sitting at my computer looking at HomeAssistant and watching a bird smoothly fly by and trees smoothly waving in the wind.

When I first navigate to that page in HA, you are 100% right. It's not great quality, and its 1fps. But after a few seconds, the "high quality mode" kicks in and I get HD camera quality and a pretty smooth fps. Is there a log I can share with you that would show you what is happening?

NickM-27 commented 1 year ago

what is your current config?

Maxxell commented 1 year ago

what is your current config?

In HA or in frigate? The only change I've made in frigate is the codec thing u suggested a minute ago.

In HA, I am using a custom frigate card. I see the little bird-like icon in the topleft corner of the stream I am talking about.

My yaml looks like this: type: custom:frigate-card cameras:

NickM-27 commented 1 year ago

oh, you didn't say you were using the frigate card, in that case it is using the jsmpeg feed from frigate which is 5 fps

Maxxell commented 1 year ago

5fps is plenty for my re-streaming purposes. So then I wont bother with the go2rtc since I can use HA to get 5fps on all my cameras using the frigate card. That works for me. I will stop making changes for now I guess. I am comfortable with my re-streaming situation and my HA notifications. Hopefully the updated plugin setting you mentioned keeps the cameras online for longer. Thank you again for this discussion.

NickM-27 commented 1 year ago

How long did it stay online before?

Maxxell commented 1 year ago

Not long. Maybe a day or two.

Maxxell commented 1 year ago

I am trying to set up FrontDoor with separate streams, but its not working. Any idea why? Does that "Roles" stuff not work anymore?

  Frontdoor:
    ffmpeg:
      hwaccel_args: preset-nvidia-h264
      inputs:
        - path: rtsp://admin:***@192.168.7.166:554/cam/realmonitor?channel=1&subtype=0
      roles: 
         - record
        - path: rtsp://admin:***@192.168.7.166:554/cam/realmonitor?channel=1&subtype=1
      roles: 
         - detect
    record: 
      enabled: True
      retain: 
        days: 2
      events:
        retain:
          default: 4
          mode: motion
    snapshots:
      enabled: True
    rtmp: 
      enabled: False
    detect:
      width: 720
      height: 576
    objects:
      track:
        - person
        - dog
NickM-27 commented 1 year ago

what are the logs?

Maxxell commented 1 year ago

I don't get any. Frigate won't event start.

I'm gonna shut down for the night. Thanks again.

NickM-27 commented 1 year ago

I don't get any. Frigate won't event start.

I'm gonna shut down for the night. Thanks again.

if frigate is failing to start, there will be logs showing the error in the config

s482dcaw commented 1 year ago

For what it's worth, I had the same problem with cameras and streams getting disconnecting and giving error messages. Here's how the problem went away (for a couple of days now, I may need to update that in the future if it reappears). In my case, I have a Coral Edge TPU connected in a USB-3 port, and an external SSD disk for storage of recordings and clips, also connected to a USB-3 port.

Solution 1: Do not use the Coral at all; use OpenVino instead. CPU is "Intel(R) Core(TM) i3-4150T CPU @ 3.00GHz". Coral's inference speed is slightly over 10ms. OpenVino's inference in this CPU is 30-40ms. If that is acceptable, try it.

Solution 2: Bind mount the standard edgetpu library to the docker container. The container has preinstalled the package "libedgetpu1-max" which has the Coral running at the maximum frequency (500 MHz). Installing on the host libedgetpu1-std which runs Coral at the reduced frequency (250 MHz) and mounting the library "/usr/lib/x86_64-linux-gnu/libedgetpu.so.1.0", so having frigate use the throttled library, also made the errors disappear. That takes the inderence speed to ~17ms over the initial ~10ms

NickM-27 commented 1 year ago

@s482dcaw sounds like your device is struggling to power both USB devices, that's when a USB powered hub is recommended.

s482dcaw commented 1 year ago

That is probable, and since the error messages are pretty similar, others may have the same problem. That's why I wrote down the details.

Maxxell commented 1 year ago

I don't get any. Frigate won't event start. I'm gonna shut down for the night. Thanks again.

if frigate is failing to start, there will be logs showing the error in the config

All of my efforts to get the 'role' feature working continue to fail. :-(
But even when I leave the role feature out of it, I still find errors in my log. Strangely, the errors seem to share the time of day... I had a series of errors start on 9/5 at 4:42. Those stopped at 4:43. Then more errors on 9/6 starting at 4:44. Those stopped at 4:46. And the camera it is complaining about is working now without any intervention on my part. So weird.

I did find that the very first error this morning is unique: 2023-09-06 04:44:19.106061924 [2023-09-06 04:44:19] frigate.video ERROR : Frontdoor: Unable to read frames from ffmpeg process. 2023-09-06 04:44:19.106095451 [2023-09-06 04:44:19] frigate.video ERROR : Frontdoor: ffmpeg process is not running. exiting capture thread... 2023-09-06 04:44:19.921246534 [2023-09-06 04:44:19] watchdog.Frontdoor ERROR : Ffmpeg process crashed unexpectedly for Frontdoor. 2023-09-06 04:44:19.921369188 [2023-09-06 04:44:19] watchdog.Frontdoor ERROR : The following ffmpeg logs include the last 100 lines prior to exit. 2023-09-06 04:44:19.921482021 [2023-09-06 04:44:19] ffmpeg.Frontdoor.detect ERROR : [segment @ 0x55ae02991bc0] Timestamps are unset in a packet for stream 0. This is deprecated and will stop working in the future. Fix your code to set the timestamps properly 2023-09-06 04:44:19.921579785 [2023-09-06 04:44:19] ffmpeg.Frontdoor.detect ERROR : rtsp://:@192.168.7.166:554/cam/realmonitor?channel=1&subtype=0: Connection timed out 2023-09-06 04:44:19.921674078 [2023-09-06 04:44:19] ffmpeg.Frontdoor.detect ERROR : Last message repeated 1 times

NickM-27 commented 1 year ago

ffmpeg.Frontdoor.detect ERROR : rtsp://:@192.168.7.166:554/cam/realmonitor?channel=1&subtype=0: Connection timed out

looks like network issues or something, camera stream timing out

Maxxell commented 1 year ago

Timestamps are unset in a packet for stream 0. This is deprecated and will stop working in the future. Fix your code to set the timestamps properly

Any insight on this error that immediately precedes it?

NickM-27 commented 1 year ago

that is not an actual error, that is just a warning that ffmpeg prints out. When a stream crashes frigate prints all ffmpeg logs

Maxxell commented 1 year ago

... so the internet to my frontdoor camera is dying every night at around 445 am? That's so strange. I suppose its possible the camera is doing a nightly 'check for updates' or something like that? Other than missing a minute or two of video each night from that one camera, is this something that can be ignored?

NickM-27 commented 1 year ago

Many cameras come with auto reboot enabled so that could be it as well.

Maxxell commented 1 year ago

Woke up this morning and noticed that I hadn't gotten any frigate notifications. Loaded up the web portal and found all 5 cameras offline. All showing the same "no frames received check error logs"... So here's my error logs: logs 2023.9.13.txt

Maxxell commented 1 year ago

Seems like it starts with this - or maybe this is just as far back as the logs are stored?

2023-09-13 11:48:59.140050340 [2023-09-13 11:48:59] ffmpeg.Garage.detect ERROR : [AVHWDeviceContext @ 0x55c5d358e7c0] cu->cuInit(0) failed -> CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected 2023-09-13 11:48:59.140051457 [2023-09-13 11:48:59] ffmpeg.Garage.detect ERROR : Device creation failed: -542398533. 2023-09-13 11:48:59.140052645 [2023-09-13 11:48:59] ffmpeg.Garage.detect ERROR : [h264 @ 0x55c5d35a7e40] No device available for decoder: device type cuda needed for codec h264. 2023-09-13 11:48:59.140053739 [2023-09-13 11:48:59] ffmpeg.Garage.detect ERROR : Device setup failed for decoder on input stream #0:0 : Generic error in an external library

NickM-27 commented 1 year ago

No device available for decoder: device type cuda needed for codec h264.

looks like the GPU was disconnected or there was some issue with the driver

Maxxell commented 1 year ago

The GPU in this device has not had an easy life. Is there an easy way to ask Frigate to run without the GPU? I'd like to try changing as little as possible and see whether the GPU is truly failing at this point. That may be the ultimate culprit of many of my issues.

NickM-27 commented 1 year ago

I mean you can just remove the hwaccel args. The logs are very clear in this instance that the GPU is not available to the container

Maxxell commented 1 year ago

Is it weird that the WebUI cant trigger a restart of frigate? I tried hitting soft restart and nothing happened. I went in and restarted the container with Portainer and all the cameras came back online. I do want to try the 'without GPU' option and see if that helps. Its a strong CPU and its only doing frigate for now.
Do I need to change anything in the Docker-Compose? I have the resources/reservations stuff in there now from this page: https://docs.frigate.video/configuration/hardware_acceleration/

NickM-27 commented 1 year ago

as long as you are not referencing the GPU in the frigate config it does not matter

Maxxell commented 1 year ago

Should I delete that line entirely or change it to this: hwaccel_args: preset-intel-qsv-h264

(The CPU in there is an i7-10700)

NickM-27 commented 1 year ago

I'd suggest using preset-vaapi instead

Maxxell commented 1 year ago

No, that broke everything. Maybe I'm doing it wrong.
If I do this:

  Backyard:
    ffmpeg:
      hwaccel_args: preset-vaapi
      inputs:

It wont read any of my cameras. But if I do this:

  Backyard:
    ffmpeg:
      inputs:

It works fine.

Maxxell commented 1 year ago

thats not formatted right hang on

NickM-27 commented 1 year ago

docker compose, logs?

Maxxell commented 1 year ago

logs 2023.9.13.2.txt

Maxxell commented 1 year ago

docker-compose.txt

NickM-27 commented 1 year ago

you aren't mapping the /dev/dri/renderD128 device to the container

Maxxell commented 1 year ago

what is /dev/dri/renderD128?