Open madasus opened 10 months ago
There's no info provided here so there is nothing to go off of. You first need to figure out why the machine is actually freezing (is it memory issue, kernel panic, etc.)
nothing is shown on the console. I'll check to see if there is anything in syslog. The last time i checked there was nothing - the whole machine was just frozen.
it can happen for many different reasons, if there is no information that can be provided then there's not really much that can be done on the frigate side. There are plenty of solutions like having a log written to a file so the cause can be seen in the logs after restarting the machine.
Also, you can try putting a memory limit on the frigate container
The next steps would be to back down frigate to a bare minimum config and slowly add parts back until you can see what is causing the issue.
Thanks - i'm following the other thread also. I also added some more debugging to Ubuntu to see if I can capture anything in the logs before the freeze.
Do you have any suggestions on where to start removing the config from? us the hwaccel param a place to start?
ffmpeg: hwaccel_args: preset-vaapi
This comment is not very helpful but I had the exact issue running the containers on kubernetes (microk8s on ubuntu). Host would crash, have to power cycle. No useful information in logs or kernel log.
I ended up removing my coral (m.2) and switching to CPU/VAAPI detection for now, It's been a few weeks without issue.
It's a long shot but could be worth trying the same to rule it out? I have not gone back to the coral as I have only a 1 camera doing detection / CPU usage is not high.
I have a similar issue, running an i5-6500T, no external accelerator, and so far I've been able to ascertain the following:
Here's my config using three random camera feeds from the Internet that I use for debugging, currently the hardware acceleration for decoding/encoding is commented out:
mqtt:
enabled: false
go2rtc:
streams:
# test_camera_1_main:
# - rtsp://admin:xxxxxxxxxxx@192.168.10.17:554/h265Preview_01_main
# test_camera_1_sub:
# - rtsp://admin:xxxxxxxxxxx@192.168.10.17:554/h264Preview_01_sub
test1:
# - ffmpeg:http://78.31.82.246/mjpg/video.mjpg#video=h264#hardware
- ffmpeg:http://78.31.82.246/mjpg/video.mjpg#video=h264
test2:
# - ffmpeg:http://webcam.zvnoordwijk.nl:82/mjpg/video.mjpg#video=h264#hardware
- ffmpeg:http://webcam.zvnoordwijk.nl:82/mjpg/video.mjpg#video=h264
test3:
# - ffmpeg:http://tacocam.tacoma.uw.edu/mjpg/video.mjpg#video=h264#hardware
- ffmpeg:http://tacocam.tacoma.uw.edu/mjpg/video.mjpg#video=h264
cameras:
# test_camera_1: # <------ Name the camera
# ffmpeg:
# output_args:
# record: preset-record-generic-audio-copy
# inputs:
# - path: rtsp://127.0.0.1:8554/test_camera_1_sub # <----- The stream you want to use for detection
# input_args: preset-rtsp-restream
# hwaccel_args: preset-vaapi
# roles:
# - detect
# - path: rtsp://127.0.0.1:8554/test_camera_1_main # <----- The stream you want to use for recording
# input_args: preset-rtsp-restream
# hwaccel_args: preset-vaapi
# roles:
# - record
# record:
# enabled: True
# detect:
# enabled: True # <---- disable detection until you have a working camera feed
# width: 640 # <---- update for your camera's resolution
# height: 480 # <---- update for your camera's resolution
# live:
# stream_name: test_camera_1_main
test1: # <------ Name the camera
ffmpeg:
inputs:
- path: rtsp://127.0.0.1:8554/test1 # <----- The stream you want to use for detection
input_args: preset-rtsp-restream
# hwaccel_args: preset-vaapi
roles:
- detect
- path: rtsp://127.0.0.1:8554/test1 # <----- The stream you want to use for recording
input_args: preset-rtsp-restream
# hwaccel_args: preset-vaapi
roles:
- record
record:
enabled: true
detect:
enabled: true # <---- disable detection until you have a working camera feed
width: 1280 # <---- update for your camera's resolution
height: 720 # <---- update for your camera's resolution
live:
stream_name: test1
objects:
track:
- person
- car
test2: # <------ Name the camera
ffmpeg:
inputs:
- path: rtsp://127.0.0.1:8554/test2 # <----- The stream you want to use for detection
input_args: preset-rtsp-restream
hwaccel_args: preset-vaapi
roles:
- detect
- path: rtsp://127.0.0.1:8554/test2 # <----- The stream you want to use for recording
input_args: preset-rtsp-restream
# hwaccel_args: preset-vaapi
roles:
- record
record:
enabled: true
detect:
enabled: true # <---- disable detection until you have a working camera feed
width: 1280 # <---- update for your camera's resolution
height: 720 # <---- update for your camera's resolution
live:
stream_name: test2
objects:
track:
- person
test3: # <------ Name the camera
ffmpeg:
inputs:
- path: rtsp://127.0.0.1:8554/test3 # <----- The stream you want to use for detection
input_args: preset-rtsp-restream
# hwaccel_args: preset-vaapi
roles:
- detect
- path: rtsp://127.0.0.1:8554/test3 # <----- The stream you want to use for recording
input_args: preset-rtsp-restream
# hwaccel_args: preset-vaapi
roles:
- record
record:
enabled: true
detect:
enabled: true # <---- disable detection until you have a working camera feed
width: 1920 # <---- update for your camera's resolution
height: 1080 # <---- update for your camera's resolution
live:
stream_name: test3
objects:
track:
- person
motion:
mask:
- 716,0,723,359,126,378,129,0
- 1920,0,1920,0,1920,731,1869,783,1804,823,1722,860,1604,855,1480,838,1314,778,1299,729,1188,676,1123,667,1061,683,1010,642,978,598,961,516,850,496,755,464,674,447,603,306,582,0
record:
retain:
days: 0
mode: all
events:
retain:
default: 14
mode: motion
objects:
person: 30
detectors:
ov:
type: openvino
device: AUTO
model:
path: /openvino-model/ssdlite_mobilenet_v2.xml
model:
width: 300
height: 300
input_tensor: nhwc
input_pixel_format: bgr
labelmap_path: /openvino-model/coco_91cl_bkgr.txt
# Include all cameras by default in Birdseye view
birdseye:
enabled: true
I tried to grab kernel crashdump via kdump, and also tried out kernel netconsole (dmesg) logging to another server running on the same network, but neither resulted in any output, which makes me think it's a driver issue that affects the CPU itself, not even a kernel crash.
Running the beta2 image in docker-compose, the beta3 image has an issue with go2rtc failing to parse the camera feed URLs.
If you have any ideas for any further troubleshooting I could do, please do let me know.
@ggidofalvy-tc Have the same issue on unraid: https://github.com/blakeblackshear/frigate/issues/8461
After round about 12h-24h the Host is crashing when using OpenVINO.
Tried different drivers in the Host, but didnt Help.
Currently thinking about to buy a Coral...
I have same issue on K3s on Debian with i3-6100U. VAAPI HW encode/decode + OpenVINO setup in config.
with obj detection turned off its rock solid. if I turn on obj detection for a single object on a single camera, whole node hangs within 3 days.
Those of us using the official helm chart cant update go2rtc or ffmpeg with custom versions.
Adding onto my previous comment:
Running Ubuntu 22.04, tried both the GA (5.15) and HWE (6.2) kernels, both exhibited the same crash behaviour.
https://github.com/blakeblackshear/frigate/issues/8338#issuecomment-1807391160 may be relevant with a couple suggestions (and other linked issue)
@ggidofalvy-tc @madasus especially if your frigate machine is headless, I would recommend removing the often-default quiet
kernel parameter/command-line-argument and adding debug
. that's what helped in my case linked above to at least narrow down the issue to the GPU, but I have made limited progress above as NickM has linked. my errors only showed up on the physical console, due to the hang.
It's certainly suspicious that what I reported in #8338 is also using a i7-6600(U) / Skylake GPU - same generation as you both - wondering if there is a driver bug / hardware quirk that other generations don't have that the i915 driver isn't handling
@kevin-david my host is headless so i'll give this a try. Will the debug then be written to syslog? how are you grabbing it?
Can you point me in the direction of where you made this change in your linux distro? (i'm using Ubuntu).
I'm glad i opened this thread as it appears this is not an isolated problem - and while not a Frigate issue but likely something that Frigate exposes due to load in the underlying hardware/software.
Thanks
M
@madasus sure - I am using proxmox, so it should be similar. In my case the message never appeared in syslog, only on the physically connected screen - I guess because the machine was hung, it wasn't able to be written to syslog. this might mean you need to temporarily connect a monitor to the machine.
To do what I was talking about, you'll want to change GRUB_CMDLINE_LINUX_DEFAULT
in the /etc/default/grub
file and run update-grub
to regenerate the configuration file, and reboot.
This describes it a little more: https://askubuntu.com/a/19487. Again in my case, I removed quiet
which resulted in messages logged to the console, and added debug
(which I'm not sure makes a huge difference, but isn't super noisy either)
I gave echo 0 | sudo tee /sys/class/drm/card0/engine/rcs0/preempt_timeout_ms
a spin, but no luck in preventing/prolonging the crash.
This is what I got in dmesg:
[525594.184400] [drm:__uc_sanitize [i915]] *ERROR* Failed to reset GuC, ret = -110
I'll keep a look out for more messages in the netconsole destination now that I rebooted again and set the debug
kernel commandline flag, I won't be applying the "fix" this time around.
@ggidofalvy-tc @madasus
I probably found a solution... Running a yolov8s model since some days and currently >48h stable without any crash. Perhaps you can try this aswell?
@Pingbo can you share your model and detector config.yml snippets? Sorry for the mild derail, I would like to see if this might be a model-specific issue, not an OpenVINO-related one. Running the beta2 branch, since beta3 has issues with go2rtc with my config.
I've been trying to get yolov8n/yolov8s running on my setup based on the notebook linked in this comment: https://github.com/blakeblackshear/frigate/issues/5184#issuecomment-1449314504
But I keep getting an error when the detector starts up:
2023-11-20 12:55:12.267683201 Traceback (most recent call last):
2023-11-20 12:55:12.267747530 File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
2023-11-20 12:55:12.267749391 self.run()
2023-11-20 12:55:12.267806449 File "/usr/lib/python3.9/multiprocessing/process.py", line 108, in run
2023-11-20 12:55:12.267808281 self._target(*self._args, **self._kwargs)
2023-11-20 12:55:12.267855642 File "/opt/frigate/frigate/object_detection.py", line 102, in run_detector
2023-11-20 12:55:12.267857527 object_detector = LocalObjectDetector(detector_config=detector_config)
2023-11-20 12:55:12.267898126 File "/opt/frigate/frigate/object_detection.py", line 53, in __init__
2023-11-20 12:55:12.267899858 self.detect_api = create_detector(detector_config)
2023-11-20 12:55:12.267941613 File "/opt/frigate/frigate/detectors/__init__.py", line 18, in create_detector
2023-11-20 12:55:12.267943162 return api(detector_config)
2023-11-20 12:55:12.267986312 File "/opt/frigate/frigate/detectors/plugins/openvino.py", line 26, in __init__
2023-11-20 12:55:12.267988059 self.ov_model = self.ov_core.read_model(detector_config.model.path)
2023-11-20 12:55:12.268047582 RuntimeError: Check 'false' failed at src/frontends/common/src/frontend.cpp:53:
2023-11-20 12:55:12.268049096 Converting input model
2023-11-20 12:55:12.268050592 Cannot create Interpolate layer /model.10/Resize id:164 from unsupported opset: opset11
My config.yml bits, attempting to run the yolov8n model:
detectors:
ov:
type: openvino
device: AUTO
model:
path: /config/openvino-model/yolov8n.xml
model:
width: 416
height: 416
input_tensor: nhwc
input_pixel_format: bgr
model_type: yolov8
labelmap_path: /openvino-model/coco_91cl_bkgr.txt
(all 3 output files are mounted /config/openvino-model, I'm reusing the labelmap from the original mobileSSD model used)
when mine freezes I managed to check the console this time and there were no messages at all being written to the console before the crash.
@Pingbo can you elaborate on how to use the model you are suggesting? is this being used instead of the Coral?
@ggidofalvy-tc
Thats how i have done it:
detectors:
ov:
type: openvino
device: GPU
model:
# path: /openvino-model/ssdlite_mobilenet_v2.xml
path: /config/yolov8s/yolov8s.xml
model:
width: 416 height: 416 input_tensor: nchw # nhwc input_pixel_format: bgr model_type: yolov8 labelmap_path: /config/coco_80cl.txt #/openvino-model/coco_91cl_bkgr.txt
@madasus
Yes this is using OpenVino as detector and not coral. As far is i know you cannot use coral and yolo models together
Thank you for the detail @Pingbo
The only thing I would clarify for others is 1. you want to put all 3 files in the .zip file from the yolo model generation in the model folder, and 2. the files that were generated for me were yolo8n.xml, so make sure your file path is correct. Hopefully this is the fix.
Edit: 2 weeks running with solid person detection using the yolov8n model on a single camera. Looks like CPU usage dropped significantly for me. Enabling it on the rest of my cameras now.
@Pingbo Thank you for the help and the detailed instructions! I've been using yolov8n for nearly two weeks now without any crashing on beta2.
I think the issue might indeed be caused by the combination of the bundled ssdlite_mobilenet_v2 model and Skylake-gen OpenVINO -- is this perhaps worth documenting somewhere?
Wanted to chime in here, im a new frigate user as of about two weeks ago. My hardware is an i7-7700 kaby lake. I am running frigate and wyze-bridge together. Wyze bridge is correctly using Intel QSV with ffmpeg and Frigate will use it fine on ffmpeg as well. However, if i tried to use any openvino detector, it would crash the container everytime. If i set a detector as cpu (not openvino cpu), the container would start and detect fine.
Today i followed these steps by @Pingbo and finally my openvino detector will start with GPU selected. My inference speed went from 45ms (cpu) to 15ms (ov gpu).
The only error i could make out from the container was
RuntimeError: The input blob size is not equal to the network input size: got 307200 expecting 270000
I tried spinning up various python openvino demos and i was getting similar errors. I was running these demos inside the container. Errors like Resulting shape '{1,3,300,3}' after preprocessing is not aligned with original parameter's shape: {1,300,300,3}, input parameter: image_tensor
. This led me to believe maybe something with the included frigate openvino model and kabylake was not working out.
Anyway, the yolov8 model from the above comment seems to have resolved my issue for now. Ive been stable for a few hours (where previously i was unable to even start the containers). I will continue to monitor. (thanks @Pingbo !!)
edit: i am on frigate version 0.12.1-367D724
Followed the advice of @Pingbo for using the yolov8 model as well. Been running for a couple weeks without any issues. My detections are more reliable as well, so that is an added bonus. Thanks @Pingbo
@Pingbo i'm trying generate a YOLO model via the link, but i have zero to no clue what i'm doing. The scripts are giving me errors. Is it possible for you to help me?
Example from first script: ImportError: cannot import name 'is_exact_shape_match' from 'pandas.core.indexers' (/usr/local/lib/python3.10/dist-packages/pandas/core/indexers/init.py)
Example from second script: ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. ipython 7.34.0 requires jedi>=0.16, which is not installed. lida 0.0.10 requires fastapi, which is not installed. lida 0.0.10 requires kaleido, which is not installed. lida 0.0.10 requires python-multipart, which is not installed. lida 0.0.10 requires uvicorn, which is not installed. llmx 0.0.15a0 requires cohere, which is not installed. llmx 0.0.15a0 requires openai, which is not installed. llmx 0.0.15a0 requires tiktoken, which is not installed.
(with some other lines and then ending with..)
WARNING: The following packages were previously imported in this runtime: [PIL,_distutils_hack,certifi,dateutil,defusedxml,google,numpy,pkg_resources,setuptools,six] You must restart the runtime in order to use newly installed versions.
By that i understand that i just have to refresh the page and try again, but it produces the same error.
@Strux-DK its not you, you just picked a bad day.
I literally just ran this yesterday to generate an 8s model to upgrade from the 8n I was using and it worked flawlessly. Today, I'm getting the same errors you are. (tried 3x)
It looks like @aeozyalcin 's awesome colab may have broken?
That said, since I just generated it yesterday, maybe give this one a try? yolov8s.zip
I have Frigate 0.13 running on a Dell Optiplex 3070 Micro i3-9100t, with a single USB coral. It runs Debian Bookworm and Frigate only. I've had probably one hang per month on average, but recently I made quite a few config changes (mainly around go2rtc streams), and it now seems to crash around once a week. I'm in the process of experimenting with kernel crash dumps to see if I can catch any more info.
As has already been said, this is highly unlikely to be a Frigate bug. I suspect the issue is either in Linux drivers, or (particularly in the case of cheap no-name mini PCs, or even micro form factor desktops like my Optiplex) hardware - not necessarily faulty hardware, but hardware that isn't designed to run full-tilt 24/7 on the CPU/iGPU while also supplying 1A power to a Coral (or two).
I never had any problems on my Dell R220 with Xeon E3s and ECC RAM, and I don't think I've seen many examples of people reporting crashes on server-class hardware. I think Frigate is actually quite an interesting test-case of a 24/7 demanding application often running on cheap consumer gear. The only other tasks most people might assign to this gear - transcoding video or gaming - doesn't run 24/7.
I had the same issues with a Poweredge T20 Xeon E3 1225 V3 CPU, hardware acceleration and coral usb active. Unable to log anything, many succesfull system checks.
Therefore I upgraded to a Dell PowerEdge T320 without hardware acceleration and with coral usb. No crashes with this system and it's running for 4 months.
Another +1 here with similar config, Skylake (i3-6100) generation CPU with OpenVINO (No Coral) in case it helps as a further reference point.
The combination of mobilenet_v2 and vaapi for hwaccel causes hard system crashes within 24 hours every time. The workarounds I've found that work are:
And the things I've tried that I tried in isolation and do NOT work (Including because other issues referenced these as possible solutions):
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Issue still exists.
Same problem in unraid with 0.13.2-6476f8a. I have setup a syslog to try to catch the what it is happening when server gets stuck.
Same problem in unraid with 0.13.2-6476f8a. I have setup a syslog to try to catch the what it is happening when server gets stuck.
Having this issue as well on unRAID... Attaching unRAID diagnostics (includes syslog and other logs/info) in case it helps track this down. Observed out of memory errors and the reaper process killing ffmpeg within the syslog. Sometimes the system runs for weeks other times it dies in a matter of days but works great otherwise. Usually, the system will hard lock and I'm unable to get any further diagnostic information but one of my plugins detected the OOM problems and this diagnostic zip was captured at that time.
and full props to yayitazale for helping me get this far over on the unRAID forums.
I had this happen on unraid and it was due to their network implementation. Switching to ipvlan fixed the issue and hasn't happened to me in months https://forums.unraid.net/topic/151441-macvlan-crashes-6126/
I had this happen on unraid and it was due to their network implementation. Switching to ipvlan fixed the issue and hasn't happened to me in months https://forums.unraid.net/topic/151441-macvlan-crashes-6126/
I have switched to ipvlan and I'm still having crashes.
I had this happen on unraid and it was due to their network implementation. Switching to ipvlan fixed the issue and hasn't happened to me in months https://forums.unraid.net/topic/151441-macvlan-crashes-6126/
Confirmed I am already on ipvlan and having this issue. Fought that battle a couple years ago.
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Another +1 here with similar config, Skylake (i3-6100) generation CPU with OpenVINO (No Coral) in case it helps as a further reference point.
The combination of mobilenet_v2 and vaapi for hwaccel causes hard system crashes within 24 hours every time. The workarounds I've found that work are:
* ✅ Disabling vaapi or * ✅ Using yolov8n
And the things I've tried that I tried in isolation and do NOT work (Including because other issues referenced these as possible solutions):
* ❌ Upgrading the kernel to 6.2 (From 5.15) * ❌ Changing shm values. Tried huge ones, makes no difference * ❌ Switching between iHD and i965 drivers * ❌ Different Frigate versions (Tried v0.11, 0.12 and the current v0.13.1)
I can confirm that using yolov8n has most likely fixed the issue on my HP ProDesk G3, before that the GPU would crash every ~2 weeks causing the whole PC to hang.
Switched from Cuda (Quadro RTX 4000) to Coral/Intel 12th Gen QSV for processing. Consumes about 100watts less power and generates a lot less heat but still crashes. If there's an improvement in uptime I haven't noticed. Though limiting the memory of this container to 4GBs seems to keep it in check.
In my case, everything up to date and running with no issues for 1 month.
Same Here Using a Raspberry Pi 4 8gb Model with USB Coral TPU
Frigate Version 0.13.2 docker container on Raspberry Pi OS
The whole machine crashes, No access from any source (web,ssh) but strangely ping works (with lots of lag) The only way to restore everything is by restarting the Pi
4 Cameras (3 detecting off office hours) Google Coral TPU over USB
UPDATE: Did some changes on the config files but no luck so far
Container:
frigate:
container_name: Frigate
privileged: true
restart: unless-stopped
image: ghcr.io/blakeblackshear/frigate:stable
shm_size: 128mb <---------------- increased to 1024mb
devices:
- /dev/bus/usb/002/004:/dev/bus/usb/002/004 # passes the USB Coral, needs to be modified for other versions
- /dev/video11:/dev/video11 # Extracted from frigate doumentation for RBPi
volumes:
- /etc/localtime:/etc/localtime:ro
- /home/djcrawleravp/docker/frigate:/config
- /home/djcrawleravp/docker/frigate/media:/media/frigate
network_mode: host
environment:
FRIGATE_RTSP_PASSWORD: "Password123"
Config File: (Recentrly Disabled Birdseye, not much improvement)
mqtt:
host: 127.0.0.1
user: mqtt
password: Password123
topic_prefix: frigate
#ffmpeg:
# hwaccel_args: preset-rpi-64-h264 <---------------- hwaccell disabled
detectors:
coral:
type: edgetpu
device: usb
birdseye:
enabled: false
mode: continuous
cameras:
oficina:
birdseye:
order: 2
ffmpeg:
inputs:
- path: rtsp://arquitec1:pass**@192.168.1.193:554/cam/realmonitor?channel=1&subtype=0
roles:
- record
- path: rtsp://arquitec1:pass**92.168.1.193:554/cam/realmonitor?channel=1&subtype=1
roles:
- detect
objects:
track:
- person
- mouse
snapshots:
enabled: true
retain:
default: 30
objects:
person: 30
quality: 100
record:
enabled: true
retain:
days: 1
mode: active_objects
events:
pre_capture: 2
post_capture: 2
retain:
default: 15
mode: active_objects
objects:
person: 15
mouse: 15
detect:
enabled: true
fps: 5
width: 1280
height: 720
motion:
mask:
- 0,296,440,172,864,113,918,164,989,187,988,253,1079,275,1102,84,1280,76,1280,0,0,0
ingreso:
birdseye:
order: 1
ffmpeg:
inputs:
- path: rtsp://arquitec1:pass**@192.168.1.193:554/cam/realmonitor?channel=2&subtype=0
roles:
- record
- path: rtsp://arquitec1:pass**@192.168.1.193:554/cam/realmonitor?channel=2&subtype=1
roles:
- detect
objects:
track:
- person
- cat
- mouse
snapshots:
enabled: true
retain:
default: 30
objects:
person: 30
quality: 100
record:
enabled: true
retain:
days: 1
mode: active_objects
events:
pre_capture: 2
post_capture: 2
retain:
default: 15
mode: active_objects
objects:
person: 15
cat: 15
mouse: 15
detect:
enabled: true
fps: 5
width: 1280
height: 720
motion:
mask:
- 158,406,0,720,0,0,1280,0,1280,39,1280,720,1013,720,1067,364,657,283,398,234
garaje:
birdseye:
order: 3
ffmpeg:
inputs:
- path: rtsp://arquitec1:pass**@192.168.1.193:554/cam/realmonitor?channel=3&subtype=0
roles:
- record
- path: rtsp://arquitec1:pass**@192.168.1.193:554/cam/realmonitor?channel=3&subtype=1
roles:
- detect
objects:
track:
- person
- cat
- mouse
- car
snapshots:
enabled: true
retain:
default: 30
objects:
person: 30
quality: 100
record:
enabled: true
retain:
days: 1
mode: active_objects
events:
pre_capture: 2
post_capture: 2
retain:
default: 15
mode: active_objects
objects:
person: 15
cat: 15
mouse: 15
detect:
enabled: true
fps: 5
width: 1280
height: 720
motion:
mask:
- 0,185,0,0,33,0,1280,0,1280,75,771,83,134,158
patio:
birdseye:
order: 4
ffmpeg:
inputs:
- path: rtsp://arquitec1:pass**@192.168.1.193:554/cam/realmonitor?channel=4&subtype=0
roles:
- record
- path: rtsp://arquitec1:pass**@192.168.1.193:554/cam/realmonitor?channel=4&subtype=1
roles:
- detect
objects:
track:
- person
- cat
- mouse
snapshots:
enabled: true
retain:
default: 30
objects:
person: 30
quality: 100
record:
enabled: true
retain:
days: 1
mode: active_objects
events:
pre_capture: 2
post_capture: 2
retain:
default: 15
mode: active_objects
objects:
person: 15
cat: 15
mouse: 15
detect:
enabled: true
fps: 5
width: 1280
height: 720
motion:
mask:
- 912,217,478,336,0,337,0,0,1280,0,1280,194
Htop while detecting: (usually below 20%when not detecting)
Frigate Side:
I have this same thing happening, Running Ubuntu on a SFF pc, with a PCI Coral. I get maybe 3 days before it goes unresponsive. I have removed all the recording and coral from the config to get down to a barebones and will add things back as I get stability.
My setup is 5 unifi cameras, gortc, and QSV for decode.
I have same problem, hard hang with no access at console or remotely.
I’ve moved from RPI4 with 4GB to Mac mini (2012) with 16GB on Ubuntu both having same issue. Hangs within 2-3 days normally but sometimes a couple of times a day. No logs and doesn’t seem related to excess activity. I have 2 coral tpu usb devices but have moved one off to another server to limit this server to just frigate.
Am running 4 cams.
Same Here Using a Raspberry Pi 4 8gb Model with USB Coral TPU
Frigate Version 0.13.2 docker container on Raspberry Pi OS
The whole machine crashes, No access from any source (web,ssh) but strangely ping works (with lots of lag) The only way to restore everything is by restarting the Pi
4 Cameras (3 detecting off office hours) Google Coral TPU over USB
UPDATE: Did some changes on the config files but no luck so far
Container:
frigate: container_name: Frigate privileged: true restart: unless-stopped image: ghcr.io/blakeblackshear/frigate:stable shm_size: 128mb <---------------- increased to 1024mb devices: - /dev/bus/usb/002/004:/dev/bus/usb/002/004 # passes the USB Coral, needs to be modified for other versions - /dev/video11:/dev/video11 # Extracted from frigate doumentation for RBPi volumes: - /etc/localtime:/etc/localtime:ro - /home/djcrawleravp/docker/frigate:/config - /home/djcrawleravp/docker/frigate/media:/media/frigate network_mode: host environment: FRIGATE_RTSP_PASSWORD: "Password123"
Config File: (Recentrly Disabled Birdseye, not much improvement)
mqtt: host: 127.0.0.1 user: mqtt password: Password123 topic_prefix: frigate #ffmpeg: # hwaccel_args: preset-rpi-64-h264 <---------------- hwaccell disabled detectors: coral: type: edgetpu device: usb birdseye: enabled: false mode: continuous cameras: oficina: birdseye: order: 2 ffmpeg: inputs: - path: rtsp://arquitec1:pass**@192.168.1.193:554/cam/realmonitor?channel=1&subtype=0 roles: - record - path: rtsp://arquitec1:pass**92.168.1.193:554/cam/realmonitor?channel=1&subtype=1 roles: - detect objects: track: - person - mouse snapshots: enabled: true retain: default: 30 objects: person: 30 quality: 100 record: enabled: true retain: days: 1 mode: active_objects events: pre_capture: 2 post_capture: 2 retain: default: 15 mode: active_objects objects: person: 15 mouse: 15 detect: enabled: true fps: 5 width: 1280 height: 720 motion: mask: - 0,296,440,172,864,113,918,164,989,187,988,253,1079,275,1102,84,1280,76,1280,0,0,0 ingreso: birdseye: order: 1 ffmpeg: inputs: - path: rtsp://arquitec1:pass**@192.168.1.193:554/cam/realmonitor?channel=2&subtype=0 roles: - record - path: rtsp://arquitec1:pass**@192.168.1.193:554/cam/realmonitor?channel=2&subtype=1 roles: - detect objects: track: - person - cat - mouse snapshots: enabled: true retain: default: 30 objects: person: 30 quality: 100 record: enabled: true retain: days: 1 mode: active_objects events: pre_capture: 2 post_capture: 2 retain: default: 15 mode: active_objects objects: person: 15 cat: 15 mouse: 15 detect: enabled: true fps: 5 width: 1280 height: 720 motion: mask: - 158,406,0,720,0,0,1280,0,1280,39,1280,720,1013,720,1067,364,657,283,398,234 garaje: birdseye: order: 3 ffmpeg: inputs: - path: rtsp://arquitec1:pass**@192.168.1.193:554/cam/realmonitor?channel=3&subtype=0 roles: - record - path: rtsp://arquitec1:pass**@192.168.1.193:554/cam/realmonitor?channel=3&subtype=1 roles: - detect objects: track: - person - cat - mouse - car snapshots: enabled: true retain: default: 30 objects: person: 30 quality: 100 record: enabled: true retain: days: 1 mode: active_objects events: pre_capture: 2 post_capture: 2 retain: default: 15 mode: active_objects objects: person: 15 cat: 15 mouse: 15 detect: enabled: true fps: 5 width: 1280 height: 720 motion: mask: - 0,185,0,0,33,0,1280,0,1280,75,771,83,134,158 patio: birdseye: order: 4 ffmpeg: inputs: - path: rtsp://arquitec1:pass**@192.168.1.193:554/cam/realmonitor?channel=4&subtype=0 roles: - record - path: rtsp://arquitec1:pass**@192.168.1.193:554/cam/realmonitor?channel=4&subtype=1 roles: - detect objects: track: - person - cat - mouse snapshots: enabled: true retain: default: 30 objects: person: 30 quality: 100 record: enabled: true retain: days: 1 mode: active_objects events: pre_capture: 2 post_capture: 2 retain: default: 15 mode: active_objects objects: person: 15 cat: 15 mouse: 15 detect: enabled: true fps: 5 width: 1280 height: 720 motion: mask: - 912,217,478,336,0,337,0,0,1280,0,1280,194
Htop while detecting: (usually below 20%when not detecting)
![]()
Frigate Side:
![]()
Disabling hardware accel and increasing shm_size to 1024 did the trick for me… running perfectly for 3 days now
I have this same thing happening, Running Ubuntu on a SFF pc, with a PCI Coral. I get maybe 3 days before it goes unresponsive. I have removed all the recording and coral from the config to get down to a barebones and will add things back as I get stability.
My setup is 5 unifi cameras, gortc, and QSV for decode.
I increased shm_size -> 1024m and have not had an issue with just camera decode (no detection or storage) Uptime for almost 7 days now. I'm going add in detection tonight.
I think for me this was due to having the media/frigate folder mounted to a samba share. I've put the physical disk into the same host machine that is running frigate, and will try to use an nfs share to let home assistant access the files over the network.
I have had memory/CPU spike issues for the past couple weeks, which crash my homeassistant every few hours. My configuration has worked for over a year without issue, so I think HASSOS must have made a change recently which caused issues. I had issues with my i5-6500T w/ 8 Gig Ram. I upgraded to new systems with a i5-8500 and 16 Gig ram and still have the same issue.
I isolated the problem to Frigate (had to disable all integrations and addons to isolate it). From reading this issue, I figured out the vaapi hardware acceleration was causing spikes. Disabling vaapi with:
hwaccel_args: " "
prevents the memory spikes and no longer have crashes. Using vaapi crashes within an hour or so. I use a coral TPU for the detection, so the increased CPU is not a big deal, especially with the upgraded CPU.
Thought I would post this issue so others would know. I have used 13.2 and 14.0-beta2 (both regular and full access) and all crash if use vaapi. So strange.
you could try using qsv and see if it behaves any differently
@NickM-27 The documentation says qsv is only for >= 10th gen. I am only on an 8th gen now.
it is a bit ambiguous because some 8th gen CPUs do support it, might be worth a try
Thanks. Appears to work, UI says intel-qsv GPU is working. I have two identical cameras and see a 18% v 6% CPU use when I put one on qsv. I will see if it is stable and works on my other cameras
edit: Still got a memory spike/crash using qsv. I will just go back to no hardware acceleration.
Describe the problem you are having
I have two docker hosts and both have a coral. I find that Frigate seems to cause the whole host to freeze completely (console is not responsive) at frequent intervals - right now I would say on average every 48 hours but its not consistent. I've moved the docker container to the other host and cleared out all the other dockers and the freeze follows Frigate.
Its likely Frigate is pushing the hosts much harder than any other docker and perhaps its finding a bug somewhere in the hardware or OS. The Devices are BeeLink devices running the latest Ubuntu.
Looking for some advice - has anyone seen this sort of behavior and identified the cause?
This has been happening for many months so it is not related to the beta Frigate or any particular Frigate (and likely this is NOT a Frigate bug)
Version
0.13 Beta 3
Frigate config file
Relevant log output
Frigate stats
No response
Operating system
Other
Install method
Docker Compose
Coral version
USB
Any other information that may be helpful
No response