Closed my-umd closed 1 year ago
To get GPU stats you must run frigate in docker as privileged. If it is still not working then it is an issue with something on the host OS. intel_gpu_top
is very finicky.
If you included the frigate logs that would be helpful to understand what intel_gpu_top is failing
Thanks @NickM-27. The provided docker run command shows it's running under the privileged mode (sorry, I could have explicitly mentioned the privileged mode to save you time). I can certainly provide logs, but just don't know what part to include. The log is huge.
Thanks @NickM-27. The provided docker run command shows it's running under the privileged mode (sorry, I could have explicitly mentioned the privileged mode to save you time). I can certainly provide logs, but just don't know what part to include. The log is huge.
just paste the whole thing
It says my log is too long (more than 65536 characters). I only pasted something from the top that I think might be helpful (sorry for the mess. I couldn't find how to add it to the log section once my issue is created):
2023-03-21 17:54:54.700689136 [INFO] Starting Frigate...
2023-03-21 17:54:57.781681302 [2023-03-21 13:54:57] frigate.app INFO : Starting Frigate (0.12.0-e454daf)
2023-03-21 17:54:57.890044958 [2023-03-21 13:54:57] frigate.config WARNING : Customizing more than a detector model path is unsupported.
2023-03-21 17:54:57.915757450 [2023-03-21 13:54:57] peewee_migrate INFO : Starting migrations
2023-03-21 17:54:57.956348058 [2023-03-21 13:54:57] peewee_migrate INFO : There is nothing to migrate
2023-03-21 17:54:57.971795627 [2023-03-21 13:54:57] frigate.comms.dispatcher ERROR : Received invalid set command: backyard/recordings/set
2023-03-21 17:54:57.971803887 [2023-03-21 13:54:57] frigate.comms.dispatcher ERROR : Received invalid set command: backyard/snapshots/set
2023-03-21 17:54:57.971812557 [2023-03-21 13:54:57] frigate.comms.dispatcher ERROR : Received invalid set command: front_door/recordings/set
2023-03-21 17:54:57.971815945 [2023-03-21 13:54:57] frigate.comms.dispatcher ERROR : Received invalid set command: front_door/snapshots/set
2023-03-21 17:54:57.971851279 [2023-03-21 13:54:57] frigate.comms.dispatcher ERROR : Received invalid set command: porch/recordings/set
2023-03-21 17:54:57.972776515 [2023-03-21 13:54:57] frigate.comms.dispatcher ERROR : Received invalid set command: porch/snapshots/set
2023-03-21 17:54:58.011811607 [2023-03-21 13:54:58] detector.ov INFO : Starting detection process: 290
2023-03-21 17:54:58.027216533 [2023-03-21 13:54:58] frigate.app INFO : Output process started: 292
2023-03-21 17:54:58.027225099 [2023-03-21 13:54:58] frigate.app INFO : Camera processor started for front_door: 294
2023-03-21 17:54:58.051078025 [2023-03-21 13:54:58] frigate.app INFO : Camera processor started for garage_outside: 295
2023-03-21 17:54:58.065876296 [2023-03-21 13:54:58] frigate.app INFO : Camera processor started for porch: 297
2023-03-21 17:54:58.075386894 [2023-03-21 13:54:58] frigate.app INFO : Camera processor started for backyard: 299
2023-03-21 17:54:58.084745602 [2023-03-21 13:54:58] frigate.app INFO : Camera processor started for garage_inside: 300
2023-03-21 17:54:58.093886931 [2023-03-21 13:54:58] frigate.app INFO : Camera processor started for under_deck: 302
2023-03-21 17:54:58.132060130 [2023-03-21 13:54:58] frigate.app INFO : Capture process started for front_door: 303
2023-03-21 17:54:58.141555383 [2023-03-21 13:54:58] frigate.app INFO : Capture process started for garage_outside: 304
2023-03-21 17:54:58.151358636 [2023-03-21 13:54:58] frigate.app INFO : Capture process started for porch: 305
2023-03-21 17:54:58.169875845 [2023-03-21 13:54:58] frigate.app INFO : Capture process started for backyard: 307
2023-03-21 17:54:58.183059738 [2023-03-21 13:54:58] frigate.app INFO : Capture process started for garage_inside: 311
2023-03-21 17:54:58.194230344 [2023-03-21 13:54:58] frigate.app INFO : Capture process started for under_deck: 312
2023-03-21 17:54:58.536163778 [2023-03-21 13:54:58] frigate.app WARNING : The current SHM size of 67.1MB is too small, recommend increasing it to at least 102.59999999999998MB.
2023-03-21 17:54:59.486560500 [2023-03-21 13:54:59] frigate.util ERROR : Unable to poll intel GPU stats: Failed to detect engines! (No such file or directory)
2023-03-21 17:54:59.486568655 (Kernel 4.16 or newer is required for i915 PMU support.)
2023-03-21 17:54:59.486570402
2023-03-21 17:54:59.596138977 [2023-03-21 13:54:59] frigate.detectors.plugins.openvino INFO : Model Input Shape: {1, 300, 300, 3}
2023-03-21 17:54:59.601609966 [2023-03-21 13:54:59] frigate.detectors.plugins.openvino INFO : Model Output-0 Shape: {1, 1, 100, 7}
2023-03-21 17:54:59.602612739 [2023-03-21 13:54:59] frigate.detectors.plugins.openvino INFO : Model has 1 Output Tensors
Looks like the driver does not support GPU stats for your CPU model
When running in a container you need to add the CAP_PERFMON
capability for this to work. Your host also needs to be on Kernel 5.8 or above.
Here is a Kubernetes example that works for me:
apiVersion: apps/v1
kind: Deployment
metadata:
name: frigate
namespace: default
labels:
app: frigate
spec:
selector:
matchLabels:
app: frigate
template:
metadata:
labels:
app: frigate
spec:
containers:
- name: frigate
image: ghcr.io/blakeblackshear/frigate:0.12.0-beta10
imagePullPolicy: Always
securityContext:
capabilities:
add: ["PERFMON"]
resources:
limits:
gpu.intel.com/i915: 1
cpu: 500m
memory: 1Gi
env:
- name: FRIGATE_RTSP_PASSWORD
value: "ThisIsFakeObviously!"
ports:
# http
- containerPort: 5000
# rtmp
- containerPort: 1935
volumeMounts:
- name: data
mountPath: /media/frigate
- name: cache
mountPath: /tmp/cache
- name: config
mountPath: /config/
- name: shm
mountPath: /dev/shm
volumes:
- name: data
hostPath:
path: /data/frigate
type: DirectoryOrCreate
- name: cache
emptyDir:
sizeLimit: 1Gi
- name: config
configMap:
name: frigate-config
- emptyDir:
medium: Memory
sizeLimit: 128Mi
name: shm
restartPolicy: Always
On Docker it should be (I didn't test):
--cap-add=PERFMON
Docker Compose should be:
cap_add:
- PERFMON
We should probably add this to the documentation
When running in a container you need to add the
CAP_PERFMON
capability for this to work. Your host also needs to be on Kernel 5.8 or above.Here is a Kubernetes example that works for me:
apiVersion: apps/v1 kind: Deployment metadata: name: frigate namespace: default labels: app: frigate spec: selector: matchLabels: app: frigate template: metadata: labels: app: frigate spec: containers: - name: frigate image: ghcr.io/blakeblackshear/frigate:0.12.0-beta10 imagePullPolicy: Always securityContext: capabilities: add: ["PERFMON"] resources: limits: gpu.intel.com/i915: 1 cpu: 500m memory: 1Gi env: - name: FRIGATE_RTSP_PASSWORD value: "ThisIsFakeObviously!" ports: # http - containerPort: 5000 # rtmp - containerPort: 1935 volumeMounts: - name: data mountPath: /media/frigate - name: cache mountPath: /tmp/cache - name: config mountPath: /config/ - name: shm mountPath: /dev/shm volumes: - name: data hostPath: path: /data/frigate type: DirectoryOrCreate - name: cache emptyDir: sizeLimit: 1Gi - name: config configMap: name: frigate-config - emptyDir: medium: Memory sizeLimit: 128Mi name: shm restartPolicy: Always
On Docker it should be (I didn't test):
--cap-add=PERFMON
Docker Compose should be:
cap_add: - PERFMON
We should probably add this to the documentation
That's what it was for HA OS so makes sense. PRs always welcome
Looks like the driver does not support GPU stats for your CPU model
Thanks @NickM-27. I can live with it as long as OpenVINO is working for my GPU. But just want to clarify two things before I close this issue:
Kernel 4.16 or newer in my log: what does the kernel refer to? OpenVINO driver? What kind of Frigate docker is (LXC or LXD)? And what version? I am not familiar with docker business.
Looks like the driver does not support GPU stats for your CPU model
Thanks @NickM-27. I can live with it as long as OpenVINO is working for my GPU. But just want to clarify two things before I close this issue:
Kernel 4.16 or newer in my log: what does the kernel refer to? OpenVINO driver?
Thst would be Linux kernel
What kind of Frigate docker is (LXC or LXD)? And what version? I am not familiar with docker business.
Frigate is published as a docker container. LXC / LXD is related to proxmox and depends how it is installed, that's unrelated to frigate AFAIK
@jav
When running in a container you need to add the
CAP_PERFMON
capability for this to work. Your host also needs to be on Kernel 5.8 or above.cap_add: - PERFMON
Thanks. I'll see what PERFMON does. I don't have it now and it seems working. What will happen if I don't have this?
@jav
When running in a container you need to add the
CAP_PERFMON
capability for this to work. Your host also needs to be on Kernel 5.8 or above.cap_add: - PERFMON
Thanks. I'll see what PERFMON does. I don't have it now and it seems working. What will happen if I don't have this?
I think you misunderstand, that user is referring to your GPU stats which you just said are not working?
Looks like the driver does not support GPU stats for your CPU model
Thanks @NickM-27. I can live with it as long as OpenVINO is working for my GPU. But just want to clarify two things before I close this issue: Kernel 4.16 or newer in my log: what does the kernel refer to? OpenVINO driver?
Thst would be Linux kernel
What kind of Frigate docker is (LXC or LXD)? And what version? I am not familiar with docker business.
Frigate is published as a docker container. LXC / LXD is related to proxmox and depends how it is installed, that's unrelated to frigate AFAIK
Ok, thanks @NickM-27
@jav
When running in a container you need to add the
CAP_PERFMON
capability for this to work. Your host also needs to be on Kernel 5.8 or above.cap_add: - PERFMON
Thanks. I'll see what PERFMON does. I don't have it now and it seems working. What will happen if I don't have this?
I think you misunderstand, that user is referring to your GPU stats which you just said are not working?
Oh, sorry for not being clear. By saying 'working', I meant Frigate is working. Sure, GPU stats is not :)
Anyone have idea how to make it work in rootless container?
Running on Kernel 6.5.11-8-pve with Intel i915 GPU in privileged container on Proxmox. intel_gpu_top
runs in container without issue, but GPU returns error as above. I've added:
cap_add:
- PERFMON
to the docker-compose.yml file but it doesn't seem to make a difference. Any thoughts?
Some i965 don't support Intel GPU stats
Some i965 don't support Intel GPU stats
I have an i915, not i965. Does same apply? as mentioned, not problem running intel_gpu_top from command line which returns the stats without a problem.
Some i965 don't support Intel GPU stats
I have an i915, not i965. Does same apply? as mentioned, not problem running intel_gpu_top from command line which returns the stats without a problem.
Just tried some more tests. Seems that intel_gpu_top does not run in the frigate container and this is where the error comes from. Running intel_gpu_top sys:/sys/devices/pci0000:00/0000:00:02.0
or intel_gpu_top
from the host command line works.
i915 is the same as iHD
in any case, intel_gpu_top is very finicky and it can work on the host but not work in docker container. For example it does not work on any synology devices.
I found that in addition to adding the "cap_add: - PERFMON" to docker compose config as discussed in this issue (but with privileged: false), I also needed to set /proc/sys/kernel/perf_event_paranoid to 3 (was set to 4 by default) on the host Linux OS.
This seems to also be discussed here: https://github.com/blakeblackshear/frigate/issues/8494 but in that case under HassOS, and needed to be changed from 3 to 2 it seems.
Hope this helps someone.
I see this under Systems>GPUS from Frigate web UI: error-gpu There was an error getting usage stats. This does not mean hardware acceleration is not working. Either your GPU does not support this or Frigate does not have proper access to get statistics. This is expected for the Home Assistant addon.
I am using OpenVINO as the detector (set to AUTO in the config file). Under System > Detectors, I see a CPU percentage listed under 'OV' and an inference speed of ~43 ms. Does this mean it still uses CPU to do the inference? But, from the sample speed under https://deploy-preview-4055--frigate-docs.netlify.app/frigate/hardware. Looks 43 ms is not too bad when checking the data for J4105 CPU. So probably it does use the GPU to do the inferencing.
Version
0.12.0-E454DAF
Frigate config file
docker-compose file or Docker CLI command
Relevant log output
Operating system
Other Linux
Install method
Docker CLI
Coral version
CPU (no coral)
Any other information that may be helpful
Intel Celeron J3455 CPU with integrated HD Graphics 500 GPU.