search

blakeblackshear / frigate

NVR with realtime local object detection for IP cameras
https://frigate.video
MIT License
18.87k stars 1.72k forks source link

No GPU information displayed #5795

Closed my-umd closed 1 year ago

my-umd commented 1 year ago

I see this under Systems>GPUS from Frigate web UI: error-gpu There was an error getting usage stats. This does not mean hardware acceleration is not working. Either your GPU does not support this or Frigate does not have proper access to get statistics. This is expected for the Home Assistant addon.

I am using OpenVINO as the detector (set to AUTO in the config file). Under System > Detectors, I see a CPU percentage listed under 'OV' and an inference speed of ~43 ms. Does this mean it still uses CPU to do the inference? But, from the sample speed under https://deploy-preview-4055--frigate-docs.netlify.app/frigate/hardware. Looks 43 ms is not too bad when checking the data for J4105 CPU. So probably it does use the GPU to do the inferencing.

Version

0.12.0-E454DAF

Frigate config file

mqtt:
  host: 192.168.xxx.xxx

detectors:
  ov:
    type: openvino
    device: AUTO
    model:
      path: /openvino-model/ssdlite_mobilenet_v2.xml

model:
  width: 300
  height: 300
  input_tensor: nhwc
  input_pixel_format: bgr
  labelmap_path: /openvino-model/coco_91cl_bkgr.txt

birdseye:
  enabled: True
  mode: continuous

ffmpeg:
  hwaccel_args: -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format yuv420p
  output_args:
    # Optional: output args for detect streams (default: shown below)
    # detect: -f rawvideo -pix_fmt yuv420p
    # Optional: output args for record streams (default: shown below)
    record: -f segment -segment_time 60 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c:v copy -c:a aac

cameras:
  front_door:
    ffmpeg:
      inputs:
        - path: rtsp://username:password@192.168.xxx.xxx/live
          roles:
            - detect
            - record
            - rtmp
    rtmp:
      enabled: False
    detect:
      enabled: True
    motion:
      mask:
        - 927,720,1246,720,1244,676,926,677
    record:
      enabled: True
      retain:
        days: 60
        mode: all

docker-compose file or Docker CLI command

# created using QNAP container station web UI.  Used runlike (https://github.com/lavie/runlike/) to reverse engineer as something like the following
docker run --name=frigate-0.12.0-beta10 --hostname=3a801f0b39f5 --mac-address=02:42:0a:00:03:02 --env=TZ=America/New_York --env=LIBVA_DRIVER_NAME=i965 --env=FRIGATE_RTSP_PASSWORD=frigate_password --volume=/share/Container/frigate_0.12.0-beta/config:/config:rw --volume=/share/share_vol2/frigate_0.12.0-beta/media:/media/frigate:rw --network=bridge --privileged --workdir=/opt/frigate -p 49176:1935 -p 49175:5000 -p 49177:8554 -p 49178:8555 -p 49178:8555/udp --label='com.qnap.qcs.network.mode=nat' --label='com.qnap.qcs.gpu=False' --log-opt max-size=10m --log-opt max-file=10 --runtime=runc --memory="4194304000" --detach=true -t ghcr.io/blakeblackshear/frigate:0.12.0-beta10

Relevant log output

N/A

Operating system

Other Linux

Install method

Docker CLI

Coral version

CPU (no coral)

Any other information that may be helpful

Intel Celeron J3455 CPU with integrated HD Graphics 500 GPU.

NickM-27 commented 1 year ago

To get GPU stats you must run frigate in docker as privileged. If it is still not working then it is an issue with something on the host OS. intel_gpu_top is very finicky.

If you included the frigate logs that would be helpful to understand what intel_gpu_top is failing

my-umd commented 1 year ago

Thanks @NickM-27. The provided docker run command shows it's running under the privileged mode (sorry, I could have explicitly mentioned the privileged mode to save you time). I can certainly provide logs, but just don't know what part to include. The log is huge.

NickM-27 commented 1 year ago

Thanks @NickM-27. The provided docker run command shows it's running under the privileged mode (sorry, I could have explicitly mentioned the privileged mode to save you time). I can certainly provide logs, but just don't know what part to include. The log is huge.

just paste the whole thing

my-umd commented 1 year ago

It says my log is too long (more than 65536 characters). I only pasted something from the top that I think might be helpful (sorry for the mess. I couldn't find how to add it to the log section once my issue is created):

2023-03-21 17:54:54.700689136  [INFO] Starting Frigate...
2023-03-21 17:54:57.781681302  [2023-03-21 13:54:57] frigate.app                    INFO    : Starting Frigate (0.12.0-e454daf)
2023-03-21 17:54:57.890044958  [2023-03-21 13:54:57] frigate.config                 WARNING : Customizing more than a detector model path is unsupported.
2023-03-21 17:54:57.915757450  [2023-03-21 13:54:57] peewee_migrate                 INFO    : Starting migrations
2023-03-21 17:54:57.956348058  [2023-03-21 13:54:57] peewee_migrate                 INFO    : There is nothing to migrate
2023-03-21 17:54:57.971795627  [2023-03-21 13:54:57] frigate.comms.dispatcher       ERROR   : Received invalid set command: backyard/recordings/set
2023-03-21 17:54:57.971803887  [2023-03-21 13:54:57] frigate.comms.dispatcher       ERROR   : Received invalid set command: backyard/snapshots/set
2023-03-21 17:54:57.971812557  [2023-03-21 13:54:57] frigate.comms.dispatcher       ERROR   : Received invalid set command: front_door/recordings/set
2023-03-21 17:54:57.971815945  [2023-03-21 13:54:57] frigate.comms.dispatcher       ERROR   : Received invalid set command: front_door/snapshots/set
2023-03-21 17:54:57.971851279  [2023-03-21 13:54:57] frigate.comms.dispatcher       ERROR   : Received invalid set command: porch/recordings/set
2023-03-21 17:54:57.972776515  [2023-03-21 13:54:57] frigate.comms.dispatcher       ERROR   : Received invalid set command: porch/snapshots/set
2023-03-21 17:54:58.011811607  [2023-03-21 13:54:58] detector.ov                    INFO    : Starting detection process: 290
2023-03-21 17:54:58.027216533  [2023-03-21 13:54:58] frigate.app                    INFO    : Output process started: 292
2023-03-21 17:54:58.027225099  [2023-03-21 13:54:58] frigate.app                    INFO    : Camera processor started for front_door: 294
2023-03-21 17:54:58.051078025  [2023-03-21 13:54:58] frigate.app                    INFO    : Camera processor started for garage_outside: 295
2023-03-21 17:54:58.065876296  [2023-03-21 13:54:58] frigate.app                    INFO    : Camera processor started for porch: 297
2023-03-21 17:54:58.075386894  [2023-03-21 13:54:58] frigate.app                    INFO    : Camera processor started for backyard: 299
2023-03-21 17:54:58.084745602  [2023-03-21 13:54:58] frigate.app                    INFO    : Camera processor started for garage_inside: 300
2023-03-21 17:54:58.093886931  [2023-03-21 13:54:58] frigate.app                    INFO    : Camera processor started for under_deck: 302
2023-03-21 17:54:58.132060130  [2023-03-21 13:54:58] frigate.app                    INFO    : Capture process started for front_door: 303
2023-03-21 17:54:58.141555383  [2023-03-21 13:54:58] frigate.app                    INFO    : Capture process started for garage_outside: 304
2023-03-21 17:54:58.151358636  [2023-03-21 13:54:58] frigate.app                    INFO    : Capture process started for porch: 305
2023-03-21 17:54:58.169875845  [2023-03-21 13:54:58] frigate.app                    INFO    : Capture process started for backyard: 307
2023-03-21 17:54:58.183059738  [2023-03-21 13:54:58] frigate.app                    INFO    : Capture process started for garage_inside: 311
2023-03-21 17:54:58.194230344  [2023-03-21 13:54:58] frigate.app                    INFO    : Capture process started for under_deck: 312
2023-03-21 17:54:58.536163778  [2023-03-21 13:54:58] frigate.app                    WARNING : The current SHM size of 67.1MB is too small, recommend increasing it to at least 102.59999999999998MB.
2023-03-21 17:54:59.486560500  [2023-03-21 13:54:59] frigate.util                   ERROR   : Unable to poll intel GPU stats: Failed to detect engines! (No such file or directory)
2023-03-21 17:54:59.486568655  (Kernel 4.16 or newer is required for i915 PMU support.)
2023-03-21 17:54:59.486570402  
2023-03-21 17:54:59.596138977  [2023-03-21 13:54:59] frigate.detectors.plugins.openvino INFO    : Model Input Shape: {1, 300, 300, 3}
2023-03-21 17:54:59.601609966  [2023-03-21 13:54:59] frigate.detectors.plugins.openvino INFO    : Model Output-0 Shape: {1, 1, 100, 7}
2023-03-21 17:54:59.602612739  [2023-03-21 13:54:59] frigate.detectors.plugins.openvino INFO    : Model has 1 Output Tensors
NickM-27 commented 1 year ago

Looks like the driver does not support GPU stats for your CPU model

javydekoning commented 1 year ago

When running in a container you need to add the CAP_PERFMON capability for this to work. Your host also needs to be on Kernel 5.8 or above.

Here is a Kubernetes example that works for me:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: frigate
  namespace: default
  labels:
    app: frigate
spec:
  selector:
    matchLabels:
      app: frigate
  template:
    metadata:
      labels:
        app: frigate
    spec:
      containers:
        - name: frigate
          image: ghcr.io/blakeblackshear/frigate:0.12.0-beta10
          imagePullPolicy: Always
          securityContext:
            capabilities:
              add: ["PERFMON"]
          resources:
            limits:
              gpu.intel.com/i915: 1
              cpu: 500m
              memory: 1Gi
          env:
            - name: FRIGATE_RTSP_PASSWORD
              value: "ThisIsFakeObviously!"
          ports:
            # http
            - containerPort: 5000
            # rtmp
            - containerPort: 1935
          volumeMounts:
            - name: data
              mountPath: /media/frigate
            - name: cache
              mountPath: /tmp/cache
            - name: config
              mountPath: /config/
            - name: shm
              mountPath: /dev/shm
      volumes:
        - name: data
          hostPath:
            path: /data/frigate
            type: DirectoryOrCreate
        - name: cache
          emptyDir:
            sizeLimit: 1Gi
        - name: config
          configMap:
            name: frigate-config
        - emptyDir:
            medium: Memory
            sizeLimit: 128Mi
          name: shm
      restartPolicy: Always

On Docker it should be (I didn't test): 

--cap-add=PERFMON

Docker Compose should be:

cap_add:
  - PERFMON

We should probably add this to the documentation

NickM-27 commented 1 year ago

When running in a container you need to add the CAP_PERFMON capability for this to work. Your host also needs to be on Kernel 5.8 or above.

Here is a Kubernetes example that works for me:


apiVersion: apps/v1

kind: Deployment

metadata:

  name: frigate

  namespace: default

  labels:

    app: frigate

spec:

  selector:

    matchLabels:

      app: frigate

  template:

    metadata:

      labels:

        app: frigate

    spec:

      containers:

        - name: frigate

          image: ghcr.io/blakeblackshear/frigate:0.12.0-beta10

          imagePullPolicy: Always

          securityContext:

            capabilities:

              add: ["PERFMON"]

          resources:

            limits:

              gpu.intel.com/i915: 1

              cpu: 500m

              memory: 1Gi

          env:

            - name: FRIGATE_RTSP_PASSWORD

              value: "ThisIsFakeObviously!"

          ports:

            # http

            - containerPort: 5000

            # rtmp

            - containerPort: 1935

          volumeMounts:

            - name: data

              mountPath: /media/frigate

            - name: cache

              mountPath: /tmp/cache

            - name: config

              mountPath: /config/

            - name: shm

              mountPath: /dev/shm

      volumes:

        - name: data

          hostPath:

            path: /data/frigate

            type: DirectoryOrCreate

        - name: cache

          emptyDir:

            sizeLimit: 1Gi

        - name: config

          configMap:

            name: frigate-config

        - emptyDir:

            medium: Memory

            sizeLimit: 128Mi

          name: shm

      restartPolicy: Always

On Docker it should be (I didn't test): 


--cap-add=PERFMON

Docker Compose should be:


cap_add:

  - PERFMON

We should probably add this to the documentation

That's what it was for HA OS so makes sense. PRs always welcome

my-umd commented 1 year ago

Looks like the driver does not support GPU stats for your CPU model

Thanks @NickM-27. I can live with it as long as OpenVINO is working for my GPU. But just want to clarify two things before I close this issue:

Kernel 4.16 or newer in my log: what does the kernel refer to? OpenVINO driver? What kind of Frigate docker is (LXC or LXD)? And what version? I am not familiar with docker business.

NickM-27 commented 1 year ago

Looks like the driver does not support GPU stats for your CPU model

Thanks @NickM-27. I can live with it as long as OpenVINO is working for my GPU. But just want to clarify two things before I close this issue:

Kernel 4.16 or newer in my log: what does the kernel refer to? OpenVINO driver?

Thst would be Linux kernel

What kind of Frigate docker is (LXC or LXD)? And what version? I am not familiar with docker business.

Frigate is published as a docker container. LXC / LXD is related to proxmox and depends how it is installed, that's unrelated to frigate AFAIK

my-umd commented 1 year ago

@jav

When running in a container you need to add the CAP_PERFMON capability for this to work. Your host also needs to be on Kernel 5.8 or above.

cap_add:
  - PERFMON

Thanks. I'll see what PERFMON does. I don't have it now and it seems working. What will happen if I don't have this?

NickM-27 commented 1 year ago

@jav

When running in a container you need to add the CAP_PERFMON capability for this to work. Your host also needs to be on Kernel 5.8 or above.

cap_add:
  - PERFMON

Thanks. I'll see what PERFMON does. I don't have it now and it seems working. What will happen if I don't have this?

I think you misunderstand, that user is referring to your GPU stats which you just said are not working?

my-umd commented 1 year ago

Looks like the driver does not support GPU stats for your CPU model

Thanks @NickM-27. I can live with it as long as OpenVINO is working for my GPU. But just want to clarify two things before I close this issue: Kernel 4.16 or newer in my log: what does the kernel refer to? OpenVINO driver?

Thst would be Linux kernel

What kind of Frigate docker is (LXC or LXD)? And what version? I am not familiar with docker business.

Frigate is published as a docker container. LXC / LXD is related to proxmox and depends how it is installed, that's unrelated to frigate AFAIK

Ok, thanks @NickM-27

my-umd commented 1 year ago

@jav

When running in a container you need to add the CAP_PERFMON capability for this to work. Your host also needs to be on Kernel 5.8 or above.

cap_add:
  - PERFMON

Thanks. I'll see what PERFMON does. I don't have it now and it seems working. What will happen if I don't have this?

I think you misunderstand, that user is referring to your GPU stats which you just said are not working?

Oh, sorry for not being clear. By saying 'working', I meant Frigate is working. Sure, GPU stats is not :)

sarim commented 1 year ago

Anyone have idea how to make it work in rootless container?

rjblake commented 8 months ago

Running on Kernel 6.5.11-8-pve with Intel i915 GPU in privileged container on Proxmox. intel_gpu_top runs in container without issue, but GPU returns error as above. I've added:

cap_add:
  - PERFMON

to the docker-compose.yml file but it doesn't seem to make a difference. Any thoughts?

NickM-27 commented 8 months ago

Some i965 don't support Intel GPU stats

rjblake commented 8 months ago

Some i965 don't support Intel GPU stats

I have an i915, not i965. Does same apply? as mentioned, not problem running intel_gpu_top from command line which returns the stats without a problem.

rjblake commented 8 months ago

Some i965 don't support Intel GPU stats

I have an i915, not i965. Does same apply? as mentioned, not problem running intel_gpu_top from command line which returns the stats without a problem.

Just tried some more tests. Seems that intel_gpu_top does not run in the frigate container and this is where the error comes from. Running intel_gpu_top sys:/sys/devices/pci0000:00/0000:00:02.0 or intel_gpu_top from the host command line works.

NickM-27 commented 8 months ago

i915 is the same as iHD

in any case, intel_gpu_top is very finicky and it can work on the host but not work in docker container. For example it does not work on any synology devices.

B0ycee commented 6 months ago

I found that in addition to adding the "cap_add: - PERFMON" to docker compose config as discussed in this issue (but with privileged: false), I also needed to set /proc/sys/kernel/perf_event_paranoid to 3 (was set to 4 by default) on the host Linux OS.

This seems to also be discussed here: https://github.com/blakeblackshear/frigate/issues/8494 but in that case under HassOS, and needed to be changed from 3 to 2 it seems.

Hope this helps someone.