search

blakeblackshear / frigate

NVR with realtime local object detection for IP cameras
https://frigate.video
MIT License
17.31k stars 1.6k forks source link

How to optimize memory needed by ffmpeg? #770

Closed lukaszolek closed 3 years ago

lukaszolek commented 3 years ago

Hi,

I have now a successful setup of 2 cameras to my frigate. Needed to set /boot/config.txt gpu_mem to 512.

There are no issues in frigate log:

 * Starting nginx nginx
   ...done.
frigate.app                    INFO    : Creating directory: /tmp/cache
frigate.app                    INFO    : Creating tmpfs of size 256m
frigate.app                    WARNING : Camera garage has rtmp enabled, but rtmp is not assigned to an input.
frigate.app                    WARNING : Camera garden_front has rtmp enabled, but rtmp is not assigned to an input.
Starting migrations
peewee_migrate                 INFO    : Starting migrations
There is nothing to migrate
peewee_migrate                 INFO    : There is nothing to migrate
frigate.mqtt                   INFO    : MQTT connected
detector.coral                 INFO    : Starting detection process: 35
frigate.app                    INFO    : Camera processor started for garage: 38
frigate.edgetpu                INFO    : Attempting to load TPU as usb
frigate.app                    INFO    : Camera processor started for garden_front: 39
frigate.app                    INFO    : Capture process started for garage: 41
frigate.app                    INFO    : Capture process started for garden_front: 43
frigate.edgetpu                INFO    : TPU found

These are two streams of video: h264 (High), yuvj420p(pc, bt709, progressive), 1280x720 [SAR 1:1 DAR 16:9], 3 fps, 50 tbr, 90k tbn, 6 tbc

However, once I try to add third stream to my configuration (I have 6 cameras in total) frigate starts throwing errors:

 * Starting nginx nginx
   ...done.
frigate.app                    INFO    : Creating directory: /tmp/cache
frigate.app                    INFO    : Creating tmpfs of size 256m
frigate.app                    WARNING : Camera garage has rtmp enabled, but rtmp is not assigned to an input.
frigate.app                    WARNING : Camera garden_front has rtmp enabled, but rtmp is not assigned to an input.
frigate.app                    WARNING : Camera tarrace_front has rtmp enabled, but rtmp is not assigned to an input.
Starting migrations
peewee_migrate                 INFO    : Starting migrations
There is nothing to migrate
peewee_migrate                 INFO    : There is nothing to migrate
detector.coral                 INFO    : Starting detection process: 36
frigate.edgetpu                INFO    : Attempting to load TPU as usb
frigate.app                    INFO    : Camera processor started for garage: 39
frigate.app                    INFO    : Camera processor started for garden_front: 41
frigate.app                    INFO    : Camera processor started for tarrace_front: 43
frigate.app                    INFO    : Capture process started for garage: 45
frigate.app                    INFO    : Capture process started for garden_front: 46
frigate.app                    INFO    : Capture process started for tarrace_front: 48
frigate.mqtt                   INFO    : MQTT connected
frigate.edgetpu                INFO    : TPU found
frigate.video                  INFO    : garden_front: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
frigate.video                  INFO    : garden_front: ffmpeg process is not running. exiting capture thread...
frigate.video                  INFO    : garage: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
frigate.video                  INFO    : garage: ffmpeg process is not running. exiting capture thread...
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x1fe4ce0] moov atom not found
/tmp/cache/garage-20210210110414.mp4: Invalid data found when processing input
frigate.events                 INFO    : bad file: garage-20210210110414.mp4
ffmpeg.garage.detect           ERROR   : [h264_v4l2m2m @ 0xc42040] capture VIDIOC_REQBUFS failed: Cannot allocate memory
ffmpeg.garage.detect           ERROR   : [h264_v4l2m2m @ 0xc42040] can't request capture buffers
ffmpeg.garage.detect           ERROR   : Error while decoding stream #0:0: Cannot allocate memory
ffmpeg.garage.detect           ERROR   : [h264_v4l2m2m @ 0xc42040] capture VIDIOC_REQBUFS failed: Cannot allocate memory
ffmpeg.garage.detect           ERROR   : [h264_v4l2m2m @ 0xc42040] can't request capture buffers
ffmpeg.garage.detect           ERROR   : Error while decoding stream #0:0: Cannot allocate memory
ffmpeg.garage.detect           ERROR   : [h264_v4l2m2m @ 0xc42040] capture VIDIOC_REQBUFS failed: Cannot allocate memory
ffmpeg.garage.detect           ERROR   : [h264_v4l2m2m @ 0xc42040] can't request capture buffers
ffmpeg.garage.detect           ERROR   : Error while decoding stream #0:0: Cannot allocate memory
ffmpeg.garage.detect           ERROR   : [h264_v4l2m2m @ 0xc42040] capture VIDIOC_REQBUFS failed: Cannot allocate memory
ffmpeg.garage.detect           ERROR   : [h264_v4l2m2m @ 0xc42040] can't request capture buffers
ffmpeg.garage.detect           ERROR   : Error while decoding stream #0:0: Cannot allocate memory
ffmpeg.garage.detect           ERROR   : [h264_v4l2m2m @ 0xc42040] capture VIDIOC_REQBUFS failed: Cannot allocate memory
ffmpeg.garage.detect           ERROR   : [h264_v4l2m2m @ 0xc42040] can't request capture buffers
ffmpeg.garage.detect           ERROR   : Error while decoding stream #0:0: Cannot allocate memory
ffmpeg.garage.detect           ERROR   : [h264_v4l2m2m @ 0xc42040] capture VIDIOC_REQBUFS failed: Cannot allocate memory
ffmpeg.garage.detect           ERROR   : [h264_v4l2m2m @ 0xc42040] can't request capture buffers
ffmpeg.garage.detect           ERROR   : Error while decoding stream #0:0: Cannot allocate memory
ffmpeg.garage.detect           ERROR   : [h264_v4l2m2m @ 0xc42040] capture VIDIOC_REQBUFS failed: Cannot allocate memory
ffmpeg.garage.detect           ERROR   : [h264_v4l2m2m @ 0xc42040] can't request capture buffers
ffmpeg.garage.detect           ERROR   : Error while decoding stream #0:0: Cannot allocate memory
ffmpeg.garage.detect           ERROR   : [h264_v4l2m2m @ 0xc42040] capture VIDIOC_REQBUFS failed: Cannot allocate memory
ffmpeg.garage.detect           ERROR   : [h264_v4l2m2m @ 0xc42040] can't request capture buffers
ffmpeg.garage.detect           ERROR   : Error while decoding stream #0:0: Cannot allocate memory
ffmpeg.garage.detect           ERROR   : Error while decoding stream #0:0: Resource temporarily unavailable
ffmpeg.garage.detect           ERROR   : malloc_consolidate(): invalid chunk size
ffmpeg.garden_front.detect     ERROR   : [h264_v4l2m2m @ 0xa650b0] output VIDIOC_REQBUFS failed: Cannot allocate memory
ffmpeg.garden_front.detect     ERROR   : [h264_v4l2m2m @ 0xa650b0] no v4l2 output context's buffers
ffmpeg.garden_front.detect     ERROR   : [h264_v4l2m2m @ 0xa650b0] can't configure decoder
ffmpeg.garden_front.detect     ERROR   : Error while opening decoder for input stream #0:0 : Inappropriate ioctl for device

I tried to raise gpu_mem higher, but then the Raspberry doesn't start properly (is stuck during boot).

Is there any way to use hwaccel_args to optimize the memory needed by ffmpeg?

blakeblackshear commented 3 years ago

Can you post your config?

lukaszolek commented 3 years ago

When I comment out hwaccel_args everything works fine, except high CPU usage.

detectors:
  coral:
    type: edgetpu
    device: usb

mqtt:
  host: 192.168.0.200
  user: frigate
  password: 98gfk349dfk309w

ffmpeg:
  hwaccel_args:
    - -c:v
    - h264_v4l2m2m
cameras:
  garage:
    ffmpeg:
      inputs:
        - path: rtsp://***:***@192.168.0.33:554/cam/realmonitor?channel=1&subtype=0&unicast=true
          roles:
            - detect
            - clips
    width: 1280
    height: 720
    fps: 5
    clips:
      enabled: True
      pre_capture: 2
      post_capture: 2
      objects:
        - person
        - car
        - cat
        - dog
      retain:
        default: 30
  garage_door:
    ffmpeg:
      inputs:
        - path: rtsp://***:***@192.168.0.33:554/cam/realmonitor?channel=2&subtype=0&unicast=true
          roles:
            - detect
            - clips
    width: 1280
    height: 720
    fps: 5
    clips:
      enabled: True
      pre_capture: 2
      post_capture: 2
      objects:
        - person
        - car
        - cat
        - dog
      retain:
        default: 30

  garden_front:
    ffmpeg:
      inputs:
        - path: rtsp://***:***@192.168.0.33:554/cam/realmonitor?channel=3&subtype=0&unicast=true
          roles:
            - detect
            - clips
    width: 1280
    height: 720
    fps: 5
    clips:
      enabled: True
      pre_capture: 2
      post_capture: 2
      objects:
        - person
        - car
        - cat
        - dog
      retain:
        default: 30

  tarrace_front:
    ffmpeg:
      inputs:
        - path: rtsp://***:***@192.168.0.33:554/cam/realmonitor?channel=5&subtype=0&unicast=true
          roles:
            - detect
            - clips
    width: 1280
    height: 720
    fps: 5
    clips:
      enabled: True
      pre_capture: 2
      post_capture: 2
      objects:
        - person
        - car
        - cat
        - dog
      retain:
        default: 30

  tarrace_back:
    ffmpeg:
      inputs:
        - path: rtsp://***:***@192.168.0.33:554/cam/realmonitor?channel=4&subtype=0&unicast=true
          roles:
            - detect
            - clips
    width: 1280
    height: 720
    fps: 5
    clips:
      enabled: True
      pre_capture: 2
      post_capture: 2
      objects:
        - person
        - car
        - cat
        - dog
      retain:
        default: 30       
  garden_back:
    ffmpeg:
      inputs:
        - path: rtsp://***:***@192.168.0.33:554/cam/realmonitor?channel=6&subtype=0&unicast=true
          roles:
            - detect
            - clips
    width: 1280
    height: 720
    fps: 5
    clips:
      enabled: True
      pre_capture: 2
      post_capture: 2
      objects:
        - person
        - car
        - cat
        - dog
      retain:
        default: 30         

clips:
  max_seconds: 300
  tmpfs_cache_size: 256m
  retain:
    default: 30
objects:
  track:
    - person
    - car
    - cat
    - dog
  filters:
    person:
      min_area: 200
      max_area: 200000
      min_score: 0.5
      threshold: 0.7
    car:
      min_area: 200
      max_area: 200000
      min_score: 0.5
      threshold: 0.7
    cat:
      min_area: 200
      max_area: 200000
      min_score: 0.5
      threshold: 0.7
    dog:
      min_area: 200
      max_area: 200000
      min_score: 0.5
      threshold: 0.7

motion:
  threshold: 10
  contour_area: 100
  delta_alpha: 0.2
  frame_alpha: 0.2
  frame_height: 180
blakeblackshear commented 3 years ago

I don't know why it is requiring so much memory. Are using an RPi4 with the 64bit OS?

hawkeye217 commented 3 years ago

I saw the same thing with ffmpeg and hardware acceleration on a Pi4 when using a large resolution camera for the detect role. Changing the gpu_mem parameter didn't have an effect, either. I had to use a lower resolution substream and that solved the issue.

Edit: The setup has 3 cams with detect roles at 640x480 and clips role at 4k. Seems like an ffmpeg issue more than a Frigate one.

lukaszolek commented 3 years ago

@blakeblackshear Yes, RPi4, but I'm not sure how to check which wersion. However acceleration for 32bit raised errors always, and 64bit settings for hwaccel_args were working for 2 streams, and failing for more.

eloebl commented 3 years ago

Glad I saw this, just set this up last night in my Pi4 kubernetes cluster using helm. I have node affinity enabled as a Coral TPU installed on this node. The node has 8gb of RAM, but keeping getting ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures errors.

      width: 3072
      height: 1728
      fps: 5

I assume the camera's resolution is too high, and I should go off a lower resolution stream off the camera?

EDIT: Also I do believe it's seeing the TPU:

frigate.mqtt                   INFO    : MQTT connected
detector.coral                 INFO    : Starting detection process: 34
frigate.edgetpu                INFO    : Attempting to load TPU as usb
frigate.edgetpu                INFO    : TPU found

EDIT2: Nevermind didn't work, dropped the stream do the 640x480 substream and set the width/height, and still having the error and it terminating.

HSB-collab commented 3 years ago

I am trying to use hardware acceleration on a Raspberry Pi 4 (it is a model B). I have put the following configuration parameters into the frigate.yml file as per the documentation: ``ffmpeg: hwaccel_args:

Error fetching information from http://ccab4aaf-frigate:5000/api/stats - Server disconnected Error fetching information from http://ccab4aaf-frigate:5000/api/stats - 502, message='Bad Gateway', url=URL('http://ccab4aaf-frigate:5000/api/stats')``

Any ideas?

stale[bot] commented 3 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

HSB-collab commented 3 years ago

I do not believe this issue is solved because like @lukaszolek and @mloebl I cannot get hwaccel to work on the Pi4 (64bit with 4GByte memory). However, I was able to get 7 cameras working with Frigate without hardware acceleration by dropping the frame rate on four of my cameras (which are configurable via a web interface - Sercomm iCamera2) to 5 fps. Their resolution is 1280x720 (which I did not change). My other 3 cameras are Wyze cameras with the RTSP firmware and there is no way to change their resolution or fps. Despite the fact that the Wyze cameras are working (I am able to detect people, record clips and see live streaming) I am getting errors from the 3 Wyze cameras (but not the Sercomm ones) as follows:

ffmpeg.Camera7_Wyze_2.detect   ERROR   : Guessed Channel Layout for Input Stream #0.1 : mono
ffmpeg.Camera7_Wyze_2.detect   ERROR   : [h264 @ 0x558d4809e0] cabac decode of qscale diff failed at 68 59
ffmpeg.Camera7_Wyze_2.detect   ERROR   : [h264 @ 0x558d4809e0] error while decoding MB 68 59, bytestream -4
ffmpeg.Camera7_Wyze_2.detect   ERROR   : rtsp://****:****@10.0.0.126/live: corrupt decoded frame in stream 0
ffmpeg.Camera7_Wyze_2.detect   ERROR   : [flv @ 0x558d4b39e0] Failed to update header with correct duration.
ffmpeg.Camera7_Wyze_2.detect   ERROR   : [flv @ 0x558d4b39e0] Failed to update header with correct filesize.

I would really love to solve this.

blakeblackshear commented 3 years ago

@lukaszolek can you try adding -fflags nobuffer to your hwaccel args like this:

ffmpeg:
  hwaccel_args:
    - -c:v
    - h264_v4l2m2m
    - -fflags
    - nobuffer

I am curious if that will reduce the gpu_mem usage for each camera.

eloebl commented 3 years ago

I did figure out my problem was not frigate/ffmpeg related, but couldn't find this issue to update it when I searched for it. TLDR: Was a network issue causing my issues.

In my case I had my cameras behind a linux box acting as a quick and dirty router to access cameras behind it on a different subnet (it has 2 nics and was the nvr box for the cameras). A few years ago had initially wanted to isolate the cameras to their own subnet off the regular network, but as time evolved and issues with the nvr and it's api access, I wanted direct access to the cameras to work with HA. In the end it appears I had some sort of misconfiguration causing routing to flake a bit. I hadn't noticed this as "seemed to be working fine" until I did some basic pings and saw some redirect errors on responses. I moved the cameras to an unused port on my firewall so I could isolate them there, and my ffpmeg errors went away. It's been very stable and no errors at all in my frigate logs anymore. I'm still running it on a PI4 in k8s, but now a k3s cluster with a Google Coral USB TPU.

stale[bot] commented 3 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.