search

blakeblackshear / frigate

NVR with realtime local object detection for IP cameras
https://frigate.video
MIT License
19.21k stars 1.76k forks source link

[Support]: Memory leak in AudioManager #7274

Closed YBonline closed 1 year ago

YBonline commented 1 year ago

Describe the problem you are having

I'm experiencing a memory leak in Audio Manager when I have the new audio events enabled. From the best I can tell, the issue seems to be related to when a camera that has audio monitoring enabled, and something causes a disconnect from said camera, it leaks memory, even if it reconnects and continues to work.

My most severe memory leak was last night, when I applied a firmware update to my network switches, gateways, etc along with some cameras that I have, causing all of my cameras to go offline multiple times for short periods of time. Frigate eventually crashed, seemingly with nothing in the logs, but when it consumed all system memory in the frigate.audio_manager process.

Today I'm trying to configure another camera, and while the camera is misbehaving and disconnecting, AudioManager seems to be growing and growing in its memory consumption. Its currently consuming 38.8055GB of memory. CPU time from audio_manager also seems to stay around 110% once the leak starts happening, which is way higher then normal.

I have not seen any memory leak without an associated camera network disconnect, and from my testing, it does not appear to be associated with any particular camera model (although the camera must have audio enabled obviously).

AudioDetector P-ID | CPU % | Avg CPU % | Memory % 1580 | 108.0% | 109% | 36.1%

Version

0.13.0-0FD1EAF

Frigate config file

ui:
  use_experimental: true

database:
  path: /db/frigate.db

birdseye:
  enabled: true
  restream: true
  mode: objects
  width: 1920
  height: 1200

mqtt:
  host: 10.10.22.11
  user: frigate
  password: frigatePASS1

detectors:
  coral1:
    type: edgetpu
    device: pci:0
  coral2:
    type: edgetpu
    device: pci:1

record:
  enabled: true
  retain:
    days: 14
    mode: motion
  events:
    retain:
      default: 45
      mode: active_objects

objects:
  track:
  - person
  - bicycle
  - car
  - cat
  - dog
  filters:
    person:
      # Optional: minimum width*height of the bounding box for the detected object (default: 0)
      #min_area: 500
      # Optional: maximum width*height of the bounding box for the detected object (default: 24000000)
      #max_area: 100000
      # Optional: minimum width/height of the bounding box for the detected object (default: 0)
      #min_ratio: 0.5
      # Optional: maximum width/height of the bounding box for the detected object (default: 24000000)
      #max_ratio: 2.0
      # Optional: minimum score for the object to initiate tracking (default: shown below)
      #min_score: 0.60
      # Optional: minimum decimal percentage for tracked object's computed score to be considered a true positive (default: shown below)
      threshold: 0.75

audio:
  enabled: true
  listen:
  - bark
  - scream
  - speech
  - yell
  - goat
  - chicken
  - skidding
  - tire_squeal
  - emergency_vehicle
  - smoke_detector
  - gunshot
  - glass
  - crack
  - shatter
  - smash
  - breaking

snapshots:
  enabled: true
  clean_copy: true
  timestamp: false
  bounding_box: true
  crop: true
  retain:
    default: 180

go2rtc:
  streams:
 [removed]

  webrtc:
    candidates:
    - 10.10.22.11:8555
    - stun:8555

cameras:
  warehouse_exterior_doorbell:
    ffmpeg:
      inputs:
      - path: rtsp://127.0.0.1:8554/warehouse_exterior_doorbell
        roles:
        - record
        - detect
        - audio
      output_args:
        record: preset-record-generic-audio-aac
      hwaccel_args: preset-nvidia-h264
    detect:
      width: 2560
      height: 1920
    motion:
      mask:
      - 0,0,2560,0,2560,986,2045,956,2034,1016,2406,1054,2332,1121,371,1118,357,910,0,846

  warehouse_exterior_nw:
    ffmpeg:
      inputs:
      - path: rtsp://127.0.0.1:8554/warehouse_exterior_nw
        roles:
        - record
        #- path: rtsp://127.0.0.1:8554/warehouse_exterior_nw_sub
        #  roles:
        - detect
      hwaccel_args: preset-nvidia-h264
    audio:
      enabled: false
    detect:
      width: 1920
      height: 1080

    motion:
      mask:
      - 684,132,1110,125,1114,68,686,75
  warehouse_exterior_sw:
    ffmpeg:
      inputs:
      - path: rtsp://127.0.0.1:8554/warehouse_exterior_sw
        roles:
        - record
      - path: rtsp://127.0.0.1:8554/warehouse_exterior_sw_sub
        roles:
        - detect
      hwaccel_args: preset-nvidia-h264
    audio:
      enabled: false
    detect:
      width: 1920
      height: 1080

    motion:
      mask:
      - 731,75,734,137,1078,132,1086,80
  warehouse_exterior_se:
    ffmpeg:
      inputs:
      - path: rtsp://127.0.0.1:8554/warehouse_exterior_se
        roles:
        - record
      - path: rtsp://127.0.0.1:8554/warehouse_exterior_se_sub
        roles:
        - detect
      output_args:
        record: preset-record-generic-audio-aac
      hwaccel_args: preset-nvidia-h264
    audio:
      enabled: false
    detect:
      width: 480
      height: 360
    motion:
      mask:
      - 480,0,480,55,109,50,0,80,0,0
#    detect:
#      width: 2592
#      height: 1944

  warehouse_exterior_front:
    ffmpeg:
      inputs:
      - path: rtsp://127.0.0.1:8554/warehouse_exterior_front_sub
        roles:
        - record
        #- path: rtsp://127.0.0.1:8554/warehouse_exterior_front_sub
        #  roles:
        - detect
        - audio
      output_args:
        record: preset-record-generic-audio-aac
      hwaccel_args: preset-nvidia-h264
    detect:
      width: 1536
      height: 576
    #birdseye:
    #  enabled: False
    #detect:
    #  width: 4608
    #  height: 1728

    motion:
      mask:
      - 635,0,0,0,0,292,356,141,617,114
  warehouse_workbench:
    ffmpeg:
      inputs:
      - path: rtsp://127.0.0.1:8554/warehouse_workbench
        roles:
        - record
      - path: rtsp://127.0.0.1:8554/warehouse_workbench_sub
        roles:
        - detect
        - audio
      output_args:
        record: preset-record-generic-audio-aac
      hwaccel_args: preset-nvidia-h264
    #detect:
    #  width: 2560
    #  height: 1920
    detect:
      width: 640
      height: 480
    motion:
      mask:
      - 220,480,100,480,98,459,217,453

  warehouse_interior_front:
    ffmpeg:
      inputs:
      - path: rtsp://127.0.0.1:8554/warehouse_interior_front
        roles:
        - record
      - path: rtsp://127.0.0.1:8554/warehouse_interior_front_sub
        roles:
        - detect
        - audio
      output_args:
        record: preset-record-generic-audio-aac
      hwaccel_args: preset-nvidia-h264
    #detect:
    #  width: 2560
    #  height: 1920
    detect:
      width: 640
      height: 480
    motion:
      mask:
      - 376,0,397,31,205,27,205,0

  warehouse_interior_office:
    ffmpeg:
      inputs:
      - path: rtsp://127.0.0.1:8554/warehouse_interior_office
        roles:
        - record
      - path: rtsp://127.0.0.1:8554/warehouse_interior_office_sub
        roles:
        - detect
        - audio
#          input_args: -ss 00:01
      output_args:
        record: preset-record-generic-audio-aac
      hwaccel_args: preset-nvidia-h264
    detect:
      width: 704
      height: 480
    motion:
      mask:
      - 682,21,683,52,418,47,405,22
#    detect:
#      width: 3840
#      height: 2160

  warehouse_interior_overview:
    ffmpeg:
      inputs:
      - path: rtsp://127.0.0.1:8554/warehouse_interior_overview
        roles:
        - record
#        - path: rtsp://127.0.0.1:8554/warehouse_interior_overview_sub
#          roles:
#            - detect
      hwaccel_args: preset-nvidia-h264
    audio:
      enabled: false
    detect:
      width: 1920
      height: 1080
#    detect:
#      width: 704
#      height: 480

  gp_clubhouse_gym_1:
    ffmpeg:
      inputs:
      - path: rtsp://127.0.0.1:8554/gp_clubhouse_gym_1
        roles:
        - record
#        - path: rtsp://127.0.0.1:8554/gp_clubhouse_gym_1_sub
#          roles:
        - detect
      hwaccel_args: preset-nvidia-h265
    detect:
      width: 2592
      height: 1944
#    detect:
#      width: 480
#      height: 360
    audio:
      enabled: false
    motion:
      mask:
      - 2592,173,2592,0,0,0,0,337
    objects:
      filters:
        bicycle:
          mask:
          - 1635,80,1886,1019,408,1059,350,177
        motorcycle:
          mask:
          - 1635,80,1886,1019,408,1059,350,177

  gp_clubhouse_gym_2:
    ffmpeg:
      inputs:
      - path: rtsp://127.0.0.1:8554/gp_clubhouse_gym_2
        roles:
        - record
#        - path: rtsp://127.0.0.1:8554/gp_clubhouse_gym_2_sub
#          roles:
        - detect
      hwaccel_args: preset-nvidia-h265
#    detect:
#      width: 480
#      height: 360
    detect:
      width: 2592
      height: 1944
    audio:
      enabled: false
    motion:
      mask:
      - 2592,0,0,0,0,385
    objects:
      filters:
        bicycle:
          mask:
          - 2391,1369,1261,677,1331,135,2542,274
        motorcycle:
          mask:
          - 2391,1369,1261,677,1331,135,2542,274

  gp_clubhouse_front_door:
    ffmpeg:
      inputs:
      - path: rtsp://127.0.0.1:8554/gp_clubhouse_front_door
        roles:
        - record
      - path: rtsp://127.0.0.1:8554/gp_clubhouse_front_door_sub
        roles:
        - detect
      hwaccel_args: preset-nvidia-h264
    detect:
      width: 480
      height: 360
#    detect:
#      width: 2592
#      height: 1944
    audio:
      enabled: false
    motion:
      mask:
      - 0,0,0,34,265,30,287,0

  gp_clubhouse_back_door:
    ffmpeg:
      inputs:
      - path: rtsp://127.0.0.1:8554/gp_clubhouse_back_door
        roles:
        - record
      - path: rtsp://127.0.0.1:8554/gp_clubhouse_back_door_sub
        roles:
        - detect
      hwaccel_args: preset-nvidia-h264
    detect:
      width: 480
      height: 360
    audio:
      enabled: false
    motion:
      mask:
      - 480,0,480,45,0,56,0,0
#    detect:
#      width: 2592
#      height: 1944

  gp_clubhouse_banquet:
    ffmpeg:
      inputs:
      - path: rtsp://127.0.0.1:8554/gp_clubhouse_banquet
        roles:
        - record
      - path: rtsp://127.0.0.1:8554/gp_clubhouse_banquet_sub
        roles:
        - detect
      hwaccel_args: preset-nvidia-h264
    detect:
      width: 480
      height: 360
    audio:
      enabled: false
    motion:
      mask:
      - 0,0,0,31,277,31,373,0
#    detect:
#      width: 2592
#      height: 1944
  gp_clubhouse_pool_porch:
    ffmpeg:
      inputs:
      - path: rtsp://127.0.0.1:8554/gp_clubhouse_pool_porch
        roles:
        - record
#        - path: rtsp://127.0.0.1:8554/gp_clubhouse_pool_porch_sub
#          roles:
        - detect
      hwaccel_args: preset-nvidia-h264
    zones:
      gp_clubhouse_pool_deck:
        coordinates: 1920,0,1920,1080,0,1080,0,0
    detect:
      width: 1920
      height: 1080
    audio:
      enabled: false
    motion:
      mask:
      - 1000,130,674,122,678,76,993,67
#    detect:
#      width: 704
#      height: 480

  gp_clubhouse_pool:
    ffmpeg:
      inputs:
      - path: rtsp://127.0.0.1:8554/gp_clubhouse_pool
        roles:
        - record
      - path: rtsp://127.0.0.1:8554/gp_clubhouse_pool_sub
        roles:
        - detect
        - audio
      output_args:
        record: preset-record-generic-audio-aac
      hwaccel_args: preset-nvidia-h265
    detect:
      width: 1920
      height: 536

    motion:
      mask:
      - 1920,0,875,0,1037,24,1856,240,1920,202
      - 0,536,551,536,965,80,731,56,0,402
    zones:
      gp_clubhouse_pool_deck:
        coordinates: 636,536,1920,536,1920,317,1607,146,989,27
  gp_clubhouse_firepit:
    ffmpeg:
      inputs:
      - path: rtsp://127.0.0.1:8554/gp_clubhouse_firepit
        roles:
        - record
#        - path: rtsp://127.0.0.1:8554/gp_clubhouse_firepit_sub
#          roles:
        - detect
        - audio
      output_args:
        record: preset-record-generic-audio-aac
      hwaccel_args: preset-nvidia-h265
    motion:
      mask:
      - 2211,0,0,0,0,495,1795,224,1795,67,2202,62
      - 2061,1203,1952,361,1771,361,1702,1182
    objects:
      filters:
        person:
          mask:
          - 2114,1213,1610,1210,1612,366,2080,361
    zones:
      gp_clubhouse_pool_deck:
        coordinates: 3840,2160,3657,599,1531,331,0,725,0,2160
    detect:
      width: 3840
      height: 2160

  gp_clubhouse_parking:
    ffmpeg:
      inputs:
      - path: rtsp://127.0.0.1:8554/gp_clubhouse_parking
        roles:
        - record
      - path: rtsp://admin:campass123@10.10.23.30:554/Streaming/channels/102
        #- path: rtsp://127.0.0.1:8554/gp_clubhouse_parking_sub
        roles:
        - detect
        - audio
      output_args:
        record: preset-record-generic-audio-aac
      hwaccel_args: preset-nvidia-h265
    detect:
      width: 1920
      height: 536
    motion:
      mask:
      - 1341,81,915,23,899,0,1920,0,1920,233
  gp636_doorbell:
    ffmpeg:
      inputs:
      - path: rtsp://127.0.0.1:8554/gp636_doorbell
        roles:
        - record
        - detect
        - audio
      output_args:
        record: preset-record-generic-audio-aac
      hwaccel_args: preset-nvidia-h264
    detect:
      width: 1920
      height: 1080
    motion:
      mask:
      - 0,0,1920,0,1920,60,928,361,926,599,0,499

  gp525_doorbell:
    ffmpeg:
      inputs:
        #- path: rtsp://admin:000000Sk@10.10.30.30:554/cam/realmonitor?channel=1&subtype=0
      - path: rtsp://127.0.0.1:8554/gp525_doorbell
        roles:
        - record
        - detect
        - audio
      output_args:
        record: preset-record-generic-audio-aac
      hwaccel_args: preset-nvidia-h264
    detect:
      width: 2560
      height: 1920
    motion:
      mask:
      - 1641,0,827,995,333,1019,173,861,0,904,0,0

  gp_entrance_75:
    ffmpeg:
      inputs:
      - path: rtsp://127.0.0.1:8554/gp_entrance_75
        roles:
        - record
        - detect
        - audio
      output_args:
        record: preset-record-generic-audio-aac
      hwaccel_args: preset-nvidia-h265
    detect:
      width: 1920
      height: 1080
#    record:
#      retain:
#        days: 5
#        mode: all
#      events:
#        retain:
#          default: 45

    motion:
      mask:
      - 1489,1059,1873,1059,1880,1008,1493,1000
  gp_entrance_archer:
    ffmpeg:
      inputs:
      - path: rtsp://127.0.0.1:8554/gp_entrance_archer
        roles:
        - record
        - detect
        - audio
      output_args:
        record: preset-record-generic-audio-aac
      hwaccel_args: preset-nvidia-h265
    detect:
      width: 1920
      height: 1080
    record:
      retain:
        days: 5
        mode: all
      events:
        retain:
          default: 45

    motion:
      mask:
      - 0,0,0,196,1311,185,1783,0
  gp_bldg_08_front:
    ffmpeg:
      inputs:
      - path: rtsp://127.0.0.1:8554/gp_bldg_08_front
        roles:
        - record
      - path: rtsp://127.0.0.1:8554/gp_bldg_08_front_sub
        roles:
        - detect
        - audio
      output_args:
        record: preset-record-generic-audio-aac
      hwaccel_args: preset-nvidia-h265
    detect:
      width: 1920
      height: 536
#      width: 5120
#      height: 1440
#    zones:
#      gp_bldg_alert:
#        coordinates: 0,961,0,1440,5120,1440,5120,1030,4090,139,1293,0

    motion:
      mask:
      - 0,208,288,101,635,77,679,0,0,0
  gp_bldg_08_rear:
    ffmpeg:
      inputs:
      - path: rtsp://127.0.0.1:8554/gp_bldg_08_rear
        roles:
        - record
      - path: rtsp://127.0.0.1:8554/gp_bldg_08_rear_sub
        roles:
        - detect
        - audio
      output_args:
        record: preset-record-generic-audio-aac
      hwaccel_args: preset-nvidia-h265
    detect:
      width: 1920
      height: 536
#      width: 5120
#      height: 1440
#    zones:
#      gp_bldg_alert:
#        coordinates: 5120,1440,0,1440,0,767,1113,393,2685,308,4223,514,5120,976

    motion:
      mask:
      - 363,91,662,43,1076,35,1351,65,1920,276,1920,0,0,0,0,226
  gp_bldg_03_side:
    ffmpeg:
      inputs:
      - path: rtsp://127.0.0.1:8554/gp_bldg_03_side
        roles:
        - record
      - path: rtsp://127.0.0.1:8554/gp_bldg_03_side_sub
        roles:
        - detect
        - audio
      output_args:
        record: preset-record-generic-audio-aac
      hwaccel_args: preset-nvidia-h265
    detect:
      width: 1920
      height: 536
    motion:
      mask:
      - 764,93,1255,111,1920,260,1920,0,0,0,0,175
#      width: 5120
#      height: 1440
    zones:
      gp_bldg_alert:
        coordinates: 0,536,0,306,240,239,1189,370,1352,536

  gp611_rear:
    ffmpeg:
      inputs:
      - path: rtsp://127.0.0.1:8554/gp611_rear
        roles:
        - record
      - path: rtsp://127.0.0.1:8554/gp611_rear_sub
        roles:
        - detect
        - audio
      output_args:
        record: preset-record-generic-audio-aac
      hwaccel_args: preset-nvidia-h264
    detect:
      width: 1536
      height: 576
    #birdseye:
    #  enabled: False
    #detect:
    #  width: 4608
    #  height: 1728

Relevant log output

NA

FFprobe output from your camera

NA

Frigate stats

{"cpu_usages":{"1":{"cmdline":"/package/admin/s6/command/s6-svscan -d4 -- /run/service","cpu":"0.0","cpu_average":"0","mem":"0.0"},"15":{"cmdline":"s6-supervise s6-linux-init-shutdownd","cpu":"0.0","cpu_average":"0","mem":"0.0"},"17":{"cmdline":"/package/admin/s6-linux-init/command/s6-linux-init-shutdownd -c /run/s6/basedir -g 3000 -C -B","cpu":"0.0","cpu_average":"0","mem":"0.0"},"24":{"cmdline":"s6-supervise go2rtc","cpu":"0.0","cpu_average":"0","mem":"0.0"},"25":{"cmdline":"s6-supervise go2rtc-log","cpu":"0.0","cpu_average":"0","mem":"0.0"},"26":{"cmdline":"s6-supervise frigate","cpu":"0.0","cpu_average":"0","mem":"0.0"},"27":{"cmdline":"s6-supervise s6rc-oneshot-runner","cpu":"0.0","cpu_average":"0","mem":"0.0"},"28":{"cmdline":"s6-supervise s6rc-fdholder","cpu":"0.0","cpu_average":"0","mem":"0.0"},"29":{"cmdline":"s6-supervise go2rtc-healthcheck","cpu":"0.0","cpu_average":"0","mem":"0.0"},"30":{"cmdline":"s6-supervise nginx","cpu":"0.0","cpu_average":"0","mem":"0.0"},"31":{"cmdline":"s6-supervise nginx-log","cpu":"0.0","cpu_average":"0","mem":"0.0"},"32":{"cmdline":"s6-supervise frigate-log","cpu":"0.0","cpu_average":"0","mem":"0.0"},"40":{"cmdline":"/package/admin/s6-2.11.3.0/command/s6-fdholderd -1 -i data/rules","cpu":"0.0","cpu_average":"0","mem":"0.0"},"42":{"cmdline":"/package/admin/s6/command/s6-ipcserverd -1 -- /package/admin/s6/command/s6-ipcserver-access -v0 -E -l0 -i data/rules -- /package/admin/s6/command/s6-sudod -t 30000 -- /package/admin/s6-rc/command/s6-rc-oneshot-run -l ../.. --","cpu":"0.0","cpu_average":"0","mem":"0.0"},"80":{"cmdline":"s6-log -b -- T 1 n0 s10000000 T /dev/shm/logs/nginx","cpu":"0.0","cpu_average":"0","mem":"0.0"},"81":{"cmdline":"s6-log -b -- T 1 n0 s10000000 T /dev/shm/logs/frigate","cpu":"0.0","cpu_average":"0","mem":"0.0"},"82":{"cmdline":"s6-log -b -- T 1 n0 s10000000 T /dev/shm/logs/go2rtc","cpu":"0.0","cpu_average":"0","mem":"0.0"},"89":{"cmdline":"/usr/local/go2rtc/bin/go2rtc -config=/dev/shm/go2rtc.yaml","cpu":"93.3","cpu_average":"84","mem":"0.0"},"97":{"cmdline":"bash ./run.user go2rtc-healthcheck","cpu":"0.0","cpu_average":"0","mem":"0.0"},"98":{"cmdline":"python3 -u -m frigate","cpu":"124.5","cpu_average":"154","mem":"5.6"},"119":{"cmdline":"nginx: master process nginx","cpu":"0.0","cpu_average":"0","mem":"0.0"},"145":{"cmdline":"nginx: worker process","cpu":"0.2","cpu_average":"0","mem":"0.0"},"146":{"cmdline":"nginx: worker process","cpu":"0.0","cpu_average":"0","mem":"0.0"},"147":{"cmdline":"nginx: worker process","cpu":"0.0","cpu_average":"0","mem":"0.0"},"148":{"cmdline":"nginx: worker process","cpu":"0.0","cpu_average":"0","mem":"0.0"},"149":{"cmdline":"nginx: worker process","cpu":"0.0","cpu_average":"0","mem":"0.0"},"152":{"cmdline":"nginx: worker process","cpu":"17.9","cpu_average":"1","mem":"0.0"},"159":{"cmdline":"nginx: worker process","cpu":"0.0","cpu_average":"0","mem":"0.0"},"170":{"cmdline":"nginx: worker process","cpu":"0.0","cpu_average":"0","mem":"0.0"},"181":{"cmdline":"nginx: worker process","cpu":"0.0","cpu_average":"0","mem":"0.0"},"193":{"cmdline":"nginx: worker process","cpu":"0.0","cpu_average":"0","mem":"0.0"},"203":{"cmdline":"nginx: worker process","cpu":"0.0","cpu_average":"0","mem":"0.0"},"217":{"cmdline":"nginx: worker process","cpu":"0.2","cpu_average":"0","mem":"0.0"},"234":{"cmdline":"nginx: worker process","cpu":"0.0","cpu_average":"0","mem":"0.0"},"257":{"cmdline":"nginx: worker process","cpu":"0.0","cpu_average":"2","mem":"0.0"},"291":{"cmdline":"nginx: worker process","cpu":"0.0","cpu_average":"0","mem":"0.0"},"339":{"cmdline":"nginx: worker process","cpu":"0.0","cpu_average":"0","mem":"0.0"},"398":{"cmdline":"nginx: worker process","cpu":"0.0","cpu_average":"0","mem":"0.0"},"459":{"cmdline":"nginx: worker process","cpu":"0.0","cpu_average":"0","mem":"0.0"},"513":{"cmdline":"nginx: worker process","cpu":"0.0","cpu_average":"0","mem":"0.0"},"560":{"cmdline":"nginx: worker process","cpu":"0.0","cpu_average":"0","mem":"0.0"},"589":{"cmdline":"nginx: worker process","cpu":"0.0","cpu_average":"0","mem":"0.0"},"616":{"cmdline":"nginx: worker process","cpu":"0.0","cpu_average":"0","mem":"0.0"},"641":{"cmdline":"nginx: worker process","cpu":"0.0","cpu_average":"0","mem":"0.0"},"687":{"cmdline":"nginx: worker process","cpu":"0.0","cpu_average":"0","mem":"0.0"},"741":{"cmdline":"nginx: worker process","cpu":"0.0","cpu_average":"0","mem":"0.0"},"793":{"cmdline":"nginx: worker process","cpu":"0.0","cpu_average":"0","mem":"0.0"},"821":{"cmdline":"nginx: worker process","cpu":"0.0","cpu_average":"0","mem":"0.0"},"854":{"cmdline":"nginx: worker process","cpu":"0.0","cpu_average":"0","mem":"0.0"},"896":{"cmdline":"nginx: worker process","cpu":"0.0","cpu_average":"0","mem":"0.0"},"931":{"cmdline":"nginx: worker process","cpu":"0.0","cpu_average":"0","mem":"0.0"},"973":{"cmdline":"nginx: worker process","cpu":"0.0","cpu_average":"8","mem":"0.0"},"1021":{"cmdline":"nginx: worker process","cpu":"0.0","cpu_average":"0","mem":"0.0"},"1301":{"cmdline":"frigate.logger       ","cpu":"0.2","cpu_average":"0","mem":"0.1"},"1303":{"cmdline":"frigate.recording_manager","cpu":"31.4","cpu_average":"19","mem":"0.7"},"1312":{"cmdline":"/usr/bin/python3 -c from multiprocessing.resource_tracker import main;main(91)","cpu":"2.3","cpu_average":"2","mem":"0.0"},"1313":{"cmdline":"frigate.detector.coral1","cpu":"9.1","cpu_average":"10","mem":"0.2"},"1314":{"cmdline":"frigate.detector.coral2","cpu":"9.3","cpu_average":"10","mem":"0.2"},"1318":{"cmdline":"frigate.output       ","cpu":"136.9","cpu_average":"127","mem":"0.2"},"1340":{"cmdline":"frigate.process:warehouse_exterior_doorbell","cpu":"31.2","cpu_average":"7","mem":"0.2"},"1342":{"cmdline":"ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 2560x1920 -i pipe: -f mpegts -s 960x720 -codec:v mpeg1video -q 8 -bf 0 pipe:","cpu":"0.0","cpu_average":"0","mem":"0.0"},"1353":{"cmdline":"frigate.process:warehouse_exterior_nw","cpu":"2.3","cpu_average":"11","mem":"0.2"},"1355":{"cmdline":"ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 1920x1080 -i pipe: -f mpegts -s 1280x720 -codec:v mpeg1video -q 8 -bf 0 pipe:","cpu":"0.0","cpu_average":"0","mem":"0.0"},"1356":{"cmdline":"frigate.process:warehouse_exterior_sw","cpu":"1.9","cpu_average":"13","mem":"0.2"},"1358":{"cmdline":"frigate.process:warehouse_exterior_se","cpu":"1.7","cpu_average":"15","mem":"0.2"},"1360":{"cmdline":"frigate.process:warehouse_exterior_front","cpu":"4.1","cpu_average":"14","mem":"0.2"},"1362":{"cmdline":"ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 1920x1080 -i pipe: -f mpegts -s 1280x720 -codec:v mpeg1video -q 8 -bf 0 pipe:","cpu":"0.0","cpu_average":"0","mem":"0.0"},"1363":{"cmdline":"frigate.process:warehouse_workbench","cpu":"8.7","cpu_average":"5","mem":"0.2"},"1366":{"cmdline":"frigate.process:warehouse_interior_front","cpu":"16.3","cpu_average":"6","mem":"0.2"},"1367":{"cmdline":"ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 480x360 -i pipe: -f mpegts -s 960x720 -codec:v mpeg1video -q 8 -bf 0 pipe:","cpu":"0.0","cpu_average":"0","mem":"0.0"},"1368":{"cmdline":"frigate.process:warehouse_interior_office","cpu":"2.9","cpu_average":"5","mem":"0.2"},"1371":{"cmdline":"frigate.process:warehouse_interior_overview","cpu":"6.0","cpu_average":"15","mem":"0.2"},"1372":{"cmdline":"frigate.process:gp_clubhouse_gym_1","cpu":"2.3","cpu_average":"9","mem":"0.2"},"1373":{"cmdline":"ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 1536x576 -i pipe: -f mpegts -s 1920x720 -codec:v mpeg1video -q 8 -bf 0 pipe:","cpu":"0.0","cpu_average":"0","mem":"0.0"},"1375":{"cmdline":"frigate.process:gp_clubhouse_gym_2","cpu":"2.3","cpu_average":"10","mem":"0.2"},"1376":{"cmdline":"frigate.process:gp_clubhouse_front_door","cpu":"1.7","cpu_average":"4","mem":"0.2"},"1379":{"cmdline":"ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 640x480 -i pipe: -f mpegts -s 960x720 -codec:v mpeg1video -q 8 -bf 0 pipe:","cpu":"0.0","cpu_average":"0","mem":"0.0"},"1380":{"cmdline":"frigate.process:gp_clubhouse_back_door","cpu":"2.1","cpu_average":"4","mem":"0.2"},"1382":{"cmdline":"frigate.process:gp_clubhouse_banquet","cpu":"1.9","cpu_average":"5","mem":"0.2"},"1384":{"cmdline":"ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 640x480 -i pipe: -f mpegts -s 960x720 -codec:v mpeg1video -q 8 -bf 0 pipe:","cpu":"0.0","cpu_average":"0","mem":"0.0"},"1385":{"cmdline":"frigate.process:gp_clubhouse_pool_porch","cpu":"8.3","cpu_average":"8","mem":"0.2"},"1387":{"cmdline":"frigate.process:gp_clubhouse_pool","cpu":"5.2","cpu_average":"7","mem":"0.2"},"1389":{"cmdline":"ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 704x480 -i pipe: -f mpegts -s 1056x720 -codec:v mpeg1video -q 8 -bf 0 pipe:","cpu":"0.0","cpu_average":"0","mem":"0.0"},"1390":{"cmdline":"frigate.process:gp_clubhouse_firepit","cpu":"2.5","cpu_average":"12","mem":"0.3"},"1392":{"cmdline":"frigate.process:gp_clubhouse_parking","cpu":"2.5","cpu_average":"7","mem":"0.2"},"1393":{"cmdline":"ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 1920x1080 -i pipe: -f mpegts -s 1280x720 -codec:v mpeg1video -q 8 -bf 0 pipe:","cpu":"0.0","cpu_average":"0","mem":"0.0"},"1394":{"cmdline":"frigate.process:gp636_doorbell","cpu":"3.3","cpu_average":"5","mem":"0.2"},"1398":{"cmdline":"frigate.process:gp525_doorbell","cpu":"3.9","cpu_average":"11","mem":"0.2"},"1399":{"cmdline":"ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 2592x1944 -i pipe: -f mpegts -s 960x720 -codec:v mpeg1video -q 8 -bf 0 pipe:","cpu":"0.0","cpu_average":"0","mem":"0.0"},"1400":{"cmdline":"frigate.process:gp_entrance_75","cpu":"2.3","cpu_average":"9","mem":"0.2"},"1402":{"cmdline":"frigate.process:gp_entrance_archer","cpu":"25.8","cpu_average":"31","mem":"0.2"},"1404":{"cmdline":"frigate.process:gp_bldg_08_front","cpu":"3.1","cpu_average":"9","mem":"0.2"},"1406":{"cmdline":"ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 2592x1944 -i pipe: -f mpegts -s 960x720 -codec:v mpeg1video -q 8 -bf 0 pipe:","cpu":"0.0","cpu_average":"0","mem":"0.0"},"1407":{"cmdline":"frigate.process:gp_bldg_08_rear","cpu":"3.1","cpu_average":"4","mem":"0.2"},"1409":{"cmdline":"frigate.process:gp_bldg_03_side","cpu":"35.1","cpu_average":"14","mem":"0.2"},"1411":{"cmdline":"ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 480x360 -i pipe: -f mpegts -s 960x720 -codec:v mpeg1video -q 8 -bf 0 pipe:","cpu":"0.0","cpu_average":"0","mem":"0.0"},"1412":{"cmdline":"frigate.process:gp611_rear","cpu":"0.0","cpu_average":"0","mem":"0.2"},"1414":{"cmdline":"frigate.capture:warehouse_exterior_doorbell","cpu":"15.3","cpu_average":"12","mem":"0.2"},"1417":{"cmdline":"ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 480x360 -i pipe: -f mpegts -s 960x720 -codec:v mpeg1video -q 8 -bf 0 pipe:","cpu":"0.0","cpu_average":"0","mem":"0.0"},"1420":{"cmdline":"ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 480x360 -i pipe: -f mpegts -s 960x720 -codec:v mpeg1video -q 8 -bf 0 pipe:","cpu":"0.0","cpu_average":"0","mem":"0.0"},"1422":{"cmdline":"frigate.capture:warehouse_exterior_nw","cpu":"7.5","cpu_average":"5","mem":"0.2"},"1425":{"cmdline":"ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 1920x1080 -i pipe: -f mpegts -s 1280x720 -codec:v mpeg1video -q 8 -bf 0 pipe:","cpu":"0.0","cpu_average":"0","mem":"0.0"},"1426":{"cmdline":"frigate.capture:warehouse_exterior_sw","cpu":"6.4","cpu_average":"5","mem":"0.2"},"1427":{"cmdline":"ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel cuda -hwaccel_output_format cuda -user_agent FFmpeg Frigate/0.13.0-0fd1eaf -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -timeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://127.0.0.1:8554/warehouse_exterior_nw -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy -an /tmp/cache/warehouse_exterior_nw-%Y%m%d%H%M%S.mp4 -r 5 -vf fps=5,scale_cuda=w=1920:h=1080:format=nv12,hwdownload,format=nv12,format=yuv420p -threads 2 -f rawvideo -pix_fmt yuv420p pipe:","cpu":"10.8","cpu_average":"9","mem":"0.2"},"1428":{"cmdline":"frigate.capture:warehouse_exterior_se","cpu":"0.6","cpu_average":"0","mem":"0.2"},"1431":{"cmdline":"ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel cuda -hwaccel_output_format cuda -user_agent FFmpeg Frigate/0.13.0-0fd1eaf -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -timeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://127.0.0.1:8554/warehouse_exterior_sw_sub -r 5 -vf fps=5,scale_cuda=w=1920:h=1080:format=nv12,hwdownload,format=nv12,format=yuv420p -threads 2 -f rawvideo -pix_fmt yuv420p pipe:","cpu":"10.6","cpu_average":"9","mem":"0.2"},"1434":{"cmdline":"frigate.capture:warehouse_exterior_front","cpu":"4.2","cpu_average":"2","mem":"0.2"},"1436":{"cmdline":"ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 1920x536 -i pipe: -f mpegts -s 2579x720 -codec:v mpeg1video -q 8 -bf 0 pipe:","cpu":"0.0","cpu_average":"0","mem":"0.0"},"1437":{"cmdline":"ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel cuda -hwaccel_output_format cuda -user_agent FFmpeg Frigate/0.13.0-0fd1eaf -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -timeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://127.0.0.1:8554/warehouse_exterior_se_sub -r 5 -vf fps=5,scale_cuda=w=480:h=360:format=nv12,hwdownload,format=nv12,format=yuv420p -threads 2 -f rawvideo -pix_fmt yuv420p pipe:","cpu":"3.7","cpu_average":"3","mem":"0.1"},"1439":{"cmdline":"frigate.capture:warehouse_workbench","cpu":"1.0","cpu_average":"1","mem":"0.2"},"1445":{"cmdline":"ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel cuda -hwaccel_output_format cuda -user_agent FFmpeg Frigate/0.13.0-0fd1eaf -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -timeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://127.0.0.1:8554/warehouse_exterior_sw -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy -an /tmp/cache/warehouse_exterior_sw-%Y%m%d%H%M%S.mp4","cpu":"0.8","cpu_average":"0","mem":"0.0"},"1446":{"cmdline":"frigate.capture:warehouse_interior_front","cpu":"1.5","cpu_average":"1","mem":"0.2"},"1450":{"cmdline":"ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 3840x2160 -i pipe: -f mpegts -s 1280x720 -codec:v mpeg1video -q 8 -bf 0 pipe:","cpu":"0.0","cpu_average":"0","mem":"0.0"},"1451":{"cmdline":"ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel cuda -hwaccel_output_format cuda -user_agent FFmpeg Frigate/0.13.0-0fd1eaf -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -timeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://127.0.0.1:8554/warehouse_workbench_sub -r 5 -vf fps=5,scale_cuda=w=640:h=480:format=nv12,hwdownload,format=nv12,format=yuv420p -threads 2 -f rawvideo -pix_fmt yuv420p pipe:","cpu":"3.9","cpu_average":"3","mem":"0.2"},"1452":{"cmdline":"frigate.capture:warehouse_interior_office","cpu":"1.2","cpu_average":"1","mem":"0.2"},"1453":{"cmdline":"ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel cuda -hwaccel_output_format cuda -user_agent FFmpeg Frigate/0.13.0-0fd1eaf -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -timeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://127.0.0.1:8554/warehouse_exterior_se -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c:v copy -c:a aac /tmp/cache/warehouse_exterior_se-%Y%m%d%H%M%S.mp4","cpu":"1.2","cpu_average":"1","mem":"0.0"},"1461":{"cmdline":"frigate.capture:warehouse_interior_overview","cpu":"6.2","cpu_average":"5","mem":"0.2"},"1462":{"cmdline":"ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel cuda -hwaccel_output_format cuda -user_agent FFmpeg Frigate/0.13.0-0fd1eaf -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -timeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://127.0.0.1:8554/warehouse_interior_front_sub -r 5 -vf fps=5,scale_cuda=w=640:h=480:format=nv12,hwdownload,format=nv12,format=yuv420p -threads 2 -f rawvideo -pix_fmt yuv420p pipe:","cpu":"4.6","cpu_average":"3","mem":"0.2"},"1465":{"cmdline":"ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 1920x536 -i pipe: -f mpegts -s 2579x720 -codec:v mpeg1video -q 8 -bf 0 pipe:","cpu":"0.0","cpu_average":"0","mem":"0.0"},"1466":{"cmdline":"ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel cuda -hwaccel_output_format cuda -user_agent FFmpeg Frigate/0.13.0-0fd1eaf -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -timeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://127.0.0.1:8554/warehouse_workbench -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c:v copy -c:a aac /tmp/cache/warehouse_workbench-%Y%m%d%H%M%S.mp4","cpu":"7.3","cpu_average":"7","mem":"0.0"},"1467":{"cmdline":"frigate.capture:gp_clubhouse_gym_1","cpu":"15.2","cpu_average":"12","mem":"0.2"},"1468":{"cmdline":"ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel cuda -hwaccel_output_format cuda -user_agent FFmpeg Frigate/0.13.0-0fd1eaf -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -timeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://127.0.0.1:8554/warehouse_interior_office_sub -r 5 -vf fps=5,scale_cuda=w=704:h=480:format=nv12,hwdownload,format=nv12,format=yuv420p -threads 2 -f rawvideo -pix_fmt yuv420p pipe:","cpu":"4.0","cpu_average":"3","mem":"0.2"},"1469":{"cmdline":"ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel cuda -hwaccel_output_format cuda -user_agent FFmpeg Frigate/0.13.0-0fd1eaf -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -timeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://127.0.0.1:8554/warehouse_interior_overview -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy -an /tmp/cache/warehouse_interior_overview-%Y%m%d%H%M%S.mp4 -r 5 -vf fps=5,scale_cuda=w=1920:h=1080:format=nv12,hwdownload,format=nv12,format=yuv420p -threads 2 -f rawvideo -pix_fmt yuv420p pipe:","cpu":"10.1","cpu_average":"10","mem":"0.2"},"1473":{"cmdline":"frigate.capture:gp_clubhouse_gym_2","cpu":"14.7","cpu_average":"12","mem":"0.2"},"1476":{"cmdline":"ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel cuda -hwaccel_output_format cuda -user_agent FFmpeg Frigate/0.13.0-0fd1eaf -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -timeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://127.0.0.1:8554/gp_clubhouse_gym_1 -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy -an /tmp/cache/gp_clubhouse_gym_1-%Y%m%d%H%M%S.mp4 -r 5 -vf fps=5,scale_cuda=w=2592:h=1944:format=nv12,hwdownload,format=nv12,format=yuv420p -threads 2 -f rawvideo -pix_fmt yuv420p pipe:","cpu":"21.2","cpu_average":"18","mem":"0.3"},"1479":{"cmdline":"ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel cuda -hwaccel_output_format cuda -user_agent FFmpeg Frigate/0.13.0-0fd1eaf -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -timeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://127.0.0.1:8554/warehouse_interior_front -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c:v copy -c:a aac /tmp/cache/warehouse_interior_front-%Y%m%d%H%M%S.mp4","cpu":"7.2","cpu_average":"7","mem":"0.0"},"1480":{"cmdline":"ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 1920x1080 -i pipe: -f mpegts -s 1280x720 -codec:v mpeg1video -q 8 -bf 0 pipe:","cpu":"0.0","cpu_average":"0","mem":"0.0"},"1483":{"cmdline":"frigate.capture:gp_clubhouse_front_door","cpu":"0.8","cpu_average":"0","mem":"0.2"},"1484":{"cmdline":"ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel cuda -hwaccel_output_format cuda -user_agent FFmpeg Frigate/0.13.0-0fd1eaf -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -timeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://127.0.0.1:8554/warehouse_interior_office -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c:v copy -c:a aac /tmp/cache/warehouse_interior_office-%Y%m%d%H%M%S.mp4","cpu":"0.6","cpu_average":"0","mem":"0.0"},"1488":{"cmdline":"ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel cuda -hwaccel_output_format cuda -user_agent FFmpeg Frigate/0.13.0-0fd1eaf -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -timeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://127.0.0.1:8554/gp_clubhouse_gym_2 -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy -an /tmp/cache/gp_clubhouse_gym_2-%Y%m%d%H%M%S.mp4 -r 5 -vf fps=5,scale_cuda=w=2592:h=1944:format=nv12,hwdownload,format=nv12,format=yuv420p -threads 2 -f rawvideo -pix_fmt yuv420p pipe:","cpu":"17.5","cpu_average":"18","mem":"0.3"},"1489":{"cmdline":"frigate.capture:gp_clubhouse_back_door","cpu":"1.1","cpu_average":"0","mem":"0.2"},"1490":{"cmdline":"ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel cuda -hwaccel_output_format cuda -user_agent FFmpeg Frigate/0.13.0-0fd1eaf -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -timeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://127.0.0.1:8554/gp_clubhouse_front_door_sub -r 5 -vf fps=5,scale_cuda=w=480:h=360:format=nv12,hwdownload,format=nv12,format=yuv420p -threads 2 -f rawvideo -pix_fmt yuv420p pipe:","cpu":"3.4","cpu_average":"3","mem":"0.2"},"1493":{"cmdline":"ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 2560x1920 -i pipe: -f mpegts -s 960x720 -codec:v mpeg1video -q 8 -bf 0 pipe:","cpu":"0.0","cpu_average":"0","mem":"0.0"},"1494":{"cmdline":"frigate.capture:gp_clubhouse_banquet","cpu":"0.8","cpu_average":"0","mem":"0.2"},"1498":{"cmdline":"ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel cuda -hwaccel_output_format cuda -user_agent FFmpeg Frigate/0.13.0-0fd1eaf -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -timeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://127.0.0.1:8554/gp_clubhouse_back_door_sub -r 5 -vf fps=5,scale_cuda=w=480:h=360:format=nv12,hwdownload,format=nv12,format=yuv420p -threads 2 -f rawvideo -pix_fmt yuv420p pipe:","cpu":"4.2","cpu_average":"3","mem":"0.1"},"1501":{"cmdline":"frigate.capture:gp_clubhouse_pool_porch","cpu":"7.2","cpu_average":"5","mem":"0.2"},"1504":{"cmdline":"ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel cuda -hwaccel_output_format cuda -user_agent FFmpeg Frigate/0.13.0-0fd1eaf -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -timeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://127.0.0.1:8554/gp_clubhouse_banquet_sub -r 5 -vf fps=5,scale_cuda=w=480:h=360:format=nv12,hwdownload,format=nv12,format=yuv420p -threads 2 -f rawvideo -pix_fmt yuv420p pipe:","cpu":"4.0","cpu_average":"3","mem":"0.2"},"1505":{"cmdline":"ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 1920x1080 -i pipe: -f mpegts -s 1280x720 -codec:v mpeg1video -q 8 -bf 0 pipe:","cpu":"0.0","cpu_average":"0","mem":"0.0"},"1506":{"cmdline":"ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel cuda -hwaccel_output_format cuda -user_agent FFmpeg Frigate/0.13.0-0fd1eaf -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -timeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://127.0.0.1:8554/gp_clubhouse_front_door -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy -an /tmp/cache/gp_clubhouse_front_door-%Y%m%d%H%M%S.mp4","cpu":"1.3","cpu_average":"1","mem":"0.0"},"1507":{"cmdline":"frigate.capture:gp_clubhouse_pool","cpu":"3.8","cpu_average":"3","mem":"0.2"},"1512":{"cmdline":"frigate.capture:gp_clubhouse_firepit","cpu":"18.1","cpu_average":"19","mem":"0.2"},"1513":{"cmdline":"ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel cuda -hwaccel_output_format cuda -user_agent FFmpeg Frigate/0.13.0-0fd1eaf -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -timeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://127.0.0.1:8554/gp_clubhouse_pool_porch -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy -an /tmp/cache/gp_clubhouse_pool_porch-%Y%m%d%H%M%S.mp4 -r 5 -vf fps=5,scale_cuda=w=1920:h=1080:format=nv12,hwdownload,format=nv12,format=yuv420p -threads 2 -f rawvideo -pix_fmt yuv420p pipe:","cpu":"10.7","cpu_average":"9","mem":"0.2"},"1516":{"cmdline":"ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel cuda -hwaccel_output_format cuda -user_agent FFmpeg Frigate/0.13.0-0fd1eaf -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -timeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://127.0.0.1:8554/gp_clubhouse_back_door -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy -an /tmp/cache/gp_clubhouse_back_door-%Y%m%d%H%M%S.mp4","cpu":"1.3","cpu_average":"1","mem":"0.0"},"1517":{"cmdline":"ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel cuda -hwaccel_output_format cuda -user_agent FFmpeg Frigate/0.13.0-0fd1eaf -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -timeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://127.0.0.1:8554/gp_clubhouse_pool_sub -r 5 -vf fps=5,scale_cuda=w=1920:h=536:format=nv12,hwdownload,format=nv12,format=yuv420p -threads 2 -f rawvideo -pix_fmt yuv420p pipe:","cpu":"7.4","cpu_average":"6","mem":"0.2"},"1520":{"cmdline":"frigate.capture:gp_clubhouse_parking","cpu":"2.9","cpu_average":"3","mem":"0.2"},"1521":{"cmdline":"ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel cuda -hwaccel_output_format cuda -user_agent FFmpeg Frigate/0.13.0-0fd1eaf -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -timeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://127.0.0.1:8554/gp_clubhouse_banquet -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy -an /tmp/cache/gp_clubhouse_banquet-%Y%m%d%H%M%S.mp4","cpu":"1.0","cpu_average":"1","mem":"0.0"},"1522":{"cmdline":"ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel cuda -hwaccel_output_format cuda -user_agent FFmpeg Frigate/0.13.0-0fd1eaf -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -timeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://127.0.0.1:8554/gp_clubhouse_firepit -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c:v copy -c:a aac /tmp/cache/gp_clubhouse_firepit-%Y%m%d%H%M%S.mp4 -r 5 -vf fps=5,scale_cuda=w=3840:h=2160:format=nv12,hwdownload,format=nv12,format=yuv420p -threads 2 -f rawvideo -pix_fmt yuv420p pipe:","cpu":"36.3","cpu_average":"36","mem":"0.4"},"1523":{"cmdline":"ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 1920x1080 -i pipe: -f mpegts -s 1280x720 -codec:v mpeg1video -q 8 -bf 0 pipe:","cpu":"0.0","cpu_average":"0","mem":"0.0"},"1526":{"cmdline":"frigate.capture:gp636_doorbell","cpu":"7.3","cpu_average":"5","mem":"0.2"},"1537":{"cmdline":"frigate.capture:gp525_doorbell","cpu":"12.2","cpu_average":"10","mem":"0.2"},"1538":{"cmdline":"ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel cuda -hwaccel_output_format cuda -user_agent FFmpeg Frigate/0.13.0-0fd1eaf -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -timeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://admin:campass123@10.10.23.30:554/Streaming/channels/102 -r 5 -vf fps=5,scale_cuda=w=1920:h=536:format=nv12,hwdownload,format=nv12,format=yuv420p -threads 2 -f rawvideo -pix_fmt yuv420p pipe:","cpu":"6.5","cpu_average":"7","mem":"0.2"},"1539":{"cmdline":"ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel cuda -hwaccel_output_format cuda -user_agent FFmpeg Frigate/0.13.0-0fd1eaf -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -timeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://127.0.0.1:8554/gp_clubhouse_pool -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c:v copy -c:a aac /tmp/cache/gp_clubhouse_pool-%Y%m%d%H%M%S.mp4","cpu":"3.8","cpu_average":"4","mem":"0.0"},"1540":{"cmdline":"ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 1920x536 -i pipe: -f mpegts -s 2579x720 -codec:v mpeg1video -q 8 -bf 0 pipe:","cpu":"0.0","cpu_average":"0","mem":"0.0"},"1541":{"cmdline":"ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel cuda -hwaccel_output_format cuda -user_agent FFmpeg Frigate/0.13.0-0fd1eaf -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -timeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://127.0.0.1:8554/gp636_doorbell -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c:v copy -c:a aac /tmp/cache/gp636_doorbell-%Y%m%d%H%M%S.mp4 -r 5 -vf fps=5,scale_cuda=w=1920:h=1080:format=nv12,hwdownload,format=nv12,format=yuv420p -threads 2 -f rawvideo -pix_fmt yuv420p pipe:","cpu":"16.7","cpu_average":"16","mem":"0.2"},"1544":{"cmdline":"frigate.capture:gp_entrance_75","cpu":"7.2","cpu_average":"5","mem":"0.2"},"1548":{"cmdline":"frigate.capture:gp_entrance_archer","cpu":"8.0","cpu_average":"5","mem":"0.2"},"1552":{"cmdline":"ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel cuda -hwaccel_output_format cuda -user_agent FFmpeg Frigate/0.13.0-0fd1eaf -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -timeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://127.0.0.1:8554/gp_entrance_75 -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c:v copy -c:a aac /tmp/cache/gp_entrance_75-%Y%m%d%H%M%S.mp4 -r 5 -vf fps=5,scale_cuda=w=1920:h=1080:format=nv12,hwdownload,format=nv12,format=yuv420p -threads 2 -f rawvideo -pix_fmt yuv420p pipe:","cpu":"19.7","cpu_average":"18","mem":"0.2"},"1553":{"cmdline":"ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 1920x536 -i pipe: -f mpegts -s 2579x720 -codec:v mpeg1video -q 8 -bf 0 pipe:","cpu":"0.0","cpu_average":"0","mem":"0.0"},"1554":{"cmdline":"frigate.capture:gp_bldg_08_front","cpu":"3.8","cpu_average":"3","mem":"0.2"},"1556":{"cmdline":"ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel cuda -hwaccel_output_format cuda -user_agent FFmpeg Frigate/0.13.0-0fd1eaf -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -timeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://127.0.0.1:8554/gp_clubhouse_parking -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c:v copy -c:a aac /tmp/cache/gp_clubhouse_parking-%Y%m%d%H%M%S.mp4","cpu":"3.6","cpu_average":"5","mem":"0.0"},"1562":{"cmdline":"frigate.capture:gp_bldg_08_rear","cpu":"4.0","cpu_average":"3","mem":"0.2"},"1563":{"cmdline":"ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel cuda -hwaccel_output_format cuda -user_agent FFmpeg Frigate/0.13.0-0fd1eaf -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -timeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://127.0.0.1:8554/gp_entrance_archer -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c:v copy -c:a aac /tmp/cache/gp_entrance_archer-%Y%m%d%H%M%S.mp4 -r 5 -vf fps=5,scale_cuda=w=1920:h=1080:format=nv12,hwdownload,format=nv12,format=yuv420p -threads 2 -f rawvideo -pix_fmt yuv420p pipe:","cpu":"20.4","cpu_average":"18","mem":"0.2"},"1566":{"cmdline":"ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel cuda -hwaccel_output_format cuda -user_agent FFmpeg Frigate/0.13.0-0fd1eaf -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -timeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://127.0.0.1:8554/gp_bldg_08_front_sub -r 5 -vf fps=5,scale_cuda=w=1920:h=536:format=nv12,hwdownload,format=nv12,format=yuv420p -threads 2 -f rawvideo -pix_fmt yuv420p pipe:","cpu":"7.6","cpu_average":"6","mem":"0.2"},"1567":{"cmdline":"frigate.capture:gp_bldg_03_side","cpu":"4.0","cpu_average":"3","mem":"0.2"},"1571":{"cmdline":"ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel cuda -hwaccel_output_format cuda -user_agent FFmpeg Frigate/0.13.0-0fd1eaf -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -timeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://127.0.0.1:8554/gp_bldg_08_rear_sub -r 5 -vf fps=5,scale_cuda=w=1920:h=536:format=nv12,hwdownload,format=nv12,format=yuv420p -threads 2 -f rawvideo -pix_fmt yuv420p pipe:","cpu":"6.6","cpu_average":"6","mem":"0.2"},"1573":{"cmdline":"frigate.capture:gp611_rear","cpu":"0.0","cpu_average":"0","mem":"0.2"},"1579":{"cmdline":"ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel cuda -hwaccel_output_format cuda -user_agent FFmpeg Frigate/0.13.0-0fd1eaf -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -timeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://127.0.0.1:8554/gp_bldg_08_front -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c:v copy -c:a aac /tmp/cache/gp_bldg_08_front-%Y%m%d%H%M%S.mp4","cpu":"4.5","cpu_average":"4","mem":"0.0"},"1580":{"cmdline":"frigate.audio_manager","cpu":"109.6","cpu_average":"109","mem":"37.1"},"1581":{"cmdline":"ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel cuda -hwaccel_output_format cuda -user_agent FFmpeg Frigate/0.13.0-0fd1eaf -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -timeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://127.0.0.1:8554/gp_bldg_03_side_sub -r 5 -vf fps=5,scale_cuda=w=1920:h=536:format=nv12,hwdownload,format=nv12,format=yuv420p -threads 2 -f rawvideo -pix_fmt yuv420p pipe:","cpu":"7.2","cpu_average":"6","mem":"0.2"},"1584":{"cmdline":"ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel cuda -hwaccel_output_format cuda -user_agent FFmpeg Frigate/0.13.0-0fd1eaf -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -timeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://127.0.0.1:8554/gp_bldg_08_rear -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c:v copy -c:a aac /tmp/cache/gp_bldg_08_rear-%Y%m%d%H%M%S.mp4","cpu":"4.5","cpu_average":"4","mem":"0.0"},"1587":{"cmdline":"ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 1920x536 -i pipe: -f mpegts -s 2579x720 -codec:v mpeg1video -q 8 -bf 0 pipe:","cpu":"0.0","cpu_average":"0","mem":"0.0"},"1588":{"cmdline":"","cpu":"0.0","cpu_average":"0","mem":"0.0"},"1594":{"cmdline":"","cpu":"0.0","cpu_average":"0","mem":"0.0"},"1595":{"cmdline":"ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel cuda -hwaccel_output_format cuda -user_agent FFmpeg Frigate/0.13.0-0fd1eaf -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -timeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://127.0.0.1:8554/gp_bldg_03_side -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c:v copy -c:a aac /tmp/cache/gp_bldg_03_side-%Y%m%d%H%M%S.mp4","cpu":"2.3","cpu_average":"2","mem":"0.0"},"1598":{"cmdline":"ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 1536x576 -i pipe: -f mpegts -s 1920x720 -codec:v mpeg1video -q 8 -bf 0 pipe:","cpu":"0.0","cpu_average":"0","mem":"0.0"},"1599":{"cmdline":"ffmpeg -hide_banner -loglevel warning -threads 2 -rtsp_transport tcp -timeout 5000000 -vn -i rtsp://127.0.0.1:8554/warehouse_workbench_sub -f s16le -ar 16000 -ac 1 -y /tmp/cache/warehouse_workbench-audio","cpu":"0.4","cpu_average":"0","mem":"0.0"},"1607":{"cmdline":"ffmpeg -hide_banner -loglevel warning -threads 2 -rtsp_transport tcp -timeout 5000000 -vn -i rtsp://127.0.0.1:8554/warehouse_interior_front_sub -f s16le -ar 16000 -ac 1 -y /tmp/cache/warehouse_interior_front-audio","cpu":"0.9","cpu_average":"0","mem":"0.0"},"1610":{"cmdline":"ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 1920x1200 -i pipe: -f mpegts -s 1920x1200 -codec:v mpeg1video -q 8 -bf 0 pipe:","cpu":"24.7","cpu_average":"25","mem":"0.1"},"1611":{"cmdline":"","cpu":"0.0","cpu_average":"0","mem":"0.0"},"1649":{"cmdline":"","cpu":"0.0","cpu_average":"0","mem":"0.0"},"1651":{"cmdline":"","cpu":"0.0","cpu_average":"0","mem":"0.0"},"1654":{"cmdline":"","cpu":"0.0","cpu_average":"0","mem":"0.0"},"1657":{"cmdline":"ffmpeg -hide_banner -loglevel warning -threads 2 -rtsp_transport tcp -timeout 5000000 -vn -i rtsp://127.0.0.1:8554/gp636_doorbell -f s16le -ar 16000 -ac 1 -y /tmp/cache/gp636_doorbell-audio","cpu":"0.6","cpu_average":"0","mem":"0.0"},"1660":{"cmdline":"","cpu":"0.0","cpu_average":"0","mem":"0.0"},"1670":{"cmdline":"ffmpeg -hide_banner -loglevel warning -threads 2 -rtsp_transport tcp -timeout 5000000 -vn -i rtsp://127.0.0.1:8554/gp_entrance_75 -f s16le -ar 16000 -ac 1 -y /tmp/cache/gp_entrance_75-audio","cpu":"2.1","cpu_average":"2","mem":"0.0"},"1675":{"cmdline":"ffmpeg -hide_banner -loglevel warning -threads 2 -rtsp_transport tcp -timeout 5000000 -vn -i rtsp://127.0.0.1:8554/gp_entrance_archer -f s16le -ar 16000 -ac 1 -y /tmp/cache/gp_entrance_archer-audio","cpu":"2.1","cpu_average":"2","mem":"0.0"},"1677":{"cmdline":"","cpu":"0.0","cpu_average":"0","mem":"0.0"},"1680":{"cmdline":"","cpu":"0.0","cpu_average":"0","mem":"0.0"},"1684":{"cmdline":"","cpu":"0.0","cpu_average":"0","mem":"0.0"},"1687":{"cmdline":"","cpu":"0.0","cpu_average":"0","mem":"0.0"},"3614":{"cmdline":"ffmpeg -hide_banner -f rawvideo -pix_fmt yuv420p -video_size 1920x1200 -r 10 -i /tmp/cache/birdseye -c:v libx264 -g 50 -profile:v high -level:v 4.1 -preset:v superfast -tune:v zerolatency -rtsp_transport tcp -f rtsp rtsp://127.0.0.1:8554/d5842d2f48319551b595c330d8368a5c","cpu":"52.4","cpu_average":"54","mem":"0.2"},"156476":{"cmdline":"ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel cuda -hwaccel_output_format cuda -user_agent FFmpeg Frigate/0.13.0-0fd1eaf -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -timeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://127.0.0.1:8554/warehouse_exterior_doorbell -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c:v copy -c:a aac /tmp/cache/warehouse_exterior_doorbell-%Y%m%d%H%M%S.mp4 -r 5 -vf fps=5,scale_cuda=w=2560:h=1920:format=nv12,hwdownload,format=nv12,format=yuv420p -threads 2 -f rawvideo -pix_fmt yuv420p pipe:","cpu":"23.1","cpu_average":"22","mem":"0.4"},"235210":{"cmdline":"ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel cuda -hwaccel_output_format cuda -user_agent FFmpeg Frigate/0.13.0-0fd1eaf -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -timeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://127.0.0.1:8554/warehouse_exterior_front_sub -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c:v copy -c:a aac /tmp/cache/warehouse_exterior_front-%Y%m%d%H%M%S.mp4 -r 5 -vf fps=5,scale_cuda=w=1536:h=576:format=nv12,hwdownload,format=nv12,format=yuv420p -threads 2 -f rawvideo -pix_fmt yuv420p pipe:","cpu":"10.4","cpu_average":"9","mem":"0.2"},"293276":{"cmdline":"ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel cuda -hwaccel_output_format cuda -user_agent FFmpeg Frigate/0.13.0-0fd1eaf -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -timeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://127.0.0.1:8554/gp525_doorbell -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c:v copy -c:a aac /tmp/cache/gp525_doorbell-%Y%m%d%H%M%S.mp4 -r 5 -vf fps=5,scale_cuda=w=2560:h=1920:format=nv12,hwdownload,format=nv12,format=yuv420p -threads 2 -f rawvideo -pix_fmt yuv420p pipe:","cpu":"15.9","cpu_average":"18","mem":"0.4"},"297234":{"cmdline":"sleep 30s","cpu":"0.0","cpu_average":"0","mem":"0.0"}},"detection_fps":48.3,"detectors":{"coral1":{"detection_start":0.0,"inference_speed":8.18,"pid":1313},"coral2":{"detection_start":0.0,"inference_speed":8.04,"pid":1314}},"gp525_doorbell":{"camera_fps":5.3,"capture_pid":1537,"detection_enabled":1,"detection_fps":0.3,"ffmpeg_pid":293276,"pid":1398,"process_fps":5.3,"skipped_fps":0.0},"gp611_rear":{"camera_fps":0.0,"capture_pid":1573,"detection_enabled":1,"detection_fps":0.0,"ffmpeg_pid":297236,"pid":1412,"process_fps":0.0,"skipped_fps":0.0},"gp636_doorbell":{"camera_fps":5.1,"capture_pid":1526,"detection_enabled":1,"detection_fps":0.1,"ffmpeg_pid":1541,"pid":1394,"process_fps":5.1,"skipped_fps":0.0},"gp_bldg_03_side":{"camera_fps":5.0,"capture_pid":1567,"detection_enabled":1,"detection_fps":15.4,"ffmpeg_pid":1581,"pid":1409,"process_fps":4.9,"skipped_fps":0.1},"gp_bldg_08_front":{"camera_fps":5.1,"capture_pid":1554,"detection_enabled":1,"detection_fps":1.4,"ffmpeg_pid":1566,"pid":1404,"process_fps":5.1,"skipped_fps":0.0},"gp_bldg_08_rear":{"camera_fps":5.0,"capture_pid":1562,"detection_enabled":1,"detection_fps":0.0,"ffmpeg_pid":1571,"pid":1407,"process_fps":5.0,"skipped_fps":0.0},"gp_clubhouse_back_door":{"camera_fps":5.1,"capture_pid":1489,"detection_enabled":1,"detection_fps":0.0,"ffmpeg_pid":1498,"pid":1380,"process_fps":5.1,"skipped_fps":0.0},"gp_clubhouse_banquet":{"camera_fps":5.0,"capture_pid":1494,"detection_enabled":1,"detection_fps":0.5,"ffmpeg_pid":1504,"pid":1382,"process_fps":5.0,"skipped_fps":0.0},"gp_clubhouse_firepit":{"camera_fps":5.1,"capture_pid":1512,"detection_enabled":1,"detection_fps":0.0,"ffmpeg_pid":1522,"pid":1390,"process_fps":5.1,"skipped_fps":0.0},"gp_clubhouse_front_door":{"camera_fps":5.0,"capture_pid":1483,"detection_enabled":1,"detection_fps":0.0,"ffmpeg_pid":1490,"pid":1376,"process_fps":5.0,"skipped_fps":0.0},"gp_clubhouse_gym_1":{"camera_fps":5.0,"capture_pid":1467,"detection_enabled":1,"detection_fps":0.5,"ffmpeg_pid":1476,"pid":1372,"process_fps":5.0,"skipped_fps":0.0},"gp_clubhouse_gym_2":{"camera_fps":5.0,"capture_pid":1473,"detection_enabled":1,"detection_fps":2.5,"ffmpeg_pid":1488,"pid":1375,"process_fps":5.0,"skipped_fps":0.0},"gp_clubhouse_parking":{"camera_fps":5.1,"capture_pid":1520,"detection_enabled":1,"detection_fps":0.3,"ffmpeg_pid":1538,"pid":1392,"process_fps":5.1,"skipped_fps":0.0},"gp_clubhouse_pool":{"camera_fps":5.0,"capture_pid":1507,"detection_enabled":1,"detection_fps":0.6,"ffmpeg_pid":1517,"pid":1387,"process_fps":5.0,"skipped_fps":0.0},"gp_clubhouse_pool_porch":{"camera_fps":5.0,"capture_pid":1501,"detection_enabled":1,"detection_fps":0.8,"ffmpeg_pid":1513,"pid":1385,"process_fps":5.0,"skipped_fps":0.0},"gp_entrance_75":{"camera_fps":5.0,"capture_pid":1544,"detection_enabled":1,"detection_fps":0.1,"ffmpeg_pid":1552,"pid":1400,"process_fps":5.0,"skipped_fps":0.0},"gp_entrance_archer":{"camera_fps":5.1,"capture_pid":1548,"detection_enabled":1,"detection_fps":8.5,"ffmpeg_pid":1563,"pid":1402,"process_fps":5.1,"skipped_fps":0.0},"gpu_usages":{"NVIDIA T1000 8GB":{"gpu":"14.0%","mem":"48.66%"}},"processes":{"audioDetector":{"pid":1580},"go2rtc":{"pid":89},"logger":{"pid":1301},"recording":{"pid":1303}},"service":{"last_updated":1690208276,"latest_version":"0.12.1","storage":{"/dev/shm":{"free":1430.7,"mount_type":"tmpfs","total":1521.0,"used":90.3},"/media/frigate/clips":{"free":140111.8,"mount_type":"zfs","total":3683179.2,"used":3543067.5},"/media/frigate/recordings":{"free":140111.8,"mount_type":"zfs","total":3683179.2,"used":3543067.5},"/tmp/cache":{"free":936.1,"mount_type":"tmpfs","total":1024.0,"used":87.9}},"temperatures":{"apex_0":49.05,"apex_1":50.55},"uptime":7310,"version":"0.13.0-0fd1eaf"},"warehouse_exterior_doorbell":{"camera_fps":5.1,"capture_pid":1414,"detection_enabled":1,"detection_fps":9.0,"ffmpeg_pid":156476,"pid":1340,"process_fps":5.1,"skipped_fps":0.0},"warehouse_exterior_front":{"camera_fps":5.1,"capture_pid":1434,"detection_enabled":1,"detection_fps":0.2,"ffmpeg_pid":235210,"pid":1360,"process_fps":5.1,"skipped_fps":0.0},"warehouse_exterior_nw":{"camera_fps":5.1,"capture_pid":1422,"detection_enabled":1,"detection_fps":0.0,"ffmpeg_pid":1427,"pid":1353,"process_fps":5.1,"skipped_fps":0.0},"warehouse_exterior_se":{"camera_fps":5.0,"capture_pid":1428,"detection_enabled":1,"detection_fps":0.0,"ffmpeg_pid":1437,"pid":1358,"process_fps":5.0,"skipped_fps":0.0},"warehouse_exterior_sw":{"camera_fps":5.0,"capture_pid":1426,"detection_enabled":1,"detection_fps":0.0,"ffmpeg_pid":1431,"pid":1356,"process_fps":5.0,"skipped_fps":0.0},"warehouse_interior_front":{"camera_fps":5.0,"capture_pid":1446,"detection_enabled":1,"detection_fps":5.0,"ffmpeg_pid":1462,"pid":1366,"process_fps":5.0,"skipped_fps":0.0},"warehouse_interior_office":{"camera_fps":5.0,"capture_pid":1452,"detection_enabled":1,"detection_fps":0.1,"ffmpeg_pid":1468,"pid":1368,"process_fps":5.0,"skipped_fps":0.0},"warehouse_interior_overview":{"camera_fps":5.1,"capture_pid":1461,"detection_enabled":1,"detection_fps":0.7,"ffmpeg_pid":1469,"pid":1371,"process_fps":5.1,"skipped_fps":0.0},"warehouse_workbench":{"camera_fps":5.1,"capture_pid":1439,"detection_enabled":1,"detection_fps":2.3,"ffmpeg_pid":1451,"pid":1363,"process_fps":5.0,"skipped_fps":0.0}}

Operating system

Debian

Install method

Docker Compose

Coral version

PCIe

Network connection

Wired

Camera make and model

NA

Any other information that may be helpful

No response

NickM-27 commented 1 year ago

I am unable to reproduce this. I see a couple things that can be cleaned up and could potentially be related to this and have created #7291

YBonline commented 1 year ago

Alright, I applied the patch. Just restarting 1 camera before the patch didn't do it, and didn't do it after, so I'll have to try to restart some switches and cause network chaos tonight when most people are sleeping to see if I can reproduce it with the patch. I've had it now happen 5 times, all after a network or camera being offline for several minutes. Will report back probably in the morning.

YBonline commented 1 year ago

I was able to restart one switch with just cameras on it, and memory is leaking with the patch.... Since the logs were much more reasonable now, I've attached a full log, but nothing is striking me as useful.

While the network switch is out, CPU usage stays at 115-125% and memory usage just continues to rise. Without network switch outages, I usually see extremely low CPU usage.

frigate_audio_memleak.txt

NickM-27 commented 1 year ago

Weird, I noticed that the audio logs are never printed. I wonder if this is part of the issue, perhaps the audio logs are not being released for whatever reason and that is what is using memory.

YBonline commented 1 year ago

I don't believe I've ever seen a log from the audio detector, outside of what you see in that log (process starting, cleanup on shutdown, etc). It does work perfectly if there's no network outages

NickM-27 commented 1 year ago

There should definitely also be logs for it "terminating existing ffmpeg". Something tells me it is not correctly stopping the ffmpeg process.

YBonline commented 1 year ago

That seems likely. /api/stats doesn't seem to list the crashed cameras audio ffmpegs ever again once they go offline, even after they restore. I've attached api/stats during the memleak

frigate_stats_audio_memleak.txt

YBonline commented 1 year ago

although maybe not, the crashed cameras don't seem to be here: root# ps ax | grep ffmpeg | grep audio 1598 ? Ssl 0:21 ffmpeg -hide_banner -loglevel warning -threads 2 -rtsp_transport tcp -timeout 5000000 -vn -i rtsp://127.0.0.1:8554/warehouse_exterior_doorbell -f s16le -ar 16000 -ac 1 -y /tmp/cache/warehouse_exterior_doorbell-audio 1602 ? Ssl 0:14 ffmpeg -hide_banner -loglevel warning -threads 2 -rtsp_transport tcp -timeout 5000000 -vn -i rtsp://127.0.0.1:8554/warehouse_exterior_front_sub -f s16le -ar 16000 -ac 1 -y /tmp/cache/warehouse_exterior_front-audio 1605 ? Ssl 0:09 ffmpeg -hide_banner -loglevel warning -threads 2 -rtsp_transport tcp -timeout 5000000 -vn -i rtsp://127.0.0.1:8554/warehouse_workbench_sub -f s16le -ar 16000 -ac 1 -y /tmp/cache/warehouse_workbench-audio 1612 ? Ssl 0:05 ffmpeg -hide_banner -loglevel warning -threads 2 -rtsp_transport tcp -timeout 5000000 -vn -i rtsp://127.0.0.1:8554/warehouse_interior_front_sub -f s16le -ar 16000 -ac 1 -y /tmp/cache/warehouse_interior_front-audio 1659 ? Ssl 0:09 ffmpeg -hide_banner -loglevel warning -threads 2 -rtsp_transport tcp -timeout 5000000 -vn -i rtsp://127.0.0.1:8554/gp636_doorbell -f s16le -ar 16000 -ac 1 -y /tmp/cache/gp636_doorbell-audio 1680 ? Ssl 0:43 ffmpeg -hide_banner -loglevel warning -threads 2 -rtsp_transport tcp -timeout 5000000 -vn -i rtsp://127.0.0.1:8554/gp611_rear -f s16le -ar 16000 -ac 1 -y /tmp/cache/gp611_rear-audio

NickM-27 commented 1 year ago

I see the problem, it is basically sitting at the pipe.open because data is never written to it

NickM-27 commented 1 year ago

@YBonline Okay I just pushed up, that should have the issue printing to logs and not getting stuck in an infinite loop

YBonline commented 1 year ago

So just applied your patch, and rebooted same switch as last time, and no infinite loop, no memory leak. However, for some reason, it appears the ffmpeg doesn't seem to want to restart for the audio side now 5 minutes after the video ffmpeg has restored. Infact, I think it might have never started at all? frigate_audio_memleak.txt

NickM-27 commented 1 year ago

It has to have started for the logs to be printed. It's also working for me with this code under normal circumstances. The code specifically checks if the process is currently running.

YBonline commented 1 year ago

Aren't the logs I attached indicating it sees the ffmpeg process is not running, and keeps trying to restart it? The AudioManager process is starting, just the audio ffmpeg seemingly isn't for some reason....

YBonline commented 1 year ago

It might be a config issue on these cameras, all the ones that are restarting seem to have never gotten an audio event, maybe a codec issue although recordings have audio? I'm investigating now

NickM-27 commented 1 year ago

Yeah it is working fine for me, as soon as the cameras are back online audio events work

YBonline commented 1 year ago

Ok, I think I know my mistake here... wanted to detail it just in case others run into it or you want to add something to the documentation about this.

When I ran into the DB locked issue, I noticed in my logs a ton of ffmpeg/AAC codec errors and had just enabled audio events when those errors started. I suspected the audio events were causing it, so I disabled it in the Frigate config, but the codec errors did not disappear, so then I ended up disabling the audio entirely on the cameras while we were working through that DB locked issue to try to keep those errors out of my logs, which worked to get those AAC codec errors gone.

Once you resolved that and I re-enabled the audio, but my AAC codec errors returned. I worked through it, and found that ffmpeg seems to want the sampling rate at 16kHz to not produce the error, and I originally had it at 48kHz. That successfully eliminated the AAC codec errors. For reference, these were with my Hikvision panoramic cameras

However, when I re-enabled the audio recordings on the cameras, I apparently only did it on the main stream, but I am detecting on the sub stream which was still inadvertently disabled.

I've fixed my configuration, rebooted the same switch again, and perfect performance! This bug appears to be fully fixed. Thank you again!

NickM-27 commented 1 year ago

Interesting, the ffmpeg profess should be setting the audio rate but maybe it doesn't like higher rate. My cameras are set to PCMU 8000 and it works without issues with audio events

Anyway, thanks for reporting and glad this could be solved.

YBonline commented 1 year ago

I'm just curious, is that a better codec/config then the AAC/16kHz rate?

Its also possible that the Hikvision is buggy with AAC/48kHz, I've found a few other settings that ffmpeg doesn't like on the video side.

On the cameras where I did not mess up the configuration, the audio events have been working perfect with the AAC codec as long as I didn't restart network equipment for several days now! I'm really hoping the detections for glass/crack/shatter/smash/breaking work well, that would make getting timely alerts a LOT easier. I've already found out "smoke_detector" gets vehicles with the backup alarm

NickM-27 commented 1 year ago

I just use it because it's compatible with mse and WebRTC.

And yeah I'm just using speech and bark on a couple cameras right now mainly to get notifications when my dog is barking or people are talking outside my door.

YBonline commented 1 year ago

Ah, ok, you know, I was wondering if it was possible within Frigate to get audio on the live streams, I guess that is why I haven't been able to?

Yup, the audio detections have been super accurate for animals. Infact, the animal audio detectors have already detected someone leaving a door open and my chickens getting inside my warehouse, along with my goats escaping their section to go join the chickens in their area lol

I'm gonna have to find some old pieces of glass and break them in front of the cameras to see if that works...