search

blakeblackshear / frigate

NVR with realtime local object detection for IP cameras
https://frigate.video
MIT License
16.02k stars 1.51k forks source link

[Support]: Re-Open my old post regarding Promox LXC out of resources #7476

Closed ranrinc closed 9 months ago

ranrinc commented 10 months ago

Describe the problem you are having

Running 7 camera with low resolutions (one cam 640x480 and the rest 640x360) Each detect will only grab very small resources until an error occurs on the logs (unpredictable) Then Frigate begin to consume memory usage until it runs out and it will continue to consume the swap memory, and the last draw is to consume the processor CPU (current setup using 8 core, 10 GB memory, and 4GB swap)

The latest configurations will take 8-16 minutes tops before the system totally freeze out.

CleanShot 2023-08-15 at 08 24 20

P.S. I replace go2rtc with the latest 1.6.2 as suggested, however, the stream comes out from my other go2rtc on a different PC. Basically, the new one did not do any streaming.

Version

0.12.0-DA3E197

Frigate config file

mqtt:
  host: 192.168.0.0
  port: 1883
  topic_prefix: fri
  client_id: fri
  user:
  password:
  stats_interval: 300

detectors:
  coral:
    type: edgetpu
    device: usb

birdseye:
  enabled: false

ffmpeg:
  input_args: preset-rtsp-restream

rtmp:
  enabled: false

ui:
  use_experimental: false
  live_mode: mse

cameras:
  front_garage:
    ffmpeg:
      inputs:
        - path: rtsp://192.168.0.0:8554/front_garage
          roles:
            - detect
    detect:
      width: 640
      height: 480
      fps: 5
    objects:
      track:
        - person
      filters:
        person:
          min_score: 0.4
          threshold: 0.5
          min_area: 70
    motion:
      mask:
        - 0,372,175,366,176,239,273,229,415,219,413,279,640,370,640,480,0,480

  front_porch:
    ffmpeg:
      inputs:
        - path: rtsp://192.168..0.0:8554/front_porch
          roles:
            - detect
    detect:
      width: 640
      height: 360
      fps: 5
    objects:
      track:
        - person
      filters:
        person:
          min_score: 0.4
          threshold: 0.5
          min_area: 70

  front_porch_2:
    ffmpeg:
      inputs:
        - path: rtsp://192.168.0.0:8554/front_porch2
          roles:
            - detect
    detect:
      width: 640
      height: 360
      fps: 5
    objects:
      track:
        - person
      filters:
        person:
          min_score: 0.4
          threshold: 0.5
          min_area: 70

  front_roof:
    ffmpeg:
      inputs:
        - path: rtsp://192.168.0.0:8554/front_roof
          roles:
            - detect
    detect:
      width: 640
      height: 360
      fps: 5
    objects:
      track:
        - person
      filters:
        person:
          min_score: 0.4
          threshold: 0.5
          min_area: 70

  back_roof:
    ffmpeg:
      inputs:
        - path: rtsp://192.168.0.0:8554/back_roof
          roles:
            - detect
    detect:
      width: 640
      height: 360
      fps: 5
    objects:
      track:
        - person
      filters:
        person:
          min_score: 0.4
          threshold: 0.5
          min_area: 70

  garden:
    ffmpeg:
      inputs:
        - path: rtsp://192.168.0.0:8554/garden
          roles:
            - detect
    detect:
      width: 640
      height: 360
      fps: 5
    objects:
      track:
        - person
      filters:
        person:
          min_score: 0.4
          threshold: 0.5
          min_area: 70

  kitchen:
    ffmpeg:
      inputs:
        - path: rtsp://192.168.0.0:8554/kitchen
          roles:
            - detect
    detect:
      width: 640
      height: 360
      fps: 5
    objects:
      track:
        - person
      filters:
        person:
          min_score: 0.4
          threshold: 0.5
          min_area: 70

Relevant log output

2023-08-15 08:20:17.092491049  [INFO] Starting Frigate...
2023-08-15 08:20:18.162489611  [2023-08-15 08:20:18] frigate.app                    INFO    : Starting Frigate (0.12.0-da3e197)
2023-08-15 08:20:18.187931841  [2023-08-15 08:20:18] frigate.app                    INFO    : Creating directory: /media/frigate/recordings
2023-08-15 08:20:18.187986179  [2023-08-15 08:20:18] frigate.app                    INFO    : Creating directory: /media/frigate/clips
2023-08-15 08:20:18.188929477  [2023-08-15 08:20:18] peewee_migrate                 INFO    : Starting migrations
2023-08-15 08:20:18.193544487  [2023-08-15 08:20:18] peewee_migrate                 INFO    : Migrate "001_create_events_table"
2023-08-15 08:20:18.193630882  [2023-08-15 08:20:18] peewee_migrate                 INFO    : sql ('CREATE TABLE IF NOT EXISTS "event" ("id" VARCHAR(30) NOT NULL PRIMARY KEY, "label" VARCHAR(20) NOT NULL, "camera" VARCHAR(20) NOT NULL, "start_time" DATETIME NOT NULL, "end_time" DATETIME NOT NULL, "top_score" REAL NOT NULL, "false_positive" INTEGER NOT NULL, "zones" JSON NOT NULL, "thumbnail" TEXT NOT NULL)',)
2023-08-15 08:20:18.193699049  [2023-08-15 08:20:18] peewee_migrate                 INFO    : sql ('CREATE INDEX IF NOT EXISTS "event_label" ON "event" ("label")',)
2023-08-15 08:20:18.193949994  [2023-08-15 08:20:18] peewee_migrate                 INFO    : sql ('CREATE INDEX IF NOT EXISTS "event_camera" ON "event" ("camera")',)
2023-08-15 08:20:18.194391471  [2023-08-15 08:20:18] peewee_migrate                 INFO    : Done 001_create_events_table
2023-08-15 08:20:18.197137904  [2023-08-15 08:20:18] peewee_migrate                 INFO    : Migrate "002_add_clip_snapshot"
2023-08-15 08:20:18.197185886  [2023-08-15 08:20:18] peewee_migrate                 INFO    : add_column ('event', 'has_clip', <BooleanField: Event.has_clip>)
2023-08-15 08:20:18.198725363  [2023-08-15 08:20:18] peewee_migrate                 INFO    : add_column ('event', 'has_snapshot', <BooleanField: Event.has_snapshot>)
2023-08-15 08:20:18.199888666  [2023-08-15 08:20:18] peewee_migrate                 INFO    : Done 002_add_clip_snapshot
2023-08-15 08:20:18.202407690  [2023-08-15 08:20:18] peewee_migrate                 INFO    : Migrate "003_create_recordings_table"
2023-08-15 08:20:18.202471317  [2023-08-15 08:20:18] peewee_migrate                 INFO    : sql ('CREATE TABLE IF NOT EXISTS "recordings" ("id" VARCHAR(30) NOT NULL PRIMARY KEY, "camera" VARCHAR(20) NOT NULL, "path" VARCHAR(255) NOT NULL, "start_time" DATETIME NOT NULL, "end_time" DATETIME NOT NULL, "duration" REAL NOT NULL)',)
2023-08-15 08:20:18.202609955  [2023-08-15 08:20:18] peewee_migrate                 INFO    : sql ('CREATE INDEX IF NOT EXISTS "recordings_camera" ON "recordings" ("camera")',)
2023-08-15 08:20:18.202735253  [2023-08-15 08:20:18] peewee_migrate                 INFO    : sql ('CREATE UNIQUE INDEX IF NOT EXISTS "recordings_path" ON "recordings" ("path")',)
2023-08-15 08:20:18.202929765  [2023-08-15 08:20:18] peewee_migrate                 INFO    : sql ('CREATE INDEX IF NOT EXISTS "recordings_start_time_end_time" ON "recordings" (start_time, end_time)',)
2023-08-15 08:20:18.203383046  [2023-08-15 08:20:18] peewee_migrate                 INFO    : Done 003_create_recordings_table
2023-08-15 08:20:18.206488680  [2023-08-15 08:20:18] peewee_migrate                 INFO    : Migrate "004_add_bbox_region_area"
2023-08-15 08:20:18.206538269  [2023-08-15 08:20:18] peewee_migrate                 INFO    : add_column ('event', 'region', <JSONField: Event.region>)
2023-08-15 08:20:18.208575027  [2023-08-15 08:20:18] peewee_migrate                 INFO    : add_column ('event', 'box', <JSONField: Event.box>)
2023-08-15 08:20:18.209132932  [2023-08-15 08:20:18] peewee_migrate                 INFO    : add_column ('event', 'area', <IntegerField: Event.area>)
2023-08-15 08:20:18.210370479  [2023-08-15 08:20:18] peewee_migrate                 INFO    : Done 004_add_bbox_region_area
2023-08-15 08:20:18.212905915  [2023-08-15 08:20:18] peewee_migrate                 INFO    : Migrate "005_make_end_time_nullable"
2023-08-15 08:20:18.212984419  [2023-08-15 08:20:18] peewee_migrate                 INFO    : drop_not_null ('event', 'end_time')
2023-08-15 08:20:18.214313040  [2023-08-15 08:20:18] peewee_migrate                 INFO    : Done 005_make_end_time_nullable
2023-08-15 08:20:18.217189660  [2023-08-15 08:20:18] peewee_migrate                 INFO    : Migrate "006_add_motion_active_objects"
2023-08-15 08:20:18.217252309  [2023-08-15 08:20:18] peewee_migrate                 INFO    : add_column ('recordings', 'objects', <IntegerField: Recordings.objects>)
2023-08-15 08:20:18.217569745  [2023-08-15 08:20:18] peewee_migrate                 INFO    : add_column ('recordings', 'motion', <IntegerField: Recordings.motion>)
2023-08-15 08:20:18.217959887  [2023-08-15 08:20:18] peewee_migrate                 INFO    : Done 006_add_motion_active_objects
2023-08-15 08:20:18.220578227  [2023-08-15 08:20:18] peewee_migrate                 INFO    : Migrate "007_add_retain_indefinitely"
2023-08-15 08:20:18.220629143  [2023-08-15 08:20:18] peewee_migrate                 INFO    : add_column ('event', 'retain_indefinitely', <BooleanField: Event.retain_indefinitely>)
2023-08-15 08:20:18.222660663  [2023-08-15 08:20:18] peewee_migrate                 INFO    : Done 007_add_retain_indefinitely
2023-08-15 08:20:18.225011366  [2023-08-15 08:20:18] peewee_migrate                 INFO    : Migrate "008_add_sub_label"
2023-08-15 08:20:18.225055437  [2023-08-15 08:20:18] peewee_migrate                 INFO    : add_column ('event', 'sub_label', <CharField: Event.sub_label>)
2023-08-15 08:20:18.225562007  [2023-08-15 08:20:18] peewee_migrate                 INFO    : Done 008_add_sub_label
2023-08-15 08:20:18.228727147  [2023-08-15 08:20:18] peewee_migrate                 INFO    : Migrate "009_add_object_filter_ratio"
2023-08-15 08:20:18.228795943  [2023-08-15 08:20:18] peewee_migrate                 INFO    : add_column ('event', 'ratio', <FloatField: Event.ratio>)
2023-08-15 08:20:18.229897854  [2023-08-15 08:20:18] peewee_migrate                 INFO    : Done 009_add_object_filter_ratio
2023-08-15 08:20:18.232747514  [2023-08-15 08:20:18] peewee_migrate                 INFO    : Migrate "010_add_plus_image_id"
2023-08-15 08:20:18.232801293  [2023-08-15 08:20:18] peewee_migrate                 INFO    : add_column ('event', 'plus_id', <CharField: Event.plus_id>)
2023-08-15 08:20:18.233295152  [2023-08-15 08:20:18] peewee_migrate                 INFO    : Done 010_add_plus_image_id
2023-08-15 08:20:18.235392604  [2023-08-15 08:20:18] peewee_migrate                 INFO    : Migrate "011_update_indexes"
2023-08-15 08:20:18.235452949  [2023-08-15 08:20:18] peewee_migrate                 INFO    : sql ('CREATE INDEX "event_start_time_end_time" ON "event" ("start_time" DESC, "end_time" DESC)',)
2023-08-15 08:20:18.235610305  [2023-08-15 08:20:18] peewee_migrate                 INFO    : sql ('DROP INDEX recordings_start_time_end_time',)
2023-08-15 08:20:18.235742028  [2023-08-15 08:20:18] peewee_migrate                 INFO    : sql ('CREATE INDEX "recordings_end_time_start_time" ON "recordings" ("end_time" DESC, "start_time" DESC)',)
2023-08-15 08:20:18.236145092  [2023-08-15 08:20:18] peewee_migrate                 INFO    : Done 011_update_indexes
2023-08-15 08:20:18.238510461  [2023-08-15 08:20:18] peewee_migrate                 INFO    : Migrate "012_add_segment_size"
2023-08-15 08:20:18.238545802  [2023-08-15 08:20:18] peewee_migrate                 INFO    : add_column ('recordings', 'segment_size', <FloatField: Recordings.segment_size>)
2023-08-15 08:20:18.240208063  [2023-08-15 08:20:18] peewee_migrate                 INFO    : Done 012_add_segment_size
2023-08-15 08:20:18.255778140  [2023-08-15 08:20:18] detector.coral                 INFO    : Starting detection process: 698
2023-08-15 08:20:20.896805716  [2023-08-15 08:20:18] frigate.app                    INFO    : Output process started: 700
2023-08-15 08:20:20.896854327  [2023-08-15 08:20:18] frigate.detectors.plugins.edgetpu_tfl INFO    : Attempting to load TPU as usb
2023-08-15 08:20:20.896890925  [2023-08-15 08:20:18] frigate.app                    INFO    : Camera processor started for front_garage: 707
2023-08-15 08:20:20.896924030  [2023-08-15 08:20:18] frigate.app                    INFO    : Camera processor started for front_porch: 708
2023-08-15 08:20:20.896956996  [2023-08-15 08:20:18] frigate.app                    INFO    : Camera processor started for front_porch_2: 710
2023-08-15 08:20:20.896991848  [2023-08-15 08:20:18] frigate.app                    INFO    : Camera processor started for front_roof: 711
2023-08-15 08:20:20.897025721  [2023-08-15 08:20:18] frigate.app                    INFO    : Camera processor started for back_roof: 713
2023-08-15 08:20:20.897059525  [2023-08-15 08:20:18] frigate.app                    INFO    : Camera processor started for garden: 714
2023-08-15 08:20:20.897091932  [2023-08-15 08:20:18] frigate.app                    INFO    : Camera processor started for kitchen: 716
2023-08-15 08:20:20.897131883  [2023-08-15 08:20:18] frigate.app                    INFO    : Capture process started for front_garage: 717
2023-08-15 08:20:20.897164080  [2023-08-15 08:20:18] frigate.app                    INFO    : Capture process started for front_porch: 721
2023-08-15 08:20:20.897193624  [2023-08-15 08:20:18] frigate.app                    INFO    : Capture process started for front_porch_2: 725
2023-08-15 08:20:20.897216602  [2023-08-15 08:20:18] frigate.app                    INFO    : Capture process started for front_roof: 730
2023-08-15 08:20:20.897276178  [2023-08-15 08:20:18] frigate.app                    INFO    : Capture process started for back_roof: 735
2023-08-15 08:20:20.897292801  [2023-08-15 08:20:18] frigate.app                    INFO    : Capture process started for garden: 741
2023-08-15 08:20:20.897331843  [2023-08-15 08:20:18] frigate.app                    INFO    : Capture process started for kitchen: 748
2023-08-15 08:20:20.903794965  [2023-08-15 08:20:20] frigate.detectors.plugins.edgetpu_tfl INFO    : TPU found
2023-08-15 08:22:38.321473382  [2023-08-15 08:22:38] watchdog.front_roof            INFO    : front_roof exceeded fps limit. Exiting ffmpeg...
2023-08-15 08:22:38.321565435  [2023-08-15 08:22:38] watchdog.front_roof            INFO    : Waiting for ffmpeg to exit gracefully...

FFprobe output from your camera

[{"return_code":0,"stderr":"","stdout":{"programs":[],"streams":[{"avg_frame_rate":"10/1","codec_long_name":"H.264/AVC/MPEG-4AVC/MPEG-4part10","height":360,"width":640},{"avg_frame_rate":"0/0","bit_rate":"64000","codec_long_name":"PCMA-law/G.711A-law"}]}}]

[{"return_code":0,"stderr":"","stdout":{"programs":[],"streams":[{"avg_frame_rate":"0/0","codec_long_name":"H.264/AVC/MPEG-4AVC/MPEG-4part10","height":480,"width":640},{"avg_frame_rate":"0/0","codec_long_name":"AAC(AdvancedAudioCoding)"}]}}]

[{"return_code":0,"stderr":"","stdout":{"programs":[],"streams":[{"avg_frame_rate":"10/1","codec_long_name":"H.264/AVC/MPEG-4AVC/MPEG-4part10","height":360,"width":640},{"avg_frame_rate":"0/0","bit_rate":"64000","codec_long_name":"PCMA-law/G.711A-law"}]}}]

[{"return_code":0,"stderr":"","stdout":{"programs":[],"streams":[{"avg_frame_rate":"10/1","codec_long_name":"H.264/AVC/MPEG-4AVC/MPEG-4part10","height":360,"width":640},{"avg_frame_rate":"0/0","bit_rate":"64000","codec_long_name":"PCMA-law/G.711A-law"}]}}]

[{"return_code":0,"stderr":"","stdout":{"programs":[],"streams":[{"avg_frame_rate":"10/1","codec_long_name":"H.264/AVC/MPEG-4AVC/MPEG-4part10","height":360,"width":640},{"avg_frame_rate":"0/0","bit_rate":"64000","codec_long_name":"PCMA-law/G.711A-law"}]}}]

[{"return_code":0,"stderr":"","stdout":{"programs":[],"streams":[{"avg_frame_rate":"10/1","codec_long_name":"H.264/AVC/MPEG-4AVC/MPEG-4part10","height":360,"width":640},{"avg_frame_rate":"0/0","bit_rate":"64000","codec_long_name":"PCMA-law/G.711A-law"}]}}]

[{"return_code":0,"stderr":"","stdout":{"programs":[],"streams":[{"avg_frame_rate":"20/1","codec_long_name":"H.264/AVC/MPEG-4AVC/MPEG-4part10","height":360,"width":640},{"avg_frame_rate":"0/0","bit_rate":"128000","codec_long_name":"PCMmu-law/G.711mu-law"}]}}]

Frigate stats

{"back_roof":{"camera_fps":5.0,"capture_pid":744,"detection_enabled":1,"detection_fps":0.0,"ffmpeg_pid":1885,"pid":712,"process_fps":5.0,"skipped_fps":0.0},"cpu_usages":{"%Cpu(s):":{"cpu":"id,","mem":"0.0"},"1":{"cpu":"0.0","mem":"0.0"},"114":{"cpu":"0.0","mem":"0.0"},"134":{"cpu":"0.0","mem":"0.0"},"135":{"cpu":"0.0","mem":"0.0"},"136":{"cpu":"0.0","mem":"0.0"},"145":{"cpu":"0.0","mem":"0.0"},"15":{"cpu":"0.0","mem":"0.0"},"16":{"cpu":"0.0","mem":"0.0"},"1609":{"cpu":"1.7","mem":"0.1"},"161":{"cpu":"0.0","mem":"0.0"},"180":{"cpu":"0.0","mem":"0.0"},"1883":{"cpu":"4.0","mem":"0.0"},"1884":{"cpu":"1.0","mem":"0.0"},"1885":{"cpu":"1.0","mem":"0.0"},"1889":{"cpu":"0.7","mem":"0.0"},"1890":{"cpu":"1.0","mem":"0.0"},"197":{"cpu":"0.0","mem":"0.0"},"2051":{"cpu":"0.0","mem":"0.0"},"2055":{"cpu":"0.0","mem":"0.0"},"2060":{"cpu":"0.0","mem":"0.0"},"226":{"cpu":"0.0","mem":"0.0"},"24":{"cpu":"0.0","mem":"0.0"},"25":{"cpu":"0.0","mem":"0.0"},"26":{"cpu":"0.0","mem":"0.0"},"269":{"cpu":"0.0","mem":"0.0"},"27":{"cpu":"0.0","mem":"0.0"},"28":{"cpu":"0.0","mem":"0.0"},"29":{"cpu":"0.0","mem":"0.0"},"30":{"cpu":"0.0","mem":"0.0"},"31":{"cpu":"0.0","mem":"0.0"},"32":{"cpu":"0.0","mem":"0.0"},"324":{"cpu":"0.0","mem":"0.0"},"374":{"cpu":"0.0","mem":"0.0"},"396":{"cpu":"0.0","mem":"0.0"},"403":{"cpu":"0.0","mem":"0.0"},"41":{"cpu":"0.0","mem":"0.0"},"42":{"cpu":"0.0","mem":"0.0"},"466":{"cpu":"0.0","mem":"0.0"},"534":{"cpu":"0.0","mem":"0.0"},"564":{"cpu":"0.0","mem":"0.0"},"690":{"cpu":"0.3","mem":"0.0"},"696":{"cpu":"0.0","mem":"0.0"},"699":{"cpu":"1.0","mem":"0.0"},"703":{"cpu":"0.7","mem":"0.0"},"704":{"cpu":"0.0","mem":"0.0"},"707":{"cpu":"0.3","mem":"0.0"},"708":{"cpu":"0.0","mem":"0.0"},"709":{"cpu":"0.0","mem":"0.0"},"710":{"cpu":"0.0","mem":"0.0"},"711":{"cpu":"0.0","mem":"0.0"},"712":{"cpu":"0.0","mem":"0.0"},"713":{"cpu":"0.0","mem":"0.0"},"714":{"cpu":"3.0","mem":"0.0"},"715":{"cpu":"0.3","mem":"0.0"},"716":{"cpu":"0.0","mem":"0.0"},"717":{"cpu":"0.3","mem":"0.0"},"720":{"cpu":"0.0","mem":"0.0"},"721":{"cpu":"0.3","mem":"0.0"},"725":{"cpu":"0.0","mem":"0.0"},"727":{"cpu":"0.0","mem":"0.0"},"731":{"cpu":"0.0","mem":"0.0"},"744":{"cpu":"0.3","mem":"0.0"},"749":{"cpu":"0.7","mem":"0.0"},"754":{"cpu":"0.3","mem":"0.0"},"80":{"cpu":"0.0","mem":"0.0"},"81":{"cpu":"0.0","mem":"0.0"},"82":{"cpu":"0.0","mem":"0.0"},"89":{"cpu":"0.0","mem":"0.0"},"97":{"cpu":"0.0","mem":"0.0"},"98":{"cpu":"1.3","mem":"0.1"},"MiB":{"cpu":"57582.8","mem":"avail"},"PID":{"cpu":"%CPU","mem":"%MEM"},"Tasks:":{"cpu":"stopped,","mem":"2"},"top":{"cpu":"users,","mem":"load"}},"detection_fps":2.2,"detectors":{"coral":{"detection_start":0.0,"inference_speed":7.9,"pid":1609}},"front_garage":{"camera_fps":5.0,"capture_pid":717,"detection_enabled":1,"detection_fps":0.0,"ffmpeg_pid":1889,"pid":703,"process_fps":5.0,"skipped_fps":0.0},"front_porch":{"camera_fps":5.0,"capture_pid":721,"detection_enabled":1,"detection_fps":0.0,"ffmpeg_pid":1890,"pid":707,"process_fps":5.0,"skipped_fps":0.0},"front_porch_2":{"camera_fps":46.6,"capture_pid":725,"detection_enabled":1,"detection_fps":0.0,"ffmpeg_pid":732,"pid":708,"process_fps":0.2,"skipped_fps":0.0},"front_roof":{"camera_fps":108.8,"capture_pid":731,"detection_enabled":1,"detection_fps":0.0,"ffmpeg_pid":1635,"pid":710,"process_fps":0.2,"skipped_fps":0.0},"garden":{"camera_fps":5.1,"capture_pid":749,"detection_enabled":1,"detection_fps":2.2,"ffmpeg_pid":1884,"pid":714,"process_fps":5.1,"skipped_fps":0.0},"kitchen":{"camera_fps":5.1,"capture_pid":754,"detection_enabled":1,"detection_fps":0.0,"ffmpeg_pid":1883,"pid":715,"process_fps":5.1,"skipped_fps":0.0},"service":{"last_updated":1692064492,"latest_version":"0.12.1","storage":{"/dev/shm":{"free":306.5,"mount_type":"tmpfs","total":314.6,"used":8.1},"/media/frigate/clips":{"free":14583.4,"mount_type":"overlay","total":20957.4,"used":5283.6},"/media/frigate/recordings":{"free":14583.4,"mount_type":"overlay","total":20957.4,"used":5283.6},"/tmp/cache":{"free":1000.0,"mount_type":"tmpfs","total":1000.0,"used":0.0}},"temperatures":{},"uptime":1789,"version":"0.12.0-da3e197"}}

Operating system

Proxmox

Install method

Docker Compose

Coral version

USB

Network connection

Wired

Camera make and model

Sonoff GK-200MP2-B, Reolink RLC-511W, Yi-cam h201c

Any other information that may be helpful

in relation to this close issue: https://github.com/blakeblackshear/frigate/issues/7441#event-10070474273

NickM-27 commented 10 months ago

Like before, need to get info on what processes or files in the container are actually using the memory.

Also, I'd still suggest using the internal go2rtc instead of external. Frigate has healthchecks and other things in place to ensure go2rtc is running as expected and shuts it down if not.

ranrinc commented 10 months ago

Like before, need to get info on what processes or files in the container are actually using the memory.

Also, I'd still suggest using the internal go2rtc instead of external. Frigate has healthchecks and other things in place to ensure go2rtc is running as expected and shuts it down if not.

It's a single Debian 12 LXC running docker and frigate and the latest go2rtc (without any stream.) Nothing else is running on it. Since it looks like RTSP is the one that keeps on crashing I move out go2rtc to another PC. So again basically that PC only runs a single LXC container (8 CPU, 10GB Memory, 4GB Swap) Plus it runs passthrough USB Google Coral TPU.

NickM-27 commented 10 months ago

Right, but we need to know what frigate process or file is using the memory.

ranrinc commented 10 months ago

Right, but we need to know what frigate process or file is using the memory.

Any tips on how I can get that informations? Again thanks for all the help.

NickM-27 commented 10 months ago

htop, df are good tools to get the info

it would also be suggested to limit the memory usage of frigate so it doesn't use all of the system memory

ranrinc commented 10 months ago

Hi @NickM-27,

Its quite late here. So I will try to see htop and df and will share the info here. As for limit the memory usage (any idea on how to do that?) thanks

NickM-27 commented 10 months ago

No idea, I don't use proxmox

ch1988 commented 10 months ago

Proxmox is based on actual Debian, I rjn some of these servers. LXC with Debian an frigate should be no problem. I also run one of these with out any problems.

Does frigate runs in a docker container - and do you use portainer for the dockers management! Maybe you could get some more information there?

jhallenb commented 10 months ago

I see the very same problem. Frigate 0.12.1 in a docker container Proxmox 8 with Frigate in a LXC Debian (with docker installed running Frigate, Portainer and some other docker containers) Coral M2 TPU RTMP disabled Using the built in go2rtc Two 1920x1080 cameras in 5fps Dedicated 4 cores and 8Gig RAM to the LXC. Running Home Assistant (HassOS)in a different VM on Proxmox The driver for the Coral is installed on the host (Proxmox) system and in the LXC.

When it happens the CPU is maxed out on the complete Proxmox system and so is the RAM. Strange thing is that Proxmox reports 0.95% CPU usage by the Frigate LXC. This is for sure incorrect since killing that LXC makes the CPU drop to 2-5% and the RAM usage back to normal (30% due to Home Assistant on a VM with dedicated RAM).

The problem goes away if I stop the Frigate docker immediate at startup. Otherwise, the problem shows up after some minutes to a day.

I have had to stop using Frigate now until this is fixed. Let me know if you need more data....but since the system lxc crashes when this happens I cannot really get much logs.

NickM-27 commented 10 months ago

So far from what I have seen there isn't anything specific that has been discovered as needing to be fixed on the frigate side.

Many proxmox users are running without issues and even more so frigate users in general. There have been a handful of reports of this type of issue for 0.12 but each report is different and ends up having a different root cause. For some users it was not related to frigate at all, another case was due to docker config issues, another case it had to do with using go2rtc 1.2.0 (the current version) and updating to the latest fixed the issue, and in another case it was caused by the hosts GPU drivers needing to be updated.

This is why we need actual specific information from the frigate processes to be able to help, each case is different.

Docker has tools to limit container memory usage so that should be used to ensure the system does not have problems, then it should not be difficult to see inside the container what is process or file is using memory.

jhallenb commented 10 months ago

@NickM-27 Thank you, can you direct me here please.

  1. Where can I find a proper build of go2rtc as you refer to?
  2. Do you have the case regarding hosts GPU. I have not updated my GPU at all. What information should I retrieve to check this?
  3. Do you have any resource for how to limit docker memory usage, it could make sense to add that to the frigate guide.
jhallenb commented 10 months ago

@NickM-27 also I managed to catch these logs. Looks like a ffmpeg crash. The GPU hardware acceleration seems like a good starting point. For now I tried to remove the ffmpeg transcoding to opus for both streams.

My ffmpeg config ffmpeg: global_args: -hide_banner -loglevel warning hwaccel_args: preset-vaapi

2023-08-17 19:15:39.177963858 [INFO] Starting go2rtc... 2023-08-17 19:15:39.257081504 21:15:39.257 INF go2rtc version 1.2.0 linux/amd64 2023-08-17 19:15:39.257286467 21:15:39.257 INF [api] listen addr=:1984 2023-08-17 19:15:39.257466872 21:15:39.257 INF [rtsp] listen addr=:8554 2023-08-17 19:15:39.257705024 21:15:39.257 INF [srtp] listen addr=:8443 2023-08-17 19:15:39.257830627 21:15:39.257 INF [webrtc] listen addr=:8555 2023-08-17 19:15:41.965913043 192.168.1.112 - - [17/Aug/2023:21:15:41 +0200] "GET /api/stats HTTP/1.1" 200 2333 "-" "HomeAssistant/2023.8.2 aiohttp/3.8.5 Python/3.11" "-" 2023-08-17 19:15:47.919654488 [2023-08-17 21:15:47] watchdog.kamera_baksidan ERROR : Ffmpeg process crashed unexpectedly for kamera_baksidan. 2023-08-17 19:15:47.919726772 [2023-08-17 21:15:47] watchdog.kamera_baksidan ERROR : The following ffmpeg logs include the last 100 lines prior to exit. 2023-08-17 19:15:47.919986326 [2023-08-17 21:15:47] ffmpeg.kamera_baksidan.detect ERROR : [h264 @ 0x55b78a5aef40] top block unavailable for requested intra mode

jhallenb commented 10 months ago

@NickM-27 Not sure if this tells you something but this is the VAINFO outut.

{"return_code":0,"stderr":"","stdout":"vainfo:VA-APIversion:1.17(libva2.10.0)nvainfo:Driverversion:InteliHDdriverforIntel(R)GenGraphics-23.1.1()nvainfo:SupportedprofileandentrypointsnVAProfileNone:tVAEntrypointVideoProcnVAProfileNone:tVAEntrypointStatsnVAProfileMPEG2Simple:tVAEntrypointVLDnVAProfileMPEG2Simple:tVAEntrypointEncSlicenVAProfileMPEG2Main:tVAEntrypointVLDnVAProfileMPEG2Main:tVAEntrypointEncSlicenVAProfileH264Main:tVAEntrypointVLDnVAProfileH264Main:tVAEntrypointEncSlicenVAProfileH264Main:tVAEntrypointFEInVAProfileH264Main:tVAEntrypointEncSliceLPnVAProfileH264High:tVAEntrypointVLDnVAProfileH264High:tVAEntrypointEncSlicenVAProfileH264High:tVAEntrypointFEInVAProfileH264High:tVAEntrypointEncSliceLPnVAProfileVC1Simple:tVAEntrypointVLDnVAProfileVC1Main:tVAEntrypointVLDnVAProfileVC1Advanced:tVAEntrypointVLDnVAProfileJPEGBaseline:tVAEntrypointVLDnVAProfileJPEGBaseline:tVAEntrypointEncPicturenVAProfileH264ConstrainedBaseline:tVAEntrypointVLDnVAProfileH264ConstrainedBaseline:tVAEntrypointEncSlicenVAProfileH264ConstrainedBaseline:tVAEntrypointFEInVAProfileH264ConstrainedBaseline:tVAEntrypointEncSliceLPnVAProfileVP8Version0_3:tVAEntrypointVLDnVAProfileVP8Version0_3:tVAEntrypointEncSlicenVAProfileHEVCMain:tVAEntrypointVLDnVAProfileHEVCMain:tVAEntrypointEncSlicenVAProfileHEVCMain:tVAEntrypointFEI"}

NickM-27 commented 10 months ago

@NickM-27 Thank you, can you direct me here please.

  1. Where can I find a proper build of go2rtc as you refer to?

https://github.com/AlexxIT/go2rtc/releases

  1. Do you have the case regarding hosts GPU. I have not updated my GPU at all. What information should I retrieve to check this?

I'll see if I can find it

  1. Do you have any resource for how to limit docker memory usage, it could make sense to add that to the frigate guide.

Not sure if it makes sense to add to the frigate guide since for most users it is not an issue, and it is just a standard docker config https://docs.docker.com/config/containers/resource_constraints/

global_args: -hide_banner -loglevel warning

you should not be overriding the global args. In this case you are removing args that reduce CPU usage

Not sure if this tells you something but this is the VAINFO outut.

What about vainfo on the host?

NickM-27 commented 10 months ago

Here is an example of updating kernel which fixed issues related to GPU https://github.com/blakeblackshear/frigate/issues/5799#issuecomment-1485571807 Here is an example where the issue was not related to frigate https://github.com/blakeblackshear/frigate/issues/6731#issuecomment-1614042925

ranrinc commented 10 months ago

Hi @NickM-27,

Been a while and still trying to fix the problem. So, far had 0 luck and ffmpeg keeps on crashing. I remove the CPU from the equations and just focus on TPU. Update go2rtc to the latest version and change LXC to VM which is more stable. Again the crash continues and now most of my screens are showing error messages.

CleanShot 2023-08-20 at 10 17 30

VM is still up and running without a problem.

Latest working error logs are as follow:

2023-08-19 22:05:13.262057813  [INFO] Starting Frigate...
2023-08-19 22:05:14.518892507  [2023-08-19 22:05:14] frigate.app                    INFO    : Starting Frigate (0.12.0-da3e197)
2023-08-19 22:05:14.549687400  [2023-08-19 22:05:14] peewee_migrate                 INFO    : Starting migrations
2023-08-19 22:05:14.553659847  [2023-08-19 22:05:14] peewee_migrate                 INFO    : There is nothing to migrate
2023-08-19 22:05:14.571846201  [2023-08-19 22:05:14] detector.coral                 INFO    : Starting detection process: 291
2023-08-19 22:05:14.572659620  [2023-08-19 22:05:14] frigate.app                    INFO    : Output process started: 293
2023-08-19 22:05:14.576910915  [2023-08-19 22:05:14] frigate.app                    INFO    : Camera processor started for front_porch: 299
2023-08-19 22:05:14.580534757  [2023-08-19 22:05:14] frigate.app                    INFO    : Camera processor started for front_porch_2: 300
2023-08-19 22:05:14.584155728  [2023-08-19 22:05:14] frigate.app                    INFO    : Camera processor started for front_roof: 302
2023-08-19 22:05:14.587749415  [2023-08-19 22:05:14] frigate.app                    INFO    : Camera processor started for back_roof: 303
2023-08-19 22:05:14.591716403  [2023-08-19 22:05:14] frigate.app                    INFO    : Camera processor started for garden: 305
2023-08-19 22:05:14.595609598  [2023-08-19 22:05:14] frigate.app                    INFO    : Camera processor started for kitchen: 306
2023-08-19 22:05:14.599469832  [2023-08-19 22:05:14] frigate.app                    INFO    : Capture process started for front_porch: 308
2023-08-19 22:05:14.603223290  [2023-08-19 22:05:14] frigate.app                    INFO    : Capture process started for front_porch_2: 309
2023-08-19 22:05:14.607668695  [2023-08-19 22:05:14] frigate.app                    INFO    : Capture process started for front_roof: 315
2023-08-19 22:05:14.611663956  [2023-08-19 22:05:14] frigate.app                    INFO    : Capture process started for back_roof: 318
2023-08-19 22:05:14.615925835  [2023-08-19 22:05:14] frigate.app                    INFO    : Capture process started for garden: 320
2023-08-19 22:05:14.622100024  [2023-08-19 22:05:14] frigate.app                    INFO    : Capture process started for kitchen: 323
2023-08-19 22:05:17.368312970  [2023-08-19 22:05:14] frigate.detectors.plugins.edgetpu_tfl INFO    : Attempting to load TPU as usb
2023-08-19 22:05:17.377504588  [2023-08-19 22:05:17] frigate.detectors.plugins.edgetpu_tfl INFO    : TPU found
2023-08-19 22:29:14.675973483  [2023-08-19 22:29:14] watchdog.back_roof             INFO    : back_roof exceeded fps limit. Exiting ffmpeg...
2023-08-19 22:29:14.677013782  [2023-08-19 22:29:14] watchdog.back_roof             INFO    : Waiting for ffmpeg to exit gracefully...
2023-08-19 22:29:14.679125223  [2023-08-19 22:29:14] watchdog.front_porch_2         INFO    : front_porch_2 exceeded fps limit. Exiting ffmpeg...
2023-08-19 22:29:14.680209738  [2023-08-19 22:29:14] watchdog.front_porch_2         INFO    : Waiting for ffmpeg to exit gracefully...
2023-08-19 22:29:51.173613801  [2023-08-19 22:29:51] watchdog.front_porch_2         INFO    : FFmpeg did not exit. Force killing...
2023-08-19 22:29:51.174342483  [2023-08-19 22:29:51] frigate.video                  ERROR   : front_porch_2: Unable to read frames from ffmpeg process.
2023-08-19 22:29:51.174384790  [2023-08-19 22:29:51] frigate.video                  ERROR   : front_porch_2: ffmpeg process is not running. exiting capture thread...
2023-08-19 22:30:14.891377624  [2023-08-19 22:30:14] frigate.watchdog               INFO    : Detection appears to be stuck. Restarting detection process...
2023-08-19 22:30:14.892502237  [2023-08-19 22:30:14] root                           INFO    : Waiting for detection process to exit gracefully...
2023-08-19 22:30:44.898228523  [2023-08-19 22:30:44] root                           INFO    : Detection process didnt exit. Force killing...
2023-08-19 22:30:44.904338651  [2023-08-19 22:30:44] root                           INFO    : Detection process has exited...
2023-08-19 22:30:44.919665272  [2023-08-19 22:30:44] detector.coral                 INFO    : Starting detection process: 1355
2023-08-19 22:30:44.944937399  [2023-08-19 22:30:44] frigate.detectors.plugins.edgetpu_tfl INFO    : Attempting to load TPU as usb
2023-08-19 22:30:47.753086218  [2023-08-19 22:30:47] frigate.detectors.plugins.edgetpu_tfl INFO    : TPU found
2023-08-19 23:06:04.787578742  [2023-08-19 23:06:04] watchdog.front_roof            INFO    : front_roof exceeded fps limit. Exiting ffmpeg...
2023-08-19 23:06:04.789801501  [2023-08-19 23:06:04] watchdog.front_roof            INFO    : Waiting for ffmpeg to exit gracefully...
2023-08-19 23:36:55.073020948  [2023-08-19 23:36:55] frigate.watchdog               INFO    : Detection appears to be stuck. Restarting detection process...
2023-08-19 23:36:55.073341040  [2023-08-19 23:36:55] root                           INFO    : Waiting for detection process to exit gracefully...
2023-08-19 23:37:25.100456941  [2023-08-19 23:37:25] root                           INFO    : Detection process didnt exit. Force killing...
2023-08-19 23:37:25.144797800  [2023-08-19 23:37:25] root                           INFO    : Detection process has exited...
2023-08-19 23:37:25.162627264  [2023-08-19 23:37:25] detector.coral                 INFO    : Starting detection process: 3674
2023-08-19 23:37:25.195499707  [2023-08-19 23:37:25] frigate.detectors.plugins.edgetpu_tfl INFO    : Attempting to load TPU as usb
2023-08-19 23:37:28.003252464  [2023-08-19 23:37:27] frigate.detectors.plugins.edgetpu_tfl INFO    : TPU found
2023-08-20 00:05:54.927661171  [2023-08-20 00:05:54] watchdog.garden                INFO    : garden exceeded fps limit. Exiting ffmpeg...
2023-08-20 00:05:54.930085987  [2023-08-20 00:05:54] watchdog.garden                INFO    : Waiting for ffmpeg to exit gracefully...
2023-08-20 00:47:04.982113894  [2023-08-20 00:47:04] watchdog.front_porch           INFO    : front_porch exceeded fps limit. Exiting ffmpeg...
2023-08-20 00:47:04.983937576  [2023-08-20 00:47:04] watchdog.front_porch           INFO    : Waiting for ffmpeg to exit gracefully...

If I return to version 0.11 then this problem will go away under RMTP.

Again I now running VM Debian 12 with USB passthrough

CleanShot 2023-08-20 at 10 22 53

It's running on Host CPU in which AMD Ryzen 7 PRO 4750G

The configuration did not changes and the only changes were from Container to VM

P.S. Since I only use TPU then I believe this have nothing to do with CPU

After restart the system working OK until its crashes again

2023-08-20 10:33:57.865312064  [INFO] Starting Frigate...
2023-08-20 10:33:58.980790260  [2023-08-20 10:33:58] frigate.app                    INFO    : Starting Frigate (0.12.0-da3e197)
2023-08-20 10:33:59.004807126  [2023-08-20 10:33:59] peewee_migrate                 INFO    : Starting migrations
2023-08-20 10:34:05.013720747  [2023-08-20 10:34:05] peewee_migrate                 INFO    : There is nothing to migrate
2023-08-20 10:34:05.031509070  [2023-08-20 10:34:05] detector.coral                 INFO    : Starting detection process: 290
2023-08-20 10:34:05.031845945  [2023-08-20 10:34:05] frigate.app                    INFO    : Output process started: 292
2023-08-20 10:34:05.035567452  [2023-08-20 10:34:05] frigate.app                    INFO    : Camera processor started for front_porch: 295
2023-08-20 10:34:05.037772703  [2023-08-20 10:34:05] frigate.detectors.plugins.edgetpu_tfl INFO    : Attempting to load TPU as usb
2023-08-20 10:34:05.039008979  [2023-08-20 10:34:05] frigate.app                    INFO    : Camera processor started for front_porch_2: 297
2023-08-20 10:34:05.042437688  [2023-08-20 10:34:05] frigate.app                    INFO    : Camera processor started for front_roof: 299
2023-08-20 10:34:05.045985101  [2023-08-20 10:34:05] frigate.app                    INFO    : Camera processor started for back_roof: 302
2023-08-20 10:34:05.049930500  [2023-08-20 10:34:05] frigate.app                    INFO    : Camera processor started for garden: 305
2023-08-20 10:34:05.055181973  [2023-08-20 10:34:05] frigate.app                    INFO    : Camera processor started for kitchen: 307
2023-08-20 10:34:05.059594967  [2023-08-20 10:34:05] frigate.app                    INFO    : Capture process started for front_porch: 308
2023-08-20 10:34:05.063430454  [2023-08-20 10:34:05] frigate.app                    INFO    : Capture process started for front_porch_2: 311
2023-08-20 10:34:05.067399049  [2023-08-20 10:34:05] frigate.app                    INFO    : Capture process started for front_roof: 314
2023-08-20 10:34:05.072343191  [2023-08-20 10:34:05] frigate.app                    INFO    : Capture process started for back_roof: 318
2023-08-20 10:34:05.075852074  [2023-08-20 10:34:05] frigate.app                    INFO    : Capture process started for garden: 323
2023-08-20 10:34:05.083207606  [2023-08-20 10:34:05] frigate.app                    INFO    : Capture process started for kitchen: 327
2023-08-20 10:34:07.855175267  [2023-08-20 10:34:07] frigate.detectors.plugins.edgetpu_tfl INFO    : TPU found
NickM-27 commented 10 months ago

Detection appears to be stuck. Restarting detection process...

Definitely looks like some issue with host hardware or software around the coral, other users reported this type of problem with proxmox as well.

Also, if you have a ryzen CPU with a GPU why are you not using hwaccel?

Starting Frigate (0.12.0-da3e197) also looks like you are running 0.12.0 not the up to date 0.12.1 (not that there were any important fixes relating to this, but best to be up to date)

ranrinc commented 10 months ago

Detection appears to be stuck. Restarting detection process...

Definitely looks like some issue with host hardware or software around the coral, other users reported this type of problem with proxmox as well.

Also, if you have a ryzen CPU with a GPU why are you not using hwaccel?

Starting Frigate (0.12.0-da3e197) also looks like you are running 0.12.0 not the up to date 0.12.1 (not that there were any important fixes relating to this, but best to be up to date)

I tried running with GPU and it make it even worst and usually, the crash is not being recovered. Instead, the server lost all its functions, and my bad in terms of setting the docker-compose. Maybe its time for me to test on Debian 11 instead <- its seems that TPU having a problem with Debian 12.

NickM-27 commented 10 months ago

I see, at this point it seems like these are proxmox related issues and I won't really be able to assist with that since I've not used it. The detection stuck issue has been reported by a number of proxmox users and if I recall correctly they found running an LXC fixed the issue.

My system runs native docker with an AMD 5700G APU without any issues, all I can suggest there is doublecheck the version of the driver that is installed on the host and make sure it is up to date.

ranrinc commented 10 months ago

Hi @NickM-27,

I temporarily want to prove a point, so this time I used the entire mini PC into Debian 11 (no more proxmox medium), with a CPU supporting ffmpeg and USB TPU.

The outcome that frigate did not crash is intriguing, however, the error from the camera is still continuing, so wondering why this happen. The reason that the system did not crash is that it has 100% resources from Mini which is AMD Ryzen 7 PRO 4750G + 64GB memory + 1TB NVME.

2023-08-22 11:43:32.905329707  [2023-08-22 11:43:32] watchdog.garden                INFO    : No frames received from garden in 20 seconds. Exiting ffmpeg...
2023-08-22 11:43:32.905468136  [2023-08-22 11:43:32] watchdog.garden                INFO    : Waiting for ffmpeg to exit gracefully...
2023-08-22 11:43:32.965555832  [2023-08-22 11:43:32] frigate.video                  ERROR   : garden: Unable to read frames from ffmpeg process.
2023-08-22 11:43:32.966185539  [2023-08-22 11:43:32] frigate.video                  ERROR   : garden: ffmpeg process is not running. exiting capture thread...
2023-08-22 11:43:42.968335664  [2023-08-22 11:43:42] watchdog.garden                ERROR   : Ffmpeg process crashed unexpectedly for garden.
2023-08-22 11:43:42.968501123  [2023-08-22 11:43:42] watchdog.garden                ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2023-08-22 11:43:42.968711840  [2023-08-22 11:43:42] ffmpeg.garden.detect           ERROR   : [rtsp @ 0x558e722ede40] RTP: PT=60: bad cseq 16fa expected=425f
2023-08-22 11:44:12.982329484  [2023-08-22 11:44:12] watchdog.garden                INFO    : garden exceeded fps limit. Exiting ffmpeg...
2023-08-22 11:44:12.982397232  [2023-08-22 11:44:12] watchdog.garden                INFO    : Waiting for ffmpeg to exit gracefully...
2023-08-22 11:44:43.013118091  [2023-08-22 11:44:43] watchdog.garden                INFO    : FFmpeg did not exit. Force killing...
2023-08-22 11:44:44.061174732  [2023-08-22 11:44:44] frigate.video                  ERROR   : garden: Unable to read frames from ffmpeg process.
2023-08-22 11:44:44.061373227  [2023-08-22 11:44:44] frigate.video                  ERROR   : garden: ffmpeg process is not running. exiting capture thread...
2023-08-22 11:44:54.062361468  [2023-08-22 11:44:54] watchdog.garden                ERROR   : Ffmpeg process crashed unexpectedly for garden.
2023-08-22 11:44:54.062515473  [2023-08-22 11:44:54] watchdog.garden                ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2023-08-22 11:44:54.062627432  [2023-08-22 11:44:54] ffmpeg.garden.detect           ERROR   : [rtsp @ 0x5614f9e29e40] RTP: PT=60: bad cseq 05f9 expected=1f2a
2023-08-22 11:44:54.062741905  [2023-08-22 11:44:54] ffmpeg.garden.detect           ERROR   : [out_0_0 @ 0x5614fa140540] 100 buffers queued in out_0_0, something may be wrong.
2023-08-22 11:44:54.062848206  [2023-08-22 11:44:54] ffmpeg.garden.detect           ERROR   : [out_0_0 @ 0x5614fa140540] 1000 buffers queued in out_0_0, something may be wrong.
2023-08-22 11:44:54.062959047  [2023-08-22 11:44:54] ffmpeg.garden.detect           ERROR   : [out_0_0 @ 0x5614fa140540] 10000 buffers queued in out_0_0, something may be wrong.
2023-08-22 11:45:34.079944408  [2023-08-22 11:45:34] watchdog.garden                INFO    : garden exceeded fps limit. Exiting ffmpeg...
2023-08-22 11:45:34.080266874  [2023-08-22 11:45:34] watchdog.garden                INFO    : Waiting for ffmpeg to exit gracefully...
2023-08-22 11:48:07.244065738  [2023-08-22 11:48:07] frigate.video                  ERROR   : garden: Unable to read frames from ffmpeg process.
2023-08-22 11:48:07.244167779  [2023-08-22 11:48:07] frigate.video                  ERROR   : garden: ffmpeg process is not running. exiting capture thread...
2023-08-22 11:48:17.219436568  [2023-08-22 11:48:17] watchdog.garden                ERROR   : Ffmpeg process crashed unexpectedly for garden.
2023-08-22 11:48:17.219567105  [2023-08-22 11:48:17] watchdog.garden                ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2023-08-22 11:48:17.219698270  [2023-08-22 11:48:17] ffmpeg.garden.detect           ERROR   : [rtsp @ 0x55c0c2f03e40] RTP: PT=60: bad cseq 222b expected=0f87
2023-08-22 11:48:17.219794933  [2023-08-22 11:48:17] ffmpeg.garden.detect           ERROR   : [out_0_0 @ 0x55c0c33bd080] 100 buffers queued in out_0_0, something may be wrong.
2023-08-22 11:48:17.219881958  [2023-08-22 11:48:17] ffmpeg.garden.detect           ERROR   : [out_0_0 @ 0x55c0c33bd080] 1000 buffers queued in out_0_0, something may be wrong.
2023-08-22 11:48:17.219988818  [2023-08-22 11:48:17] ffmpeg.garden.detect           ERROR   : [out_0_0 @ 0x55c0c33bd080] 10000 buffers queued in out_0_0, something may be wrong.
2023-08-22 11:48:17.220063550  [2023-08-22 11:48:17] ffmpeg.garden.detect           ERROR   : [out_0_0 @ 0x55c0c33bd080] 100000 buffers queued in out_0_0, something may be wrong.
2023-08-22 11:51:07.275970662  [2023-08-22 11:51:07] watchdog.garden                INFO    : No frames received from garden in 20 seconds. Exiting ffmpeg...
2023-08-22 11:51:07.276157632  [2023-08-22 11:51:07] watchdog.garden                INFO    : Waiting for ffmpeg to exit gracefully...
2023-08-22 11:51:07.324508142  [2023-08-22 11:51:07] frigate.video                  ERROR   : garden: Unable to read frames from ffmpeg process.
2023-08-22 11:51:07.325529040  [2023-08-22 11:51:07] frigate.video                  ERROR   : garden: ffmpeg process is not running. exiting capture thread...
2023-08-22 11:51:17.327096519  [2023-08-22 11:51:17] watchdog.garden                ERROR   : Ffmpeg process crashed unexpectedly for garden.
2023-08-22 11:51:17.327259184  [2023-08-22 11:51:17] watchdog.garden                ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2023-08-22 11:51:17.327351237  [2023-08-22 11:51:17] ffmpeg.garden.detect           ERROR   : rtsp://127.0.0.1:8554/garden: Connection timed out
2023-08-22 11:51:17.327497978  [2023-08-22 11:51:17] ffmpeg.garden.detect           ERROR   : [rtsp @ 0x55c0e4f95e40] RTP: PT=60: bad cseq 23f4 expected=17fc
2023-08-22 11:51:37.344901833  [2023-08-22 11:51:37] watchdog.garden                INFO    : garden exceeded fps limit. Exiting ffmpeg...
2023-08-22 11:51:37.344953866  [2023-08-22 11:51:37] watchdog.garden                INFO    : Waiting for ffmpeg to exit gracefully...
2023-08-22 11:53:51.318751417  [2023-08-22 11:53:51] frigate.video                  ERROR   : garden: Unable to read frames from ffmpeg process.
2023-08-22 11:53:51.318763989  [2023-08-22 11:53:51] frigate.video                  ERROR   : garden: ffmpeg process is not running. exiting capture thread...
2023-08-22 11:54:01.292201942  [2023-08-22 11:54:01] watchdog.garden                ERROR   : Ffmpeg process crashed unexpectedly for garden.
2023-08-22 11:54:01.301933477  [2023-08-22 11:54:01] watchdog.garden                ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2023-08-22 11:54:01.303732149  [2023-08-22 11:54:01] ffmpeg.garden.detect           ERROR   : [rtsp @ 0x557a37898e40] RTP: PT=60: bad cseq 190f expected=289a
2023-08-22 11:54:01.303817358  [2023-08-22 11:54:01] ffmpeg.garden.detect           ERROR   : [out_0_0 @ 0x557a37d54f80] 100 buffers queued in out_0_0, something may be wrong.
2023-08-22 11:54:01.303937419  [2023-08-22 11:54:01] ffmpeg.garden.detect           ERROR   : [out_0_0 @ 0x557a37d54f80] 1000 buffers queued in out_0_0, something may be wrong.
2023-08-22 11:54:01.304067397  [2023-08-22 11:54:01] ffmpeg.garden.detect           ERROR   : [out_0_0 @ 0x557a37d54f80] 10000 buffers queued in out_0_0, something may be wrong.
2023-08-22 11:54:01.304157285  [2023-08-22 11:54:01] ffmpeg.garden.detect           ERROR   : [out_0_0 @ 0x557a37d54f80] 100000 buffers queued in out_0_0, something may be wrong.
2023-08-22 11:55:11.340653782  [2023-08-22 11:55:11] watchdog.garden                INFO    : garden exceeded fps limit. Exiting ffmpeg...
2023-08-22 11:55:11.340747093  [2023-08-22 11:55:11] watchdog.garden                INFO    : Waiting for ffmpeg to exit gracefully...
2023-08-22 11:55:11.387289704  [2023-08-22 11:55:11] frigate.video                  ERROR   : garden: Unable to read frames from ffmpeg process.
2023-08-22 11:55:11.387397123  [2023-08-22 11:55:11] frigate.video                  ERROR   : garden: ffmpeg process is not running. exiting capture thread...
2023-08-22 11:55:21.388961393  [2023-08-22 11:55:21] watchdog.garden                ERROR   : Ffmpeg process crashed unexpectedly for garden.
2023-08-22 11:55:21.389129366  [2023-08-22 11:55:21] watchdog.garden                ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2023-08-22 11:55:21.389232315  [2023-08-22 11:55:21] ffmpeg.garden.detect           ERROR   : [rtsp @ 0x561ab77d5e40] RTP: PT=60: bad cseq 13fa expected=3000
2023-08-22 12:27:13.647997910  [2023-08-22 12:27:13] watchdog.front_porch           INFO    : front_porch exceeded fps limit. Exiting ffmpeg...
2023-08-22 12:27:13.651179838  [2023-08-22 12:27:13] watchdog.front_porch           INFO    : Waiting for ffmpeg to exit gracefully...

Also, the problem happens only from single brand Sonoff Camera only, so unsure why.

NickM-27 commented 10 months ago

garden exceeded fps limit. Exiting ffmpeg...

Is this a wifi camera? It seems to be sending bad data

ranrinc commented 10 months ago

garden exceeded fps limit. Exiting ffmpeg...

Is this a wifi camera? It seems to be sending bad data

No all my camera is connected via cable

NickM-27 commented 10 months ago

May want to try connecting to that camera directly without go2rtc and see if it makes a difference

github-actions[bot] commented 9 months ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.