search

blakeblackshear / frigate

NVR with realtime local object detection for IP cameras
https://frigate.video
MIT License
17.99k stars 1.64k forks source link

[Support]: Frigate stopped working #7672

Closed wilcomir closed 1 year ago

wilcomir commented 1 year ago

Describe the problem you are having

As of tonight, the ffmpeg part of frigate on my install stopped working.

I have a go2rtc setup with 6 cameras, and I verified through their app that the stream is solid. Frigate go2rtc shows the streams correctly, but the mse restreams (and homeassistant) are not working anymore.

I examined the logs, attached below, and I see a number of warnings/errors that seemingly were not there before. I am attaching both the config & the logs, keep in mind that I have snipped the config file to use only a camera, and the issue persists.

What is most surprising to me is that I did not update or change anything, it just started failing hard seemingly all by itself.

Version

0.12.1

Frigate config file

database:
  path: /db/frigate.db

mqtt:
  enabled: True
  host: ip
  port: 1234
  user: user
  password: password

birdseye:
  enabled: True
  mode: objects
  quality: 8

detectors:
  coral:
    type: edgetpu
    device: usb

detect:
  enabled: True

record:
  enabled: True
  expire_interval: 720
  retain:
    days: 0
  events:
    pre_capture: 5
    post_capture: 5
    objects:
      - cat
    retain:
      default: 365
      mode: active_objects

snapshots:
  enabled: True
  timestamp: True
  bounding_box: False
  crop: False
  retain:
    default: 0
    objects:
      cat: 3650

timestamp_style:
  position: "tl"
  format: "%a - %Y/%m/%d %H:%M:%S"
  color:
    red: 255
    green: 255
    blue: 255
  thickness: 2
  effect: shadow

ffmpeg:
  hwaccel_args: preset-vaapi

objects:
  track:
    - person
    - cat

go2rtc:
  streams:
    kitchen:
      - rtsp://usr:psw@ip/live0
    kitchen_cabinets:
      - rtsp://usr:psw@ip/live0
    bedroom:
      - rtsp://usr:psw@ip/live0
    living:
      - rtsp://usr:psw@ip/live0
    porch:
      - rtsp://usr:psw@ip/live0
    front:
      - rtsp://usr:psw@ip/live0

cameras:
  kitchen:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/kitchen
          input_args: preset-rtsp-restream
          roles:
            - detect
            - record
    zones:
      cat_feeder:
        coordinates: 1280,250,1280,433,1219,650,841,609,866,386,1056,397,1092,222
        objects:
          - cat
    objects:
      filters:
        peson:
          min_area: 100000
          min_score: 0.6

Relevant log output

2023-09-04 09:06:44.779746075  [INFO] Starting Frigate...
2023-09-04 09:06:45.515769581  [2023-09-04 09:06:45] frigate.app                    INFO    : Starting Frigate (0.12.1-367d724)
2023-09-04 09:06:45.539462345  [2023-09-04 09:06:45] peewee_migrate                 INFO    : Starting migrations
2023-09-04 09:06:45.542886260  [2023-09-04 09:06:45] peewee_migrate                 INFO    : There is nothing to migrate
2023-09-04 09:06:45.552800064  [2023-09-04 09:06:45] detector.coral                 INFO    : Starting detection process: 685
2023-09-04 09:06:48.192418175  [2023-09-04 09:06:45] frigate.app                    INFO    : Output process started: 687
2023-09-04 09:06:48.192488237  [2023-09-04 09:06:45] frigate.detectors.plugins.edgetpu_tfl INFO    : Attempting to load TPU as usb
2023-09-04 09:06:48.192593153  [2023-09-04 09:06:45] frigate.app                    INFO    : Camera processor started for kitchen: 692
2023-09-04 09:06:48.192595344  [2023-09-04 09:06:45] frigate.app                    INFO    : Capture process started for kitchen: 694
2023-09-04 09:06:48.196013322  [2023-09-04 09:06:48] frigate.detectors.plugins.edgetpu_tfl INFO    : TPU found
2023-09-04 09:06:50.250426421  [2023-09-04 09:06:50] frigate.video                  ERROR   : kitchen: Unable to read frames from ffmpeg process.
2023-09-04 09:06:50.250494790  [2023-09-04 09:06:50] frigate.video                  ERROR   : kitchen: ffmpeg process is not running. exiting capture thread...
2023-09-04 09:06:50.721947442  [2023-09-04 09:06:50] frigate.record                 WARNING : Failed to probe corrupt segment /tmp/cache/kitchen-20230904090650.mp4: 0 - b'[mov,mp4,m4a,3gp,3g2,mj2 @ 0x55babc1b66c0] moov atom not found\n/tmp/cache/kitchen-20230904090650.mp4: Invalid data found when processing input\n'
2023-09-04 09:06:50.721950575  [2023-09-04 09:06:50] frigate.record                 WARNING : Discarding a corrupt recording segment: /tmp/cache/kitchen-20230904090650.mp4
2023-09-04 09:07:05.584410434  [2023-09-04 09:07:05] watchdog.kitchen               ERROR   : Ffmpeg process crashed unexpectedly for kitchen.
2023-09-04 09:07:05.584426628  [2023-09-04 09:07:05] watchdog.kitchen               ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2023-09-04 09:07:05.584491749  [2023-09-04 09:07:05] ffmpeg.kitchen.detect          ERROR   : [segment @ 0x564e05a32140] Timestamps are unset in a packet for stream 0. This is deprecated and will stop working in the future. Fix your code to set the timestamps properly
2023-09-04 09:07:05.584535218  [2023-09-04 09:07:05] ffmpeg.kitchen.detect          ERROR   : [h264 @ 0x564e05a21880] Failed to create decode context: 2 (resource allocation failed).
2023-09-04 09:07:05.584590328  [2023-09-04 09:07:05] ffmpeg.kitchen.detect          ERROR   : [h264 @ 0x564e05a21880] Failed setup for format vaapi: hwaccel initialisation returned error.
2023-09-04 09:07:05.584620051  [2023-09-04 09:07:05] ffmpeg.kitchen.detect          ERROR   : Impossible to convert between the formats supported by the filter 'Parsed_fps_0' and the filter 'auto_scale_0'
2023-09-04 09:07:05.584656350  [2023-09-04 09:07:05] ffmpeg.kitchen.detect          ERROR   : Error reinitializing filters!
2023-09-04 09:07:05.584693524  [2023-09-04 09:07:05] ffmpeg.kitchen.detect          ERROR   : Failed to inject frame into filter network: Function not implemented
2023-09-04 09:07:05.584734512  [2023-09-04 09:07:05] ffmpeg.kitchen.detect          ERROR   : Error while processing the decoded data for stream #0:0
2023-09-04 09:07:10.250046254  [2023-09-04 09:07:10] frigate.video                  ERROR   : kitchen: Unable to read frames from ffmpeg process.
2023-09-04 09:07:10.250113504  [2023-09-04 09:07:10] frigate.video                  ERROR   : kitchen: ffmpeg process is not running. exiting capture thread...
2023-09-04 09:07:10.723302242  [2023-09-04 09:07:10] frigate.record                 WARNING : Failed to probe corrupt segment /tmp/cache/kitchen-20230904090710.mp4: 0 - b'[mov,mp4,m4a,3gp,3g2,mj2 @ 0x5619c0c806c0] moov atom not found\n/tmp/cache/kitchen-20230904090710.mp4: Invalid data found when processing input\n'
2023-09-04 09:07:10.723305741  [2023-09-04 09:07:10] frigate.record                 WARNING : Discarding a corrupt recording segment: /tmp/cache/kitchen-20230904090710.mp4
2023-09-04 09:07:15.589098577  [2023-09-04 09:07:15] watchdog.kitchen               ERROR   : Ffmpeg process crashed unexpectedly for kitchen.
2023-09-04 09:07:15.589260248  [2023-09-04 09:07:15] watchdog.kitchen               ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2023-09-04 09:07:15.589271011  [2023-09-04 09:07:15] ffmpeg.kitchen.detect          ERROR   : [segment @ 0x5647c0420980] Timestamps are unset in a packet for stream 0. This is deprecated and will stop working in the future. Fix your code to set the timestamps properly
2023-09-04 09:07:15.589327835  [2023-09-04 09:07:15] ffmpeg.kitchen.detect          ERROR   : [h264 @ 0x5647c039f740] Failed to create decode context: 2 (resource allocation failed).
2023-09-04 09:07:15.589370516  [2023-09-04 09:07:15] ffmpeg.kitchen.detect          ERROR   : [h264 @ 0x5647c039f740] Failed setup for format vaapi: hwaccel initialisation returned error.
2023-09-04 09:07:15.589399477  [2023-09-04 09:07:15] ffmpeg.kitchen.detect          ERROR   : Impossible to convert between the formats supported by the filter 'Parsed_fps_0' and the filter 'auto_scale_0'
2023-09-04 09:07:15.589433397  [2023-09-04 09:07:15] ffmpeg.kitchen.detect          ERROR   : Error reinitializing filters!
2023-09-04 09:07:15.589474346  [2023-09-04 09:07:15] ffmpeg.kitchen.detect          ERROR   : Failed to inject frame into filter network: Function not implemented
2023-09-04 09:07:15.589527363  [2023-09-04 09:07:15] ffmpeg.kitchen.detect          ERROR   : Error while processing the decoded data for stream #0:0
2023-09-04 09:07:20.251962048  [2023-09-04 09:07:20] frigate.video                  ERROR   : kitchen: Unable to read frames from ffmpeg process.
2023-09-04 09:07:20.251994038  [2023-09-04 09:07:20] frigate.video                  ERROR   : kitchen: ffmpeg process is not running. exiting capture thread...
2023-09-04 09:07:20.723229566  [2023-09-04 09:07:20] frigate.record                 WARNING : Failed to probe corrupt segment /tmp/cache/kitchen-20230904090720.mp4: 0 - b'[mov,mp4,m4a,3gp,3g2,mj2 @ 0x5653cc7516c0] moov atom not found\n/tmp/cache/kitchen-20230904090720.mp4: Invalid data found when processing input\n'
2023-09-04 09:07:20.723232417  [2023-09-04 09:07:20] frigate.record                 WARNING : Discarding a corrupt recording segment: /tmp/cache/kitchen-20230904090720.mp4
2023-09-04 09:07:25.592107035  [2023-09-04 09:07:25] watchdog.kitchen               ERROR   : Ffmpeg process crashed unexpectedly for kitchen.
2023-09-04 09:07:25.592110433  [2023-09-04 09:07:25] watchdog.kitchen               ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2023-09-04 09:07:25.592112124  [2023-09-04 09:07:25] ffmpeg.kitchen.detect          ERROR   : [segment @ 0x565019e13500] Timestamps are unset in a packet for stream 0. This is deprecated and will stop working in the future. Fix your code to set the timestamps properly
2023-09-04 09:07:25.592119209  [2023-09-04 09:07:25] ffmpeg.kitchen.detect          ERROR   : [h264 @ 0x565019ef32c0] Failed to create decode context: 2 (resource allocation failed).
2023-09-04 09:07:25.592135083  [2023-09-04 09:07:25] ffmpeg.kitchen.detect          ERROR   : [h264 @ 0x565019ef32c0] Failed setup for format vaapi: hwaccel initialisation returned error.
2023-09-04 09:07:25.592136425  [2023-09-04 09:07:25] ffmpeg.kitchen.detect          ERROR   : Impossible to convert between the formats supported by the filter 'Parsed_fps_0' and the filter 'auto_scale_0'
2023-09-04 09:07:25.592152242  [2023-09-04 09:07:25] ffmpeg.kitchen.detect          ERROR   : Error reinitializing filters!
2023-09-04 09:07:25.592223276  [2023-09-04 09:07:25] ffmpeg.kitchen.detect          ERROR   : Failed to inject frame into filter network: Function not implemented
2023-09-04 09:07:25.592225421  [2023-09-04 09:07:25] ffmpeg.kitchen.detect          ERROR   : Error while processing the decoded data for stream #0:0
2023-09-04 09:07:30.250210443  [2023-09-04 09:07:30] frigate.video                  ERROR   : kitchen: Unable to read frames from ffmpeg process.
2023-09-04 09:07:30.250254175  [2023-09-04 09:07:30] frigate.video                  ERROR   : kitchen: ffmpeg process is not running. exiting capture thread...
2023-09-04 09:07:30.724092008  [2023-09-04 09:07:30] frigate.record                 WARNING : Failed to probe corrupt segment /tmp/cache/kitchen-20230904090730.mp4: 0 - b'[mov,mp4,m4a,3gp,3g2,mj2 @ 0x55f8121706c0] moov atom not found\n/tmp/cache/kitchen-20230904090730.mp4: Invalid data found when processing input\n'
2023-09-04 09:07:30.724095404  [2023-09-04 09:07:30] frigate.record                 WARNING : Discarding a corrupt recording segment: /tmp/cache/kitchen-20230904090730.mp4
2023-09-04 09:07:35.599173848  [2023-09-04 09:07:35] watchdog.kitchen               ERROR   : Ffmpeg process crashed unexpectedly for kitchen.
2023-09-04 09:07:35.599236201  [2023-09-04 09:07:35] watchdog.kitchen               ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2023-09-04 09:07:35.599253335  [2023-09-04 09:07:35] ffmpeg.kitchen.detect          ERROR   : [segment @ 0x55601a3bbdc0] Timestamps are unset in a packet for stream 0. This is deprecated and will stop working in the future. Fix your code to set the timestamps properly
2023-09-04 09:07:35.599458936  [2023-09-04 09:07:35] ffmpeg.kitchen.detect          ERROR   : [h264 @ 0x55601a4cbac0] Failed to create decode context: 2 (resource allocation failed).
2023-09-04 09:07:35.599461814  [2023-09-04 09:07:35] ffmpeg.kitchen.detect          ERROR   : [h264 @ 0x55601a4cbac0] Failed setup for format vaapi: hwaccel initialisation returned error.
2023-09-04 09:07:35.599463011  [2023-09-04 09:07:35] ffmpeg.kitchen.detect          ERROR   : Impossible to convert between the formats supported by the filter 'Parsed_fps_0' and the filter 'auto_scale_0'
2023-09-04 09:07:35.599463908  [2023-09-04 09:07:35] ffmpeg.kitchen.detect          ERROR   : Error reinitializing filters!
2023-09-04 09:07:35.599537323  [2023-09-04 09:07:35] ffmpeg.kitchen.detect          ERROR   : Failed to inject frame into filter network: Function not implemented
2023-09-04 09:07:35.599538235  [2023-09-04 09:07:35] ffmpeg.kitchen.detect          ERROR   : Error while processing the decoded data for stream #0:0
2023-09-04 09:07:40.320165343  [2023-09-04 09:07:40] frigate.video                  ERROR   : kitchen: Unable to read frames from ffmpeg process.
2023-09-04 09:07:40.320169130  [2023-09-04 09:07:40] frigate.video                  ERROR   : kitchen: ffmpeg process is not running. exiting capture thread...
2023-09-04 09:07:40.724059539  [2023-09-04 09:07:40] frigate.record                 WARNING : Failed to probe corrupt segment /tmp/cache/kitchen-20230904090740.mp4: 0 - b'[mov,mp4,m4a,3gp,3g2,mj2 @ 0x55ef073f76c0] moov atom not found\n/tmp/cache/kitchen-20230904090740.mp4: Invalid data found when processing input\n'
2023-09-04 09:07:40.724119673  [2023-09-04 09:07:40] frigate.record                 WARNING : Discarding a corrupt recording segment: /tmp/cache/kitchen-20230904090740.mp4

FFprobe output from your camera

[{"return_code":0,"stderr":"","stdout":{"programs":[],"streams":[{"avg_frame_rate":"15/1","codec_long_name":"H.264/AVC/MPEG-4AVC/MPEG-4part10","height":1080,"width":1920},{"avg_frame_rate":"0/0","codec_long_name":"AAC(AdvancedAudioCoding)"}]}}]

Frigate stats

{"cpu_usages":{"%Cpu(s):":{"cpu":"id,","mem":"0.1"},"1":{"cpu":"0.0","mem":"0.0"},"109":{"cpu":"1.3","mem":"0.7"},"111":{"cpu":"0.0","mem":"0.0"},"119":{"cpu":"0.0","mem":"0.0"},"123":{"cpu":"0.0","mem":"0.0"},"124":{"cpu":"0.0","mem":"0.0"},"125":{"cpu":"0.0","mem":"0.0"},"126":{"cpu":"0.0","mem":"0.0"},"127":{"cpu":"0.0","mem":"0.0"},"131":{"cpu":"0.0","mem":"0.0"},"138":{"cpu":"0.0","mem":"0.0"},"144":{"cpu":"0.0","mem":"0.0"},"15":{"cpu":"0.0","mem":"0.0"},"151":{"cpu":"0.0","mem":"0.0"},"158":{"cpu":"0.0","mem":"0.0"},"16":{"cpu":"0.0","mem":"0.0"},"165":{"cpu":"0.0","mem":"0.0"},"166":{"cpu":"0.0","mem":"0.0"},"167":{"cpu":"0.0","mem":"0.0"},"168":{"cpu":"0.0","mem":"0.0"},"169":{"cpu":"0.0","mem":"0.0"},"174":{"cpu":"0.0","mem":"0.0"},"1952":{"cpu":"0.0","mem":"0.0"},"1953":{"cpu":"0.0","mem":"0.1"},"1961":{"cpu":"0.0","mem":"0.0"},"1962":{"cpu":"0.0","mem":"0.0"},"1964":{"cpu":"0.0","mem":"0.0"},"1970":{"cpu":"0.0","mem":"0.0"},"24":{"cpu":"0.0","mem":"0.0"},"25":{"cpu":"0.0","mem":"0.0"},"26":{"cpu":"0.0","mem":"0.0"},"27":{"cpu":"0.0","mem":"0.0"},"28":{"cpu":"0.0","mem":"0.0"},"29":{"cpu":"0.0","mem":"0.0"},"30":{"cpu":"0.0","mem":"0.0"},"31":{"cpu":"0.0","mem":"0.0"},"32":{"cpu":"0.0","mem":"0.0"},"41":{"cpu":"0.0","mem":"0.0"},"42":{"cpu":"0.0","mem":"0.0"},"678":{"cpu":"0.0","mem":"0.2"},"684":{"cpu":"0.0","mem":"0.0"},"685":{"cpu":"0.0","mem":"0.3"},"687":{"cpu":"0.0","mem":"0.3"},"692":{"cpu":"0.0","mem":"0.3"},"694":{"cpu":"0.0","mem":"0.3"},"698":{"cpu":"0.0","mem":"0.1"},"700":{"cpu":"0.0","mem":"0.1"},"80":{"cpu":"0.0","mem":"0.0"},"81":{"cpu":"0.0","mem":"0.0"},"82":{"cpu":"0.0","mem":"0.0"},"89":{"cpu":"0.0","mem":"0.1"},"MiB":{"cpu":"11494.5","mem":"avail"},"PID":{"cpu":"%CPU","mem":"%MEM"},"Tasks:":{"cpu":"stopped,","mem":"0"},"top":{"cpu":"users,","mem":"load"}},"detection_fps":0.0,"detectors":{"coral":{"detection_start":0.0,"inference_speed":10.0,"pid":685}},"gpu_usages":{"intel-vaapi":{"gpu":"0.0 %","mem":"- %"}},"kitchen":{"camera_fps":0.0,"capture_pid":694,"detection_enabled":1,"detection_fps":0.0,"ffmpeg_pid":1953,"pid":692,"process_fps":0.0,"skipped_fps":0.0},"service":{"last_updated":1693812032,"latest_version":"0.12.1","storage":{"/dev/shm":{"free":1073.2,"mount_type":"tmpfs","total":1073.7,"used":0.5},"/media/frigate/clips":{"free":9659159.2,"mount_type":"nfs4","total":12001868.8,"used":2340362.9},"/media/frigate/recordings":{"free":9659159.2,"mount_type":"nfs4","total":12001868.8,"used":2340362.9},"/tmp/cache":{"free":1000.0,"mount_type":"tmpfs","total":1000.0,"used":0.0}},"temperatures":{},"uptime":827,"version":"0.12.1-367d724"}}

Operating system

Debian

Install method

Docker Compose

Coral version

USB

Network connection

Wired

Camera make and model

Eufy Indoor Cam

Any other information that may be helpful

Thanks for your help - I am fully available for any additional information that might be missing.

NickM-27 commented 1 year ago

Failed setup for format vaapi: hwaccel initialisation returned error.

Your hwaccel has stopped working, some kind of issue on the host likely related to drivers being updated or something else. First thing would be restarting the entire device.

wilcomir commented 1 year ago

@NickM-27 thanks a bunch for the second pair of eyes.

I did update a few things on my host over the weekend, and I did not reboot it. After I rebooted the host, everything is back to normal. Why this happened hours after I updated, is still a mystery.

Thanks again for your kind response and help.