search

blakeblackshear / frigate

NVR with realtime local object detection for IP cameras
https://frigate.video
MIT License
16.34k stars 1.54k forks source link

[Support]: What is using up a lot of disk space in frigate container? #2657

Closed Ming-A closed 2 years ago

Ming-A commented 2 years ago

Describe the problem you are having

For some reason my frigate container is using about 22gb which makes my disk full, I have mounted clips and recording to both my samba share.

Version

0.9.4-26AE608

Frigate config file

mqtt:
  host: IP
  user: USER
  password: PASSWORD
database:
  path: /media/frigate/database/frigate.db
birdseye:
  enabled: True
detect:
  width: 1920
  height: 1080
  fps: 5
  enabled: True
record:
  enabled: True
  retain_days: 28
  events:
    max_seconds: 300
    # Optional: Restrict recordings to objects that entered any of the listed zones (default: no required zones)
    required_zones: []
    # Optional: Retention settings for recordings of events
    retain:
      # Required: Default retention days (default: shown below)
      default: 10
snapshots:
  enabled: True
  # Optional: Restrict snapshots to objects that entered any of the listed zones (default: no required zones)
  required_zones: []
  retain:
    # Required: Default retention days (default: shown below)
    default: 10
cameras:
    carport-front:
        ffmpeg:
          inputs:
            - path: rtsp://[]:[]@10.69.25.10:554/Streaming/Channels/101?transportmode=unicast&profile=Profile_1
              roles:
                - detect
                - rtmp
        zones:
          insidecarport:
            coordinates: 1920,1080,1920,933,1841,747,1704,495,1351,257,0,276,0,1080
            objects:
              - person
              - motorcycle
          outsidecarport:
            coordinates: 713,260,1255,240,1684,436,1920,866,1920,0,0,0,0,346
            objects:
              - car
              - person
              - motorcycle

detectors:
  cpu1:
    type: cpu
  cpu2:
    type: cpu

Relevant log output

[2022-01-16 07:01:10] ws4py                          INFO    : Terminating websocket [Local => 127.0.0.1:8082 | Remote => 127.0.0.1:54580]

[2022-01-16 07:01:43] ws4py                          INFO    : Terminating websocket [Local => 127.0.0.1:5002 | Remote => 127.0.0.1:33568]

[2022-01-16 07:01:52] ws4py                          INFO    : Managing websocket [Local => 127.0.0.1:5002 | Remote => 127.0.0.1:34266]

[2022-01-16 07:01:55] ws4py                          INFO    : Managing websocket [Local => 127.0.0.1:8082 | Remote => 127.0.0.1:54950]

[2022-01-16 07:01:56] ws4py                          INFO    : Terminating websocket [Local => 127.0.0.1:8082 | Remote => 127.0.0.1:54950]

[2022-01-16 07:01:58] ws4py                          INFO    : Managing websocket [Local => 127.0.0.1:8082 | Remote => 127.0.0.1:54966]

[2022-01-16 07:02:29] ws4py                          INFO    : Terminating websocket [Local => 127.0.0.1:8082 | Remote => 127.0.0.1:54966]

[2022-01-16 07:02:29] ws4py                          INFO    : Terminating websocket [Local => 127.0.0.1:5002 | Remote => 127.0.0.1:34266]

[2022-01-16 07:19:13] frigate.mqtt                   INFO    : MQTT connected

[2022-01-16 11:20:45] frigate.video                  INFO    : carport-front: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures

[2022-01-16 11:20:45] frigate.video                  INFO    : carport-front: ffmpeg process is not running. exiting capture thread...

[2022-01-16 11:20:46] watchdog.carport-front         ERROR   : FFMPEG process crashed unexpectedly for carport-front.

[2022-01-16 11:20:46] watchdog.carport-front         ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.

[2022-01-16 11:20:46] watchdog.carport-front         ERROR   : You may have invalid args defined for this camera.

[2022-01-16 11:20:46] ffmpeg.carport-front.detect    ERROR   : [swscaler @ 0x5634c8c7b800] deprecated pixel format used, make sure you did set range correctly

[2022-01-16 11:20:46] ffmpeg.carport-front.detect    ERROR   : [h264 @ 0x5634c8d2b400] cabac decode of qscale diff failed at 93 28

[2022-01-16 11:20:46] ffmpeg.carport-front.detect    ERROR   : [h264 @ 0x5634c8d2b400] error while decoding MB 93 28, bytestream 1035

[2022-01-16 11:20:46] ffmpeg.carport-front.detect    ERROR   : rtsp://username:password@10.69.25.10:554/Streaming/Channels/101?transportmode=unicast&profile=Profile_1: corrupt decoded frame in stream 0

[2022-01-16 11:20:46] ffmpeg.carport-front.detect    ERROR   : [flv @ 0x5634c8b59f40] Failed to update header with correct duration.

[2022-01-16 11:20:46] ffmpeg.carport-front.detect    ERROR   : [flv @ 0x5634c8b59f40] Failed to update header with correct filesize.

[2022-01-16 13:50:02] frigate.video                  INFO    : carport-front: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures

[2022-01-16 13:50:02] frigate.video                  INFO    : carport-front: ffmpeg process is not running. exiting capture thread...

[2022-01-16 13:50:07] watchdog.carport-front         ERROR   : FFMPEG process crashed unexpectedly for carport-front.

[2022-01-16 13:50:07] watchdog.carport-front         ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.

[2022-01-16 13:50:07] watchdog.carport-front         ERROR   : You may have invalid args defined for this camera.

[2022-01-16 13:50:07] ffmpeg.carport-front.detect    ERROR   : [segment @ 0x555797940e40] Timestamps are unset in a packet for stream 0. This is deprecated and will stop working in the future. Fix your code to set the timestamps properly

[2022-01-16 13:50:07] ffmpeg.carport-front.detect    ERROR   : [flv @ 0x55579795fbc0] Timestamps are unset in a packet for stream 0. This is deprecated and will stop working in the future. Fix your code to set the timestamps properly

[2022-01-16 13:50:07] ffmpeg.carport-front.detect    ERROR   : [segment @ 0x555797940e40] Non-monotonous DTS in output stream 0:0; previous: 0, current: 0; changing to 1. This may result in incorrect timestamps in the output file.

[2022-01-16 13:50:07] ffmpeg.carport-front.detect    ERROR   : [swscaler @ 0x555797e948c0] deprecated pixel format used, make sure you did set range correctly

[2022-01-16 13:50:07] ffmpeg.carport-front.detect    ERROR   : [h264 @ 0x555797af0780] Increasing reorder buffer to 1

[2022-01-16 13:50:07] ffmpeg.carport-front.detect    ERROR   : [h264 @ 0x555797a86480] error while decoding MB 82 14, bytestream -12

[2022-01-16 13:50:07] ffmpeg.carport-front.detect    ERROR   : rtsp://username:password@10.69.25.10:554/Streaming/Channels/101?transportmode=unicast&profile=Profile_1: corrupt decoded frame in stream 0

[2022-01-16 13:50:07] ffmpeg.carport-front.detect    ERROR   :     Last message repeated 1 times

[2022-01-16 13:50:07] ffmpeg.carport-front.detect    ERROR   : [flv @ 0x55579795fbc0] Failed to update header with correct duration.

[2022-01-16 13:50:07] ffmpeg.carport-front.detect    ERROR   : [flv @ 0x55579795fbc0] Failed to update header with correct filesize.

[2022-01-16 14:21:36] frigate.video                  INFO    : carport-front: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures

[2022-01-16 14:21:36] frigate.video                  INFO    : carport-front: ffmpeg process is not running. exiting capture thread...

[2022-01-16 14:21:37] watchdog.carport-front         ERROR   : FFMPEG process crashed unexpectedly for carport-front.

[2022-01-16 14:21:37] watchdog.carport-front         ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.

[2022-01-16 14:21:37] watchdog.carport-front         ERROR   : You may have invalid args defined for this camera.

[2022-01-16 14:21:37] ffmpeg.carport-front.detect    ERROR   : [segment @ 0x5560ed87e3c0] Timestamps are unset in a packet for stream 0. This is deprecated and will stop working in the future. Fix your code to set the timestamps properly

[2022-01-16 14:21:37] ffmpeg.carport-front.detect    ERROR   : [flv @ 0x5560ed87b580] Timestamps are unset in a packet for stream 0. This is deprecated and will stop working in the future. Fix your code to set the timestamps properly

[2022-01-16 14:21:37] ffmpeg.carport-front.detect    ERROR   : [segment @ 0x5560ed87e3c0] Non-monotonous DTS in output stream 0:0; previous: 0, current: 0; changing to 1. This may result in incorrect timestamps in the output file.

[2022-01-16 14:21:37] ffmpeg.carport-front.detect    ERROR   : [swscaler @ 0x5560eddb30c0] deprecated pixel format used, make sure you did set range correctly

[2022-01-16 14:21:37] ffmpeg.carport-front.detect    ERROR   : [h264 @ 0x5560ed897900] left block unavailable for requested intra mode

[2022-01-16 14:21:37] ffmpeg.carport-front.detect    ERROR   : [h264 @ 0x5560ed897900] error while decoding MB 0 59, bytestream 21592

[2022-01-16 14:21:37] ffmpeg.carport-front.detect    ERROR   : [h264 @ 0x5560ed9dfc00] Increasing reorder buffer to 1

[2022-01-16 14:21:37] ffmpeg.carport-front.detect    ERROR   : rtsp://username:password@10.69.25.10:554/Streaming/Channels/101?transportmode=unicast&profile=Profile_1: corrupt decoded frame in stream 0

[2022-01-16 14:21:37] ffmpeg.carport-front.detect    ERROR   : [flv @ 0x5560ed87b580] Failed to update header with correct duration.

[2022-01-16 14:21:37] ffmpeg.carport-front.detect    ERROR   : [flv @ 0x5560ed87b580] Failed to update header with correct filesize.

[2022-01-16 16:30:56] frigate.video                  INFO    : carport-front: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures

[2022-01-16 16:30:56] frigate.video                  INFO    : carport-front: ffmpeg process is not running. exiting capture thread...

[2022-01-16 16:30:57] watchdog.carport-front         ERROR   : FFMPEG process crashed unexpectedly for carport-front.

[2022-01-16 16:30:57] watchdog.carport-front         ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.

[2022-01-16 16:30:57] watchdog.carport-front         ERROR   : You may have invalid args defined for this camera.

[2022-01-16 16:30:57] ffmpeg.carport-front.detect    ERROR   : [segment @ 0x55912aab9f40] Timestamps are unset in a packet for stream 0. This is deprecated and will stop working in the future. Fix your code to set the timestamps properly

[2022-01-16 16:30:57] ffmpeg.carport-front.detect    ERROR   : [flv @ 0x55912aab8580] Timestamps are unset in a packet for stream 0. This is deprecated and will stop working in the future. Fix your code to set the timestamps properly

[2022-01-16 16:30:57] ffmpeg.carport-front.detect    ERROR   : [segment @ 0x55912aab9f40] Non-monotonous DTS in output stream 0:0; previous: 0, current: 0; changing to 1. This may result in incorrect timestamps in the output file.

[2022-01-16 16:30:57] ffmpeg.carport-front.detect    ERROR   : [swscaler @ 0x55912afeff40] deprecated pixel format used, make sure you did set range correctly

[2022-01-16 16:30:57] ffmpeg.carport-front.detect    ERROR   : [h264 @ 0x55912aad5580] error while decoding MB 100 58, bytestream -13

[2022-01-16 16:30:57] ffmpeg.carport-front.detect    ERROR   : [h264 @ 0x55912abda640] Increasing reorder buffer to 1

[2022-01-16 16:30:57] ffmpeg.carport-front.detect    ERROR   : rtsp://username:password@10.69.25.10:554/Streaming/Channels/101?transportmode=unicast&profile=Profile_1: corrupt decoded frame in stream 0

[2022-01-16 16:30:57] ffmpeg.carport-front.detect    ERROR   : [h264 @ 0x55912ab94600] error while decoding MB 43 21, bytestream -5

[2022-01-16 16:30:57] ffmpeg.carport-front.detect    ERROR   : rtsp://username:password@10.69.25.10:554/Streaming/Channels/101?transportmode=unicast&profile=Profile_1: corrupt decoded frame in stream 0

FFprobe output from your camera

ffprobe version 4.3.1 Copyright (c) 2007-2020 the FFmpeg developers
  built with gcc 9 (Ubuntu 9.3.0-17ubuntu1~20.04)
  configuration: --disable-debug --disable-doc --disable-ffplay --enable-shared --enable-avresample --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-gpl --enable-libfreetype --enable-libvidstab --enable-libmfx --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libxcb --enable-libx265 --enable-libxvid --enable-libx264 --enable-nonfree --enable-openssl --enable-libfdk_aac --enable-postproc --enable-small --enable-version3 --enable-libzmq --extra-libs=-ldl --prefix=/opt/ffmpeg --enable-libopenjpeg --enable-libkvazaar --enable-libaom --extra-libs=-lpthread --enable-vaapi --extra-cflags=-I/opt/ffmpeg/include --extra-ldflags=-L/opt/ffmpeg/lib
  libavutil      56. 51.100 / 56. 51.100
  libavcodec     58. 91.100 / 58. 91.100
  libavformat    58. 45.100 / 58. 45.100
  libavdevice    58. 10.100 / 58. 10.100
  libavfilter     7. 85.100 /  7. 85.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  7.100 /  5.  7.100
  libswresample   3.  7.100 /  3.  7.100
  libpostproc    55.  7.100 / 55.  7.100
Input #0, rtsp, from 'rtsp://[]:[]@10.69.25.10:554/Streaming/Channels/101?transportmode=unicast':
  Metadata:
    title           : Media Presentation
  Duration: N/A, start: 0.080000, bitrate: N/A
    Stream #0:0: Video: h264, yuvj420p(pc, bt709, progressive), 1920x1080, 25 tbr, 90k tbn, 180k tb

Frigate stats

No response

Operating system

Other

Install method

Docker Compose

Coral version

CPU (no coral)

Network connection

Wired

Camera make and model

HIKVISION DS-2CD1023G0E-I(L)

Any other information that may be helpful

Docker-Compose container:



services:
  frigate:
    image: blakeblackshear/frigate:stable-amd64
    container_name: frigate3
    privileged: true
    shm_size: "64mb"
    devices:
      - /dev/bus/usb:/dev/bus/usb # passes the USB Coral, needs to be modified for other versions
      - /dev/apex_0:/dev/apex_0 # passes a PCIe Coral, follow driver instructions here https://coral.ai/docs/m2/get-started/#2a-on-linux
      - /dev/dri/renderD128 # for intel hwaccel, needs to be updated for your hardware
    environment:
       FRIGATE_RTSP_PASSWORD: "password"

    restart: unless-stopped
    security_opt:
      - no-new-privileges:true
    networks:
      - proxy
    volumes:
      - /etc/localtime:/etc/localtime
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /mnt/CCTV/frigate/clips:/media/frigate/clips
      - /home/mingsoonang/frigate/database:/media/frigate/database
      - /mnt/CCTV/frigate/recordings:/media/frigate/recordings
      - /home/mingsoonang/frigate/config:/config
      - /mnt/CCTV:/media/frigate
    ports:
      - "5000:5000"
      - "1935:1935" # RTMP feeds  

    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.frigate.entrypoints=http"
      - "traefik.http.routers.frigate.rule=Host(`frigate.local.website.web`)"
      - "traefik.http.middlewares.frigate-https-redirect.redirectscheme.scheme=https"
      - "traefik.http.routers.frigate.middlewares=frigate-https-redirect"
      - "traefik.http.routers.frigate-secure.entrypoints=https"
      - "traefik.http.routers.frigate-secure.rule=Host(`frigate.local.website.web`)"
      - "traefik.http.routers.frigate-secure.tls=true"
      - "traefik.http.routers.frigate-secure.service=frigate"
      - "traefik.http.services.frigate.loadbalancer.server.port=5000"
      - "traefik.docker.network=proxy"

networks:
  proxy:
    external: true```
Ming-A commented 2 years ago

image image

Ming-A commented 2 years ago

image managed to clean space by deleting the cache file, but is there a way to have the cache clean it self after some time so i dont have to manually delete it?

Bod1985 commented 2 years ago

Disclaimer in case I get anything wrong - I am not an expert.

I have experienced cache filling when my camera is incorrectly configured and ffmpeg crashes as the recordings are being corrupted so don’t get flushed to the correct folder. I would troubleshoot your camera first. Perhaps start by disabling RTMP - especially if this is a H265 camera?

To do so, remove the rtmp role and inside the camera definition add:

rtmp:
  enabled: False

The cache should clear itself regularly, it may not be relevant but the default config recommends a RAM disk for cache as defined by the following docker-compose volume:

- type: tmpfs # Optional: 1GB of memory, reduces SSD/SD Card wear
  target: /tmp/cache
  tmpfs:
    size: 1000000000
Ming-A commented 2 years ago

Disclaimer in case I get anything wrong - I am not an expert.

I have experienced cache filling when my camera is incorrectly configured and ffmpeg crashes as the recordings are being corrupted so don’t get flushed to the correct folder. I would troubleshoot your camera first. Perhaps start by disabling RTMP - especially if this is a H265 camera?

To do so, remove the rtmp role and inside the camera definition add:

rtmp:
  enabled: False

The cache should clear itself regularly, it may not be relevant but the default config recommends a RAM disk for cache as defined by the following docker-compose volume:

- type: tmpfs # Optional: 1GB of memory, reduces SSD/SD Card wear
  target: /tmp/cache
  tmpfs:
    size: 1000000000

currently need RTMP since I'm using it with home assistant, and its an h264 camera

Bod1985 commented 2 years ago

Disclaimer in case I get anything wrong - I am not an expert. I have experienced cache filling when my camera is incorrectly configured and ffmpeg crashes as the recordings are being corrupted so don’t get flushed to the correct folder. I would troubleshoot your camera first. Perhaps start by disabling RTMP - especially if this is a H265 camera? To do so, remove the rtmp role and inside the camera definition add:

rtmp:
  enabled: False

The cache should clear itself regularly, it may not be relevant but the default config recommends a RAM disk for cache as defined by the following docker-compose volume:

- type: tmpfs # Optional: 1GB of memory, reduces SSD/SD Card wear
  target: /tmp/cache
  tmpfs:
    size: 1000000000

currently need RTMP since I'm using it with home assistant, and its an h264 camera

I’m not suggesting it needs to be a permanent change, I’m only suggesting to rule it out

Ming-A commented 2 years ago

Hm I guess I can give it a try, how do I compare the difference though to see if it has affected it? And usually how long?

Previously I wouldn't say I had any issues with storage for frigate, just only recently when I recreated the container to work with traefik for reverse proxy, might be missing something maybe in the docker-compose, I also seen in the issues GitHub that some of the cache problem is related to day light saving? Not sure if it's related though to mine

blakeblackshear commented 2 years ago

Do you see lots of files in /tmp/cache inside the container? Maybe the recording maintenance job failed and you have a bunch of lingering files in there. Try clearing them out. If you map the /tmp/cache location to somewhere on the host, the container should not grow in size since every location that it writes to will be mapped to volumes.

You can also try deleting the container to let it recreate a fresh one.

Ming-A commented 2 years ago

Do you see lots of files in /tmp/cache inside the container? Maybe the recording maintenance job failed and you have a bunch of lingering files in there. Try clearing them out. If you map the /tmp/cache location to somewhere on the host, the container should not grow in size since every location that it writes to will be mapped to volumes.

You can also try deleting the container to let it recreate a fresh one.

from the screenshot above there were a lot of files in /tmp/cache. I have cleared it and gained a lot of space back. Currently been 12+ hours and it doesn't seem to be using much yet. I didn't map a /tmp/cache location, if it increases a lot again then I might map it to a volume and see how that goes.

stale[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

reza8iucs commented 1 month ago

I'm facing the same issue. My Frigate container grows by 250 MB daily. The media library is on an external HDD, /tmp/cache isn't large, and frigate.db is 53 MB, growing normally by 2-3 MB per day. I have two cameras on a busy street. What could be causing this large container size increase?