rhatguy
commented
3 years ago Your screenshot of the output of "df -h" shows that /tmp/cache is a 2GB filesystem and is 100% full. I run a similar setup to you with 5 Reolink cameras but only a 1GB /tmp/cache filesystem. I would suggest trying to see why there is so much space utilized in /tmp/cache. If there are a bunch of video files in there maybe run ffprobe against a few of them, so you know which stream they are coming from. You might also check if any files are ever being cleaned from /tmp/cache or if they're just building up until it crashes. There should be an always rotating set of files in there.
I'm not as familiar with how the new caching system works in the 0.9 RCs yet, but in my live system I only see one or two files for each camera. Maybe one of your retention settings is causing frigate to retain more files than needed? Not sure about that. We need to know what is taking up the 2GB of space, as that seems quite high.
@.***:/tmp/cache# df -h . Filesystem Size Used Avail Use% Mounted on tmpfs 954M 59M 895M 7% /tmp/cache
On Fri, Sep 10, 2021 at 2:38 AM Jason @.***> wrote:
Describe the bug Frigate has crashed again after a couple days running, throwing these errors for all cameras every 10 seconds:
[2021-09-10 16:17:52] ffmpeg.Deck.record ERROR : Could not write header for output file #0 (incorrect codec parameters ?): No space left on device, [2021-09-10 16:17:52] ffmpeg.Deck.record ERROR : , [2021-09-10 16:17:52] watchdog.Deck INFO : Terminating the existing ffmpeg process..., [2021-09-10 16:17:52] watchdog.Deck INFO : Waiting for ffmpeg to exit gracefully..., [2021-09-10 16:17:52] ffmpeg.Laundry.record ERROR : Could not write header for output file #0 (incorrect codec parameters ?): No space left on device, [2021-09-10 16:17:52] ffmpeg.Laundry.record ERROR : , [2021-09-10 16:17:52] watchdog.Laundry INFO : Terminating the existing ffmpeg process..., [2021-09-10 16:17:52] watchdog.Laundry INFO : Waiting for ffmpeg to exit gracefully..., [2021-09-10 16:18:02] ffmpeg.Entrance.record ERROR : Could not write header for output file #0 (incorrect codec parameters ?): No space left on device, [2021-09-10 16:18:02] ffmpeg.Entrance.record ERROR : , [2021-09-10 16:18:02] watchdog.Entrance INFO : Terminating the existing ffmpeg process..., [2021-09-10 16:18:02] watchdog.Entrance INFO : Waiting for ffmpeg to exit gracefully..., [2021-09-10 16:18:02] ffmpeg.Driveway.record ERROR : Could not write header for output file #0 (incorrect codec parameters ?): No space left on device, [2021-09-10 16:18:02] ffmpeg.Driveway.record ERROR : , [2021-09-10 16:18:02] watchdog.Driveway INFO : Terminating the existing ffmpeg process..., [2021-09-10 16:18:02] watchdog.Driveway INFO : Waiting for ffmpeg to exit gracefully..., [2021-09-10 16:18:02] ffmpeg.Rumpus.record ERROR : Could not write header for output file #0 (incorrect codec parameters ?): No space left on device, [2021-09-10 16:18:02] ffmpeg.Rumpus.record ERROR : , [2021-09-10 16:18:02] watchdog.Rumpus INFO : Terminating the existing ffmpeg process..., [2021-09-10 16:18:02] watchdog.Rumpus INFO : Waiting for ffmpeg to exit gracefully..., [2021-09-10 16:18:02] ffmpeg.Balcony.record ERROR : Could not write header for output file #0 (incorrect codec parameters ?): No space left on device, [2021-09-10 16:18:02] ffmpeg.Balcony.record ERROR : , [2021-09-10 16:18:02] watchdog.Balcony INFO : Terminating the existing ffmpeg process..., [2021-09-10 16:18:02] watchdog.Balcony INFO : Waiting for ffmpeg to exit gracefully..., [2021-09-10 16:18:02] ffmpeg.Deck.record ERROR : Could not write header for output file #0 (incorrect codec parameters ?): No space left on device, [2021-09-10 16:18:02] ffmpeg.Deck.record ERROR : , [2021-09-10 16:18:02] watchdog.Deck INFO : Terminating the existing ffmpeg process..., [2021-09-10 16:18:02] watchdog.Deck INFO : Waiting for ffmpeg to exit gracefully...,
@.***:/opt/frigate# df -h Filesystem Size Used Avail Use% Mounted on overlay 233G 20G 201G 9% / tmpfs 64M 0 64M 0% /dev shm 300M 26M 275M 9% /dev/shm /dev/sda2 233G 20G 201G 9% /labelmap.txt 192.168.30.8:/storage/frigate 1.8T 449G 1.4T 26% /media/frigate tmpfs 2.0G 2.0G 0 100% /tmp/cache tmpfs 7.7G 0 7.7G 0% /proc/asound tmpfs 7.7G 0 7.7G 0% /proc/acpi tmpfs 7.7G 0 7.7G 0% /sys/firmware
Version of frigate 0.9.0-6c28613
Config file
database: path: /media/frigate/database/frigate.db detectors: coral: type: edgetpu device: usb mqtt: host: 192.168.1.5 port: 1883 user: mqtt password: "{FRIGATE_MQTT_PASSWORD}" birdseye: mode: continuous width: 1792 height: 1344 detect: width: 896 height: 672 fps: 5 ffmpeg: hwaccel_args: -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 input_args: -avoid_negative_ts make_zero -flags low_delay -fflags nobuffer -fflags +genpts+discardcorrupt -rw_timeout 5000000 -strict experimental -use_wallclock_as_timestamps 1 -f live_flv output_args: record: -f segment -segment_time 30 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy # -an motion: mask:
265,650,265,625,630,625,630,650 record: enabled: true retain_days: 1 events: enabled: true pre_capture: 3 retain: default: 15 snapshots: enabled: true clean_copy: false bounding_box: true retain: default: 15 cameras: Driveway: ffmpeg: inputs:
- path: http://192.168.1.71/flv?port=1935&app=bcs&stream=channel0_ext.bcs&user={FRIGATE_USER}&password={FRIGATE_PASSWORD} roles:
- detect
- path: http://192.168.1.71/flv?port=1935&app=bcs&stream=channel0_main.bcs&user={FRIGATE_USER}&password={FRIGATE_PASSWORD} roles:
- record objects: filters: person: mask: 510,120,540,120,540,160,510,160 track:
- person
- vehicle
- dog rtmp: enabled: false timestamp_style: format: "%d/%m/%Y %H:%M:%S"
Entrance: ffmpeg: inputs:
- path: http://192.168.1.72/flv?port=1935&app=bcs&stream=channel0_ext.bcs&user={FRIGATE_USER}&password={FRIGATE_PASSWORD} roles:
- detect
- path: http://192.168.1.72/flv?port=1935&app=bcs&stream=channel0_main.bcs&user={FRIGATE_USER}&password={FRIGATE_PASSWORD} roles:
- record objects: track:
- person
- dog rtmp: enabled: false timestamp_style: format: "%d/%m/%Y %H:%M:%S"
Rumpus: ffmpeg: inputs:
- path: http://192.168.1.73/flv?port=1935&app=bcs&stream=channel0_ext.bcs&user={FRIGATE_USER}&password={FRIGATE_PASSWORD} roles:
- detect
- path: http://192.168.1.73/flv?port=1935&app=bcs&stream=channel0_main.bcs&user={FRIGATE_USER}&password={FRIGATE_PASSWORD} roles:
- record objects: track:
- person
- dog rtmp: enabled: false timestamp_style: format: "%d/%m/%Y %H:%M:%S"
Deck: ffmpeg: inputs:
- path: http://192.168.1.74/flv?port=1935&app=bcs&stream=channel0_ext.bcs&user={FRIGATE_USER}&password={FRIGATE_PASSWORD} roles:
- detect
- path: http://192.168.1.74/flv?port=1935&app=bcs&stream=channel0_main.bcs&user={FRIGATE_USER}&password={FRIGATE_PASSWORD} roles:
- record objects: mask:
- 265,650,265,625,630,625,630,650
- 0,585,29,592,60,617,74,672,0,672 track:
- person rtmp: enabled: false timestamp_style: format: "%d/%m/%Y %H:%M:%S"
Laundry: ffmpeg: inputs:
- path: http://192.168.1.75/flv?port=1935&app=bcs&stream=channel0_ext.bcs&user={FRIGATE_USER}&password={FRIGATE_PASSWORD} roles:
- detect
- path: http://192.168.1.75/flv?port=1935&app=bcs&stream=channel0_main.bcs&user={FRIGATE_USER}&password={FRIGATE_PASSWORD} roles:
- record objects: track:
- person rtmp: enabled: false timestamp_style: format: "%d/%m/%Y %H:%M:%S"
Balcony: ffmpeg: inputs:
- path: http://192.168.1.76/flv?port=1935&app=bcs&stream=channel0_ext.bcs&user={FRIGATE_USER}&password={FRIGATE_PASSWORD} roles:
- detect
- path: http://192.168.1.76/flv?port=1935&app=bcs&stream=channel0_main.bcs&user={FRIGATE_USER}&password={FRIGATE_PASSWORD} roles:
- record objects: track:
- person
- vehicle rtmp: enabled: false timestamp_style: format: "%d/%m/%Y %H:%M:%S"
Frigate stats (After restart, as Frigate's web ui was not responding)
{"Balcony":{"camera_fps":5.1,"capture_pid":256,"detection_fps":0.0,"pid":235,"process_fps":5.1,"skipped_fps":0.0},"Deck":{"camera_fps":5.1,"capture_pid":247,"detection_fps":3.2,"pid":233,"process_fps":5.1,"skipped_fps":0.0},"Driveway":{"camera_fps":5.1,"capture_pid":237,"detection_fps":0.0,"pid":228,"process_fps":5.1,"skipped_fps":0.0},"Entrance":{"camera_fps":5.1,"capture_pid":241,"detection_fps":0.0,"pid":230,"process_fps":5.1,"skipped_fps":0.0},"Laundry":{"camera_fps":5.1,"capture_pid":252,"detection_fps":0.0,"pid":234,"process_fps":5.1,"skipped_fps":0.0},"Rumpus":{"camera_fps":5.0,"capture_pid":243,"detection_fps":0.0,"pid":231,"process_fps":5.0,"skipped_fps":0.0},"detection_fps":3.2,"detectors":{"coral":{"detection_start":0.0,"inference_speed":13.15,"pid":219}},"service":{"storage":{"/dev/shm":{"free":306.5,"mount_type":"tmpfs","total":314.6,"used":8.1},"/media/frigate/clips":{"free":1447499.7,"mount_type":"nfs4","total":1921292.6,"used":473256.0},"/media/frigate/recordings":{"free":1447499.8,"mount_type":"nfs4","total":1921292.6,"used":473256.0},"/tmp/cache":{"free":1993.8,"mount_type":"tmpfs","total":2147.5,"used":153.6}},"uptime":98,"version":"0.9.0-6c28613"}}
Computer Hardware
- OS: Debian 11
- Install method: Docker Compose
- Coral Version: USB
- Network Setup: Wired
Camera Info:
- Manufacturer: Reolink
- Model: RLC-520A
- Resolution: 5Mp
- FPS: 5
docker-compose.yml
services: frigate: container_name: frigate-0.9 image: blakeblackshear/frigate:0.9.0-6c28613-amd64 hostname: frigate restart: always shm_size: '300m' devices:
- /dev/dri/renderD128:/dev/dri/renderD128
- /dev/bus/usb:/dev/bus/usb volumes:
- /etc/localtime:/etc/localtime:ro
- /usr/docker/frigate-0.9/config.yml:/config/config.yml:ro
- /usr/docker/frigate-0.9/labelmap.txt:/labelmap.txt:ro
- /usr/docker/frigate-0.9/database:/media/frigate/database
- /mnt/nas/frigate:/media/frigate
- type: tmpfs target: /tmp/cache tmpfs: size: '2gb' network_mode: bridge ports:
- 0.0.0.0:5000:5000 environment: FRIGATE_MQTT_PASSWORD: "" FRIGATE_USER: "" FRIGATE_PASSWORD: "" LIBVA_DRIVER_NAME: ""
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/blakeblackshear/frigate/issues/1755, or unsubscribe https://github.com/notifications/unsubscribe-auth/AC3I766FWGRSW7NU6O6JKMDUBGRUTANCNFSM5DYVMENA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
nic0dk
jasonpstokes
stamandr
blakeblackshear
d3structivepyr0
Describe the bug Frigate has crashed again after a couple days running, throwing these errors for all cameras every 10 seconds.
Version of frigate 0.9.0-6c28613
Config file
Frigate stats (After restart, as Frigate's web ui was not responding)
Computer Hardware
Camera Info:
docker-compose.yml