search

ep1cman / unifi-protect-backup

Python tool to backup unifi event clips in realtime
MIT License
577 stars 25 forks source link

Not following RCLONE_RETENTION #46

Closed jacobwilsonnet closed 1 year ago

jacobwilsonnet commented 1 year ago

Description

When I set retention via environment variable, it doesn't appear to follow that. In my docker compose file, I have the retention set to 5d. However, when the nightly delete happens, it follows the default 7d retention. Aka, folders and all the files that are up to 7 days old stay around. Folders and files older than 7 days are deleted nightly. It's just not following the overwritten value of 5d.

Docker compose file: 

version: '3'
services:
  unifi-protect-backup-box:
    restart: always
    image: ghcr.io/ep1cman/unifi-protect-backup:latest
    environment:
      - UFP_USERNAME=removed
      - UFP_PASSWORD=removed
      - UFP_ADDRESS=removed
      - UFP_SSL_VERIFY=false
      - RCLONE_RETENTION=5d
    volumes:
      - /unifi-protect-backup/data:/data

What I Did

cont-init: info: /etc/cont-init.d/99-custom-files exited 0
s6-rc: info: service legacy-cont-init successfully started
s6-rc: info: service init-mods: starting
s6-rc: info: service init-mods successfully started
s6-rc: info: service legacy-services: starting
services-up: info: copying legacy longrun unifi-protect-backup (no readiness notification)
s6-rc: info: service legacy-services successfully started
s6-rc: info: service 99-ci-service-check: starting
[ls.io-init] done.
s6-rc: info: service 99-ci-service-check successfully started
2022-08-22 18:48:18 [DEBUG]:unifi_protect_backup.unifi_protect_backup:  Config:
2022-08-22 18:48:18 [DEBUG]:unifi_protect_backup.unifi_protect_backup:    address='removed'
2022-08-22 18:48:18 [DEBUG]:unifi_protect_backup.unifi_protect_backup:    port=443
2022-08-22 18:48:18 [DEBUG]:unifi_protect_backup.unifi_protect_backup:    username='removed'
2022-08-22 18:48:18 [DEBUG]:unifi_protect_backup.unifi_protect_backup:    password=REDACTED
2022-08-22 18:48:18 [DEBUG]:unifi_protect_backup.unifi_protect_backup:    verify_ssl=False
2022-08-22 18:48:18 [DEBUG]:unifi_protect_backup.unifi_protect_backup:    rclone_destination='local:/data'
2022-08-22 18:48:18 [DEBUG]:unifi_protect_backup.unifi_protect_backup:    retention='5d'
2022-08-22 18:48:18 [DEBUG]:unifi_protect_backup.unifi_protect_backup:    rclone_args=''
2022-08-22 18:48:18 [DEBUG]:unifi_protect_backup.unifi_protect_backup:    ignore_cameras=()
2022-08-22 18:48:18 [DEBUG]:unifi_protect_backup.unifi_protect_backup:    verbose=1
2022-08-22 18:48:18 [DEBUG]:unifi_protect_backup.unifi_protect_backup:    detection_types=['motion', 'person', 'vehicle', 'ring']
2022-08-22 18:48:18 [DEBUG]:unifi_protect_backup.unifi_protect_backup:    file_structure_format='{camera_name}/{event.start:%Y-%m-%d}/{event.end:%Y-%m-%dT%H-%M-%S} {detection_type}.mp4'
2022-08-22 18:48:18 [INFO]:unifi_protect_backup.unifi_protect_backup:   Starting...
2022-08-22 18:48:18 [INFO]:unifi_protect_backup.unifi_protect_backup:   Checking rclone configuration...
2022-08-22 18:48:18 [DEBUG]:unifi_protect_backup.unifi_protect_backup:  rclone found: /usr/bin/rclone
2022-08-22 18:48:18 [DEBUG]:unifi_protect_backup.unifi_protect_backup:  ffprobe found: /usr/bin/ffprobe
2022-08-22 18:48:18 [INFO]:unifi_protect_backup.unifi_protect_backup:   Connecting to Unifi Protect...
2022-08-22 18:48:19 [INFO]:unifi_protect_backup.unifi_protect_backup:   Found cameras:
2022-08-22 18:48:19 [INFO]:unifi_protect_backup.unifi_protect_backup:    - removed camera name
2022-08-22 18:48:19 [INFO]:unifi_protect_backup.unifi_protect_backup:    - removed camera name
2022-08-22 18:48:19 [INFO]:unifi_protect_backup.unifi_protect_backup:    - removed camera name
2022-08-22 18:48:19 [INFO]:unifi_protect_backup.unifi_protect_backup:    - removed camera name
2022-08-22 18:48:19 [INFO]:unifi_protect_backup.unifi_protect_backup:    - removed camera name
2022-08-22 18:48:19 [INFO]:unifi_protect_backup.unifi_protect_backup:    - removed camera name
2022-08-22 18:48:19 [INFO]:unifi_protect_backup.unifi_protect_backup:    - removed camera name
2022-08-22 18:48:19 [INFO]:unifi_protect_backup.unifi_protect_backup:    - removed camera name
2022-08-22 18:48:19 [INFO]:unifi_protect_backup.unifi_protect_backup:    - removed camera name
2022-08-22 18:48:19 [INFO]:unifi_protect_backup.unifi_protect_backup:   Setting up purge task...
2022-08-22 18:48:19 [INFO]:unifi_protect_backup.unifi_protect_backup:   Listening for events...

image (7 days)

jacobwilsonnet commented 1 year ago

Nothing noticeable in the logs when it does its nightly delete.. I turned up the logging and also manually added an entrypoint with--retention and -vvv to my compose file:

    entrypoint: 'unifi-protect-backup --retention 5d -vvv'

I will see if it deletes correctly tonight

ep1cman commented 1 year ago

I just tested the functionality and the files are being deleted as expected. What might be happening however is that empty directories are being left behind. Can you check if this is the case?

jacobwilsonnet commented 1 year ago

Hi! thanks for getting back to me and testing the functionality. I think I found my issue. I was basing the timeframe off my timezone and not UTC. Looking at the timestamps on the files, it's starting to make sense. Based on my timezone, the delete runs at 8pm ET. So I would expect 5 days before that to delete. Which it does, for example, this file just barely under 5 days old. I was looking at my ET time on the filename and saying that's bigger than 5 days. So when the service ran at 8pm last night, it wasn't old enough... idk if that makes sense. Sorry for wasting your time, appreciate your quick response and love what you've created!

-rw-r--r-- 1 911 911 7460026 Aug 17 00:01 '2022-08-16T20-01-14 motion.mp4'
ep1cman commented 1 year ago

You can set the container time zone with the "TZ" env variable. Refer to https://en.wikipedia.org/wiki/List_of_tz_database_time_zones for a list of possible timezones

With that set, the timestamps of the files should match the timestamps of the file names