search

offen / docker-volume-backup

Backup Docker volumes locally or to any S3, WebDAV, Azure Blob Storage, Dropbox or SSH compatible storage
https://offen.github.io/docker-volume-backup/
Mozilla Public License 2.0
1.98k stars 84 forks source link

Prune/Deletion of old Backups not working with local storage, old Backups getting modified #481

Open JeuJeus opened 2 hours ago

JeuJeus commented 2 hours ago

Describe the bug I'm utilizing docker-volume-backup for the backup management of several container volumes. I have configured the auto prune of backups after 30 days of retention. Whilst the nightly cron is displaying that a growing number of backups is present, 58++, none is deleted/pruned.

This is potentially happening due to the backup prune validation changing the modified date - therefore not pruning the backups? Is the "age" of the backups defined by their name or the modfiied time here?

To Reproduce Steps to reproduce the behavior:

  1. Configure a nightly backup with automatic deletion via docker-compose 
    volumes:
portainer_data:
external: true
[...]
services:
portainer-backup:
restart: always
image: offen/docker-volume-backup:v2
env_file: stack.env
environment:
  - BACKUP_FILENAME=portainer-backup-%Y-%m-%dT%H-%M-%S.{{ .Extension }}
  - BACKUP_PRUNING_PREFIX=portainer-backup-
  - BACKUP_STOP_DURING_BACKUP_LABEL=portainer-backup
  - GZIP_PARALLELISM=4
  - BACKUP_STOP_DURING_BACKUP_LABEL=30
  - BACKUP_RETENTION_DAYS=portainer-backup
  - NOTIFICATION_URLS=smtp://$REDACTED
volumes:
  - /home/jeujeus/dockerBackup:/archive
  - portainer_data:/backup/portainer:ro
  - /var/run/docker.sock:/var/run/docker.sock:ro
[...]
  2. Validate the backup is working (observe that the time of several backups is modified to be 30 days old whilst the backups themselve are not?) 
    
$ ls -al | grep portainer              
-rw-r--r--   1 jeujeus jeujeus    928166 Sep 11 21:05 portainer-backup-2024-08-13T00-00-00.tar.gz
-rw-r--r--   1 jeujeus jeujeus    928172 Sep 11 21:05 portainer-backup-2024-08-14T00-00-00.tar.gz
-rw-r--r--   1 jeujeus jeujeus    928499 Sep 11 21:05 portainer-backup-2024-08-15T00-00-00.tar.gz
-rw-r--r--   1 jeujeus jeujeus    928096 Sep 11 21:05 portainer-backup-2024-08-16T00-00-00.tar.gz
-rw-r--r--   1 jeujeus jeujeus    928070 Sep 11 21:05 portainer-backup-2024-08-17T00-00-00.tar.gz
-rw-r--r--   1 jeujeus jeujeus    928046 Sep 11 21:05 portainer-backup-2024-08-18T00-00-00.tar.gz
-rw-r--r--   1 jeujeus jeujeus    928051 Sep 11 21:05 portainer-backup-2024-08-19T00-00-00.tar.gz
-rw-r--r--   1 jeujeus jeujeus    927597 Sep 11 21:05 portainer-backup-2024-08-20T00-00-00.tar.gz
-rw-r--r--   1 jeujeus jeujeus    927644 Sep 11 21:05 portainer-backup-2024-08-21T00-00-00.tar.gz
-rw-r--r--   1 jeujeus jeujeus    927716 Sep 11 21:05 portainer-backup-2024-08-22T00-00-00.tar.gz
-rw-r--r--   1 jeujeus jeujeus    927665 Sep 11 21:05 portainer-backup-2024-08-23T00-00-00.tar.gz
-rw-r--r--   1 jeujeus jeujeus    927677 Sep 11 21:05 portainer-backup-2024-08-24T00-00-00.tar.gz
-rw-r--r--   1 jeujeus jeujeus    927551 Sep 11 21:05 portainer-backup-2024-08-25T00-00-00.tar.gz
-rw-r--r--   1 jeujeus jeujeus    927672 Sep 11 21:05 portainer-backup-2024-08-26T00-00-00.tar.gz
-rw-r--r--   1 jeujeus jeujeus    927616 Sep 11 21:05 portainer-backup-2024-08-27T00-00-00.tar.gz
-rw-r--r--   1 jeujeus jeujeus    924862 Sep 11 21:05 portainer-backup-2024-08-28T00-00-00.tar.gz
-rw-r--r--   1 jeujeus jeujeus    922381 Sep 11 21:05 portainer-backup-2024-08-29T00-00-00.tar.gz
-rw-r--r--   1 jeujeus jeujeus    922576 Sep 11 21:05 portainer-backup-2024-08-30T00-00-00.tar.gz
-rw-r--r--   1 jeujeus jeujeus    922374 Sep 11 21:05 portainer-backup-2024-08-31T00-00-00.tar.gz
-rw-r--r--   1 jeujeus jeujeus    923860 Sep 11 21:05 portainer-backup-2024-09-01T00-00-00.tar.gz
-rw-r--r--   1 jeujeus jeujeus    923938 Sep 11 21:05 portainer-backup-2024-09-02T00-00-00.tar.gz
-rw-r--r--   1 jeujeus jeujeus    923814 Sep 11 21:05 portainer-backup-2024-09-03T00-00-00.tar.gz
-rw-r--r--   1 jeujeus jeujeus    923930 Sep 11 21:05 portainer-backup-2024-09-04T00-00-00.tar.gz
-rw-r--r--   1 jeujeus jeujeus    924104 Sep 11 21:05 portainer-backup-2024-09-05T00-00-00.tar.gz
-rw-r--r--   1 jeujeus jeujeus    924472 Sep 11 21:05 portainer-backup-2024-09-06T00-00-00.tar.gz
-rw-r--r--   1 jeujeus jeujeus    923806 Sep 11 21:05 portainer-backup-2024-09-07T00-00-00.tar.gz
-rw-r--r--   1 jeujeus jeujeus    923528 Sep 11 21:05 portainer-backup-2024-09-08T00-00-00.tar.gz
-rw-r--r--   1 jeujeus jeujeus    923630 Sep 11 21:05 portainer-backup-2024-09-09T00-00-00.tar.gz
-rw-r--r--   1 jeujeus jeujeus    923606 Sep 11 21:05 portainer-backup-2024-09-10T00-00-00.tar.gz
-rw-r--r--   1 jeujeus jeujeus    927095 Sep 11 21:05 portainer-backup-2024-09-11T00-00-00.tar.gz
-rw-r--r--   1 jeujeus jeujeus    926719 Sep 13 02:00 portainer-backup-2024-09-13T00-00-00.tar.gz
-rw-r--r--   1 jeujeus jeujeus    923048 Sep 14 02:00 portainer-backup-2024-09-14T00-00-00.tar.gz
-rw-r--r--   1 jeujeus jeujeus    924589 Sep 15 02:00 portainer-backup-2024-09-15T00-00-00.tar.gz
-rw-r--r--   1 jeujeus jeujeus    924311 Sep 16 02:00 portainer-backup-2024-09-16T00-00-00.tar.gz
-rw-r--r--   1 jeujeus jeujeus    924278 Sep 17 02:00 portainer-backup-2024-09-17T00-00-00.tar.gz
-rw-r--r--   1 jeujeus jeujeus    924585 Sep 18 02:00 portainer-backup-2024-09-18T00-00-00.tar.gz
-rw-r--r--   1 jeujeus jeujeus    923988 Sep 19 02:00 portainer-backup-2024-09-19T00-00-00.tar.gz
-rw-r--r--   1 jeujeus jeujeus    925118 Sep 20 02:00 portainer-backup-2024-09-20T00-00-00.tar.gz
-rw-r--r--   1 jeujeus jeujeus    924922 Sep 21 02:00 portainer-backup-2024-09-21T00-00-00.tar.gz
-rw-r--r--   1 jeujeus jeujeus    924917 Sep 22 02:00 portainer-backup-2024-09-22T00-00-00.tar.gz
-rw-r--r--   1 jeujeus jeujeus    924490 Sep 23 02:00 portainer-backup-2024-09-23T00-00-00.tar.gz
-rw-r--r--   1 jeujeus jeujeus    924066 Sep 24 02:00 portainer-backup-2024-09-24T00-00-00.tar.gz
-rw-r--r--   1 jeujeus jeujeus    920138 Sep 25 02:00 portainer-backup-2024-09-25T00-00-00.tar.gz
-rw-r--r--   1 jeujeus jeujeus    919652 Sep 26 02:00 portainer-backup-2024-09-26T00-00-00.tar.gz
-rw-r--r--   1 jeujeus jeujeus    919629 Sep 27 02:00 portainer-backup-2024-09-27T00-00-00.tar.gz
-rw-r--r--   1 jeujeus jeujeus    919698 Sep 28 02:00 portainer-backup-2024-09-28T00-00-00.tar.gz
-rw-r--r--   1 jeujeus jeujeus    919126 Sep 29 02:00 portainer-backup-2024-09-29T00-00-00.tar.gz
-rw-r--r--   1 jeujeus jeujeus    920546 Sep 30 02:00 portainer-backup-2024-09-30T00-00-00.tar.gz
-rw-r--r--   1 jeujeus jeujeus    920544 Okt  1 02:00 portainer-backup-2024-10-01T00-00-00.tar.gz
-rw-r--r--   1 jeujeus jeujeus    920503 Okt  2 02:00 portainer-backup-2024-10-02T00-00-00.tar.gz
-rw-r--r--   1 jeujeus jeujeus    918378 Okt  3 02:00 portainer-backup-2024-10-03T00-00-00.tar.gz
-rw-r--r--   1 jeujeus jeujeus    918477 Okt  4 02:00 portainer-backup-2024-10-04T00-00-00.tar.gz
-rw-r--r--   1 jeujeus jeujeus    919057 Okt  5 02:00 portainer-backup-2024-10-05T00-00-00.tar.gz
-rw-r--r--   1 jeujeus jeujeus    918340 Okt  6 02:00 portainer-backup-2024-10-06T00-00-00.tar.gz
-rw-r--r--   1 jeujeus jeujeus    918183 Okt  7 02:00 portainer-backup-2024-10-07T00-00-00.tar.gz
-rw-r--r--   1 jeujeus jeujeus    916387 Okt  8 02:00 portainer-backup-2024-10-08T00-00-00.tar.gz
-rw-r--r--   1 jeujeus jeujeus    922904 Okt  9 02:00 portainer-backup-2024-10-09T00-00-00.tar.gz
-rw-r--r--   1 jeujeus jeujeus    922301 Okt 10 02:00 portainer-backup-2024-10-10T00-00-00.tar.gz
-rw-r--r--   1 jeujeus jeujeus    920884 Okt 11 02:00 portainer-backup-2024-10-11T00-00-00.tar.gz
3. Observe that no backup file is deleted - whilst several should be

ime=2024-10-10T05:27:20.515Z level=INFO msg="Successfully scheduled backup from environment with expression @daily" time=2024-10-11T00:00:00.100Z level=INFO msg="Now running script on schedule @daily" time=2024-10-11T00:00:00.112Z level=INFO msg="Stopping 1 out of 35 running container(s) as they were labeled docker-volume-backup.stop-during-backup=portainer-backup." time=2024-10-11T00:00:00.450Z level=INFO msg="Created backup of /backup at /tmp/portainer-backup-2024-10-11T00-00-00.tar.gz." time=2024-10-11T00:00:00.722Z level=INFO msg="Restarted 1 container(s)." time=2024-10-11T00:00:00.724Z level=INFO msg="Stored copy of backup /tmp/portainer-backup-2024-10-11T00-00-00.tar.gz in /archive." storage=Local time=2024-10-11T00:00:00.724Z level=INFO msg="None of 59 existing backups were pruned." storage=Local time=2024-10-11T00:00:00.725Z level=INFO msg="Removed tar file /tmp/portainer-backup-2024-10-11T00-00-00.tar.gz."



**Expected behavior**
Any Backup file older than 30 days should be deleted.
This should be reflected in the logs.
Also, this should be included in the error notification mails.

**Version (please complete the following information):**
 - Image Version: v2.43.0
 - Docker Version: Docker version 27.2.1, build 9e34c9b
 - Docker Compose Version (if applicable): v2.29.2
m90 commented 2 hours ago

I'm not entirely sure I understand who is changing the mod time in your setup yet. Is that you testing the prune feature and backdating files or is this happening without you doing anything?

Eligibility for pruning is happening by looking at files' mtimes: https://github.com/offen/docker-volume-backup/blob/52787a1e42024ca300d505566da9c16f8e897478/internal/storage/local/local.go#L116

Looking at the logs it seems no files match (yet), so there is nothing to be done. I.e. to debug this we'd need to find out if your files mtimes are changed inadvertently, and in case yes, who is doing that. I can't see such a behavior running the same image version.

JeuJeus commented 2 hours ago

That is indeed a good point. Is there some kind of debug flag i can activate to increase log verbosity?

I have for now removed all logs where the modification date does not match the timestamp in their filename. Will have a look and try to find out if there is some other application potentially interfering.

m90 commented 23 minutes ago

Is there some kind of debug flag i can activate to increase log verbosity?

Not really. From what I understand the easiest way to audit files for changes is using auditd: https://serverfault.com/questions/320716/find-out-which-process-is-changing-a-file