virt-backup clean does not clean backups

[deajan]

Hello,

While using virt-backup with the following config:

debug: True

threads: 4

uri: "qemu:///system"

username:
passphrase:

default:
  daily: 1

groups:
  test:
    target: /data/backups

    packager: directory

    autostart: True

    daily: 1

    quiesce: True

    hosts:
      - host: opnsense.warp.local
        quiesce: False
      - "r:^.*"

The command virt-backup clean does not remove earlier backups:

Backups removed for group test: 0
Broken backups removed for group test: 0

Anything I configured wrong ?

[aruhier]

Hi, By default, hourly and every other time options are set to "5". Can you list all the backups you have please? (It's mostly the timestamp I'm interested in)

Also, what happens if you set hourly to 1?

[deajan]

Here's an extract of my backups, knowing that I simply ran virt-backup twice in a row.

drwxr-xr-x  2 root root  216 Jul 16 13:26 .
drwxr-xr-x 12 root root  267 Jul 16 13:26 ..
-rw-r--r--  1 root root 6.5K Jul 16 13:03 20200716-125650_4_opnsense.mach.local.json
-rw-r--r--  1 root root  21G Jul 16 13:03 20200716-125650_4_opnsense.mach.local_sda.qcow2
-rw-r--r--  1 root root 6.5K Jul 16 13:26 20200716-131651_4_opnsense.mach.local.json
-rw-r--r--  1 root root  21G Jul 16 13:26 20200716-131651_4_opnsense.mach.local_sda.qcow2

My goal is to keep only one copy since I then sync that copy remotely.

I've set hourly: 1 but it doesn't make any difference into cleaning.

[aruhier]

I see. It's weird because I thought I fixed this issue here: https://github.com/aruhier/virt-backup/commit/ff4ef00d61667013951bdab961596acf0f738cdf And my test cover the case where 2 backups are in the same hour: https://github.com/aruhier/virt-backup/blob/master/tests/helper/virt_backup.py#L180

I'll try to reproduce the problem on my side.

[aruhier]

Ok I see the problem. It's located here: https://github.com/aruhier/virt-backup/blob/master/virt_backup/groups/complete.py#L250

Hourly and daily, when set to 1 but with 2 backups in the same day, will not retain the same backup, and therefore add both in the ones to keep.

Hourly will select 20200716-125650 and 20200716-131651, but will keep only 20200716-131651 because of https://github.com/aruhier/virt-backup/blob/master/virt_backup/groups/complete.py#L260 (n = 1, so it will take the last item of this list). However, Daily will group the 2 backups as they are of the same day, so it will append here the 2 backups: https://github.com/aruhier/virt-backup/blob/master/virt_backup/groups/complete.py#L271 Then this list is sorted here: https://github.com/aruhier/virt-backup/blob/master/virt_backup/groups/complete.py#L260 and the first item is took (20200716-125650, because of the backups[0]).

I did that because I wanted to keep the oldest backup for the wanted period: firsts backups of the day, firsts backups of the week, etc. I'll try to rework this logic. For now, if you want to fix your issue, set hourly: 0 in your configfile, and it should work.

[deajan]

Thanks for your answer. I've tested hourly: 0 but this only cleans the specified hosts, not those backed up via the regex above :( I've tried to comment out the host part to leave only the regex, eg:

      #- host: opnsense.warp.local
      # quiesce: False
      - "r:^.*"

The result is the same.

[prg318]

It would be awesome to have some additional clarification on how this works. I came here from the docs and while it makes sense after reading this entire issue, it would be awesome to have a simplified explanation in the docs. Perhaps once I've used this software a little more and have a better understanding, I could contribute some of the docs for this.

I've just started using this today and I'm very happy that this exists so I don't need to maintain my shell scripts that did this. Kudos to a handy project that I'm sure many people will enjoy

[aruhier]

Sorry I totally forgot about this issue. I'll try to get back to it in the next days of this week.

Thanks a lot for your comment, and I understand and agree that this logic is confusing and need clarification.