House Keeping Not Working

[mrtwister76]

Hi,

I'm running a backup command like:

/mnt/backups/barc/cv4pve-barc/eve4pve-barc create --vmid=104 --label='daily' --path=/mnt/backups/barc --compress=pigz --keep=3 --debug

However, previous diffs never get removed or merged into the diff0. New ones are created instead.

This is what my current backup target looks like after running the command 13 times (I think :):

drwxr-xr-x 2 4294967294 root 54 Sep 18 12:21 . drwxr-xr-x 3 4294967294 root 3 Sep 18 11:21 .. -rw-r--r-- 1 4294967294 root 926M Sep 18 11:29 20200918112139ceph_vm_pool.vm-104-disk-0.img.zz -rw-r--r-- 1 4294967294 root 44 Sep 18 11:29 20200918112139ceph_vm_pool.vm-104-disk-0.img.zz.sha1 -rw-r--r-- 1 4294967294 root 11 Sep 18 11:29 20200918112139ceph_vm_pool.vm-104-disk-0.img.zz.sha1.size -rw-r----- 1 4294967294 root 254 Sep 18 11:29 20200918112139.conf -rw-r--r-- 1 4294967294 root 74 Sep 18 11:40 20200918114022ceph_vm_pool.vm-104-disk-0.diff.zz -rw-r--r-- 1 4294967294 root 44 Sep 18 11:40 20200918114022ceph_vm_pool.vm-104-disk-0.diff.zz.sha1 -rw-r--r-- 1 4294967294 root 3 Sep 18 11:40 20200918114022ceph_vm_pool.vm-104-disk-0.diff.zz.sha1.size -rw-r----- 1 4294967294 root 254 Sep 18 11:40 20200918114022.conf -rw-r--r-- 1 4294967294 root 72 Sep 18 11:41 20200918114055ceph_vm_pool.vm-104-disk-0.diff.zz -rw-r--r-- 1 4294967294 root 44 Sep 18 11:41 20200918114055ceph_vm_pool.vm-104-disk-0.diff.zz.sha1 -rw-r--r-- 1 4294967294 root 3 Sep 18 11:41 20200918114055ceph_vm_pool.vm-104-disk-0.diff.zz.sha1.size -rw-r----- 1 4294967294 root 254 Sep 18 11:41 20200918114055.conf -rw-r--r-- 1 4294967294 root 73 Sep 18 11:41 20200918114118ceph_vm_pool.vm-104-disk-0.diff.zz -rw-r--r-- 1 4294967294 root 44 Sep 18 11:41 20200918114118ceph_vm_pool.vm-104-disk-0.diff.zz.sha1 -rw-r--r-- 1 4294967294 root 3 Sep 18 11:41 20200918114118ceph_vm_pool.vm-104-disk-0.diff.zz.sha1.size -rw-r----- 1 4294967294 root 254 Sep 18 11:41 20200918114118.conf -rw-r--r-- 1 4294967294 root 72 Sep 18 11:41 20200918114136ceph_vm_pool.vm-104-disk-0.diff.zz -rw-r--r-- 1 4294967294 root 44 Sep 18 11:41 20200918114136ceph_vm_pool.vm-104-disk-0.diff.zz.sha1 -rw-r--r-- 1 4294967294 root 3 Sep 18 11:41 20200918114136ceph_vm_pool.vm-104-disk-0.diff.zz.sha1.size -rw-r----- 1 4294967294 root 254 Sep 18 11:41 20200918114136.conf -rw-r--r-- 1 4294967294 root 73 Sep 18 11:42 20200918114221ceph_vm_pool.vm-104-disk-0.diff.zz -rw-r--r-- 1 4294967294 root 44 Sep 18 11:42 20200918114221ceph_vm_pool.vm-104-disk-0.diff.zz.sha1 -rw-r--r-- 1 4294967294 root 3 Sep 18 11:42 20200918114221ceph_vm_pool.vm-104-disk-0.diff.zz.sha1.size -rw-r----- 1 4294967294 root 254 Sep 18 11:42 20200918114221.conf -rw-r--r-- 1 4294967294 root 72 Sep 18 11:43 20200918114253ceph_vm_pool.vm-104-disk-0.diff.zz -rw-r--r-- 1 4294967294 root 44 Sep 18 11:43 20200918114253ceph_vm_pool.vm-104-disk-0.diff.zz.sha1 -rw-r--r-- 1 4294967294 root 3 Sep 18 11:43 20200918114253ceph_vm_pool.vm-104-disk-0.diff.zz.sha1.size -rw-r----- 1 4294967294 root 254 Sep 18 11:43 20200918114253.conf -rw-r--r-- 1 4294967294 root 74 Sep 18 11:53 20200918115311ceph_vm_pool.vm-104-disk-0.diff.zz -rw-r--r-- 1 4294967294 root 44 Sep 18 11:53 20200918115311ceph_vm_pool.vm-104-disk-0.diff.zz.sha1 -rw-r--r-- 1 4294967294 root 3 Sep 18 11:53 20200918115311ceph_vm_pool.vm-104-disk-0.diff.zz.sha1.size -rw-r----- 1 4294967294 root 254 Sep 18 11:53 20200918115311.conf -rw-r--r-- 1 4294967294 root 72 Sep 18 11:54 20200918115356ceph_vm_pool.vm-104-disk-0.diff.zz -rw-r--r-- 1 4294967294 root 44 Sep 18 11:54 20200918115356ceph_vm_pool.vm-104-disk-0.diff.zz.sha1 -rw-r--r-- 1 4294967294 root 3 Sep 18 11:54 20200918115356ceph_vm_pool.vm-104-disk-0.diff.zz.sha1.size -rw-r----- 1 4294967294 root 254 Sep 18 11:54 20200918115356.conf -rw-r--r-- 1 4294967294 root 73 Sep 18 11:58 20200918115846ceph_vm_pool.vm-104-disk-0.diff.zz -rw-r--r-- 1 4294967294 root 44 Sep 18 11:58 20200918115846ceph_vm_pool.vm-104-disk-0.diff.zz.sha1 -rw-r--r-- 1 4294967294 root 3 Sep 18 11:58 20200918115846ceph_vm_pool.vm-104-disk-0.diff.zz.sha1.size -rw-r----- 1 4294967294 root 254 Sep 18 11:58 20200918115846.conf -rw-r--r-- 1 4294967294 root 75 Sep 18 12:03 20200918120328ceph_vm_pool.vm-104-disk-0.diff.zz -rw-r--r-- 1 4294967294 root 44 Sep 18 12:03 20200918120328ceph_vm_pool.vm-104-disk-0.diff.zz.sha1 -rw-r--r-- 1 4294967294 root 3 Sep 18 12:03 20200918120328ceph_vm_pool.vm-104-disk-0.diff.zz.sha1.size -rw-r----- 1 4294967294 root 254 Sep 18 12:03 20200918120328.conf -rw-r--r-- 1 4294967294 root 73 Sep 18 12:20 20200918122007ceph_vm_pool.vm-104-disk-0.diff.zz -rw-r--r-- 1 4294967294 root 44 Sep 18 12:20 20200918122007ceph_vm_pool.vm-104-disk-0.diff.zz.sha1 -rw-r--r-- 1 4294967294 root 3 Sep 18 12:20 20200918122007ceph_vm_pool.vm-104-disk-0.diff.zz.sha1.size -rw-r----- 1 4294967294 root 254 Sep 18 12:20 20200918122007.conf -rw-r--r-- 1 4294967294 root 72 Sep 18 12:21 20200918122123ceph_vm_pool.vm-104-disk-0.diff.zz -rw-r--r-- 1 4294967294 root 44 Sep 18 12:21 20200918122123ceph_vm_pool.vm-104-disk-0.diff.zz.sha1 -rw-r--r-- 1 4294967294 root 3 Sep 18 12:21 20200918122123ceph_vm_pool.vm-104-disk-0.diff.zz.sha1.size -rw-r----- 1 4294967294 root 254 Sep 18 12:21 20200918122123.conf

The backup & eve4pve-barc are located on a local NFS share.

I'm just testing at the moment, so no changes are being made to the backed up VM, which is also not booted. Would this make a difference?

Any help appreciated.

Thanks!

[mrtwister76]

Any help appreciated :)

[franklupo]

Test new version

[mrtwister76]

The fixes in the new version don't fix this issue.

From initial investigation, this issue looks like it's being caused by a file name mismatch, possibly caused by the checksumming feature that was added recently.

I'll try to get some time to look at it today :)

[lephisto]

I have it, along with some other fixes in my fork repo.

• Housekeeping is fixed (which occured when disks were added during the lifetime of a vm)
• Bug with rbd volumes containing numbers fixed
• Made snapshot procedure more stable, once a backup has failed for reasons.

[lephisto]

We might stash the changes here and merge it. It would be cool if someone else could test on it.

https://github.com/lephisto/cv4pve-barc/

[mrtwister76]

My fixes for this were pulled in yesterday. There are some more waiting to be pulled which were added earlier today which should get everything working again :)

[lephisto]

Still, look at my changes regarding housekeeping. There's a regresson in housekeeping which can cause problems if you add more disks while a machine is already in the backup logic.