search

bit-team / backintime

Back In Time - An easy-to-use backup tool for GNU/Linux using rsync in the back
https://backintime.readthedocs.io
GNU General Public License v2.0
2.09k stars 204 forks source link

Deleting files doesn't cause a new snapshot, file deletions are not logged #136

Open Germar opened 9 years ago

Germar commented 9 years ago

I don't know if it's intentional, but for me it's a bug. When full rsync mode is enabled, if I delete a file or a folder from source, it won't be deleted in the next snapshot. Also, if the deletion is the only change in source, BIT says there's no difference and hence doesn't take any snapshot.

BTW: isn't there a way to get a snapshot in full rsync mode even if there's no change? The checkbox for this gets disabled when you choose "full rsync mode".

Imported from Launchpad using lp2gh.

Germar commented 9 years ago

(by germar) I can't confirm this here. Deleted files or folders are gone in next snapshot. And this would be a serious bug otherwise

Could you please describe your setup? Local or remote drive, source and destination filesystem, rsync version on both if remote...

'Check for changes' is disabled with 'Full rsync mode' because there is no such check. With normal mode this will deactivate the rsync --dry-run. But with 'Full rsync mode' BIT will create a new snapshot anyways and delete it afterwards if there was no changes. This is still faster than before.

Germar commented 9 years ago

(by mauromol) It's a local backup. In my test case, both source and destination folders are on the same ext4 filesytem, but I observe this even on an older Debian Wheezy on a NAS box with BIT 1.0.10 installed on it, backing up from an ext4 volume to another ext4 filesystem on a different device.

Here is my test-case profile:

profile4.name=test profile4.qt4.last_path=/ profile4.qt4.places.SortColumn=1 profile4.qt4.places.SortOrder=0 profile4.qt4.settingsdialog.exclude.SortColumn=1 profile4.qt4.settingsdialog.exclude.SortOrder=0 profile4.qt4.settingsdialog.include.SortColumn=1 profile4.qt4.settingsdialog.include.SortOrder=0 profile4.snapshots.automatic_backup_anacron_period=1 profile4.snapshots.automatic_backup_anacron_unit=20 profile4.snapshots.automatic_backup_day=1 profile4.snapshots.automatic_backup_mode=0 profile4.snapshots.automatic_backup_time=0 profile4.snapshots.automatic_backup_weekday=7 profile4.snapshots.backup_on_restore.enabled=true profile4.snapshots.bwlimit.enabled=false profile4.snapshots.bwlimit.value=3000 profile4.snapshots.check_for_changes=true profile4.snapshots.continue_on_errors=false profile4.snapshots.copy_links=false profile4.snapshots.copy_unsafe_links=false profile4.snapshots.cron.ionice=true profile4.snapshots.cron.nice=true profile4.snapshots.custom_backup_time=8,12,18,23 profile4.snapshots.dont_remove_namedsnapshots=true profile4.snapshots.exclude.1.value=.gvfs profile4.snapshots.exclude.10.value=/dev/ profile4.snapshots.exclude.11.value=/run/ profile4.snapshots.exclude.12.value=/etc/mtab profile4.snapshots.exclude.13.value=/var/cache/apt/archives/.deb profile4.snapshots.exclude.14.value=lost+found/ profile4.snapshots.exclude.15.value=/tmp/ profile4.snapshots.exclude.16.value=/var/tmp/ profile4.snapshots.exclude.17.value=/var/backups/ profile4.snapshots.exclude.18.value=.Private profile4.snapshots.exclude.2.value=.cache/ profile4.snapshots.exclude.3.value=.thumbnails profile4.snapshots.exclude.4.value=[Tt]rash profile4.snapshots.exclude.5.value=.backup profile4.snapshots.exclude.6.value=~ profile4.snapshots.exclude.7.value=.dropbox_ profile4.snapshots.exclude.8.value=/proc/ profile4.snapshots.exclude.9.value=/sys/ profile4.snapshots.exclude.bysize.enabled=false profile4.snapshots.exclude.bysize.value=500 profile4.snapshots.exclude.size=18 profile4.snapshots.full_rsync=true profile4.snapshots.include.1.type=0 profile4.snapshots.include.1.value=/home/mauro/tmp/test bit/src profile4.snapshots.include.size=1 profile4.snapshots.local.nocache=false profile4.snapshots.local.password.save=false profile4.snapshots.local.password.use_cache=true profile4.snapshots.local_encfs.path=/home/mauro/tmp/test bit/dst profile4.snapshots.log_level=3 profile4.snapshots.min_free_inodes.enabled=true profile4.snapshots.min_free_inodes.value=2 profile4.snapshots.min_free_space.enabled=true profile4.snapshots.min_free_space.unit=20 profile4.snapshots.min_free_space.value=1 profile4.snapshots.mode=local profile4.snapshots.no_on_battery=false profile4.snapshots.notify.enabled=true profile4.snapshots.path=/home/mauro/tmp/test bit/dst profile4.snapshots.path.host=hppb profile4.snapshots.path.profile=4 profile4.snapshots.path.user=mauro profile4.snapshots.preserve_acl=false profile4.snapshots.preserve_xattr=false profile4.snapshots.remove_old_snapshots.enabled=true profile4.snapshots.remove_old_snapshots.unit=80 profile4.snapshots.remove_old_snapshots.value=10 profile4.snapshots.rsync_options.enabled=false profile4.snapshots.rsync_options.value= profile4.snapshots.smart_remove=false profile4.snapshots.smart_remove.keep_all=2 profile4.snapshots.smart_remove.keep_one_per_day=7 profile4.snapshots.smart_remove.keep_one_per_month=24 profile4.snapshots.smart_remove.keep_one_per_week=4 profile4.snapshots.ssh.cipher=default profile4.snapshots.ssh.host= profile4.snapshots.ssh.ionice=false profile4.snapshots.ssh.nice=false profile4.snapshots.ssh.nocache=false profile4.snapshots.ssh.path= profile4.snapshots.ssh.port=22 profile4.snapshots.ssh.private_key_file=/home/mauro/.ssh/id_dsa profile4.snapshots.ssh.user=mauro profile4.snapshots.use_checksum=false profile4.snapshots.user_backup.ionice=false

Here is my test case:

cd /home/mauro/test\ bit mkdir src mkdir -p dst/backintime/hppb/mauro/4 cd src touch one mkdir test touch test/two backintime --profile-id 4 --backup-job

Back In Time Version: 1.1.3-4

Back In Time comes with ABSOLUTELY NO WARRANTY. This is free software, and you are welcome to redistribute it under certain conditions; type `backintime --license' for details.

INFO: Lock INFO: Inhibit Suspend started. Reason: take snapshot INFO: on process begins INFO: Profileid: 4 Fontconfig warning: "/etc/fonts/conf.d/50-user.conf", line 14: reading configurations from ~/.fonts.conf is deprecated. please move it to /home/mauro/.config/fontconfig/fonts.conf manually INFO: Call rsync to take the snapshot INFO: Command "rsync -rtDHh --links -pEgo --info=progress2 --no-i-r --delete --delete-excluded -v --chmod=Du+wx --exclude="/home/mauro/tmp/test bit/dst" --exclude="/home/mauro/.local/share/backintime" --exclude=".local/share/backintime/mnt" --include="/home/mauro/tmp/test bit/src/" --include="/home/mauro/tmp/test bit/" --include="/home/mauro/tmp/" --include="/home/mauro/" --include="/home/" --exclude=".gvfs" --exclude=".cache/" --exclude=".thumbnails" --exclude="[Tt]rash" --exclude=".backup" --exclude="~" --exclude=".dropbox" --exclude="/proc/" --exclude="/sys/" --exclude="/dev/" --exclude="/run/" --exclude="/etc/mtab" --exclude="/var/cache/apt/archives/.deb" --exclude="lost+found/" --exclude="/tmp/" --exclude="/var/tmp/" --exclude="/var/backups/" --exclude=".Private" --include="/home/mauro/tmp/test bit/src/*" --exclude="_" / "/home/mauro/tmp/test bit/dst/backintime/hppb/mauro/4/new_snapshot/backup/" -i --out-format="BACKINTIME: %i %n%L" 2>&1" returns 0 INFO: Save config file cp: l'obiettivo "bit/dst/backintime/hppb/mauro/4/new_snapshot/backup/.." non è una directory WARNING: Command "cp /home/mauro/.config/backintime/config /home/mauro/tmp/test bit/dst/backintime/hppb/mauro/4/new_snapshot/backup/.." returns 256 INFO: Create info file INFO: Remove backups older than: 20050301-000000 INFO: Keep min free disk space: 1024 Mb INFO: Keep min 2% free inodes INFO: [qt4systrayicon] begin loop INFO: Unlock INFO: Release inhibit Suspend INFO: [qt4systrayicon] end loop

ls -Rx ../dst/backintime/hppb/mauro/4/2*/ ../dst/backintime/hppb/mauro/4/20150321-141346-456/: backup info takesnapshot.log.bz2 ../dst/backintime/hppb/mauro/4/20150321-141346-456/backup: home ../dst/backintime/hppb/mauro/4/20150321-141346-456/backup/home: mauro ../dst/backintime/hppb/mauro/4/20150321-141346-456/backup/home/mauro: tmp ../dst/backintime/hppb/mauro/4/20150321-141346-456/backup/home/mauro/tmp: test bit ../dst/backintime/hppb/mauro/4/20150321-141346-456/backup/home/mauro/tmp/test bit: src ../dst/backintime/hppb/mauro/4/20150321-141346-456/backup/home/mauro/tmp/test bit/src: one test ../dst/backintime/hppb/mauro/4/20150321-141346-456/backup/home/mauro/tmp/test bit/src/test: two

rm test/two backintime --profile-id 4 --backup-job

Back In Time Version: 1.1.3-4

Back In Time comes with ABSOLUTELY NO WARRANTY. This is free software, and you are welcome to redistribute it under certain conditions; type `backintime --license' for details.

INFO: Lock INFO: Inhibit Suspend started. Reason: take snapshot INFO: on process begins INFO: Profileid: 4 INFO: Call rsync to take the snapshot INFO: Command "rsync -rtDHh --links -pEgo --info=progress2 --no-i-r --delete --delete-excluded -v --chmod=Du+wx --exclude="/home/mauro/tmp/test bit/dst" --exclude="/home/mauro/.local/share/backintime" --exclude=".local/share/backintime/mnt" --include="/home/mauro/tmp/test bit/src/" --include="/home/mauro/tmp/test bit/" --include="/home/mauro/tmp/" --include="/home/mauro/" --include="/home/" --exclude=".gvfs" --exclude=".cache/" --exclude=".thumbnails" --exclude="[Tt]rash" --exclude=".backup" --exclude="~" --exclude=".dropbox" --exclude="/proc/" --exclude="/sys/" --exclude="/dev/" --exclude="/run/" --exclude="/etc/mtab" --exclude="/var/cache/apt/archives/.deb" --exclude="lost+found/" --exclude="/tmp/" --exclude="/var/tmp/" --exclude="/var/backups/" --exclude=".Private" --include="/home/mauro/tmp/test bit/src/*" --exclude="_" / "/home/mauro/tmp/test bit/dst/backintime/hppb/mauro/4/new_snapshot/backup/" --link-dest="../../20150321-142752-455/backup" -i --out-format="BACKINTIME: %i %n%L" 2>&1" returns 0 INFO: Command "find "/home/mauro/tmp/test bit/dst/backintime/hppb/mauro/4/new_snapshot" -type d -exec chmod u+wx "{}" +" returns 0 INFO: Command "rm -rf "/home/mauro/tmp/test bit/dst/backintime/hppb/mauro/4/new_snapshot"" returns 0 INFO: Nothing changed, no back needed INFO: Command "rm -rf "/home/mauro/tmp/test bit/dst/backintime/hppb/mauro/4/20150321-142818-628"" returns 0 WARNING: No new snapshot INFO: Remove backups older than: 20050301-000000 INFO: Keep min free disk space: 1024 Mb INFO: Keep min 2% free inodes Fontconfig warning: "/etc/fonts/conf.d/50-user.conf", line 14: reading configurations from ~/.fonts.conf is deprecated. please move it to /home/mauro/.config/fontconfig/fonts.conf manually INFO: [qt4systrayicon] begin loop INFO: Unlock INFO: Release inhibit Suspend INFO: [qt4systrayicon] end loop

ls -Rx ../dst/backintime/hppb/mauro/4/2*/ ../dst/backintime/hppb/mauro/4/20150321-142752-455/: backup info takesnapshot.log.bz2

../dst/backintime/hppb/mauro/4/20150321-142752-455/backup: home

../dst/backintime/hppb/mauro/4/20150321-142752-455/backup/home: mauro

../dst/backintime/hppb/mauro/4/20150321-142752-455/backup/home/mauro: tmp

../dst/backintime/hppb/mauro/4/20150321-142752-455/backup/home/mauro/tmp: test bit

../dst/backintime/hppb/mauro/4/20150321-142752-455/backup/home/mauro/tmp/test bit: src

../dst/backintime/hppb/mauro/4/20150321-142752-455/backup/home/mauro/tmp/test bit/src: one test

../dst/backintime/hppb/mauro/4/20150321-142752-455/backup/home/mauro/tmp/test bit/src/test: two

As you can see, I just deleted file "two", but BIT didn't find any change, and hence took no new snapshot. If I uncheck the "full rync mode" I don't observe this behaviour and the new snapshot is taken correctly.

BTW: regarding the full rsync mode and no new snapshot if there are no changes: what I meant to say is that I would find very useful if I had the option to prevent BIT from deleting the new snapshot if no changes were detected (at least if either the option "keep partial snapshots in case of failure" is checked OR if there were no errors at all). In this way, if I backup every day, I will find a snapshot for every day, even if two days are exactly equal. Right now, if I don't find a snapshot, I can't know if it is missing because the backup failed (I don't want to keep partial snapshots in case of errors!) or rather because there were no changes from the previous day... I must check the logs to see.

Germar commented 9 years ago

(by mauromol) As an additional comment: this test case doesn't show that if I have a deletion in folder "test" and other changes in another folder (not in "test"), then a new snapshot IS actually taken, but the deletion of files in "test" is not applied to this new snapshot. I observed this in real cases on the NAS box I mentioned.

Germar commented 9 years ago

(by mauromol) Forgot to write about the rsync version: the test case was conducted on a Linux Mint 17.1 (based on Ubuntu 14.04): rsync version 3.1.0 protocol version 31 The NAS box is using a backported rsync version: rsync version 3.1.1 protocol version 31

Germar commented 9 years ago

(by germar) Okay, I was able to reproduce it with your config. Thanks

First think I recognized is that paths for 'save config file' are not quoted correctly and so break with spaces. But that's not the root of this problem. I'll skip new release again and fix this first ;D

BTW BIT 1.0.10 didn't have 'full rsync mode'. So that must be something different.

Germar commented 9 years ago

(by germar) Okay, here is what happens:

I'm not sure if there is a way to tell rsync to print out a message when there was more files in --link-dest than local which could be used to determine there actually was a change.

But I'll add an option (as you suggested) to keep snapshots with 'Full rsync mode' regardless if there were changes or not.

Germar commented 9 years ago

(by mauromol) IMHO if we're unlucky and in full rsync mode there's no way to know if there are changes or not in this particular case, having BIT create a new snapshot every time even if it might be equal to the previous one when full rsync mode is enabled, is a compromise that is far better than having a "wrong" snapshot or suppress the full rsync mode at all. On the contrary, as I said earlier I would prefer to have a snapshot preserved in any case, independently from this problem.

Germar commented 9 years ago

(by germar) I added the above mentioned option (http://bazaar.launchpad.net/~bit-team/backintime/trunk/revision/1072). It is default off.

So with next release 1.1.4 you can use the 'take snapshot regardless of changes' option and I'll try to find a way to be noticed of deleted files for over-next release.

Germar commented 9 years ago

(by mauromol) Hi Germar, please note that the English phrase for 'take snapshot regardless of changes' has a typo ("reagardless" instead of "regardless"). I couldn't find a way to change it myself through LaunchPad.

Germar commented 9 years ago

(by germar) I didn't found any suitable way of detecting deleted files with 'Full rsync mode'. Only way would be to either go back and run a --dry-run first or to create hard-links with 'cp -aRl' again. Which both would make it very slow again.

There is already a bug report for rsync about this https://bugzilla.samba.org/show_bug.cgi?id=5263 and a discussion on mailing list https://

emtiu commented 2 years ago

This is easily reproduced, and it's a serious bug.

  1. Create a test profile that backs up a few files: foo, bar and baz. Take a snapshot.
  2. Delete baz and take a new snapshot. --> No snapshot is taken.
  3. Create a new file qux. Take a new snapshot. --> The deletion of baz is not logged.
buhtz commented 1 year ago

Just stumbled across this myself today (again). I deleted folders. But BIT tells me that there is no need of a new snapshot.

I wonder why rsync doesn't handle this. Why need BIT to decide this?

EDIT: snapshots.py::Snapshots.takeSnapshot() seems to be involved here. The code is hard to read and full of cryptic TODO comments and object names. I will not go any further here for the time being and put the matter to one side. We don't have a time window for the next release yet. But I would hypothesize that this won't get fixed until in the next release.