New permissions handling in 1.2.0 causes re-backup of files that haven't changed

[kosal75]

I have just updated from 1.1.24 to 1.2.0 (common, gnome, qt4), and making a snapshot takes hours. When downgrading to 1.1.24, all is normal (5-6 minutes, as always).

Ubuntu 18.10 with kernel 4.18.0-18.

[ghost]

Hi Germar, hope all is good! 😀 Like @kosal75 I too got updated today, and ran into many issues. These were:

• My previous snapshots (over 2 years worth) were no longer "considered". Although they were visible in the BiT-GUI, a new update triggered a completely new "virgin" rsync of all files on all disks defined previously (see below), whether changed, new or not.
• Having said that, $HOME/.config/backintime/config was found and recognised though and executed without errors, i.e. only files/ dirs defined in $HOME/.config/backintime/config previously were backup-ed.
• As @kosal75 writes, the rsync is extremely slow. With my backup disk on a type C connection, write speeds are below < 1MB/s. Normally (and after downgrade) I get > 200 MB/s.
• After the BiT/ rsync has been running for a while, my backup disk is unmounted autonomously. As a result BiT, which still runs, throws endless error notifications.

I too downgraded to one version lower available on my ppa: sudo apt install backintime-qt4=1.1.12-2 backintime-common=1.1.12-2 backintime-gnome=1.1.12-2 sudo apt-mark hold backintime-common backintime-qt4 backintime-gnome

After this, peace and quiet has returned, all is good, and BiT performs with its previous accuracy, speed, and without flaws.

FYI: currently I'm on Xubuntu 18.04 LTS HWE (kernel 4.18.0-18).

Hope this helps...

[kosal75]

I have found one more issue after downgrading: the cron settings were completely reset by 1.2.0.

Sent from my Samsung Galaxy S7 On 2019. április 28. 20:04:49 BelladonnavGFH notifications@github.com wrote:

Hi Germar, hope all is good! Like @kosal75 I too got updated today, and ran into many issues. These were: My previous snapshots (over 2 years worth) were no longer recognised/ found. Although they were visible in the BiT-GUI, a new update triggered a new "virgin" rsync of all files on all disks. Having said that, $HOME/.config/backintime/config was found and recognised though and executed without errors. As @kosal75 writes, the rsync is extremely slow. With my backup disk on a type C connection, write speeds are blow < 1MB/s After the BiT/ rsync has been running for a while, my backup disk is unmounted. As a result BiT throws endless notification errors. I too downgraded to one version lower available on my ppa: sudo apt install backintime-qt4=1.1.12-2 backintime-common=1.1.12-2 backintime-gnome=1.1.12-2 sudo apt-mark hold backintime-common backintime-qt4 backintime-gnome After this, all is good, and BiT performs with its previous accuracy, speed, and without flaws. FYI: currently I'm on Xubuntu 18.04 LTS HWE (kernel 4.18.0-18). Hope this helps... — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[pehlm]

Thanks @BelladonnavGFH for your tip! I haven't tried to do a backup with 1.2.0 and cannot say anything about the slowness, but the old backups wasn't, as you said, recognized any longer and I panicked. I'm also on Xubuntu 18.04 and it wasn't even possible to rezise the window, it was super big! Not possible to get it smaller. Thanks to you I have it working again. Even if I was forced to get an older version 1.1.12, the version before 1.2.0 was 1.1.24. To the developers: I'm hoping you can get the new version to work, Back In Time is an extremely good backup-program!

[ghost]

No thanks @pehlm. If it now works for you, perfect! As a clarification: if 1.1.24 works for you, don't let me stop you using that one. The only reason I went to 1.1.12 is because that is the only version available on the ppa(s) I use(next to 1.2.0 that is of course). And as you mention, I assume this is only a temp thing, hoping to upgrade when the issues are resolved. Because, yes, I agree, BiT is very good...

[hannes101]

I can confirm the slow backup, although all my backups were found and no full backup was done. Although looking at the logs I was able to see that apparently there were many files backuped, which actually didn't change.

[catmatist]

In the Ubuntu bionic ppa for bit "stable", when bit 1.2 was added, 1.1.24 was removed. Can it be put back, please? There were many bugs fixed between the version in the official Ubuntu repositories (1.1.12?) and 1.1.24, which is why I was using the ppa. I was fortunate enough to see the change coming before it was applied, and disabled the ppa in order to keep the version I've been using until I have time to figure out if 1.2 will work for me. Which will take a lot longer if I have no easy way to get back to 1.1.24 on a system that I try 1.2 on. Thank you for working on this very useful software.

[jns-]

After updating to 1.2.0 BiT does a (nearly) full backup because file permissions are handled differently. Before 1.2.0 all destination file permissions were set to -rw-r--r--. In 1.2.0 rsync is executed with --perms option which tells rsync to preserve the source file permission. That's why so many files seem to be changed.

[colinl]

@jns is it actually backing them up again (and consuming extra disc space) or is it just changing the file permissions, which takes time but does not use any extra disc space?

[jns-]

I think it's backing up the files again. My last (and first 1.2.0) backup took roughly 1 hour and created a snapshot [WITH ERRORS]. However, no error messages were written to the log file. After deleting that snapshot disk usage dropped from 35% to 19% (~300G). Unfortunately I did not check file inodes before deleting that snapshot. Considering the amount of freed disk space, I assume BiT made a complete backup, except files where permissions matched -rw-r--r--.

[catmatist]

In 1.1.24 (and I don't know how many earlier versions), there is a per-profile setting for "Full rsync mode". If you turned it on, it copied permissions to the backup directory with rsync instead of saving them in a file. I think I saw a change log comment somewhere that said 1.2 dropped support for the previous default mode and only supports "Full rsync mode". I wonder if the transition is smoother for backups that were already using "Full rsync mode"? Is there any documentation anywhere on how to make the switch to the new version without tears?

[Germar]

Hey all, sorry for the inconvenience with version 1.2

Like @jns- already pointed out, BiT will now let rsync change the permissions directly in the snapshot. At the first snapshot with the new version this will make rsync calculate checksums for all source and all snapshot files to make sure, they are definitely the same. Which of course takes a long time. Normally rsync only compares modification-time, size and permission to tell if a file has changed. With the next snapshot rsync should be fast as before or even faster because of the optimized code.

To confirm you could compare the inodes of files that didn't change. Take a look at this FAQ: How can I check if my snapshots are incremental (using hard-links)?

Also you can check the infos rsync provides for each file it processed. From man rsync:

The attribute that is associated with each letter is as follows:

o      A  c means either that a regular file has a different checksum
       (requires --checksum) or that a symlink,  device,  or  special
       file  has a changed value.  Note that if you are sending files
       to an rsync prior to 3.0.1, this change flag will  be  present
       only for checksum-differing regular files.

o      A  s means the size of a regular file is different and will be
       updated by the file transfer.

o      A t means the modification time  is  different  and  is  being
       updated  to  the sender’s value (requires --times).  An alter‐
       nate value of T means that the modification time will  be  set
       to the transfer time, which happens when a file/symlink/device
       is updated without --times and when a symlink is  changed  and
       the  receiver  can’t set its time.  (Note: when using an rsync
       3.0.0 client, you might see the s flag combined with t instead
       of the proper T flag for this time-setting failure.)

o      A  p means the permissions are different and are being updated
       to the sender’s value (requires --perms).

o      An o means the owner is different and is being updated to  the
       sender’s value (requires --owner and super-user privileges).

o      A  g  means the group is different and is being updated to the
       sender’s value (requires --group and the authority to set  the
       group).

o      The u slot is reserved for future use.

o      The a means that the ACL information changed.

o      The x means that the extended attribute information changed.

If you have lots of files that didn't change but are marked with c this could indicate a failing harddrive. You should run fsck and consider replacing the drive.

[kosal75]

And what happens on a drive not supporting Linux file permissions? I also use samba mount.

Sent from my Samsung Galaxy S7 On 2019. május 2. 20:29:38 Germar Reitze notifications@github.com wrote:

Hey all, sorry for the inconvenience with version 1.2Like @jns- already pointed out, BiT will now let rsync change the permissions directly in the snapshot. At the first snapshot with the new version this will make rsync calculate checksums for all source and all snapshot files to make sure, they are definitely the same. Which of course takes a long time. Normally rsync only compares modification-time, size and permission to tell if a file has changed. With the next snapshot rsync should be fast as before or even faster because of the optimized code. To confirm you could compare the inodes of files that didn't change. Take a look at this FAQ: How can I check if my snapshots are incremental (using hard-links)? Also you can check the infos rsync provides for each file it processed. From man rsync: The attribute that is associated with each letter is as follows: o A c means either that a regular file has a different checksum (requires --checksum) or that a symlink, device, or special file has a changed value. Note that if you are sending files to an rsync prior to 3.0.1, this change flag will be present only for checksum-differing regular files. o A s means the size of a regular file is different and will be updated by the file transfer. o A t means the modification time is different and is being updated to the sender’s value (requires --times). An alter‐ nate value of T means that the modification time will be set to the transfer time, which happens when a file/symlink/device is updated without --times and when a symlink is changed and the receiver can’t set its time. (Note: when using an rsync 3.0.0 client, you might see the s flag combined with t instead of the proper T flag for this time-setting failure.) o A p means the permissions are different and are being updated to the sender’s value (requires --perms). o An o means the owner is different and is being updated to the sender’s value (requires --owner and super-user privileges). o A g means the group is different and is being updated to the sender’s value (requires --group and the authority to set the group). o The u slot is reserved for future use. o The a means that the ACL information changed. o The x means that the extended attribute information changed.

If you have lots of files that didn't change but are marked with c this could indicate a failing harddrive. You should run fsck and consider replacing the drive. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[colinl]

@Germar I just ran a test on Ubuntu 18.04 backing up to local drive. I created a new backup with version 1.1.24 on default settings then upgraded (via the bit team stable ppa) to 1.2.0 and ran another backup. In the log this showed every file with [C] cf...p..... and the size of the backup doubled (measured with ncdu and inode values checked just to be sure). I am not sure from your previous post whether that is expected or not.

[Germar]

And what happens on a drive not supporting Linux file permissions?

BiT still stores permissions inside the fileinfo.bz2 like in previous versions

[Germar]

In the log this showed every file with [C] cf...p..... and the size of the backup doubled (measured with ncdu and inode values checked just to be sure). I am not sure from your previous post whether that is expected or not.

No, thats not expected. Please hang on, I need to test this. Will take a bit as I need to prepare a test VM first which is an Ubuntu 16.04 updated to 18.04. Otherwise I'm not able to install 1.1.24 in 18.04

[Germar]

Hmmm

$ mkdir src dst1 dst2 dst3
$ cp backintime-1.1.24/CHANGES src/
$ rsync --no-p -t -r -v -i src/ dst1/
sending incremental file list
.d..t...... ./
>f+++++++++ CHANGES

sent 39,299 bytes  received 38 bytes  78,674.00 bytes/sec
total size is 39,185  speedup is 1.00
$ rsync --no-p -t -r -v -i src/ dst2/ --link-dest=../dst1/
sending incremental file list
.d..t...... ./

sent 63 bytes  received 19 bytes  164.00 bytes/sec
total size is 39,185  speedup is 477.87
$ rsync -p -t -r -v -i src/ dst3/ --link-dest=../dst2/
sending incremental file list
.d..t...... ./

sent 67 bytes  received 19 bytes  172.00 bytes/sec
total size is 39,185  speedup is 455.64
$ 

All fine so far. But this is stange:

$ mkdir src dst1 dst2 dst3
$ # copy the same file as before into src but this time using nautilus
$ rsync --no-p -t -r -v -i src/ dst1/
sending incremental file list
.d..t...... ./
>f+++++++++ CHANGES

sent 39,301 bytes  received 38 bytes  78,678.00 bytes/sec
total size is 39,185  speedup is 1.00
$ rsync --no-p -t -r -v -i src/ dst2/ --link-dest=../dst1/
sending incremental file list
.d..t...... ./

sent 69 bytes  received 19 bytes  176.00 bytes/sec
total size is 39,185  speedup is 445.28
$ rsync -p -t -r -v -i src/ dst3/ --link-dest=../dst2/
sending incremental file list
.d..t...... ./
cf...p..... CHANGES     <<<----- this is, what actually happens

sent 72 bytes  received 22 bytes  188.00 bytes/sec
total size is 39,185  speedup is 416.86
$ 

EDIT: removed the last part as it was wrong...

[jns-]

No, enabling Preserve ACL makes no difference in my case. Disk usage increases rapidly.

Investigating a random file that did not change since 2017:

BiT last log entry

[I] Take snapshot (rsync: BACKINTIME: cf...p..... home/user/out.log)

[C] cf...p..... home/user/out.log

Running

$ stat /backuppath/backintime/MACHINE/user/1/*/backup/home/user/out.log

results [truncated]

  File: '/backuppath/backintime/MACHINE/user/1/20190430-071835-513/backup/home/user/out.log'
  Size: 1124        Blocks: 8          IO Block: 4096   regular file
Device: fc00h/64512d    Inode: 83624689    Links: 25
Access: (0644/-rw-r--r--)  Uid: ( 1000/     jns)   Gid: ( 1000/     jns)
Access: 2019-05-03 07:23:28.065557278 +0200
Modify: 2017-10-04 09:11:08.408810922 +0200
Change: 2019-05-01 13:48:55.184458078 +0200
 Birth: -
  File: '/backuppath/backintime/MACHINE/user/1/last_snapshot/backup/home/user/out.log'
  Size: 1124        Blocks: 8          IO Block: 4096   regular file
Device: fc00h/64512d    Inode: 83624689    Links: 25
Access: (0644/-rw-r--r--)  Uid: ( 1000/     jns)   Gid: ( 1000/     jns)
Access: 2019-05-03 07:23:28.065557278 +0200
Modify: 2017-10-04 09:11:08.408810922 +0200
Change: 2019-05-01 13:48:55.184458078 +0200
 Birth: -
  File: '/backuppath/backintime/MACHINE/user/1/new_snapshot/backup/home/user/out.log'
  Size: 1124        Blocks: 8          IO Block: 4096   regular file
Device: fc00h/64512d    Inode: 83758208    Links: 1
Access: (0664/-rw-rw-r--)  Uid: ( 1000/     jns)   Gid: ( 1000/     jns)
Access: 2019-05-03 07:23:28.065557278 +0200
Modify: 2017-10-04 09:11:08.408810922 +0200
Change: 2019-05-03 07:23:28.065557278 +0200
 Birth: -

Inode of that file is 83624689 for all past snapshots and changes to 83758208 in new_snapshot. As I mentioned before the problem seems to be the destination permissions set to (0644/-rw-r--r--) for pre 1.2.0 snapshots and setting it to the actual source file permission (0664/-rw-rw-r--) in 1.2.0 snapshots.

[jns-]

What brought me back to business is following: I changed permissions of files in the last_snapshot folder to match source file settings by extracting that information from fileinfo.bz2. After that backing up with 1.2.0 onto a <1.2.0 snapshot was blazingly fast and no considerable disk space was used.

I do not yet know if this procedure creates any regrettable side effects!

[ghost]

@jns- Whaa! You lost me there... What exactly did you do? If I take a look at fileinfo.bz2 I see: 33188 bella bella /media/bella/TikiStorage/foo.jpg So what exactly did you do now? Did you change it in fileinfo.bz2, or did you change the permissions of the files themselves in last_snapshot? And if you did the latter, I assume you didn't do this by hand, one by one? Please tell, would love to try it myself in a VM...

[Germar]

No, enabling Preserve ACL makes no difference in my case. Disk usage increases rapidly.

Ja, I realized my mistake after laying my head on the pillow last night. So I turned out again and deleted the last part of my comment :laughing: Today I'd say you should rather ignore my last post at all :see_no_evil:

I did some further testing which now shows the root of the problem:

$ mkdir src dst1
$ echo "bar" > src/foo
$ cp -al src/foo dst1/foo
$ ls -lai src/foo dst1/foo
6183473 -rw-r--r-- 2 germar germar 4 Mai  3 22:36 dst1/foo
6183473 -rw-r--r-- 2 germar germar 4 Mai  3 22:36 src/foo
$ chmod 664 src/foo 
$ ls -lai src/foo dst1/foo
6183473 -rw-rw-r-- 2 germar germar 4 Mai  3 22:36 dst1/foo
6183473 -rw-rw-r-- 2 germar germar 4 Mai  3 22:36 src/foo
$ 

So, changing permissions of one hard-link will change permissions in all hard-links. Which, after knowing the fact, is totally logical because a hard-link is just a pointer to the inode. It doesn't store anything else. Not even permissions. And that's why rsync creates a new inode even if there was just a change in permissions...

[jns-]

In my previous posts I used the term 'pre 1.2.0' for BiT versions older than 1.2.0. From now on let's assume it was version 1.1.4 for the sake of better readability.

My intention was to prevent BiT 1.2.0 to create a full backup on top of my 1.1.4 snapshot history. The idea was to modify the 1.1.4 snapshot history outside of BiT and make it look ok to BiT 1.2.0 next time it is started.

Like @Germar explains, hard links are just pointers to files on the disk and do not store any additional information. Permissions are part of the file the hard link is pointing to.

BiT crates a snapshot folder called last_snapshot. From that folder the latest version of all backup files can be accessed (via hard links), no matter in what snapshot the files were written to the disk. The original source file permissions are stored in fileinfo.bz2 for every snapshot taken. The permissions are somewhat hidden in the first number on each line.

For example in

33204 user group /home/user/file.txt

33204 in octal representation is 100664. The last 3 digits represent the original file permission, which corresponds to -rw-rw-r--. All we have to do is to grab those last 3 digits and chmod 664 /bit_backup_path/last_snapshot/backup/home/user/file.txt. After that the backed up file has the same permissions like its source file and BiT 1.2.0 no longer thinks it needs backing it up again.

What I did in detail:

1. Be very careful and consider backing up your backup files. That should be done with something like rsync -a -H --delete --progress /bit_backup/ /backup_of_bit_backup/ -H is important to preserve the BiT hard link structure.

2. Start BiT and delete all snapshots created with BiT 1.2.0. Quit BiT. Now you have a 'clean' 1.1.4 snapshot history.

3. Decompress fileinfo.bz2 from last_snapshot folder to fileinfo.txt and put that file into your home directory.

4. Now, modify all permissions of files in in last_snapshot according to information grabbed from fileinfo.bz2.

5. Start BiT 1.2.0 and take a snapshot. It should now create a regular incremental snapshot.

I used

cat fileinfo.txt | while read line; do perm=`echo "${line}" | cut -d" " -f1`; path_=`echo ${line} | cut -d" " -f4-`; perm=$(([##8]perm)); perm="${perm:$((${#perm}-3)):3}"; echo chmod "$perm" "/bit_backup_path/last_snapshot/backup$path_"; done

for step 4. (works in Z shell, probably not in bash). The last echo before chmod has to be removed to effectively change file permissions. Otherwise the command is just echoed to your console.

I recommend to test permission modification first on a subset of backup files by truncating fileinfo.txt. Probably you can find some useless files in your backups to experiment with, for example .Fonts folder in your home directory or similar.

Running the one liner may take some time (~350GB corresponding to 200k files, took ~15min on my laptop). Optimize it according to your needs and show us something way more elegand :smirk:

Note 1: As I mentioned in a post above, I do not know if there are any side effects.

Note 2: This procedure does not clean up your complete snapshot history. It only affects the latest version of your backup files. If you restore a file from far in the past the permissions will most probably be wrong.

[colinl]

@jns I can confirm that procedure works for me on Ubuntu, thanks. I had to first install zsh to run it (which is no big deal), it would be nice if someone more knowledgeable that myself could do an sh or bash version in order to avoid this necessity. I can suggest a slight enhancement which avoids manually unpacking the file. The procedure is: Ensure bzip2 is installed (sudo apt-get install bzip2) Ensure zsh is installed (sudo apt-get install zsh) Run zsh from a terminal then

cd path/to/backups/last_snapshot
cat fileinfo.bz2 | bzip2 -d | while read line; do perm=`echo "${line}" | cut -d" " -f1`; path_=`echo ${line} | cut -d" " -f4-`; perm=$(([##8]perm)); perm="${perm:$((${#perm}-3)):3}"; echo chmod "$perm" "backup$path_"; done

As with @jns's version that will just echo all the commands it will perform, once happy that looks correct then repeat the command but without the last echo so

cat fileinfo.bz2 | bzip2 -d | while read line; do perm=`echo "${line}" | cut -d" " -f1`; path_=`echo ${line} | cut -d" " -f4-`; perm=$(([##8]perm)); perm="${perm:$((${#perm}-3)):3}"; chmod "$perm" "backup$path_"; done

You can do that by hitting up arrow then left arrow to the echo text and delete the word echo.

[ghost]

Thanks @jns- and @colinl for sharing this! I'm sure I'll give it a try too (as soon as I can get these pesky little humans squatting my box to play SuperTuxKart to go away).

[Germar]

Thanks @jns- and @colinl for the fix. I'm planing to implement this to run automatically. But as I'm on vacation for the next weeks this will take some time...

If you restore a file from far in the past the permissions will most probably be wrong.

BiT will still restore permissions from fileinfo.bz2 after restore. So permissions will be correct no matter how old the snapshot was.

[wiregrasscoder]

Building on the work of @jns- and @colinl, I added user and group. For my last 1.1.4 backup, this was necessary to prevent another full backup.

cat fileinfo.bz2 | bzip2 -d | while read line; do perm=`echo "${line}" | cut -d" " -f1`; path_=`echo ${line} | cut -d" " -f4-`; user_=`echo "${line}" | cut -d" " -f2`; group_=`echo "${line}" | cut -d" " -f3`;  perm=$(([##8]perm)); perm="${perm:$((${#perm}-3)):3}"; chmod "$perm" "backup$path_"; chown "$user_":"$group_" "backup$path_"; done

[wiregrasscoder]

Upon further review, something is wonky beyond just correcting the permissions, owners, and groups in the last good 1.1.4 snapshot. I've gone through by hand and confirmed that the permissions, owners, and groups in my source files are identical to those files in last_snapshot, but nonetheless, some files continue to be backed up in full (files that haven't changed in years). Running the above command did prevent many of the files from being backed up in full again, but it was not a panacea.

[colinl]

@wiregrasscoder Do the problematic files have unusual permissions in the source? I am seeing it with r--r--r--. If so perhaps you are falling over issue #994 I would also be interested in knowing whether anyone else is seeing issue #993. I find that if a delete a file and make a snapshot that it does not delete the file from the snapshot.

[wiregrasscoder]

@colinl I have not seen the r--r--r-- issue. The permissions on the files in my scenario do have setuid and setgid applied. Also I'm also seeing that after a certain point during the snapshot, the snapshot size begins to grow very quickly, and log entries stop, so I'm not sure what's happening.

[Cyber1000]

Hi, same problem here: updated from 1.1.24 to 1.2.0. My config file contains "profile1.snapshots.preserve_acl=false" as stated in example-config https://github.com/bit-team/backintime/blob/master/common/config-example-local Is this still used with 1.2.0?

In https://github.com/bit-team/backintime/blob/master/common/tools.py I found a no_perms flag, how can I use this, is this something I could write in my config? profile1.snapshots.no_perms=false (not tested, just a first guess)?

Would the behaviour of 1.2.0 with this config-entry be the same as with pre 1.2.0 (without groups, users, perms).

It would be fine for me to have permissions not synced.

Thanks!

[Cyber1000]

Ok seems that snapshots.py does this within takeSnapshot:

rsync_prefix = tools.rsyncPrefix(self.config, no_perms = False)

no_perms = False, so I think no chance to keep the old behaviour.

[Cyber1000]

Ok I found a solution to have the same rights like in 1.1.24, I used this config:

profile1.snapshots.rsync_options.enabled=true profile1.snapshots.rsync_options.value=--no-perms --no-group --no-owner

This results in an rsync like this:

rsync --recursive --times --devices --specials --hard-links --human-readable --links --perms --executability --group --owner --info=progress2 --no-inc-recursive --no-perms --no-group --no-owner --delete --delete-excluded -v -i --out-format=BACKINTIME: %i %n%L --link-dest=../../20190608-223022-818/backup --chmod=Du+wx

Ok lets say it's not a perfect solution and more like a hack, cause I have --perms --group --owner and also --no-perms --no-group --no-owner as parameters, but the latter seem to overrule the others. So I have the same behaviour as in 1.1.24, as far as I use this in a docker-container and only update it occasionally, this is a controllable problem.

But I think it would be better if there were a configurable property, if you want perms/users/groups or not.

[colinl]

@Germar can you confirm whether the change you suggested earlier to prevent a full backup occurring the first time after upgrade was implemented

[agmartin2]

@colinl I have not seen the r--r--r-- issue. The permissions on the files in my scenario do have setuid and setgid applied. Also I'm also seeing that after a certain point during the snapshot, the snapshot size begins to grow very quickly, and log entries stop, so I'm not sure what's happening.

Note that last 4 digits of octal representation of Bit permission string are needed, not just last 3. No idea about the second part of your problem.

I have been playing with a perl script implementing all the above. It is only minimally tested, so use with a lot of care, preferably in a sandbox first. It is supposed to be invoked from backup root (e.g., last_snapshot/backup here) having fileinfo.bz2 one dir above. It should not do anything unless --real-run option is passed, otherwise should just show some info about what is to be done.

Remember, I still did not test it in a real big backup!!.

fix-bit-permissions.zip

[agmartin2]

Finally changed the config file as proposed by Cyber1000 to keep 1.1 behavior. Had no time to handle possible problems in a real large backup, so I did not go that way yet.

In parallel I was working with the perl script, including more options and some fixes like handling filenames with whitespaces (fix-BiT-permissions --help should show available options and info). It is still tested only in a minimal backup. Current version of the perl script attached as

fix-BiT-permissions.txt

Would like to play more with it once I find time.

[geertdobbels]

Hi,

I am on Fedora31, BiT 1.2.0., and the receiving end of my backups is a remote disk mounted on my system with cifs. In the past (pre-1.2.0) this was not a problem but now I have the same problem as mentioned above: all files get a "cf...p......." because BiT wants to set the permissions, but this does not work on a mount. Result: backups take forever, remote disk gets full in no time... which basically renders BiT useless.

I tried Cyber1000's solution of adding "--no-perms --no-group --no-owner" but it does not make any difference, I still get "cf...p......." on all of my files.

This is how BiT configures rsync on my current configuration (I deleted most includes and excludes to shorten the command):

rsync --recursive --times --devices --specials --hard-links --human-readable --copy-links --perms --executability --group --owner --info=progress2 --no-inc-recursive --bwlimit=3000 --delete --delete-excluded -v -i --out-format=BACKINTIME: %i %n%L --link-dest=../../20200206-150002-371/backup --chmod=Du+wx --include=/home/ --exclude=.gvfs --exclude=.cache* /backup/backintime/desktop-zub/grt/work/new_snapshot/backup

Any suggestions on how to fix this ?
The perl script on fileinfo.bz is of no use in this case since the cifs mount does not let me change permissions at all. I would like to avoid having to go back to a previous version

[Cyber1000]

@geertdobbels You get this even with "--no-perms --no-group --no-owner"? I'm using bit with docker and a ubuntu/debian like docker-image so things may differ. Btw I'm using bit 1.2.1 if that matters (but I don't think so)

Would you mind posting your bit-config (with the no-perms stuff in, perhaps there is an error/typo whatever)? You may anonymize (not delete) sensitive data, but I don't think there is any.

[geertdobbels]

Cyber1000, Thanks for your reply. I confirm that even with "--no-perms --no-group --no-owner" the problem persists. I made the changes in the config file (see attachment) as you suggested. Once the config file is modified, one can see the additional options in BiT in Settings/Expert Options on the last line (Paste additional options)

configcopy.txt

[Cyber1000]

That looks fine to me.

You may try to exchange common/tools.py with this file: https://gist.github.com/Cyber1000/b81eea9890b259fa6cab3454d08ad315 The only change I did (from line 599 onwards) was to comment the lines out which set the perms-flags and let only the no-perms in.

❗️ That's not the best solution I know, but it should help in short term. Just be aware that you would need to do that on (nearly) every git pull of BiT and that it may not work in future versions (cause the file may change massively)

If that doesn't work there must be something I missed in your profile file.

[geertdobbels]

Cyber1000, this looks promising. I created a small new backup profile, created a backup and immediately after that, took a new snapshot. It quickly said "Nothing changed, no new snapshot necessary", so it looks good. I did this test of creating a new small backup profile before the change, and it would give me a "cf...p......." on all files. Right now, I started a bigger backup (over 100G) and will check if it still works on that one. I will comment on the results.

[agmartin2]

Hi,

I am on Fedora31, BiT 1.2.0., and the receiving end of my backups is a remote disk mounted on my system with cifs. In the past (pre-1.2.0) this was not a problem but now I have the same problem as mentioned above: all files get a "cf...p......." because BiT wants to set the permissions, but this does not work on a mount. Result: backups take forever, remote disk gets full in no time... which basically renders BiT useless.

I tried Cyber1000's solution of adding "--no-perms --no-group --no-owner" but it does not make any difference, I still get "cf...p......." on all of my files.

This is how BiT configures rsync on my current configuration (I deleted most includes and excludes to shorten the command):

rsync --recursive --times --devices --specials --hard-links --human-readable --copy-links --perms --executability --group --owner --info=progress2 --no-inc-recursive --bwlimit=3000 --delete --delete-excluded -v -i --out-format=BACKINTIME: %i %n%L --link-dest=../../20200206-150002-371/backup --chmod=Du+wx --include=/home/ --exclude=.gvfs --exclude=.cache* /backup/backintime/desktop-zub/grt/work/new_snapshot/backup

Hi,

Is this what you get after adding the "--no-perms --no-group --no-owner" stuff? It seems not there. Was the "--no-perms --no-group --no-owner" string in the actual rsync call?

[Cyber1000]

@agmartin2 good point I was wondering for the same reasons, like I said back in June 2019 (https://github.com/bit-team/backintime/issues/988#issuecomment-504665090), I had --perms and --no-perms (and the others) together in one command, where --no-perms seems to win ... Perhaps something is wrong in the config, though it looked valid to me.

[geertdobbels]

@agmartin you are right ! I must have copied this line from the wrong profile, but anyway, I doublechecked it right now... with the original "tools.py" and the config files modified to include --no-perms etc... it does not work. The screenshot is from a small backup profile with a couple of files in it. I made a backup from scratch and right after that, whithout touching the files, took another snapshot. As you can see in the image, --no-perms.... is there and still I get all files with "cf...p....." Immediately after that, I replaced the tools.py file again with the one provided by Cyber1000 and after starting a snapshot it said "Done, no backup needed".

So definitely, in my configuration combining --perms and --no-perms does not work, or it seems that in my case --perms wins from --no-perms..... @Cyber1000 I checked your solution with a big backup and it worked fine: only really changed files were considered. Thanks. This solution is OK for me right now, I do use BiT on a daily basis for my work but I usually do not upgrade that often, so it should be OK.

[agmartin2]

@agmartin2 good point I was wondering for the same reasons, like I said back in June 2019 (#988 (comment)), I had --perms and --no-perms (and the others) together in one command, where --no-perms seems to win ...

AFAIK last occurrence should win, thus in "rsync ... --perms --no-perms" --no-perms should win. See --no-OPTION in rsync manual.

[direc85]

I just ran into this issue with up-to-date Manjaro Cinnamon as backup source and FreeNAS 11.3-U1 as the backup destination. Adding "--no-perms --no-group --no-owner" to rsync parameters made the backup proceed with expected speed. Just using "--no-perms" wasn't enough, apparently.

(Another secondary issue was that my notifications applet was overflowed with "operation not permitted" messages, but perhaps "backintime-qt --debug" made them appear; that doesn't really matter actually.)

[ghost]

If you restore a file from far in the past the permissions will most probably be wrong.

BiT will still restore permissions from fileinfo.bz2 after restore. So permissions will be correct no matter how old the snapshot was.

@Germar Please don't remove this functionality in future versions. I agree with Cyber1000 that the decision if rsync considers permissions / users / groups should be configurable be the user.

[b3nmore]

At a quick glance it seems that the code already contains the possibility to use old style permission handling (i.e. running rsync with no-perms -no-group -no-owner). However it is hard-coded never to use it.

In the attached archive (permission_handling.zip) you'll find a modified version, which allows to choose between old and new permission handling via a checkbox in the expert settings tab (it just makes that part usable by setting an option). If you like to test it just replace the corresponding files of your backintime installation. Please use them with caution and only if you are sure what you are doing (although it should be fairly safe to do so).

[Cyber1000]

@b3nmore you are right I found the hard-coded part too some time ago, but didn't have time to make a PR. I would say no one would complain if you make a PR 😄 , if you have the time. I'd look through the PR though @Germar would have the last word.

Still no real time on myself to look through the code, at least you should have both options for users who want the new behaviour.

[b3nmore]

@b3nmore you are right I found the hard-coded part too some time ago, but didn't have time to make a PR. I would say no one would complain if you make a PR smile , if you have the time. I'd look through the PR though @Germar would have the last word.

Still no real time on myself to look through the code, at least you should have both options for users who want the new behaviour.

Here you go #1086. Please note that at least cosmetic enhancement are most likely necessary...

[Cyber1000]

@b3nmore I'll have a look at it this weekend ...

[ghost]

I implemented the PR #1086 and dpkg-repack'ed my packages. You find them here. I've been testing the changes today and everything seems to work fine. Thanks to @b3nmore for the patch!

988 #1093

Originally posted by @SirHackselot in https://github.com/bit-team/backintime/issues/994#issuecomment-753548928