Add to FAQ: How to use extended filesystem attributes (xattr) to exclude files/directories

[dlippold]

Currently one can exclude directories (and files) only by their name and path (defined by a pattern). It would be nice if it would be possible to exclude a directory also by a user defined extended attribute, e.g. by setfattr -n user.backup-exclude testdir or alternatively by setfattr -n user.backup -v no testdir.

That would make it possible that users exclude directories (and maybe files) from the backup without the need to modify the configuration of BiT.

Btw: You can list all files below directory topdir which have the extended attribute user.backup-exclude set by the following command (in bash): getfattr -n user.backup-exclude -R topdir 2> /dev/null | fgrep '# file:' | cut -d ':' -f 2

[buhtz]

I am not aware of setfattr and getfattr. Can you please describe your use case more generaly? For which cases is this usefull?

On the current (low) level of my understand something like this depend on the abilities of rsync (and it's --exclude option) which is used for all snapshots. Do you have informations about that rsync cat take extended attributes into account?

[dlippold]

I have installed BiT on the RHEL/CentOS servers of our small working group. For that I used the rpm package from Fedora. I configured BiT to backup the home directories of all users every night.

Sometimes a user want to create a temporary directory with very large files. That directory should not be part of the backup (because it would take unnecessarily much space on the backup disks). Currently that user has to write a mail to me with the path of that directory and I can add that directory to the list of excluded directories in BiT. If I don't read the mail before the next nightly backup the directory will be in the backup nevertheless.

It would be much easier if the user could exclude the directory from the backup by himself. For that he would only have to execute the command setfattr -n user.backup-exclude tmpdir (if the name of the directory is tmpdir).

Even on single user systems (desktop, workstation) it would be more convenient from my point of view to execute the command above instead of starting BiT and add the directory to the list of excluded directories.

To make that possible in BiT an option has to be added to BiT by which one can define the name of a user extended attribute. Then BiT can do the following (let us say user.backup-exclude is the specified name and topdir is the directoy for the backup):

• ExcludedFiles=$(mktemp)

• getfattr -n user.backup-exclude -R topdir 2> /dev/null | fgrep '# file:' | cut -d ':' -f 2 > $ExcludedFiles

or for full paths:

• getfattr -n user.backup-exclude -R XaTest 2> /dev/null | fgrep '# file:' | cut -d ':' -f 2 | xargs readlink -e > $ExcludedFiles

• add --exclude-from=$ExcludedFiles to the parameters of rsync

• rm $ExcludedFiles

If the user would want to add the directory after all to the backup, he could execute the command setfattr -x user.backup-exclude tmpdir

[buhtz]

I see your point. Thank you for your detailed description. Nice feature. @Germar should add this to the wishlist and keep this Issue open.

But keep in mind that currently there are absolute no ressources to implement things like that. I am sorry.

[Germar]

You can easily do this with a user-callback

#!/bin/bash

profile_id="$1"
profile_name="$2"
reason="$3"
ExcludedFiles=~/.local/share/backintime/cusom_excludes

case $reason in
    1) #Backup process begins
        getfattr -n user.backup-exclude -R XaTest 2> /dev/null | fgrep '# file:' | cut -d ':' -f 2 | xargs readlink -e > $ExcludedFiles
        ;;
    2) #Backup process ends
        rm $ExcludedFiles
        ;;
esac

and add --exclude-from="~/.local/share/backintime/cusom_excludes" in Paste additional options to rsync under Expert Options

[buhtz]

@dlippold Did this fixed your problem? Can we close the Issue?

[dlippold]

Thanks for the tip about user-callback. But I found that there are problems with the file/directory names.

A minor problem is that rsync uses the entries in the file of the parameter --exclude-from as patterns by default. That can be changed by adding a slash to the beginning of a file/directory. Then the entry is regarded as a file/directory relative to the current directory. For example if rsync is executed in /home and there is an entry /user/test.txt in the file exclude.txt then the parameter --exclude-from=exclude.txt excludes the file /home/user/test.txt (i.e. the leading slash is removed from the entry and the rest is regared as a relative file name).

Therefore I changed the script to the following form (and saved it as /root/.config/backintime/user-callback).

#!/bin/bash

profile_id="$1"
profile_name="$2"
reason="$3"
UserExcludeAttrib="user.nobackup"
Excluded="/root/.local/share/backintime/user_excludes.txt"

case $reason in
    1) #Backup process begins
        getfattr -n $UserExcludeAttrib -R . 2> /dev/null | fgrep '# file:' | sed 's/.*:[ ]*/\//' > $Excluded
        ;;
    2) #Backup process ends
        rm $Excluded
        ;;
esac

The major problem is that the script is executed by BiT (as far as I can see) in the directory from which BiT was started and it is executed only once. It seems to be necessary that BiT changes to the directory which has to be backuped and execute the above script from there (for every directory which has to be backuped).

[Germar]

rsync will always sync the root of your filesystem /. So you can run your script in /, too. Just add cd / before getfattr

[dlippold]

Thanks for the hint. After adding the command cd / to the script and after adding --exclude-from="/root/.local/share/backintime/user_excludes.txt" to the additonal rsync options it seems to work perfectly.

I think the issue can be closed.

Maybe you can add the script as an example for a user-callback script (and to explain how to exclude files/directories by user attributes from the backup) to the manual.

[dlippold]

Because the additional rsync option --exclude-from="/root/.local/share/backintime/user_excludes.txt" is also added to the restore command the file /root/.local/share/backintime/user_excludes.txt must exist when executing a restore. Therefore there must be a line touch $Excluded after the line rm $Excluded in the callback script.

Here is my current callback script /root/.config/backintime/user-callback:

#!/bin/bash

profile_id="$1"
profile_name="$2"
reason="$3"
UserExcludeAttrib="user.nobackup"
Excluded="/root/.local/share/backintime/user_excludes.txt"

case $reason in
    1) # Backup process begins
       /usr/local/bin/list-nobackup / | /usr/bin/cut -c 2- > $Excluded
       ;;
    2) # Backup process ends
       /bin/rm $Excluded
       /usr/bin/touch $Excluded
       ;;
esac

Therein the script /usr/local/bin/list-nobackup is used which is:

#!/bin/bash

# filename: list-nobackup
# author  : Dietmar Lippold
# version : 20.01.2018

# Lists recursively all files an directories which are excluded from the
# backup (for the case that the file or directory is included in the
# configuration of the # backup) for a given directory. If only a filename
# is given, the filename is output if it is excluded.

# Usage:
# list-nobackup file_or_directory

AttribName="user.nobackup"

/usr/bin/getfattr --absolute-names -hR -m $AttribName $1 | /bin/grep '# file:' | /bin/sed 's/.*:[ ]*//' | sed 's/\/\//\//'

Furthermore I have a script /usr/local/bin/mark-nobackup:

#!/bin/bash

# filename: mark-nobackup
# author  : Dietmar Lippold
# version : 20.01.2018

# Marks a file or directory so that it should not be in the backup (for the
# case that the file or directory is included in the configuration of the
# backup). The marking can be removed by the script "mark-backup".

# Usage:
# mark-nobackup file_or_directory

AttribName="user.nobackup"

/usr/bin/setfattr -n $AttribName $1

And a script /usr/local/bin/mark-backup:

#!/bin/bash

# filename: mark-backup
# author  : Dietmar Lippold
# version : 20.01.2018

# Marks a file or directory so that it should be in the backup (for the case
# that the file or directory is included in the configuration of the backup).
# Because that is the default for every file and directory, this script only
# needs to be executed after the script mark-nobackup was executed for that
# file or directory.

# Usage:
# mark-backup file_or_directory

AttribName="user.nobackup"

/usr/bin/setfattr -x $AttribName $1

[emtiu]

Thanks @dlippold for your work on the user-callback script! It's great to have it documented here.

I still think this would be a very convenient feature for backintime, so let's keep this open as a worthwhile feature request :)

[buhtz]

I would add this to the FAQ and link this Issue there. I see no resources in years to get something like this done.