search

borgbackup / borg

Deduplicating archiver with compression and authenticated encryption.
https://www.borgbackup.org/
Other
11.15k stars 742 forks source link

Include patterns with excluded parent dir show entries in backup for all cache directories in parent dir #3991

Closed taladar closed 6 years ago

taladar commented 6 years ago

Have you checked borgbackup docs, FAQ, and open Github issues?

Yes

Is this a BUG / ISSUE report or a QUESTION?

BUG report

System information. For client/server mode post info for both machines.

Your borg version (borg -V).

Borg version 1.1.6 Linux64 on both machines

Operating system (distribution) and version.

Both Debian Jessie

Hardware / network configuration, and filesystems used.

/var/lib/docker is ext4

How much data is handled by borg?

Full borg commandline that lead to the problem (leave away excludes and passwords)

Excludes trimmed down (just regular --exclude to unrelated directories), destination and backup name anonymized, otherwise exactly as used

ionice -c 3 /etc/scripts/borg create -s --remote-path /etc/scripts/borg-linux64-1.1.6 --show-rc --exclude-caches --keep-exclude-tags \ --pattern=+/var/lib/docker/volumes \ --pattern=-/var/lib/docker \ user@backuphost:/backup_path::backupname /

Describe the problem you're observing.

Directories under /var/lib/docker/overlay2 that contain a CACHEDIR.TAG appear in the archive, though not the CACHEDIR.TAG files themselves (which would make sense due to --keep-exclude-tags), e.g.

drwxr-xr-x root root 0 Sun, 2018-07-08 01:33:05 var/lib/docker/overlay2/4716af2331e06a6a89d5de837cee369760e07ebc02579dbc3d6f6523c9904b0e/diff/var/cache/fontconfig drwxr-xr-x root root 0 Sun, 2018-03-18 00:23:42 var/lib/docker/overlay2/fcd658b1daddbd8d0787a1a17b08aa0ff00091da9201362cfd1f9354e5df512a/diff/var/cache/fontconfig drwxr-xr-x root root 0 Mon, 2017-07-10 11:13:21 var/lib/docker/overlay2/2891a096e4caa4ec612fb91598ef1193ed239472195dcd110fc9354566c310fa/diff/var/cache/fontconfig drwxr-xr-x root root 0 Sun, 2018-07-08 02:05:09 var/lib/docker/overlay2/5bf47430958e0ff9b68326b68314c3822713f4faa3b9c9fc6d0ea78d2c99290e/diff/var/cache/fontconfig drwxr-sr-x man root 0 Sun, 2018-07-08 02:06:50 var/lib/docker/overlay2/5bf47430958e0ff9b68326b68314c3822713f4faa3b9c9fc6d0ea78d2c99290e/diff/var/cache/man
drwxr-sr-x man root 0 Sun, 2018-07-08 02:09:24 var/lib/docker/overlay2/324e46dba0ccfb3fe1a7b1ed6218bc9f076bff4709e22f732044028d56093d8e/diff/var/cache/man/cs
drwxr-sr-x man root 0 Sun, 2018-07-08 02:09:24 var/lib/docker/overlay2/324e46dba0ccfb3fe1a7b1ed6218bc9f076bff4709e22f732044028d56093d8e/diff/var/cache/man/da drwxr-sr-x man root 0 Sun, 2018-07-08 02:09:24 var/lib/docker/overlay2/324e46dba0ccfb3fe1a7b1ed6218bc9f076bff4709e22f732044028d56093d8e/diff/var/cache/man/de
drwxr-sr-x man root 0 Sun, 2018-07-08 02:09:24 var/lib/docker/overlay2/324e46dba0ccfb3fe1a7b1ed6218bc9f076bff4709e22f732044028d56093d8e/diff/var/cache/man/es
drwxr-sr-x man root 0 Sun, 2018-07-08 02:09:24 var/lib/docker/overlay2/324e46dba0ccfb3fe1a7b1ed6218bc9f076bff4709e22f732044028d56093d8e/diff/var/cache/man/fi
drwxr-sr-x man root 0 Sun, 2018-07-08 02:09:24 var/lib/docker/overlay2/324e46dba0ccfb3fe1a7b1ed6218bc9f076bff4709e22f732044028d56093d8e/diff/var/cache/man/fr
drwxr-sr-x man root 0 Sun, 2018-07-08 02:09:24 var/lib/docker/overlay2/324e46dba0ccfb3fe1a7b1ed6218bc9f076bff4709e22f732044028d56093d8e/diff/var/cache/man/hu
drwxr-sr-x man root 0 Sun, 2018-07-08 02:09:24 var/lib/docker/overlay2/324e46dba0ccfb3fe1a7b1ed6218bc9f076bff4709e22f732044028d56093d8e/diff/var/cache/man/id
drwxr-sr-x man root 0 Sun, 2018-07-08 02:09:24 var/lib/docker/overlay2/324e46dba0ccfb3fe1a7b1ed6218bc9f076bff4709e22f732044028d56093d8e/diff/var/cache/man/it

Can you reproduce the problem? If so, describe how. If not, describe troubleshooting steps you took before opening the issue.

The problem appears in all (2) archives since I started using the patterns for /var/lib/docker and /var/lib/docker/volumes

ThomasWaldmann commented 6 years ago

To reproduce, this is easier:

(borg-env) tw@tux:~/w/borg$ mkdir inp
(borg-env) tw@tux:~/w/borg$ mkdir inp/excl
(borg-env) tw@tux:~/w/borg$ mkdir inp/excl/incl
(borg-env) tw@tux:~/w/borg$ mkdir inp/incl
(borg-env) tw@tux:~/w/borg$ mkdir inp/excl/check
(borg-env) tw@tux:~/w/borg$ echo 'Signature: 8a477f597d28d172789f06886806bc55' > inp/excl/check/CACHEDIR.TAG

(borg-env) tw@tux:~/w/borg$ borg init -e none repo

(borg-env) tw@tux:~/w/borg$ borg create repo::arch inp --list --exclude-caches --keep-exclude-tags --pattern='+inp/excl/incl' --pattern='-inp/excl'
x inp/excl
d inp/excl/incl
x inp/excl/check
x inp/excl/check/CACHEDIR.TAG
x inp/excl/check
d inp/incl
d inp

(borg-env) tw@tux:~/w/borg$ borg list repo::arch
drwxrwxr-x tw     tw            0 Sat, 2018-08-04 19:25:49 inp
drwxrwxr-x tw     tw            0 Sat, 2018-08-04 19:25:34 inp/excl/incl
drwxrwxr-x tw     tw            0 Sat, 2018-08-04 19:30:08 inp/excl/check
drwxrwxr-x tw     tw            0 Sat, 2018-08-04 19:25:49 inp/incl

Notable: inp/excl/check is archived, inp/excl/check/CACHEDIR.TAG is not archived.

ThomasWaldmann commented 6 years ago

The question now is whether the bug is

taladar commented 6 years ago

Well, considering nothing else in that explicitly excluded tree is archived I would say the cache directories and their CACHEDIR.TAG files should not be archived either if they are under an explicitly excluded path.