Borg 1.4.0 peeks into excluded directories

dukelsky commented 2 months ago

Have you checked borgbackup docs, FAQ, and open GitHub issues?

Yes

Is this a BUG / ISSUE report or a QUESTION?

ISSUE

System information. For client/server mode post info for both machines.

borg 1.4.0 at both the server and the client. I tried both self built rpm package and borg-linux-glibc231 from Releases on the client machine.

Fedora Linux KDE 40 (client), AlmaLinux 9.4 (server)

Client: ASUS PRIME B550-PLUS motherboard, AMD Ryzen 7 5700G, 64 GB RAM.

How much data is handled by borg?

~600 GB

Full borg commandline that lead to the problem

/usr/bin/borg --iec --show-rc create --list --filter=- --dry-run --compression zstd --patterns-from=$PATTERN_FILE borg@nas.example.org:/path/to/repo::rootfs-{now:%Y-%m-%d} / >> $LOG 2>&1

cat $PATTERN_FILE
-/dev/
-/home/*/.cache/
-/media/
-/mnt/
-/proc/
-/root/.cache/
-/run/
-/sys/
-/tmp/
-/var/cache/
-/var/crash/
-/var/lib/libvirt/images/
-/var/lib/mock/
-/var/lib/sddm/.cache/
-/var/log/
-/var/spool/
-/**/tmp/

Describe the problem you're observing.

Most excluded directories are not backed up, but I find the following in the log:

...
/proc/6915/task/6915/fd/10: stat: [Errno 2] No such file or directory: '10'
/proc/6915/task/6915/fdinfo/10: stat: [Errno 2] No such file or directory: '10'
/proc/6915/fd/7: stat: [Errno 2] No such file or directory: '7'
/proc/6915/fdinfo/7: stat: [Errno 2] No such file or directory: '7'
- /proc
- /var/log
- /var/lock
...
- /media
/run/user/1000/doc: stat: [Errno 13] Permission denied: 'doc'
- /run

Can you reproduce the problem?

The problem is reproducible, and it does not matter whether --dry-run is used or not. I did not try the previous versions, since this is the first time I used borgbackup.

ThomasWaldmann commented 2 months ago

I suspect your patterns don't match like you think they do. See the patterns help page and use these commands to see what's excluded and what not:

borg list REPO::archive <- check how the paths look like: no leading slash!
borg create --dry-run --list ... <- use this to test your adapted patterns

dukelsky commented 2 months ago

I removed the leading slash in all patterns, but the result is the same: borg peeks into /proc and /run.

dukelsky commented 2 months ago

I removed --filter=- from the command line, and now I see that borg lists everything in all "excluded" directories including /sys, /proc, /run etc.

x /proc/1872/task/1939/fdinfo/251
x /proc/1872/task/1939/fdinfo/253
x /proc/1872/task/1939/fdinfo/254
x /proc/1872/task/1939/fdinfo/256
x /proc/1872/task/1939/fdinfo/258
x /proc/1872/task/1939/fdinfo/284
x /proc/1872/task/1939/fdinfo/255
/proc/1872/task/1939/fdinfo/255: stat: [Errno 2] No such file or directory: '255'

But why?

Aztorius commented 2 months ago

I removed --filter=- from the command line, and now I see that borg lists everything in all "excluded" directories including /sys, /proc, /run etc.
x /proc/1872/task/1939/fdinfo/251
x /proc/1872/task/1939/fdinfo/253
x /proc/1872/task/1939/fdinfo/254
x /proc/1872/task/1939/fdinfo/256
x /proc/1872/task/1939/fdinfo/258
x /proc/1872/task/1939/fdinfo/284
x /proc/1872/task/1939/fdinfo/255
/proc/1872/task/1939/fdinfo/255: stat: [Errno 2] No such file or directory: '255'
But why?

Borg lists everything because an include rule may match on a subdirectory or file.

I believe what you are looking for is the exclude no-recurse pattern prefix '!' instead of the basic exclude '-'. See the doc : https://borgbackup.readthedocs.io/en/stable/usage/help.html

From the doc :

Use the prefix !, followed by a pattern, to define an exclusion that does not recurse into subdirectories. This saves time, but prevents include patterns to match any files in subdirectories.

dukelsky commented 2 months ago

Thanks for the tip.

ThomasWaldmann commented 2 months ago

Just wanted to note that using ! instead of - is:

advisable for dirs like /proc and /sys on Linux where no includes below the excluded dir are expected
but it is not the root cause why your patterns file did not work. the root cause is likely that you did not match the paths of all files that you wanted to exclude.

dukelsky commented 2 months ago

I want to include the special directories /dev, /proc, /run, /sys, but not their contents. I tried this:

+ re:^(dev|proc|run|sys|tmp)$
! re:^(dev|proc|run|sys|tmp)

But despite my expectations, there were directories one level lower in the list.

x /sys/fs
x /sys/devices
x /sys/dev
x /sys/bus
x /sys/class
x /sys/firmware
x /sys/hypervisor
x /sys/kernel
x /sys/power
x /sys/module
x /sys/block
- /sys

Aztorius commented 2 months ago

If you want to include /sys but not its content, I think you have done the right thing. Borg needs to list the content to exclude it with the rules.

borgbackup / borg