--pattern having different outcome in crontab

[BlackAsLight]

Have you checked borgbackup docs, FAQ, and open GitHub issues?

Yes

Is this a BUG / ISSUE report or a QUESTION?

bug/issue

System information. For client/server mode post info for both machines.

Your borg version (borg -V).

borg 1.2.8

Operating system (distribution) and version.

MacOS v14.5

Hardware / network configuration, and filesystems used.

M1 Chip, backing up to remote raspberry pi via ssh

How much data is handled by borg?

~ 30-40 GBs

Full borg commandline that lead to the problem (leave away excludes and passwords)

/opt/homebrew/bin/borg create \
    -psvC auto,zstd,22 \
    --list \
    --filter=-AMU \
    --dry-run \
    --exclude-caches \
    --exclude-nodump \
    -e '**/.*' \
    -e ~/BorgBackup \
    -e ~/Movies \
    -e ~/Music \
    -e ~/Pictures \
    -e '**/target' \
    --pattern=+/Users/soul/Library/Mobile\ Documents/com\~apple\~CloudDocs \
    --pattern=+/Users/soul/Library/CloudStorage \
    --pattern=-/Users/soul/Library \
    "Borg:/mnt/Archive/SilverMini::SilverMini-$datetime.zst" \
    ~ \
    >> $logfile 2>&1

Describe the problem you're observing.

When I run this script manually in the terminal, it excludes everything in \~/Library, except for my iCloud Drive (\~/Library/Mobile Documents/com\~apple\~CloudDocs), DropBox, and One Drive (\~/Library/CloudStorage), but when I run it via a cronjob, it doesn't respect these patterns and includes everything inside \~/Library.

Can you reproduce the problem? If so, describe how. If not, describe troubleshooting steps you took before opening the issue.

I can consistently reproduce the problem by retrying to run the script via a crontab.

Include any warning/errors/backtraces from the system logs

Ignoring --stats. It is not supported when using --dry-run.
Creating archive at "Borg:/mnt/Archive/SilverMini::SilverMini-2024-05-31T10-08.zst"
/Users/soul/Library/Application Support/CallHistoryDB: dir_open: [Errno 1] Operation not permitted: 'CallHistoryDB'
/Users/soul/Library/Application Support/Knowledge: dir_open: [Errno 1] Operation not permitted: 'Knowledge'
/Users/soul/Library/Application Support/CallHistoryTransactions: dir_open: [Errno 1] Operation not permitted: 'CallHistoryTransactions'
/Users/soul/Library/Application Support/com.apple.TCC: dir_open: [Errno 1] Operation not permitted: 'com.apple.TCC'
/Users/soul/Library/Application Support/AddressBook: dir_open: [Errno 1] Operation not permitted: 'AddressBook'
/Users/soul/Library/Application Support/com.apple.sharedfilelist: dir_open: [Errno 1] Operation not permitted: 'com.apple.sharedfilelist'
/Users/soul/Library/Application Support/CloudDocs/session/db: dir_open: [Errno 1] Operation not permitted: 'db'
/Users/soul/Library/Application Support/FileProvider: dir_open: [Errno 1] Operation not permitted: 'FileProvider'
Keyboard interrupt

[ThomasWaldmann]

Guess that is not a borg problem, but caused by a different environment, like e.g.:

• different shell causing different behaviour relating to tilde expansion, quoting and escaping?
• different environment variables? like e.g. $HOME.

borg does not check whether it is run from cron or not, so if it is invoked with the same cli args and the same env vars, it would behave the same.

[BlackAsLight]

Within both contexts, the --exclude arguments work as expected, but these pattern ones seem to be ignored in the cronjob context.

I was unable to figure out how to use the tilde with the --pattern and I find the documentation around --pattern to be quite confusing, making me have to guess and go through trial and error to know if I have the syntax correct.

This command is ran in a shell script in both manual and cronjob contexts with the bash environment set. And while I know cronjob still has less variables set, I don't see why it would be ignoring that last pattern when it would respect the --exclude equivalent. -e /Users/soul/Library.

[ThomasWaldmann]

IIRC, borg does nothing with the tilde.

Usually, tilde expansion is the task of the shell.

[BlackAsLight]

Even a pattern like this --pattern="-**/Library" seems to work when I manually do it, but doesn't when tried via crontab.

[BlackAsLight]

I figured out what the problem was. --pattern="-**/Library" was giving the desired result, as so was --pattern=-/Users/soul/Library. I was just mistaking errors from it trying to access areas it can't in a crontab due to lack of permissions, with it trying to backup those areas.

I settled on using --pattern="-$HOME/Library" as the tilde wasn't working with a --pattern.

Edit: I wish there was a way to include directories without having it go recursively in other sub directories that it should know aren't needed.

[BlackAsLight]

I've figured out that this argument -e 're:^Users/soul/Library/(?!Mobile Documents/com~apple~CloudDocs|CloudStorage).+$' works a lot better although a bit harder to read. It also seems to not go down these other subdirectories throwing all sorts of errors.