OS X: support for new Catalina security mechanisms

greenmoss commented 4 years ago

Have you checked borgbackup docs, FAQ, and open Github issues?

Yes

Is this a BUG / ISSUE report or a QUESTION?

ISSUE

System information. For client/server mode post info for both machines.

Your borg version (borg -V).

1.1.13

Operating system (distribution) and version.

Mac OS X Catalina: 10.15.6

Hardware / network configuration, and filesystems used.

Apple MacBook Pro, backing up to Linux via SSH. FS is "APFS" encrypted.

How much data is handled by borg?

about 200 GB

Full borg commandline that lead to the problem (leave away excludes and passwords)

borg create --checkpoint-interval 300 --compression lz4 --exclude-caches --filter AME --list --show-rc --stats --verbose user@backupserver:/borg/myhost::{now} /opt /Users /usr/local

Describe the problem you're observing.

OS X Catalina does not allow us to assign "full-disk access" to bash scripts. Thus, the backup solution from the documentation fails to back up multiple important directories such as /Users/*/Documents, etc.

There are currently a few workarounds to address this, for example:

Write a Golang wrapper script which invokes the borg backup bash script (example)
Use Platypus to create an OS X Application bundle which invokes the borg backup bash script

This ticket is a request to help out OS X users. The quick fix might be to add this as a FAQ to the documentation, and mention how to get past it. I could write something up and attach it as a PR for the documentation. Would such a PR be welcome?

P.S. it would be even nicer to have an easier way for OS X users to approach this that bypasses Golang wrapper scripts. However the only solution I can think of in this case is a full-blown compiled/signed app of some kind.

Can you reproduce the problem? If so, describe how. If not, describe troubleshooting steps you took before opening the issue.

Yes, running borg from bash script always causes the problem. Running from a Golang wrapper resolves it.

Include any warning/errors/backtraces from the system logs

I'll have to dig around to get the exact error message. However, copying from here a typical error looks like

/users/xxx/Pictures/Photos Library.photoslibrary: scandir: [Errno 1] Operation not permitted: '/users/xxx/Pictures/Photos Library.photoslibrary’

ThomasWaldmann commented 4 years ago

How about filing this issue against macOS?

borg is doing nothing wrong here, it is just the OS behaving stupidly.

ThomasWaldmann commented 4 years ago

If someone wants to contribute a simple binary wrapper, I'm fine with that also.

Simple means adding no dependencies (thus: not in Go, but in C [which we already use within borg project]).

Hmm, I remember that when installing python cli apps on Windows, setuptools(?) creates a *.exe - maybe the same or a similar mechanism could be used on macOS?

greenmoss commented 4 years ago

Agreed; there were some choice words uttered under my breath when I understood the implications of this new OS X policy. I'll see if I can find anyone at Apple who can push back on their devs. Based on past behavior though, I doubt they'll change their mind about this.

The key is to make a script that the user can configure on their own using their desired options, then compile on their own. Having that "static" executable is required so that it can be granted whole-disk access via the OS X security control panel. It seems the script-file-invoking-Python, or script-file-invoking-Bash solution is specifically not accepted by the security mechanism/control panel.

I'm guessing average users would have trouble with a compiled C option. I know I would :D. I like the idea of a Python executable. I haven't messed with that before, but I'm certainly happy to dig into it further. Any pointers anyone could offer would be most welcome.

m3nu commented 4 years ago

Usually Borg would just inherit permissions from the Terminal or iTerm2 app when running it there. This would also validate related libraries the first time they are run. I'm not sure you can even white-list a single binary that's not an app.

As an alternative, have you looked at Vorta. It's a full macOS app that can be white-listed permission-wise and simply runs Borg internally.

greenmoss commented 4 years ago

@m3nu I used the Golang wrapper strategy and am running it from launchd. So far, so good.

I have not tried Vorta before. It looks great, and is certainly another possibility.

@ThomasWaldmann what if I made a documentation PR:

https://github.com/borgbackup/borg/blob/master/docs/installation.rst in the OS X documentation
https://github.com/borgbackup/borg/blob/master/docs/faq.rst with a new Why am I getting 'Operation not permitted' errors when backing up on OS X?

ThomasWaldmann commented 4 years ago

Similar to the sleep issue: this is a speciality of the OS, so out of scope of our docs.

As people might stumble over it, we can include a short hint though (e.g. a pointer to where more information can be found).

BTW "Operation not permitted" likely can happen due to a lot of reasons, so be careful.

starcraft66 commented 4 years ago

I'm pretty sure you can just give borg the full filesystem access permission somewhere in the osx settings to avert this problem.

cd2357 commented 4 years ago

What worked for me was to give cron the Full Disk Access permission as described here.

Would probably also work the same for Terminal, if called directly.

greenmoss commented 4 years ago

This is what I have so far:

cd to a temp directory

add python wrapper:

$ cat run-shell.py
#!/usr/bin/env python3
import subprocess, sys
process = subprocess.run(sys.argv[1:])
sys.exit(process.returncode)

brew install pyinstaller
pyinstaller --name run-shell --onefile --distpath . --clean run-shell.py

The results:

drwxr-xr-x 3 me staff   96 Sep  8 20:38 build
-rwxr-xr-x 1 me staff 6.7M Sep  8 20:39 run-shell
-rwxr-xr-x 1 me staff  114 Sep  8 20:30 run-shell.py
-rw-r--r-- 1 me staff  870 Sep  8 20:39 run-shell.spec

6.7 Megabytes, from a 114-byte wrapper file, so that Apple can feel warm and fuzzy.

Well, at least it works:

$ ./run-shell echo hello there
hello there
$ echo $?
0
$ ./run-shell ls nothing-here
ls: cannot access 'nothing-here': No such file or directory
$ echo $?
2

It takes a second or two to kick off, which is maybe OK:

$ time ./run-shell echo super fast
super fast

real    0m1.460s
user    0m0.111s
sys 0m0.169s

Note: intended usage would be something like /usr/local/bin/run-shell /path/to/your/borg/script.sh

m3nu commented 4 years ago

I don't think the intention by Apple was to have a wrapper that can run literally any command. Would suggest to just assign permissions to Python, the Borg binary, Terminal, Vorta or Cron. As mentioned in the post above.

In addition, this wrapper will only work on your machine, since it's not signed.

greenmoss commented 4 years ago

What worked for me was to give cron the Full Disk Access permission as described here.

Would probably also work the same for Terminal, if called directly.

Confirmed the cron whole-disk-access solution works. That is actually much simpler and easier. @ThomasWaldmann OK to just paste that link above into the docs somewhere for Mac OS users?

FWIW, the wrapper script does not work!

ThomasWaldmann commented 4 years ago

https://osxdaily.com/2020/04/27/fix-cron-permissions-macos-full-disk-access/

that one? if so, a PR would be fine with a few short phrases (what is needed) plus link (how to do that) as @cd2357 posted it.

greenmoss commented 4 years ago

https://osxdaily.com/2020/04/27/fix-cron-permissions-macos-full-disk-access/

that one? if so, a PR would be fine with a few short phrases (what is needed) plus link (how to do that) as @cd2357 posted it.

I opened #5508; does it look ok?

greenmoss commented 4 years ago

Note: also #5509 for maint-1.1 branch

uasi commented 2 years ago

I'm super late to the party, but for those who are still interested in an easy way to run borg from launchd without cron or third-party tools: you can just grant Full Disk Access to /usr/bin/env and use it as a wrapper.

<key>ProgramArguments</key>
<array>
  <string>env</string>
  <string>/path/to/borg</string>
  <string>create</string>
  ...  
</array>

rhaps0dy commented 1 year ago

A perhaps more secure way to do it without third-party tools is using Swift. The following program just runs a shell script:

import Cocoa

let task = Process()
task.launchPath = "/bin/sh"
task.arguments = ["/path/to/shell/script/borg-backup.sh"]
task.launch()
task.waitUntilExit()

compile it with swiftc file.swift and give the resulting binary full-disk permissions

bwesen commented 1 year ago

Is there an easier fix if you don't want to run it through cron but just invoke on the cmdline?

ThomasWaldmann commented 1 year ago

You could give the Terminal app full disk permissions.

borgbackup / borg