macOS sleep causes Broken pipe error

deermo commented 4 years ago

Have you checked borgbackup docs, FAQ, and open Github issues?

Yes

Is this a BUG / ISSUE report or a QUESTION?

Issue.

System information. For client/server mode post info for both machines.

Your borg version (borg -V).

1.1.13 (using Vorta GUI)

Operating system (distribution) and version.

Client: macOS 10.15.6 Server: Hetzner Storage Box

Hardware / network configuration, and filesystems used.

Network: Wifi

How much data is handled by borg?

300-400 GB

Full borg commandline that lead to the problem (leave away excludes and passwords)

/usr/local/bin/borg create --list --info --log-json --json --filter=AM -C zstd,8 --exclude-from ... --exclude-if-present .nobackup ssh://PATH /PATH

Describe the problem you're observing.

I observed that my backups to a network share were frequently interrupted. The errors from the log file were always

- vorta.borg.borg_thread - WARNING - Remote: client_loop: send disconnect: Broken pipe
- vorta.borg.borg_thread - ERROR - Connection closed by remote host

I thus digged into these issues, and my client SSH config now reads

Host XYZ
    ServerAliveInterval 30
    ServerAliveCountMax 240
    IPQoS=throughput

On the server, I've also set the ClientAliveInterval parameter. (Thanks to this, BorgBase/Vorta FAQ and #636. The IPQoS option is not mentioned already and could help too?)

Altough I felt this improved things, the issue continued. I could finally reproduce the same error message when I forced my MacBook to sleep early and had the "Prevent computer from sleeping automatically when the display is off" (orange rectangle) unchecked (thus NOT preventing the computer from sleeping, i.e. keeping the computer awake although display is off). grafik

Using a tool like Amphetamine to force macOS to "stay awake" also resolved the issue.

Can you reproduce the problem? If so, describe how. If not, describe troubleshooting steps you took before opening the issue.

Set Display Sleep to 1 Minute, uncheck the option highlighted in the graphic ("Prevent computer from sleeping automatically when the display is off")
Start a borg backup
Wait (it takes a few more minutes than the sleep interval)

Discussion

This issue raised a few aspects for me (somehow related, somehow seperate), which I want to share:

Although this behaviour does not seem to be technically a borg bug, I had a hard time tracing the bug. This is partially caused by the error message, so I wonder if it is possible to change the reporting, so that the user notices that the macOS sleep is the issue and not the SSH session (on macOS, it is sometimes hard to tell if the display is just turned off or if the computer is sleeping)
It could be an option to stop the mac from sleeping while backing up (alternatively, this could be a Vorta thing?). The caffeinate command could be an easy option here.
I am unsure if these SSH options help. The IPQoS=throughput is not mentioned in #636 and could help?
It might help other users to update the Borg and Vorta FAQ at and include the hint that the macOS hibernation might be the cause of a "SSH error" message

ThomasWaldmann commented 4 years ago

Guess you must not let your machine sleep while trying to do a backup. How to avoid that depends a lot on your OS and is out of scope of borg.

Even the network connection is out of scope of borg, because the borg client just creates an ssh client process and communicates over pipes with the ssh process (thus you get the error message "broken pipe" if your ssh process terminated).

So, if borg tells you "broken pipe", this is as precise as it gets. The ssh process maybecould know more details about this, but not borg.

melissaboiko commented 4 years ago

It’s very much true that this is not borg’s fault but OS X’s. Still, it is a hard problem to trace, with a lot of false positive results from the various SSH configuration issues. In my opinion it would be nice if Borg documentation or manuals mentioned in passing that OS X, on its default configuration, will break in this way, and that caffeinate can solve it.

ThomasWaldmann commented 4 years ago

Guess we could have a short hint in the FAQ, PRs welcome.

As this is the borg documentation and not the documentation of the OS or ssh or networking in general, it has to be just a hint.

Gu1nness commented 3 years ago

Taking this one as well

borgbackup / borg