Very slow backup speed when creating archive from sshfs to external hdd

AntonOellerer commented 1 year ago

Have you checked borgbackup docs, FAQ, and open GitHub issues?

Yes

Is this a BUG / ISSUE report or a QUESTION?

QUESTION

System information. For client/server mode post info for both machines.

Your borg version (borg -V)

borg 1.2.3

Operating system (distribution) and version.

fedora 37

Hardware / network configuration, and filesystems used.

hetzner storage box, (xfs?)/mounted via sshfs as fuse.sshfs linux laptop (AMD Ryzen 7 4800H, 32 GB RAM), btrfs + luks external hdd (WDC WD20SDRW), xfs + luks

How much data is handled by borg?

300GB

Full borg commandline that lead to the problem (leave away excludes and passwords)

borg create -v --stats --progress --files-cache=ctime,size --timestamp=2023-02-20 /backup/borg/nextcloud_datbkp::2023-02-20 .

Describe the problem you're observing.

To backup the content of the remote storage box, I used rsync until last year to save the data on an external hdd semiregularly. Since I wanted to use incremential backups, I decided to switch to borgbackup, created a new repo on the hdd, made a first archive from the rsync backup (also with --files-cache=ctime,size) which took about 2 hours, and then started to back up the content of the storage box, which turned out to be very slow (~10GB per hour), even when just deduplication of existing files was done, and the network speed would allow for much faster download times.

Can you reproduce the problem? If so, describe how. If not, describe troubleshooting steps you took before opening the issue.

Yes, it always happens

Include any warning/errors/backtraces from the system logs

None

CRUD benchmark results

From the storage box to the hdd:

C-Z-BIG           4.06 MB/s (10 * 100.00 MB all-zero files: 246.47s)
R-Z-BIG         490.09 MB/s (10 * 100.00 MB all-zero files: 2.04s)
U-Z-BIG          37.79 MB/s (10 * 100.00 MB all-zero files: 26.46s)
D-Z-BIG        6327.17 MB/s (10 * 100.00 MB all-zero files: 0.16s)
C-R-BIG           3.89 MB/s (10 * 100.00 MB random files: 256.86s)
R-R-BIG         105.67 MB/s (10 * 100.00 MB random files: 9.46s)
U-R-BIG          36.07 MB/s (10 * 100.00 MB random files: 27.72s)
D-R-BIG         307.78 MB/s (10 * 100.00 MB random files: 3.25s)
C-Z-MEDIUM        2.12 MB/s (1000 * 1.00 MB all-zero files: 471.65s)
R-Z-MEDIUM      655.21 MB/s (1000 * 1.00 MB all-zero files: 1.53s)
U-Z-MEDIUM       14.95 MB/s (1000 * 1.00 MB all-zero files: 66.89s)
D-Z-MEDIUM     5369.50 MB/s (1000 * 1.00 MB all-zero files: 0.19s)
C-R-MEDIUM        1.76 MB/s (1000 * 1.00 MB random files: 567.90s)
R-R-MEDIUM      111.18 MB/s (1000 * 1.00 MB random files: 8.99s)
U-R-MEDIUM       15.75 MB/s (1000 * 1.00 MB random files: 63.50s)
D-R-MEDIUM      113.18 MB/s (1000 * 1.00 MB random files: 8.84s)
C-Z-SMALL         0.10 MB/s (10000 * 10.00 kB all-zero files: 976.49s)
R-Z-SMALL       188.45 MB/s (10000 * 10.00 kB all-zero files: 0.53s)
U-Z-SMALL         0.14 MB/s (10000 * 10.00 kB all-zero files: 727.17s)
D-Z-SMALL       395.59 MB/s (10000 * 10.00 kB all-zero files: 0.25s)
C-R-SMALL         0.09 MB/s (10000 * 10.00 kB random files: 1093.80s)
R-R-SMALL        81.39 MB/s (10000 * 10.00 kB random files: 1.23s)
U-R-SMALL         0.14 MB/s (10000 * 10.00 kB random files: 706.49s)
D-R-SMALL        80.20 MB/s (10000 * 10.00 kB random files: 1.25s)

I did an additional test from the laptop to the hdd:

C-Z-BIG         420.36 MB/s (10 * 100.00 MB all-zero files: 2.38s)
R-Z-BIG         453.48 MB/s (10 * 100.00 MB all-zero files: 2.21s)
U-Z-BIG        2637.48 MB/s (10 * 100.00 MB all-zero files: 0.38s)
D-Z-BIG        3449.62 MB/s (10 * 100.00 MB all-zero files: 0.29s)
C-R-BIG          76.27 MB/s (10 * 100.00 MB random files: 13.11s)
R-R-BIG         101.26 MB/s (10 * 100.00 MB random files: 9.88s)
U-R-BIG        2287.09 MB/s (10 * 100.00 MB random files: 0.44s)
D-R-BIG         318.11 MB/s (10 * 100.00 MB random files: 3.14s)
C-Z-MEDIUM      478.53 MB/s (1000 * 1.00 MB all-zero files: 2.09s)
R-Z-MEDIUM      653.02 MB/s (1000 * 1.00 MB all-zero files: 1.53s)
U-Z-MEDIUM     2893.54 MB/s (1000 * 1.00 MB all-zero files: 0.35s)
D-Z-MEDIUM     3734.19 MB/s (1000 * 1.00 MB all-zero files: 0.27s)
C-R-MEDIUM       69.29 MB/s (1000 * 1.00 MB random files: 14.43s)
R-R-MEDIUM      106.06 MB/s (1000 * 1.00 MB random files: 9.43s)
U-R-MEDIUM     2837.86 MB/s (1000 * 1.00 MB random files: 0.35s)
D-R-MEDIUM      107.29 MB/s (1000 * 1.00 MB random files: 9.32s)
C-Z-SMALL        32.45 MB/s (10000 * 10.00 kB all-zero files: 3.08s)
R-Z-SMALL       196.30 MB/s (10000 * 10.00 kB all-zero files: 0.51s)
U-Z-SMALL        90.66 MB/s (10000 * 10.00 kB all-zero files: 1.10s)
D-Z-SMALL       370.42 MB/s (10000 * 10.00 kB all-zero files: 0.27s)
C-R-SMALL        17.68 MB/s (10000 * 10.00 kB random files: 5.66s)
R-R-SMALL       102.73 MB/s (10000 * 10.00 kB random files: 0.97s)
U-R-SMALL        85.75 MB/s (10000 * 10.00 kB random files: 1.17s)
D-R-SMALL        93.16 MB/s (10000 * 10.00 kB random files: 1.07s)

I can provide additional benchmarks (like storage box -> laptop) if needed

infectormp commented 1 year ago

hetzner may use a rate limit, so you need to run the benchmark with local network where you do not have the rate limit.

AntonOellerer commented 1 year ago

No, hetzner does not use a rate limit, but I can try to test it with a local server when I have the opportunity

ThomasWaldmann commented 1 year ago

Not sure if we can do anything here, just too much variables / unknowns to point to any specific issue:

internet connection speed / latency
sshfs speed / issues
linux storing lots of data to USB devices sometimes can be also rather slow and even slow down the whole system
hetzner server speed / load
LUKS

enkore commented 1 year ago

AIUI you're mounting a remote internet server using sshfs and then creating a backup of the files in that sshfs mount to a local disk. This can perform somewhat okay if the average file size is big enough and the BDP is <<2 MB (because ssh), but sshfs makes no attempt at latency hiding and couldn't do much if it tried - so this is very dependent on the latency of the sshfs mount. This setup will always perform badly with many files.

RonnyPfannschmidt commented 1 year ago

is there anything preventing the installation/usage of borg on the remote server?

at first glance it seems hetzner storage boxes have borg 1.2 avaliable - so maybe something cna be done there to run brok directly rather than doing the remote filesystem dance (which is costly)

someone-somenet-org commented 2 months ago

Hello :)

Your benchmarks seem a little off:

HDDs usually dont exceed 200 MB/s.
More that 125 MB/s exceeds 1 GBit/s.

Especially the Laptop->local hdd benchmarks seem like you are just writing into your RAM and dont do fsync, so you are not really testing your hdd write speeds.

ThomasWaldmann commented 2 months ago

@someone-somenet-org See the code about what these numbers mean.

AntonOellerer commented 2 months ago

@RonnyPfannschmidt the problem is that the borg running on the storage box can only be used for backing up to the box, not the other way around.

borgbackup / borg