search

borgbackup / borg

Deduplicating archiver with compression and authenticated encryption.
https://www.borgbackup.org/
Other
11.12k stars 741 forks source link

How can I speed up the backup? #3773

Closed monkiki closed 6 years ago

monkiki commented 6 years ago

Hello and thanks for this great backup software.

I've used to for a backup of a big bunch of files and I'm not sure if the time it takes is normal o it can be improved. These are the numbers:

Duration: 6 hours 30 minutes 47.62 seconds
Number of files: 683894

I've used the --nobsdflags but I don't see any difference.

ThomasWaldmann commented 6 years ago

You didn't tell the total size and your hw setup, but in general, speed depends on a lot of factors.

If that is your first backup, there is good news: subsequent backups with little changes will be much faster.

monkiki commented 6 years ago

Sadly this is not the first backup :(

                            Original size      Compressed size    Deduplicated size
This archive:               86.99 GB             72.59 GB              4.03 GB
All archives:                1.54 TB              1.24 TB            143.86 GB

The backup is done agains another server in the same network.

ThomasWaldmann commented 6 years ago

Check with borg create -v --list ... whether borg thinks all files are added/modified.

If so, see the FAQ about what to check.

borkd commented 6 years ago

@monkiki - what kind of storage do you use for source data and destination repo, and how are the machines networked? Is there anything else competing for resources on the destination machine?

I understand backup at ~30 files/s may not be ideal. What throughput do you get if network is taken out from the equation? If you copy your destination repo onto an external disk and run it locally, for example.

monkiki commented 6 years ago

I'm doing some tests, and will update the issue once I get the numbers.

Thanks!

borkd commented 6 years ago

One more thing - your config file. Defaults or custom? Are you running in append_only?

monkiki commented 6 years ago

I suppose I have no custom config file because I don't have idea about you're talking about. And I have not used any modifier to use that "append_only" mode. Please, tell me where can I find more information about these two items.

The original borg command line was this:

borg create -v --stats ${BACKUP_REPO}::'{now:%Y-%m-%d}' .
borkd commented 6 years ago

How fast was the local backup? For a quicker comparison you could create two test repos, one on a local machine and one on the remote server, and run a simple benchmark against them.

In general, RTD is best place to check for answers. Append only can be specified during init, or at a later time.

monkiki commented 6 years ago

I've rsync(ed) all these files to a newer server and the backup time in this new server has been reduced. I'm still testing, but the numbers are much better right now. I will update again this issues with both server configurations and times.

I was not aware of the benchmark command neither the "append-only" option, thanks!

ThomasWaldmann commented 6 years ago

@monkiki do I see it right, you did not use any compression for borg create?

monkiki commented 6 years ago

The default one, which seems lzh according to the documentation.

ThomasWaldmann commented 6 years ago

We don't use lzh.

Also, I am noticing again and again that this bug report is lacking a lot of infos and we have to guess...

So, please add (by editing the top post):

monkiki commented 6 years ago

Server A (the slow one)

Hardware configuration

Operating System

Benchmark

Using Borg v1.15

C-Z-BIG          38.46 MB/s (10 * 100.00 MB all-zero files: 26.00s)
R-Z-BIG         223.92 MB/s (10 * 100.00 MB all-zero files: 4.47s)
U-Z-BIG         289.99 MB/s (10 * 100.00 MB all-zero files: 3.45s)
D-Z-BIG        1533.56 MB/s (10 * 100.00 MB all-zero files: 0.65s)
C-R-BIG          23.51 MB/s (10 * 100.00 MB random files: 42.53s)
R-R-BIG          40.33 MB/s (10 * 100.00 MB random files: 24.80s)
U-R-BIG         367.23 MB/s (10 * 100.00 MB random files: 2.72s)
D-R-BIG          41.14 MB/s (10 * 100.00 MB random files: 24.31s)
C-Z-MEDIUM       35.79 MB/s (1000 * 1.00 MB all-zero files: 27.94s)
R-Z-MEDIUM      236.98 MB/s (1000 * 1.00 MB all-zero files: 4.22s)
U-Z-MEDIUM      685.34 MB/s (1000 * 1.00 MB all-zero files: 1.46s)
D-Z-MEDIUM     1545.48 MB/s (1000 * 1.00 MB all-zero files: 0.65s)
C-R-MEDIUM       15.54 MB/s (1000 * 1.00 MB random files: 64.37s)
R-R-MEDIUM       26.91 MB/s (1000 * 1.00 MB random files: 37.16s)
U-R-MEDIUM      857.36 MB/s (1000 * 1.00 MB random files: 1.17s)
D-R-MEDIUM       50.12 MB/s (1000 * 1.00 MB random files: 19.95s)
C-Z-SMALL         7.84 MB/s (10000 * 10.00 kB all-zero files: 12.76s)
R-Z-SMALL        94.38 MB/s (10000 * 10.00 kB all-zero files: 1.06s)
U-Z-SMALL        42.94 MB/s (10000 * 10.00 kB all-zero files: 2.33s)
D-Z-SMALL       168.50 MB/s (10000 * 10.00 kB all-zero files: 0.59s)
C-R-SMALL         4.36 MB/s (10000 * 10.00 kB random files: 22.95s)
R-R-SMALL        20.28 MB/s (10000 * 10.00 kB random files: 4.93s)
U-R-SMALL        18.80 MB/s (10000 * 10.00 kB random files: 5.32s)
D-R-SMALL        19.91 MB/s (10000 * 10.00 kB random files: 5.02s)
monkiki commented 6 years ago

Server B

Hardware configuration

CPU: Quad core Intel Core i7-7700 cache: 8192 KB (4200 MHz) Drive: model: TOSHIBA_MG04ACA2 size: 1.82 TiB Network Card: Intel Ethernet Connection I219-LM driver: e1000e

Operating System

Kernel: 4.4.0-119-generic x86_64 Distro: Ubuntu 16.04.4 LTS

Benchmark

Using Borg v1.15

C-Z-BIG          44.92 MB/s (10 * 100.00 MB all-zero files: 22.26s)
R-Z-BIG         282.17 MB/s (10 * 100.00 MB all-zero files: 3.54s)
U-Z-BIG         325.46 MB/s (10 * 100.00 MB all-zero files: 3.07s)
D-Z-BIG        1760.23 MB/s (10 * 100.00 MB all-zero files: 0.57s)
C-R-BIG          29.18 MB/s (10 * 100.00 MB random files: 34.27s)
R-R-BIG          57.36 MB/s (10 * 100.00 MB random files: 17.43s)
U-R-BIG         394.16 MB/s (10 * 100.00 MB random files: 2.54s)
D-R-BIG          74.38 MB/s (10 * 100.00 MB random files: 13.44s)
C-Z-MEDIUM       63.14 MB/s (1000 * 1.00 MB all-zero files: 15.84s)
R-Z-MEDIUM      307.63 MB/s (1000 * 1.00 MB all-zero files: 3.25s)
U-Z-MEDIUM     1399.73 MB/s (1000 * 1.00 MB all-zero files: 0.71s)
D-Z-MEDIUM     1912.28 MB/s (1000 * 1.00 MB all-zero files: 0.52s)
C-R-MEDIUM       33.29 MB/s (1000 * 1.00 MB random files: 30.04s)
R-R-MEDIUM       87.50 MB/s (1000 * 1.00 MB random files: 11.43s)
U-R-MEDIUM     1285.36 MB/s (1000 * 1.00 MB random files: 0.78s)
D-R-MEDIUM       50.88 MB/s (1000 * 1.00 MB random files: 19.65s)
C-Z-SMALL        11.00 MB/s (10000 * 10.00 kB all-zero files: 9.09s)
R-Z-SMALL       137.11 MB/s (10000 * 10.00 kB all-zero files: 0.73s)
U-Z-SMALL        50.47 MB/s (10000 * 10.00 kB all-zero files: 1.98s)
D-Z-SMALL        67.30 MB/s (10000 * 10.00 kB all-zero files: 1.49s)
C-R-SMALL         7.18 MB/s (10000 * 10.00 kB random files: 13.93s)
R-R-SMALL        84.81 MB/s (10000 * 10.00 kB random files: 1.18s)
U-R-SMALL        25.91 MB/s (10000 * 10.00 kB random files: 3.86s)
D-R-SMALL        32.60 MB/s (10000 * 10.00 kB random files: 3.07s)
borkd commented 6 years ago

@ThomasWaldmann - I have submitted a PR for review https://github.com/borgbackup/borg/pull/3783 to help with issues like this one in the future. Change is Github specific and doesn't touch any of the code. PR was based on master, so Travis results will reflect the current status of master.

ThomasWaldmann commented 6 years ago

https://github.com/borgbackup/borg/issues/3773#issuecomment-382039042 so, what did it output?

Still missing: the precise borg commandline.

Your hw and the benchmarks look fine.

monkiki commented 6 years ago

Very simple invocation:

borg create -v --stats ${BACKUP_REPO}::'{now:%Y-%m-%d_%H:%M}' /home

Anyway I've realised that this server have disk performance issues because a simple "du -hs /home" takes several hours, while the same data moved to the other server takes a few minutes.