hjmangalam / parsyncfp

follow-on to parsync (parallel rsync) with better startup perf
Other
161 stars 19 forks source link

how to select / system root as the starting point ? #29

Closed MasterCATZ closed 4 years ago

MasterCATZ commented 4 years ago

every-time I try it is looking at the home folder of user

looking for a way to make a system backup local / to local /mnt

./parsyncfp -v 2 --maxload=5.5 --NP=10 --rsyncopts="-aAXv --exclude={"/dev/","/proc/","/sys/","/tmp/","/run/","/mnt/","/media/","ZFSRaidz","SnapRaid*","/lost+found"}" "--startdir=/" /SnapRaidArray/DATA/Backup/Nov2019

hjmangalam commented 4 years ago

The rsync opts are up to you (and I'm not sure the rsync-options are going to work as you describe. Not saying they won't but it doesn't work for me. (Perhaps try the -– exclude-from option and provide a file of dirs to exclude?)

The main thing that you seem to be having trouble with is specifying the top-level dirs.

You can do this with [ --startdir=/ . ] which will include all the top-level dirs and then use the -–exclude-from [file] or you can use explicit names: --startdir=/ bin opt var boot etc lib sbin home lib64 root usr # etc

Does that work for you?

hjm

On Monday, November 11, 2019 5:22:02 PM PST MasterCATZ wrote:

every-time I try it is looking at the home folder of user

looking for a way to make a system backup local / to local /mnt

./parsyncfp -v 2 --maxload=5.5 --NP=10 --rsyncopts="-aAXv --exclude={"/dev/","/proc/","/sys/","/tmp/","/run/","/mnt/","/medi a/","ZFSRaidz","SnapRaid*","/lost+found"}" "--startdir=/" /SnapRaidArray/DATA/Backup/Nov2019

Harry Mangalam, Info[1]


[1] http://moo.nac.uci.edu/~hjm/hjm.sig.html

MasterCATZ commented 4 years ago

THANK YOU now I just need to work out why its running slower, looks like its pausing after each event

--startdir=/ .

that / space dot made all the difference :D

issue for me was all the small files this helped knock through the google cache fast

./parsyncfp -v 2 --maxload=5.5 --NP=10 --rsyncopts="-aAXv --exclude={"/dev/","/proc/","/sys/","/tmp/","/run/","/mnt/","/media/","/ZFSRaidz","/SnapRaid","/.snapshots","/lost+found"}" --startdir=/ . /SnapRaidArray/DATA/Backup/Nov2019

I thought I stuffed up my btrfs system after doing a snapshot roll back so was going to format and try again

but apparently it was just cleaning up after I deleted the 1000 odd snapshots that was causing the terrible lag on system

INFO: Starting the 1st [10] rsyncs .. home/aio/.config/google-chrome/unkfile [/root/.parsyncfp/fpcache/f.7] home/aio/.config/google-chrome/EMAIL/ home/aio/.config/google-chrome/EMAIL/Cookies-journal home/aio/.config/google-chrome/EMAIL/Network Action Predictor home/aio/.config/google-chrome/EMAIL/Network Action Predictor-journal home/aio/.config/google-chrome/EMAIL/Network Persistent State

sent 354,199 bytes received 12 bytes 47,228.13 bytes/sec total size is 10,739,170,759 speedup is 30,318.57 home/aio/.config/google-chrome/EMAIL/Local Extension Settings/he/f.9] home/aio/.config/google-chrome/EMAIL/Local Storage/ home/aio/.dropbox/ home/aio/.dropbox/logs/ home/aio/.dropbox/logs/1/1-1331-5dc9ece8.tmp home/aio/.dropbox/metrics/store.bin home/aio/.local/share/Steam/ home/aio/.local/share/Steam/logs/appinfo_log.txtarsyncfp/fpcache/f.9] home/aio/.local/share/Steam/logs/configstore_log.txt home/aio/.cache/google-chrome/EMAIL/Cache/ | Elapsed | 1m | [enp13s0] MB/s | Running || Susp'd | Chunks [2019-11-12] Time | time(m) | Load | TCP / RDMA out | PIDs || PIDs | [UpTo] of [ToDo] home/aio/.local/share/Steam/config/htmlcache/ home/aio/.config/google-chrome/Profile 4/ home/aio/.config/google-chrome/Profile 4/Cookies home/aio/.config/google-chrome/Profile 4/Cookies-journal home/aio/.config/google-chrome/Profile 4/TransportSecurity home/aio/.config/google-chrome/Profile 4/Local Storage/ home/aio/.config/google-chrome/Profile 4/Local Storage/leveldb/001368.log

sent 7,121,254 bytes received 210 bytes 527,515.85 bytes/sec total size is 4,812,151,766 speedup is 675.73 13.06.08 0.07 6.66 0.09 / 0.00 9 <> 0 [10] of [11]

sent 11,815,855 bytes received 119 bytes 675,198.51 bytes/sec total size is 12,295,477,634 speedup is 1,040.58 home/aio/.config/google-chrome/Local State home/aio/.config/google-chrome/BrowserMetrics/BrowserMetrics-5DCA05D6-3CAC.pma 13.06.11 0.12 7.33 0.09 / 0.00 7 <> 1 [10] of [12] home/aio/.cache/google-chrome/EMAIL/Cache/index-dir/ home/aio/.cache/google-chrome/EMAIL/Cache/index-dir/the-real-index

sent 1,670,036 bytes received 47 bytes 90,274.76 bytes/sec total size is 427,820,160 speedup is 256.17 13.06.15 0.18 7.86 0.08 / 0.00 5 <> 2 [10] of [12] home/aio/.config/google-chrome/EMAIL/Cookies home/aio/.config/google-chrome/EMAIL/Favicons-journal home/aio/.config/google-chrome/EMAIL/History-journal home/aio/.config/google-chrome/EMAIL/Preferences home/aio/.config/google-chrome/EMAIL/TransportSecurity home/aio/.config/google-chrome/EMAIL/Visited Links 13.06.18 0.23 7.86 0.09 / 0.00 3 <> 3 [10] of [12] 13.06.22 0.30 8.59 0.12 / 0.00 2 <> 4 [10] of [14] 13.06.25 0.35 8.55 0.10 / 0.00 1 <> 5 [10] of [25] 13.06.28 0.40 8.55 0.10 / 0.00 0 <> 6 [10] of [26] WARN: No more running rsync PIDs left. 13.06.31 0.45 8.34 0.03 / 0.00 0 <> 6 [10] of [26] WARN: No more running rsync PIDs left. 13.06.35 0.52 9.12 0.09 / 0.00 0 <> 6 [10] of [33] WARN: No more running rsync PIDs left. 13.06.38 0.57 9.12 0.08 / 0.00 0 <> 6 [10] of [72] WARN: No more running rsync PIDs left. 13.06.41 0.62 8.87 0.09 / 0.00 0 <> 6 [10] of [134] WARN: No more running rsync PIDs left. 13.06.44 0.67 8.87 0.11 / 0.00 0 <> 6 [10] of [134] WARN: No more running rsync PIDs left. 13.06.47 0.72 8.48 0.11 / 0.00 0 <> 6 [10] of [139] WARN: No more running rsync PIDs left. 13.06.50 0.77 8.44 0.10 / 0.00 0 <> 6 [10] of [140] WARN: No more running rsync PIDs left. 13.06.53 0.82 8.44 0.10 / 0.00 0 <> 6 [10] of [140] WARN: No more running rsync PIDs left. 13.06.56 0.87 8.08 0.09 / 0.00 0 <> 6 [10] of [142] WARN: No more running rsync PIDs left. 13.07.00 0.93 8.00 0.10 / 0.00 0 <> 6 [10] of [147] WARN: No more running rsync PIDs left. 13.07.03 0.98 8.00 0.09 / 0.00 0 <> 6 [10] of [153] WARN: No more running rsync PIDs left. 13.07.06 1.03 7.67 0.09 / 0.00 0 <> 6 [10] of [164] WARN: No more running rsync PIDs left. 13.07.09 1.08 7.67 0.10 / 0.00 0 <> 6 [10] of [188] WARN: No more running rsync PIDs left. 13.07.12 1.13 7.38 0.10 / 0.00 0 <> 6 [10] of [271] WARN: No more running rsync PIDs left. | Elapsed | 1m | [enp13s0] MB/s | Running || Susp'd | Chunks [2019-11-12] Time | time(m) | Load | TCP / RDMA out | PIDs || PIDs | [UpTo] of [ToDo] 13.07.15 1.18 7.67 0.09 / 0.00 0 <> 6 [10] of [288] WARN: No more running rsync PIDs left. 13.07.18 1.23 7.67 0.10 / 0.00 0 <> 6 [10] of [292] WARN: No more running rsync PIDs left. 13.07.21 1.28 7.70 0.09 / 0.00 0 <> 6 [10] of [293] WARN: No more running rsync PIDs left. 13.07.25 1.35 7.48 0.10 / 0.00 0 <> 6 [10] of [293] WARN: No more running rsync PIDs left. 13.07.28 1.40 7.48 0.09 / 0.00 0 <> 6 [10] of [294] WARN: No more running rsync PIDs left. 13.07.31 1.45 7.04 0.11 / 0.00 0 <> 6 [10] of [295] WARN: No more running rsync PIDs left. 13.07.34 1.50 7.04 0.10 / 0.00 0 <> 6 [10] of [295] WARN: No more running rsync PIDs left. 13.07.37 1.55 6.72 0.07 / 0.00 0 <> 6 [10] of [295] WARN: No more running rsync PIDs left. 13.07.40 1.60 6.74 0.09 / 0.00 0 <> 6 [10] of [301] WARN: No more running rsync PIDs left.