search

hjmangalam / parsyncfp

follow-on to parsync (parallel rsync) with better startup perf
Other
161 stars 19 forks source link

parsyncfp seems to work, but there is no progress output #20

Closed annymous closed 5 years ago

annymous commented 5 years ago

I am following "Good example 1". I see no errors, and files get transferred. But there is no output after:

Thanks for using parsyncfp. Tell me how to make it better.
<hjmangalam@gmail.com>

This is in a Debian 9.2 x64 VPS. I'm using multiple interfaces that are aggregated with MPTCP. And I can see that there is multipathing.

hjmangalam commented 5 years ago

On Monday, April 22, 2019 3:17:59 PM PDT annymous wrote:

I am following "Good example 1". I see no errors, and files get transferred. But there is no output after: Thanks for using parsyncfp. Tell me how to make it better. hjmangalam@gmail.com

Umm - you shouldn't see anything after that. That's the goodbye message. What were you expecting?

hjm

This is in a Debian 9.2 x64 VPS. I'm using multiple interfaces that are aggregated with MPTCP. And I can see that there is multipathing.

Harry Mangalam, Info[1]

[1] http://moo.nac.uci.edu/~hjm/hjm.sig.html

annymous commented 5 years ago

Thanks. I was expecting the status updates

 Time  | time(m) |  Load  |     TCP / RDMA  out  |   PIDs  ||  PIDs   | [UpTo] of [ToDo]

But now I see them, with more files in the test folder. With two files, I saw no status output. But with nine files, I see expected status output. Anyway, thanks again.

hjmangalam commented 5 years ago

Do note that parsyncfp is meant for transferring LOTS of files and that the admin overhead for transferring even a few hundred small files will make it much slower than a naked rsync.

rsync is an effing miraculous piece of software. :)

hjm

On Monday, April 22, 2019 6:04:31 PM PDT annymous wrote:

Thanks. I was expecting the status updates

 Time  | time(m) |  Load  |     TCP / RDMA  out  |   PIDs  ||  PIDs

| [UpTo] of [ToDo]

But now I see them, with more files in the test folder. With two files, I saw no status output. But with nine files, I see expected status output. Anyway, thanks again.

Harry Mangalam, Info[1]

[1] http://moo.nac.uci.edu/~hjm/hjm.sig.html

annymous commented 5 years ago

Yes, I get that. And so is parsyncfp :)

My application is unusual, I guess. I'm working with VPS that are linked with multiple tinc networks, with tinc running as Tor onion services. And then aggregated with MPTCP. It's not so much that I'll be transferring huge files, but I need multiple TCP transfers to saturate the links. So far, using parsyncfp, I've gotten ~30 Mbps with NP=4 and ~40 Mbps with NP=8. With just rsync, it's ~10 Mbps. And with 10 scp instances, it's also ~30 Mbps. I know that's pitiful, but for Tor it's fast.