search

ericaltendorf / plotman

Chia plotting manager
Apache License 2.0
910 stars 280 forks source link

Simultaneous Sync #766

Open toboladao opened 3 years ago

toboladao commented 3 years ago

It is possible to start rsync simultaneously when finishing two plots. Am I assigning two dirs to the created plots? However, the Archive only starts when it finishes sending a plot. For me this has been a small problem, because due to the distance between the servers I can't reach the maximum transfer speed using just one rsync. I did the test manually and running two rsync I can achieve the same transfer rates on each as when using just one.

nizoo91 commented 3 years ago

This is something I requested as well, unfortunately has not been implemented yet. I don't even want to move them on networks but rather on the same server, all of my HDDs are externals hooked up to a USB hub.

I wrote a simple bash script to do this, would be your best to do the same yourself; :/

altendky commented 3 years ago

@toboladao, what is your full configuration file? What transfer rate are you seeing? By default, there is an 80MB/s bandwidth limit set for the rsyncd archival target definition.

Do note that there is https://github.com/rjsears/chia_plot_manager. By sharing this I don't mean that plotman shouldn't support multiple archive jobs, but it doesn't presently and I want to make sure people are aware of the other tools.

jwest75674 commented 3 years ago

I similarly would love parallel archival. Even with the default BW limit removed, some users like myself would benefit.

I have a 56GB/s infiniband/Mellanox ethernet backbone between my machines, however saturating this link cannot be done single-threaded. To get close to leveraging this throughput, multiple parallel transfers is required.

I too have written a script, but native functionality for plotman would be awesome.