search

hjmangalam / parsyncfp

follow-on to parsync (parallel rsync) with better startup perf
Other
164 stars 19 forks source link

workaround: rsync processes quitting early at the start #8

Closed sebastian-wasilewski closed 6 years ago

sebastian-wasilewski commented 6 years ago

Parsyncfp is really a great tool! I have been trying to run it with high NP (64-128 and higher) and I noticed an issue that some of rsync processes often quit early at the start.

The issue seems to be limited to the initial rsync processes, so I think it may be caused by a large number of connections being opened with the remote SSH server at the same time. I have added a short, random delay before each process starts and so far I do not get those errors any more.

Please, would you accept adding this workaround to the master branch?

sebastian-wasilewski commented 6 years ago

The reason for random delay was to avoid all processes hitting the SSH server at once. If I put e.g. sleep(0.2) then all workers will sleep for 0.2s and then hit the SSH server.

I guess we could try to lower the rand(1.0) to 0.5s. I will do some testing with that. Also, another option would be to wrap that with an if statement, so the random delay would be only introduced if e.g. NP > 8.

hjmangalam commented 6 years ago

Good point of course, I wasn't thinking clearly on the weekend ;). We can tune this going forward.