search

hjmangalam / parsyncfp

follow-on to parsync (parallel rsync) with better startup perf
Other
161 stars 19 forks source link

Simultaneous parsyncfp execution with different cache directories #18

Open Krobar opened 5 years ago

Krobar commented 5 years ago

I'm seeing 1000s of these types of messages when running Parsyncfps simultaneously: INFO: No more running rsync PIDs left [No running PIDs; will start more next cycle]. All rsyncs are suspended [4808 18703 18706 18709 18716 18756 18759 18763 4762 4773 4785 4797 4819 4830 4841 8799 8856 18694 18697 18700].

I though specifying separate cache directories would allow simultaneous running, is simultaneous running supported or not?

hjmangalam commented 5 years ago

On Wednesday, March 20, 2019 5:56:47 AM PDT Krobar wrote:

I'm seeing 1000s of these types of messages when running Parsyncfps simultaneously: INFO: No more running rsync PIDs left [No running PIDs; will start more next cycle]. All rsyncs are suspended [4808 18703 18706 18709 18716 18756 18759 18763 4762 4773 4785 4797 4819 4830 4841 8799 8856 18694 18697 18700].

I though specifying separate cache directories would allow simultaneous running, is simultaneous running supported or not?

This is actually the expected behavior. What has happened is that the loadavg has increased to the point where all the running rsyncs have been suspended to avoid overrunning the set loadavg. (the default is (NP+2), so when running multiple instances, this will tend to be overrun quite easily, since the different instances do not coordinate.

In order to avoid this, you can explicitly set the maxload to a very large number. However, I'll look at reducing the verbosity of the error messages and making it more informative.

Harry Mangalam, Info[1]

[1] http://moo.nac.uci.edu/~hjm/hjm.sig.html

Krobar commented 5 years ago

Thank you for the fast response, will give this a try.

hjmangalam commented 5 years ago

On Wednesday, March 20, 2019 9:11:16 AM PDT Krobar wrote:

Thank you for the fast response, will give this a try.

Thanks. Let me know if it works as advertised with this fix.

Harry Mangalam, Info[1]

[1] http://moo.nac.uci.edu/~hjm/hjm.sig.html

novosirj commented 4 years ago

I've run into this myself; this should fix the issue. What would probably help is if this were to say that it's suspending because of load average, if it doesn't already (I don't see it in this output).