search

mschubert / clustermq

R package to send function calls as jobs on LSF, SGE, Slurm, PBS/Torque, or each via SSH
https://mschubert.github.io/clustermq/
Apache License 2.0
146 stars 27 forks source link

Warn if common data is very large #189

Closed mschubert closed 4 years ago

mschubert commented 4 years ago

There have been multiple reports of timeouts, at least some of which used very large data to be sent via the network (or even SSH); e.g. https://github.com/mschubert/clustermq/issues/146

We should introduce a threshold over which we display a warning that this may not be a good idea.

Aariq commented 3 years ago

I'm getting one of these timeout errors, but I think in my case the common data is not too large, but the model object created by one of the workers is too large, and it's stalling when being sent back to my machine over SSH. Will this situation still trigger this warning? Is there a way to check that this is in fact the problem I'm experiencing?

mschubert commented 3 years ago

This will not trigger this warning, and unfortunately the only check I can think of right now is to make the object smaller and see if it works again