mschubert
commented
5 years ago Any chance you can make a reproducible example that I can test on my computing cluster?
Generally, if half of your workers start and some finish their work, clustermq will give them new work instead of waiting for the remaining workers to start. Then they of course run longer, and will need a longer wall time. (A possibility here could be to have an option that allows a worker to process a maximum of n calls.)
After all these jobs were long expired, the master process does not show an error, warning, or message.
Yes, this is still an unresolved issue. Right now we expect the user to check log files on Slurm timeouts or segfaults, and the memory limits set in the template do not work on all systems.
In the future, we need some way to track if workers crash for unknown reasons, not relying on a clean R shutdown or any particular scheduler. The best way to do this I think is sending worker heartbeats, but that's a bit tricky to implement and I'm not sure how to best do it yet (related: #33).
I presume the above is the result of an internal clustermq time out.
Workers will stop if they do not receive new work after 10 minutes of idle waiting. You can set this using worker(..., timeout=600) in your template, but I don't see why this would be required (unless you send common and/or iterated data that is over 1 Gb each and your network is slow).
Note the large number of workers who hit the SLURM time limit without completing any work.
This is strange. I can only see this happen if you send huge amounts of data over a slow network, but then no workers should get (m)any calls done.
Are you sure the workers that didn't complete any work were not the ones with the clustermq timeout rather than the Slurm timeout? Because if your Slurm timeout is bigger than 10 minutes + buffer (like your 5 hours), they should always hit the clustermq timeout first.
28/400 workers ended with the following out of memory error
Can you try adding a call to ulimit at the beginning of your function and see if this works? This way it will produce an R error that will get sent back to the master process.
I used this in clustermq in the past, but since it's not on CRAN had to rely on the shell's ulimit instead (which should do the same, but sometimes fails).
3/400 workers ended with the following (not sure what this rare error means)
Are you running compiled code that crashes in some rare cases? I.e., do the ones that crash with this error finish if run locally?
pat-s
I ran a large 400 x 5hr
clustermqjob with SLURM viadrake.I kept track of what was going on using worker-level log files.
235/400 workers ended with the following:
with stats on number of jobs completed:
I presume the above is the result of an internal
clustermqtime out.135/400 workers ended with the following:
with stats on number of jobs completed:
Note the large number of workers who hit the SLURM time limit without completing any work.
28/400 workers ended with the following out of memory error:
with stats on number of jobs completed:
3/400 workers ended with the following (not sure what this rare error means):
with stats on number of jobs completed:
After all these jobs were long expired, the master process does not show an error, warning, or message. I'm not quite sure what the right solutions to these problems are, however. Perhaps the master tries to keep an eye on all the workers' statuses jointly through SLURM/log files, potentially showing a message when something like the above 4 errors occurs. Then once every worker has expired somehow, throw an error.
cc @wlandau