search

mllg / batchtools

Tools for computation on batch systems
https://mllg.github.io/batchtools/
GNU Lesser General Public License v3.0
172 stars 51 forks source link

Recognise abuse behavior #235

Open kokyriakidis opened 5 years ago

kokyriakidis commented 5 years ago

Hi!

Is it possible and if yes, in which part of batchtools, to set limits of job submission number, rate, etc.

I am a new user and I am trying to get it work on my University's HPC system

mllg commented 5 years ago

You cannot control the submission rate directly. batchtools will try to submit any jobs passed to submitJobs() as fast as possible (but sequentially). If a submission fails, batchtools will try again after x seconds (see argument sleep in submitJobs()). Of course, you can chunk jobs together or use array jobs to reduce load from the scheduler.

You can control the maximum number of jobs per user, see https://mllg.github.io/batchtools/reference/submitJobs.html#limiting-the-number-of-jobs.

Does this help?

kokyriakidis commented 5 years ago

Thank you for your reply!

Can it somehow check if it fails and why (check the HPC's error code), and then if it is due to resource limits, try to increase them by a certain amount? As you can do with Nextflow.

mllg commented 5 years ago

You can grep the error logs for the error message, retrieve the respective job ids and re-submit those jobs with different resource constraints. There is no automated way to do this.

kokyriakidis commented 5 years ago

sleep | [function(i) | numeric(1)] Can I set how many times to try to submit (not after how many seconds), before it gets cancelled? Also, can you give an example code of the function that can be used in sleep command?

mllg commented 5 years ago

See https://github.com/mllg/batchtools/blob/master/R/sleep.R for the default function(s).

kokyriakidis commented 5 years ago

I cannot see how I can set how many TIMES to run. Using a function only changes the time if I understand correctly.

Maybe the answer to my question is in waitForJobs function https://github.com/mllg/batchtools/blob/3b0b1a9a59e377bb4d827e355d6955d66849c9e6/R/waitForJobs.R