search

DataBiosphere / dsub

Open-source command-line tool to run batch computing tasks and workflows on backend services such as Google Cloud.
Apache License 2.0
265 stars 44 forks source link

Limit number of concurrent jobs in batch mode #190

Open dimenwarper opened 4 years ago

dimenwarper commented 4 years ago

First off, this is a superb tool, thanks a lot!

Question: Is it possible to limit the number of concurrent jobs allowed in batch mode? Would be great in situations where you share your CPU quota with other services (e.g. app engine) and don't want them to be disrupted.

wnojopra commented 4 years ago

There's currently no direct option to limit the number of concurrent jobs, but I can see that it would be useful in scenarios like you mention. I'll leave this issue open as a potential feature request.

A current solution I can think of involves the use of the --after flag. Say you have a batch of 100 tasks, but want to limit to 50 concurrently. You'd need to separate the tasks file into two and run dsub on the first 50:

JOBID_FIRST_FIFTY=$(dsub ...)

Then pass these values to the --after flag for launching the second batch:

dsub ... --after "${JOBID_FIRST_FIFTY}"

Admittedly this isn't perfect since it has to wait for the first 50 to finish completely before starting the next 50. You could be more granular if you wanted to. See the docs on --after for more detail.

wnojopra commented 4 years ago

Another potential solution would be to have a separate cloud project for dsub, with its own quotas.

dimenwarper commented 4 years ago

Awesome, thanks for the quick response! I wasn't aware of --after so I'll try that (I already have a bash script that constantly checks my quotas and launches a new batch when available but it's super hacky)