Yelp / mrjob

Run MapReduce jobs on Hadoop or Amazon Web Services
http://packages.python.org/mrjob/
Other
2.62k stars 586 forks source link

max_clusters_in_pool option #2192

Closed coyotemarin closed 3 years ago

coyotemarin commented 4 years ago

We should add an option to limit the number of clusters in a pool. If a pooled job can't find a cluster to join and the pool already has max_clusters_in_pool clusters active, the job will wait rather than launching its own cluster.