Yelp / mrjob

Run MapReduce jobs on Hadoop or Amazon Web Services
http://packages.python.org/mrjob/
Other
2.62k stars 586 forks source link

add support for concurrent steps on EMR #2189

Closed coyotemarin closed 4 years ago

coyotemarin commented 4 years ago

Added the max_concurrent_steps option, which allows us to launch or join pooled clusters that can run multiple steps at once (the StepConcurrencyLevel). Fixes #2185.

To make this work, I had to change the locking semantics for clusters than run steps concurrently. Rather than requiring a status of WAITING, we allow either WAITING or RUNNING but check the number of active steps on the cluster. The good thing about this is that we can release the lock as soon as we add steps, since other jobs attempting to join the cluster will see our new pending step immediately (unlike with the cluster state, which can take several seconds to flip from WAITING to RUNNING after we add our step).

This change builds on #2187 (the add_steps_in_batch option), which ensures that multi-step jobs submit steps one at a time on clusters that can run steps concurrently.

88manpreet commented 4 years ago

+1

coyotemarin commented 4 years ago

Thanks!