search

Yelp / mrjob

Run MapReduce jobs on Hadoop or Amazon Web Services
http://packages.python.org/mrjob/
Other
2.62k stars 586 forks source link

add_steps_in_batch option #2187

Closed coyotemarin closed 4 years ago

coyotemarin commented 4 years ago

Now that EMR can run steps concurrently, it can be a problem to submit all of the steps of a multi-step job concurrently. This change makes it so that for AMIs that can run steps concurrently (5.28.0 and later), the runner submits multi-step jobs one step at a time. This behavior can be changed explicitly with the add_steps_in_batch option.

This is a first step toward full support for concurrent steps (#2185).

88manpreet commented 4 years ago

+1

coyotemarin commented 4 years ago

Thanks!