Now that EMR can run steps concurrently, it can be a problem to submit all of the steps of a multi-step job concurrently. This change makes it so that for AMIs that can run steps concurrently (5.28.0 and later), the runner submits multi-step jobs one step at a time. This behavior can be changed explicitly with the add_steps_in_batch option.
This is a first step toward full support for concurrent steps (#2185).
Now that EMR can run steps concurrently, it can be a problem to submit all of the steps of a multi-step job concurrently. This change makes it so that for AMIs that can run steps concurrently (5.28.0 and later), the runner submits multi-step jobs one step at a time. This behavior can be changed explicitly with the
add_steps_in_batch
option.This is a first step toward full support for concurrent steps (#2185).