Added the max_concurrent_steps option, which allows us to launch or join pooled clusters that can run multiple steps at once (the StepConcurrencyLevel). Fixes #2185.
To make this work, I had to change the locking semantics for clusters than run steps concurrently. Rather than requiring a status of WAITING, we allow either WAITING or RUNNING but check the number of active steps on the cluster. The good thing about this is that we can release the lock as soon as we add steps, since other jobs attempting to join the cluster will see our new pending step immediately (unlike with the cluster state, which can take several seconds to flip from WAITING to RUNNING after we add our step).
This change builds on #2187 (the add_steps_in_batch option), which ensures that multi-step jobs submit steps one at a time on clusters that can run steps concurrently.
Added the
max_concurrent_steps
option, which allows us to launch or join pooled clusters that can run multiple steps at once (theStepConcurrencyLevel
). Fixes #2185.To make this work, I had to change the locking semantics for clusters than run steps concurrently. Rather than requiring a status of
WAITING
, we allow eitherWAITING
orRUNNING
but check the number of active steps on the cluster. The good thing about this is that we can release the lock as soon as we add steps, since other jobs attempting to join the cluster will see our new pending step immediately (unlike with the cluster state, which can take several seconds to flip fromWAITING
toRUNNING
after we add our step).This change builds on #2187 (the
add_steps_in_batch
option), which ensures that multi-step jobs submit steps one at a time on clusters that can run steps concurrently.