Feature Request: More granular control over pipelines

[seanabraham]

Pipelines as they currently exist are really useful and with wait steps there is some degree of dependencies you can specify within a pipeline itself. Triggering other pipelines is a good way to set up more complex dependencies, but we don't always want to trigger other pipelines. I'd like some way to define an arbitrary DAG in a pipeline. Even if we can't trigger a full on DAG, it might be nice to have some more control. Maybe most problems can be solved with some kind of grouping? Here's an example use case:

We need to run Steps A, B, C and D but D depends on B. Right now, the way we do that is with Pipeline 1:

A
B
C
-- wait
D

This works but usually has D wait longer than it needs to if B is faster than max(A, C). It also causes D to be unnecessarily skipped sometimes if A or C fail. We could presumably do

Pipeline 1:

A
-- trigger Pipeline 2
C

Pipeline 2:

B
-- wait
D

But I think the UI for triggering other pipelines can be a bit confusing so I'd prefer to avoid that. Also, if we needed a more complex dependency setup that wouldn't work (like if we had another step E that depended on both A and B for example).

[lox]

Maybe most problems can be solved with some kind of grouping?

That is exactly what we discussed a while back. It's a pretty fundamental change, so it's not likely to happen any time soon, but it's definitely something that we are pondering.

[jeffparsons]

I'm pretty enthusiastic about this, too.

The vast majority of my pipelines are configured dynamically, so any kind of interface that would make this possible within a single pipeline would be really handy, even if it wasn't the prettiest thing ever imagined.

I'm thinking something like a "wait for jobs labelled XXXXXXXX" feature, where job ABC won't start unless all other jobs with some arbitrary specified label have completed successfully. I assume you have your own ideas about how you'd solve this problem in a way that fits more cohesively into your overall feature set — just giving an example of what kind of interface would get the job done for me. 😃

Any kind of grouping that linearises my dependencies is going to leave a lot of "critical path length" improvements on the table, because I'll always have jobs waiting around artificially when all their actual dependencies have finished. Currently the only way around this for my case would be to create ~100 different pipelines, which would be a bit of a pain to maintain and wrangle.

To be clear, this is far from an urgent need for me. It would just be nice to be able to more precisely communicate my build dependencies to Buildkite to really cut down on that critical path length! 🏎

[felixfbecker]

So essentially, like CircleCI 2.0 workflows

[lox]

So essentially, like CircleCI 2.0 workflows

Except more awesome? 😉