Step type to wait for existing job completion

[jeffparsons]

Trigger steps are great. We use them extensively. But sometimes it would be nice to be able to wait for an existing job to finish instead.

Scenario:

We have a big high-level controller pipeline. Let's call it "do all the things". It triggers, among other things, builds of another pipeline called "do one specific thing", depending on whether there are any relevant code changes since the last successful build of "do one specific thing" (ordered by commit ancestry).

Sometimes there are several concurrent builds of "do all the things" (sometimes different branches, sometimes for the main branch) that each then trigger their own build of "do one specific thing". In cases where there is already a build of "do one specific thing" for the same commit (regardless of the branch), then it would be nice to reuse that build by waiting on it, rather than triggering a new build.

Workaround:

Occupy an agent with a simple script that polls for completion of the job we're interested in.

[lox]

Interesting idea! So it would basically just be a wait step that accepted a specific job id? I can see how that would handle the fan-in aspect, but it might not cover the "avoid doing the same specific thing if it's been done before" 🤔

It would be neat to be able to skip a given step if it had executed before with the same commit hash. Perhaps they might be different things.

[jeffparsons]

Interesting idea! So it would basically just be a wait step that accepted a specific job id?

Yeah, that's exactly what I had in mind: wait for a specific job ID.

I can see how that would handle the fan-in aspect, but it might not cover the "avoid doing the same specific thing if it's been done before" :thinking:

I wouldn't exactly describe my use-case as "fan-in", though I can see how it would be useful for that, too.

In my case it's more like... I already have all the logic to trigger the appropriate builds on other pipelines, and from there it'd be trivial (thanks to your awesome API!) for me to detect existing builds that are equivalent to ones I'm about to kick off, and instead wait for those.

It would be neat to be able to skip a given step if it had executed before with the same commit hash. Perhaps they might be different things.

Yeah, that would be pretty nifty. But I'm perfectly happy to have that sort of logic live in my build scripts. I'm really after a much more "dumb" wait-for-existing-build step type. If you were excited about building a more elaborate feature, I guess I wouldn't complain. :grin:

In all seriousness, though, one of the things I like most about Buildkite is that it gives me a great toolkit of simple orthogonal features for me to build my own system from, rather than trying to anticipate my every need with higher-level features. So while I'd be overjoyed to have any version of this, I actually consider the "dumber" version to be better, because it gives me more explicit control.

[toolmantim]

I'm not entirely sure that it helps, but our Dependent Pipeline Example used to use the Builds API to do something similar: https://github.com/buildkite/dependent-pipeline-example/pull/3/files#diff-0fd391c97e31f77f48053a2e5ed70070

[jeffparsons]

@toolmantim If I'm not mistaken, that's pretty much what I was hinting at under the "workaround" heading in my original post. Does that sound right to you, or is there something more subtle going on in that example that I've missed?

[toolmantim]

@jeffparsons oh yep, that's exactly what you suggested 😅

[jeffparsons]

@toolmantim @lox If you're willing to give me read-only access to your backend repo, I'm happy to submit a PR implementing this! 😁

[toolmantim]

@jeffparsons 😁 sorry!

A neat workaround might be to start some kind of transient agent with a UUID based queue value, which can do the waiting without tying up any resources/tasks?

[jeffparsons]

@jeffparsons 😁 sorry!

Bummer. I had to try. 😛

A neat workaround might be to start some kind of transient agent with a UUID based queue value, which can do the waiting without tying up any resources/tasks?

Yeah, I think we'll do something like that for now. Or, since we have a couple of other use cases for trivial-but-long-running tasks, I might just spin up a tiny box with 100+ agents and have those kinds of jobs only ever run on that box. Or something like that.

By the way, it looks like we're practically around the corner from each other. (We're in Abbotsford.) Friday afternoons are usually pretty chill around here; if any of you would like to drop by for a beer some time, we'd love to meet you!

[toolmantim]

By the way, it looks like we're practically around the corner from each other. (We're in Abbotsford.) Friday afternoons are usually pretty chill around here; if any of you would like to drop by for a beer some time, we'd love to meet you!

Sounds great! Drop us an email… although I'm about to take 2 months 👶🏼 leave.