buildkite / feedback

Got feedback? Please let us know!
https://buildkite.com
25 stars 24 forks source link

⏭⏰ Feature: Skip timeout #308

Open therealbnut opened 6 years ago

therealbnut commented 6 years ago

Some of our builds are expensive (10 minutes). We want to skip un-necessary builds because of this. However if our build has been running for 9 minutes it's a real shame to cancel it.

It'd be great if you had a skip-timeout. A build would not be skippable if this timeout is exceeded.

I'm happy for alternatives, if they solve the same problem.

keithpitt commented 6 years ago

Howdy, and thanks for the idea!

Hrm - I think I understand what you're after... What would you deem as an "un-necessary" build? We've already got some controls in place that allow you to skip/cancel builds depending on the branch:

screen shot 2018-01-22 at 10 26 57 am

Here you can say "just cancel/skip anything not on a master branch" (so just development branches really).

lox commented 6 years ago

Interesting idea! It's almost the inverse of a timeout, perhaps a skip upper bound or threshold?

therealbnut commented 6 years ago

Yeah lol, I didn't think timeout was best, "skippability timeout" seemed a bit 💩 too, skipableness.

therealbnut commented 6 years ago

@keithpitt thanks, we already use those a lot, they're great thank you :)

My use case is a bit more specific though:

Cancelling an already running build could waste most of the work. The build wouldn't have been useful for allowing a merge, but it may have been useful to diagnose an issue.


What I'm after is probably better described as a "cancel threshold" (thanks @lox).

I think whether something should be safely cancelled depends on whether it's something that throws away useful work. The longer a build runs, the more likely you are to be throwing away useful work, so a time threshold makes sense to me.

lox commented 6 years ago

The other way to solve this would be to let agents know that a build is being skipped and let them choose to not skip it. Currently the agents get a "build cancelled" and 10 seconds to terminate the build.

keithpitt commented 6 years ago

@therealbnut ah I see. So essentially you want to tell BK: "Hey, only cancel this build automatically if it's just stated. If it's at the 9 minute mark, don't cancel it - just let it finish because it's pretty much almost done" - is that right?

@lox that's a good idea, but that brings up another issue of "No, I really need to cancel this thing". When ever I've thought about increasing that threshold, it raises the question of "well, if I hit cancel - do I need to hit cancel again to really cancel it" if that makes sense?

lox commented 6 years ago

Yup, I hear ya, I've had the same thoughts.

therealbnut commented 6 years ago

@keithpitt that's correct. I like the signal approach, maybe you send a SIGKILL on a manual/automated cancel, but you can opt-in to instead receive a SIGTERM on an automated cancel?

therealbnut commented 6 years ago

Although I'm not sure if it's easy for the child process to indicate that it exited because it was cancelled or because the tests failed.