search

buildkite / feedback

Got feedback? Please let us know!
https://buildkite.com
25 stars 24 forks source link

notification for stuck builds #333

Closed ygrek closed 6 years ago

ygrek commented 6 years ago

Usecase: Imagine some interactive command in build script is introduced - after some time all agents are stuck for the same reason. This goes unnoticed for a long time because there are no messages and everybody assumes builds are going fine (we don't check buildkite status, only when there is broken build notification in slack).

Would be nice to receive notifications for stuck builds (e.g. into the same notification channel as configured for build failures) after some configured time (say 2h or smth). Also I note builds get autocanceled after 3 days - this can work too if only it was possible to configure the autocancel period and get notification for autocanceled builds.

jufemaiz commented 6 years ago

Ref: https://github.com/buildkite/feedback/issues/181 for visual cues too.

gtirloni commented 6 years ago

The build should fail after timeout_in_minutes and a notification is generated. Would that fix the issue with stuck builds?

ygrek commented 6 years ago

Where is timeout_in_minutes configured? I saw build getting auto canceled after 3 days, but there was no notification afair

gtirloni commented 6 years ago

It's currently a per-step configuration: https://buildkite.com/docs/pipelines/command-step#command-step-attributes.

There's a feature request for a per-pipeline setting (if I understood it right): https://github.com/buildkite/feedback/issues/170

ygrek commented 6 years ago

Yes, this request is an exact duplicate. I will double-check that I am receiving notifications in case of timeout. Thank you!