Wait for in progress github checks before merging

[styfle]

Some repositories use GitHub Actions to dynamically create jobs based on what files changed.

In that case, we can't set those checks to "required" because it would block PRs that don't create those jobs.

However, this caused Kodiak to permaturally merge a PR while the checks were still in progress.

It would be great if there was a way to say "don't merge while any checks are in progress".

[chdsbd]

Hey @styfle,

Thanks for the suggestion!

On a separate project, we ran into a similar issue with GitHub Actions when using the "on.push.paths" configuration option. As a workaround, we're using https://github.com/fkirc/skip-duplicate-actions, which seems to work: https://github.com/sbdchd/squawk/blob/f8588d1daf1d09a57a1dc8d1efed49375ed0df32/.github/workflows/js.yml

We use CircleCI in Kodiak, which makes skipping jobs much easier and we have a small script to handle that: https://github.com/chdsbd/kodiak/blob/ea26b8cdf1f1e64a5e4bcaa71d10fda051b8b08c/.circleci/config.yml#L16-L19

Anyway, I think we could add an option to support not merging until all checks have finished, but wouldn't that mean if a job failed, the PR would still be merged?

[styfle]

Anyway, I think we could add an option to support not merging until all checks have finished, but wouldn't that mean if a job failed, the PR would still be merged?

Hmm good point. So maybe the new option should be something like "wait for optional checks to pass" in addition to required checks.

[chdsbd]

One catch I just realized is that the GitHub API only provides a list of required status checks, not optional status checks.

So there isn't a way for Kodiak to know the optional status checks unless we provide them manually.

Some solution would probably interact with the code here: https://github.com/chdsbd/kodiak/blob/5bae1332c6752d1a865072d6bbe99facef46a909/bot/kodiak/evaluation.py#L827