search

llvm / llvm-iwg

The LLVM Infrastructure Working Group
https://foundation.llvm.org/docs/infrastructure-wg/
Other
17 stars 14 forks source link

Webhook for Bazel Buildkite pipeline #55

Closed GMNGeoffrey closed 3 years ago

GMNGeoffrey commented 3 years ago

As part of adding an unsupported Bazel configuration, I've set up a Buildkite pipeline that runs the Bazel build on the main branch. Currently it's configured to run every 10 minutes (the shortest cron Buildkite allows), but it would be much nicer to have it instead trigger on pushes to main via webhook. Adding a webhook to a repository requires admin access. I don't think it has any real security implications for a public repository. Is this something someone could help me add?

Note that since this is an unsupported build, I'm not posting statuses on the GitHub commits.

Side note: I wish GitHub had a pubsub model so anyone could subscribe to updates to a public repository. Would also make automatically syncing forks much easier

ChristianKuehnel commented 3 years ago

@tstellar @asl do you have the permission to set that up?

This is probably also related to #40 .

tstellar commented 3 years ago

How are the existing buildkite jobs configured?

GMNGeoffrey commented 3 years ago

I didn't set any of those up, but just poking around the settings: The one building the release branch https://buildkite.com/llvm-project/llvm-releases appears to have webhooks set up, which you can see by looking attaching "/webhook" to a build: https://buildkite.com/llvm-project/llvm-releases/builds/1104/webhook.

The one that runs on the main branch appears to be on a 4 hour cron (https://buildkite.com/llvm-project/llvm-main/settings/schedules). The ones doing pre-merge checks are pointed at the https://buildkite.com/llvm-project/premerge-checks/settings/repository repository and doing some kind of triggering from phabricator that I know nothing about (but probably https://buildkite.com/docs/integrations/phabricator)

The native Buildkite GitHub integration has one webhook URL per pipeline though, so this would be a separate webhook.

tstellar commented 3 years ago

It seems strange that you would need a new webhook for each pipeline, is there any way to configure it so the entire buildkite instance shares the same webhook?

GMNGeoffrey commented 3 years ago

I think that's a result of the way the Buildkite models things. Each pipeline can potentially be for a different repository (or even different source control system) and so the integrations are per-pipeline, not per-organization. Buildkite has the ability for one pipeline to trigger another: https://buildkite.com/docs/pipelines/trigger-step, but if I'm reading the docs right this would require that the author of a commit themselves have permissions to trigger pipelines, so having a single pipeline that redistributes webhooks wouldn't work. I can talk to support to see if there's another way to have one webhook trigger multiple pipelines, but I'm pretty sure there's not a configuration setting or anything like that (because we have quite a few Buildkite pipelines in my project and I was annoyed about having to add the webhook each time). From the documentation:

If you want to use the same repository in multiple pipelines (including pipelines in different Buildkite organizations), you need to configure a separate webhook for each pipeline.

https://buildkite.com/docs/integrations/github#using-one-repository-in-multiple-pipelines-and-organizations

tstellar commented 3 years ago

@GMNGeoffrey Ok, what's the webhook URL? If you have the ability to customize the URL it would be great if you could make it self-documenting: so that it has bazel in its name.

GMNGeoffrey commented 3 years ago

The setup instructions including the payload are at https://buildkite.com/llvm-project/upstream-bazel-rbe/settings/setup/github. You should be able to see it since you're a member of the Buildkite org. I'm guessing there's a reason Buildkite hides it behind auth, so not posting it directly here.

Unfortunately, there doesn't seem to be any customization available. I'll email Buildkite support with this set of questions and see if we can work out something nicer longer term. I agree that one UUID webhook URL per pipeline is pretty annoying...

tstellar commented 3 years ago

@GMNGeoffrey I've added the webhook. Let me know if it is working.

GMNGeoffrey commented 3 years ago

It worked: https://buildkite.com/llvm-project/upstream-bazel-rbe/builds/977/webhook. Thanks!

GMNGeoffrey commented 3 years ago

I'm still in conversations with support about how to have pipelines trigger each other, but one option that might reduce the manual maintenance here would be the one for auto-creating webhooks that I just discovered:

https://buildkite.com/changelog/108-webhooks-can-be-auto-created-for-github-repositories