Pipelines for multiple repos

[ineffyble]

It would be great to be able to define a pipeline with a set of jobs, but have that work for multiple repos.

For example, a simple pipeline that ran a syntax check and style guide check against all our PHP Drupal modules. Because the alternative is creating a pipeline per repo, for 50+ repos, all to run the same basic command.

Alternatively, the ability to have a pipeline defined purely via YML in the repo (with a generic webhook URL), so that there's no explicit need to "create" something for each repo.

[lox]

This sounds like exactly what we built plugins for! https://buildkite.com/docs/agent/plugins

[GaryPWhite]

Hey everyone, I know there's plugins to make this happen, wondering if there's a recommended one or if this feature has moved at all

[lox]

Hey @GaryPWhite can you tell us a bit more about what you are trying to do? There are a ton of ways to support this currently, I don't think we have anything else planned in this space.

[rmnl]

I'm following up on this issue, because I cannot find a plugin that would do what I'd like to do, but perhaps I'm looking in the wrong direction.

I'd like to have one generic pipeline that checks code formatting and runs linters for all of our repositories. This single pipeline has one Buildkite webhook URL that could be added to the Github webhooks for all repositories that require these checks. Ideally the Buildkite pipeline would then use the Github repository that is specified in the webhook's payload.

I'm having trouble trying to get this to work with plugins because the repository setting is a required and fixed pipeline setting.

I can use the checkout hook to prevent the initial checkout and then use commands with environment variables like BUILDKITE_PULL_REQUEST_REPO to checkout the correct repository, but reporting statuses back to Github is another issue. This behavior is also specified in a separate tab of pipeline settings (and it uses the initially specified repository not the one in the webhook payload).

Is there a way to do this with buildkite and if so, what would be it?

[toolmantim]

👋🏼 A quick update on this issue: as of Buildkite Agent 3.7.0 (released 20 Dec 2018) you can now modify the BUILDKITE_REPO environment variable (https://buildkite.com/docs/pipelines/environment-variables) using an environment or pre-checkout agent hook (https://buildkite.com/docs/agent/v3/hooks) — you're able to dynamically switch the repo value depending on any of the information available in the build environment. And if you want to skip checkout altogether, you can export BUILDKITE_REPO=''.

[wgroenewald]

👋🏼 A quick update on this issue: as of Buildkite Agent 3.7.0 (released 20 Dec 2018) you can now modify the BUILDKITE_REPO environment variable (https://buildkite.com/docs/pipelines/environment-variables) using an environment or pre-checkout agent hook (https://buildkite.com/docs/agent/v3/hooks) — you're able to dynamically switch the repo value depending on any of the information available in the build environment. And if you want to skip checkout altogether, you can export BUILDKITE_REPO=''.

I'm trying to do something similar, using one pipeline for multiple repos. I can disable the initial checkout, but I'm not quite sure how I would get the payload of the webhook to ensure that the BUILDKITE_REPO value can be set to the repo which triggered the build. Any ideas?

[mallyvai]

^ I'd be interested in this too. The use-case is common linters (think security, style...) that can be applied to repos across the org, even ones that are not currently using Buildkite.

[sj26]

how I would get the payload of the webhook

I think this is a blocker. I've added a feature request to our forum here: https://forum.buildkite.community/t/access-webhook-which-triggered-build-from-command-steps/1378

[poponuts]

Any updates on this? I need a codebaae repo to deploy to an environment then run the tests sourced from another repo within the same pipeline. Bitbucket pipelines and other CI’s are able to achieve this.

[ThomasHemery]

Somewhat necro-ing this thread but I am currently looking into a similar issue. At my company we have a number of docker based solutions that are built, tested and deployed via gitlab-ci. We have a single repository containing all the gitlab-ci definitions for re-usable build, test and release steps, the projects themselves only contain a single config file that pulls in the required files from the central, re-usable repository.

We are now migrating to github and looking for an alternative CI solution. Buildkite looks promising but I am unable to find a good way to achieve a similar setup, i.e. having one central repository containing pipeline definitions that can be re-used in many downstream projects. trigger and buildkite-agent pipeline upload both offer elements of the behaviour I need, but each has limitations that I think prevent them working. I cannot trigger a pipeline without having that pipeline run only in it's own repository, I cannot circumvent this it seems. I also cannot upload a pipeline yml file from anywhere but the local files. Support for uploading a yml file from a remote repo would be a swift and very helpful solution to this problem.

Is there any other way to achieve what I am looking for? Without creating a public plugin? Our code is closed source and we most likely having our pipelines be open publically would not sit well with management. We also cannot afford the extreme duplication / complexity that would come from having the pipelines be replicated in all of our downstream repos.

[mallyvai]

Tom - If you don't want a public plugin, it's possible to easily create private plugins from private repos that live in your github org. You just need a specific naming scheme on the repo name. The BK docs have recipes around this.

Sent from Android

On Wed, May 17, 2023, 12:28 PM Tom @.***> wrote:

Somewhat necro-ing this thread but I am currently looking into a similar issue. At my company we have a number of docker based solutions that are built, tested and deployed via gitlab-ci. We have a single repository containing all the gitlab-ci definitions for re-usable build, test and release steps, the projects themselves only contain a single config file that pulls in the required files from the central, re-usable repository.

We are now migrating to github and looking for an alternative CI solution. Buildkite looks promising but I am unable to find a good way to achieve a similar setup, i.e. having one central repository containing pipeline definitions that can be re-used in many downstream projects. trigger and buildkite-agent pipeline upload both offer elements of the behaviour I need, but each has limitations that I think prevent them working. I cannot trigger a pipeline without having that pipeline run only in it's own repository, I cannot circumvent this it seems. I also cannot upload a pipeline yml file from anywhere but the local files. Support for uploading a yml file from a remote repo would be a swift and very helpful solution to this problem.

Is there any other way to achieve what I am looking for? Without creating a public plugin? Our code is closed source and we most likely having our pipelines be open publically would not sit well with management. We also cannot afford the extreme duplication / complexity that would come from having the pipelines be replicated in all of our downstream repos.

— Reply to this email directly, view it on GitHub https://github.com/buildkite/feedback/issues/238#issuecomment-1551716943, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAIXOT5J2QPWV6IA6Z7HYDXGT4EXANCNFSM4DFDX5UA . You are receiving this because you commented.Message ID: @.***>