vito commented 7 years ago

Feature Request

What challenge are you facing?

When working on a project with parallel tracks of work, it does not intuitively make sense to integrate them into one single pipeline. Conceptually, a single pipeline is operating on versions of something going towards the same goal.

Pipelines are sort of the "top" of the Concourse level of semantics, so this raises the question of where I go from here. Once multiple branches are introduced, the needs vary based on the nature of the branches:

Feature branches

Feature branches are versioning different "products" until the point of re-integration. For example, a feature branch introducing authentication should not be versioned sequentially with other branches, even if they all forked from the same ref. That is to say, v1.3.0-rc.2, v1.3.0-rc.3, and v1.3.0-rc.4 should not alternate between having and not having a particular feature.

When starting a feature branch, it's likely that its pipeline would fit the same mold as its origin, except for things like deploying to production. It may also need its own staging/testing environment.

Pull requests

Pull requests are very similar to feature branches, with a couple key differences:

The reporting of its status back to the creator of the pull request.
Security risk in running potentially untrusted code from a pull requester.
Risk in overloading the Concourse cluster from pull requests submitted by a meanie poo.

This may call out the need to have a custom starting point for pull requests, rather than basing them on the "origin" pipeline. It may also call out the need for placing restrictions on the capabilities of their containers.

Release branches

A release branch is similar to a feature branch, with the following differences:

It will likely never be re-integrated, and might just be manually destroyed at some point.
It more strictly has the behavior of basing itself off of an original pipeline (the point at which the release is cut).
It more likely will stray from the master branch, and the differences may be more pronounced over time.
It might also want its dependencies to be "pinned" at the point of branching.
Images used for tasks and resource types might also want to be "pinned" in some way.

chendrix commented 7 years ago

One of the major reasons people ask for #413 is because they are using the PR resource and it does not currently play well with "spatial" versions.

edude03 commented 7 years ago

I'll share my use case for this feature below to future the discussion.

I'm looking to have an environment created for each PR opened on Github then have that environment torn down when the branch is merged. In my particular case, it would require cooperation between various different github repos and would look something like this.

User opens PR
Pipeline is triggered that builds a docker container and pushes it to Dockerhub.
A pipeline discovers that a new image was pushed to dockerhub and generates a kubernetes manifest with the image pointing to the newly built container.
The pipeline checks in the new manifest.
A pipeline detects that a new manifest was checked in and triggers a deploy.

All of this is currently doable with Concourse CI but the current infrastructure leaves a few gaps:

How can users see where in their pipeline their branch is (has it been deployed? did it build successfully?)
How do I keep the branches across the repos in sync, for example, if the PR is called "my-new-feature" how do I create a branch in the manifests repo called "my new feature"?
How do I retrigger the pipeline if something fails?

One thing I've considered is having a pipeline that generates concourse CI manifests for each PR so that users can see the status of their feature as it goes along the pipeline.

charlieoleary commented 7 years ago

Adding context in regards to #413.

We use PRs to merge short-lived branches back into master. All of our flows do utilize a single pipeline for running tests and builds along side the pull_request resource and it works well. The problem that we're having is when a test or build fails for some miscellaneous reason (this isn't always because of a code issue) there is no easy way to re-trigger. Unfortunately the only way to re-trigger is to make a commit, and often times this ends up being an empty commit. It's not particularly intuitive.

jtarchie commented 7 years ago

@charlieoleary, I'd say this use case still doesn't work for you. Even though you had a flakey test run on a feature branch pipeline, you wouldn't be able to re-trigger it. Join my campaign for #413.

-- Paid for by the committee that supports #413

chendrix commented 7 years ago

the committee that supports #413 doesn't understand Concourse fundamentals 😝

vito commented 7 years ago

@jtarchie @charlieoleary With each PR being a separate pipeline, the likelihood that you would need to re-trigger a particular build vs. re-triggering the job like you do today would be much lower. You would only need build re-triggering if for whatever reason you wanted to re-run an older iteration of the pull request.

So, you wouldn't need an empty commit to re-trigger the PR - you'd just re-trigger the job. And you'd also be able to have a full pipeline for running pull-requests, rather than cramming everything into a single job. There are many people using pull requests in a way that warrants having multiple stages, and perhaps environment management - it's a much bigger need than simply re-triggering a single build, and if we can make one significant camp happy while making a large portion of another camp happy (#413), that seems like a wise place to investigate.

I think there are legitimate arguments for #413, but re-triggering pull requests is not and should not be one of them. That's all we're trying to clear up.

charlieoleary commented 7 years ago

@vito I don't know if misunderstanding, but I definitely don't want separate pipelines for each PR. I have a full pipeline that runs my pull requests, and everything isn't crammed into a single job. Can you expand more or include some visuals / arch diagrams on what you're proposing here?

vito commented 7 years ago

@charlieoleary In today's model it makes sense that you wouldn't want separate pipelines. Pipelines are managed manually and configuring one for each pull request would be a pain. This issue is to explore alternative approaches to satisfy models like pull requests, feature branches, and release branches. This may mean having a pipeline-that-generates-pipelines as a Concourse abstraction, or some other primitive (or just UX) that caters towards this need.

So I'd ask, what exactly is it about having a separate pipeline for each PR that you don't like? Again, that's not to suggest that it should be "good enough" or even the recommended approach - we're in feedback collection mode here, so we are literally just looking for answers to questions like that.

Today's pull-request resource works okay-ish for one job, but beyond that it begins to quickly collide with Concourse pipeline semantics, in many more ways than #413 can solve:

Resources normally represent the same logical entity progressing incrementally through versions, but with pull requests the nature of what's even being tested or built changes from version to version. This is the root of basically all of the following problems.
There's no guarantee that every PR runs, as Concourse by default picks the latest version available. You have to add version: every to every get step to even make sure your job runs with them. It's a smell that a resource requires configuration outside of the resource itself to work properly.
Because what's being tested changes so dramatically from build-to-build, re-triggering with the latest version is not enough to resolve flakes. You would have to go to the build that ran against the PR and be able to re-trigger it with the same PR. Thus, many people running pull requests end up wanting #413.
Going beyond one job, things quickly become confusing. You'll have to apply version: every everywhere for progression to happen. Looking at the pipeline, the status from job to job will be meaningless, as it could be showing the status of different PRs.
You have to sacrifice parallelism, or you'll get bitten by https://github.com/concourse/concourse/issues/736#issuecomment-258926683.
Even if we did implement #413, re-triggering a build in the middle of the pipeline won't result in the rest of the pipeline running if it's already run with a more recent PR. This won't be fixed. version: every should never go back in time and test older versions than what's already been tested. That is the path to darkness.

So the tl;dr is that cramming multiple streams of work into a single pipeline simply does not make sense, and should not be the primary motivating factor for #413. The support for pull requests will be woefully inadequate for pipelines beyond a single job unless we do something more.

charlieoleary commented 7 years ago

@vito Thanks for the more in-depth explanation. We have enjoyed the single pipeline model because it's easy to find everything we need about builds, but with how you're describing it this does sound like it could be a better solution. It almost sounds like this this solution you could truly store pipeline info in a repo similar to a JenkinsFile, which currently isn't really possible. If each build for a repo followed a template-like pipeline, I could see this being very useful and also understand why it would be superior to #413.

cjcjameson commented 7 years ago

Sometimes teams / projects would agree that the threshold for a PR pipeline passing is lower than that for the master pipeline (because of long / expensive tests that should only be run after the code integrates)

codesuki commented 7 years ago

@vito is there any plan to implement 1 pipeline instance per commit? So that all, not only the most recent, commits are available for deployment? It would be great to have the UI show a miniature pipeline per commit / PR. That way it would be easier to track which commit was bad etc. Sorry if I missed an issue where this is discussed, but I couldn't find any clear direction for that. For me personally, that would improve concourse usability by a lot.

vito commented 7 years ago

@codesuki That sounds a lot like what we proposed in https://github.com/concourse/concourse/issues/413#issuecomment-311399474

codesuki commented 7 years ago

@vito Just read the comment. That sounds perfect. Is there a roadmap?

timrchavez commented 7 years ago

There seems to be radio silence on this issue. I was told that Pivotal was in the process of rotating engineers on/off Concourse which probably explains why, but still hoping for an update, anyway.

vito commented 7 years ago

@timrchavez I can and probably should write an entire blog post about the things our team has been going though as Concourse gets bigger and bigger. Things are looking a lot better now. The long and short of it is we've reached a critical mass of really important things we need to do, and this whole time have tried to tackle them as one big team, and it just wasn't working. The scope was too large and there was always something emergent or important to do - we had no breathing room for big features like this.

We tried many things to make this work but at the end of the day found that we were spreading ourselves too thin. So now we're trying smaller, more focused teams, basically splitting up and hopefully growing those teams now that they have smaller scope and are more able to onboard people effectively.

We're taking that process slowly and steadily, so it may be a little while before we can sink our teeth in to this issue - it lies in the "Core" team, which will be one of the last to split off. But I can at least say that it's the number one priority within that team, once we get there.

If you're curious, here's the plan for the teams and their focus:

(Annoyingly, GitHub makes organization-wide projects private, so a screenshot is the best that I can do.)

cdaringe commented 6 years ago

this is all still a little meta for me to grok, so... pictures! suppose we're doing concurrent dev the following:

a:master
├── b:new_feature
│   └── c:master
└── d:master
    └── f:new_feature

are we talking about a pipeline that automatically templates from the default, then looks to optimistically match against new_feature? if so, that would be raaaad! it would solve the need to run full system integration tests w/ new bits all the way up your software system

cforce commented 6 years ago

We justs have to drop Concourse from our tools list in favour to Jenkins for that feature ;(

Globegitter commented 6 years ago

I want to add a different nuance to the Pull Requests workflow: Phabricator does not have any upstream branches for Pull Requests - the pull requests (diff) is generated locally (pointed either against local or upstream master) and then a patch with some metadata is being sent to phabricator which displays the diff based on that. Furthermore it has a concept of "staging areas", where it pushes the state of the branch that the PR is based on to specified upstream repository as a tag in a specific, unique format. This tag is not based on any upstream branch. Phabricator also provides it's own endpoint to get a list of PRs.

Just thought it is worth adding this as additional input.

Seraf commented 6 years ago

Hello,

I would like to know if there is some progress on this feature ?

My goal is the same as people commented here : being able to create a new environment (or at least actions like that), when a new branch is opened (not especially tied to a PR), and being able to destroy it when this branch is deleted.

As the branch is currently something hardcoded in the pipeline, I don't know how it's possible to handle it ? Should it be a dynamically generated pipeline using https://github.com/concourse/concourse-pipeline-resource ?

It sounds a bit hard to maintain/overkill. Could be a really interesting feature as others CI implemented it already and then, it's a usecase our customers are giving us ...

Thanks !

marco-m commented 6 years ago

I think that this ticket has been opened before the "Spaces" concept (see https://medium.com/concourse-ci/sneak-peek-spatial-resources-d0eed9bb3fa, https://medium.com/concourse-ci/designing-for-space-in-concourse-3037344644c6 and the Space epic here on github).

I want to share an insight we got while operating Concourse that might help people in the meantime:

We support self-service pipelines per feature branch as follows. We have only one pipeline configuration per project/repo, that we call a template. It is parametrized with ((params)). Some params, the non security-sensitive ones, are provided by a YAML file stored in the repo and read with fly --load-vars-from. The security-sensitive ones are set with Concourse integration with a credential manager.

Among the ((params)), the trick is to set also the git resource branch name. We have a wrapper to fly that reads the current git branch, does some other consistency checks and instantiates the feature branch pipeline, naming it mix a mix of repo name and branch name. This is why we call it self-service, the developer does

git checkout -b foo
fly_helper set-pipeline

and the pipeline is there.

To reduce load, again we use ((param)) to make feature branch pipelines non polling the git repo, while the master branch pipeline polls it.

The teardown is done with a special reaper pipeline, that runs periodically, scans all the pipelines and destroys the ones for which the branch doesn't exist anymore (meaning that the branch has been merged).

vito commented 6 years ago

Thanks for chiming in @marco-m! Yeah, we forgot to give an update on this issue - it's led to concourse/concourse#1707 which we've been pretty public about in blog posts and such, but folks that are just following this issue may not have seen it yet.

We're working on this right now (see the epic/spaces label), with #2386 wrapping up soon. It'll take more time as we still have to finalize concourse/rfcs#1 and fully implement the new spaces-aware pipeline UI.

stale[bot] commented 5 years ago

Beep boop! This issue has been idle for long enough that it's time to check in and see if it's still important.

If it is, what is blocking it? Would anyone be interested in submitting a PR or continuing the discussion to help move things forward?

If no activity is observed within the next week, this issue will be ~~exterminated~~ closed, in accordance with our stale issue process.

vito commented 5 years ago

Woah, chill stale-bot!

I'll keep this issue open as a long-term goal in the Spatial Resources project. It doesn't feel right to close it since it's still capturing a valid need.

For anyone following this issue, check out Core roadmap: towards v10, which outlines our new strategy for addressing this feature.

avif commented 4 years ago

@vito It seems to me that now with the "across" feature out we can accomplish this (experimental & without UI support) - is that true?

If so, can you provide an example of a multi-pipeline configured to trigger on each commit?

Very excited for this, been waiting for 2+ years!

vito commented 4 years ago

@avif We're not quite there yet, but the pieces are coming together! The across step currently only supports a statically provided list of values. For true multi-branch/PR workflows we'll need that to be dynamic, pointing to a var source which represents the set of branches or PRs so that it may change at runtime as branches/PRs come and go.

I've added a section to the README tracking all the work: The road to Concourse v10

Once dynamic var sources/across step is done, running pipelines across all branches could be done by having a pipeline which sets a pipeline instance for each branch:

var_sources:
- name: branches
  type: git
  source:
    uri: https://github.com/concourse/concourse
    branches: feature/*

resources:
- name: ci
  type: git
  source:
    uri: https://github.com/concourse/ci

jobs:
- name: set-pipelines
  plan:
  - get: ci
  - set_pipeline: feature
    file: ci/pipelines/feature.yml
    instance_vars: {branch: ((branch.name))}
    across:
    - var: branch
      source: branches
      trigger: true

This build would trigger whenever the set of branches changes, and pipelines will automatically be archived when branches go away. (The details for this mechanic are documented here.)

Once Projects arrives (concourse/rfcs#32), the above would make more sense as a project living in the ci repo. That would look something like this:

var_sources:
- name: branches
  type: git
  source:
    uri: https://github.com/concourse/concourse
    branches: feature/*

plan:
- set_pipeline: feature
  instance_vars: {branch: ((branch.name))}
  across:
  - var: branch
    source: branches
    trigger: true

Hope this helps! Sorry it's taking so long - it took a while to figure out the right way to do this, but now we're making good progress.

avif commented 4 years ago

Thanks for explaining this, can't wait! And no worries about the delay, I love what you came up with.

kardashov commented 3 years ago

Hi. Are there any updates on this feature? Absence of multi-branch workflows is actually the only thing which stops my company from full migration of Bamboo workloads to Concourse. Thanks.

vito commented 3 years ago

@kardashov Still working on it - we're keeping the README updated: https://github.com/concourse/concourse#the-road-to-concourse-v10

aoldershaw commented 3 years ago

The var_sources-based solution is still a work-in-progress (and is the eventual goal here), but we're happy to report that many multi-branch workflows are now possible with the release of 7.4.0 (and in particular, the dynamic across step)! This approach makes use of some experimental features, and requires special-purpose resource types whose versions represent a list of values (e.g. git branches, GitHub PRs, etc.)

We have a write-up on how to achieve a "build, test, and deploy to staging" workflow across a dynamic set of feature branches: https://concourse-ci.org/multi-branch-workflows.html. This is just the tip of the iceberg as to what should be possible with some of the new "v10" features, and we're excited to hear about the workflows that you're all able to experiment with (and what problems you run into along the way)

We're also currently experimenting with this new model for PR-based workflows - you can take a look at the relevant job config here, and the set of PR instanced pipelines on our Concourse deployment here

concourse / concourse

Multi-branch workflows #1172

Feature Request

What challenge are you facing?

Feature branches

Pull requests

Release branches