Trigger job with custom parameters

[cameron-r]

Feature Request

What challenge are you facing?

I want to build a pipeline that has a deploy step that can deploy to many different environments, eg:

build image -> test image -> deploy

where the 'deploy' job can deploy to one of several environments. We have a bunch of servers running that different teams use (around 20). Currently, it seems like with concourse, to be able to quickly deploy to any one of those 20 environments, we need to create 20 separate deploy jobs, which is not the ideal (especially when I think about having pipelines for multiple microservices as we build them out).

A Modest Proposal

The ideal for us would be if when someone kicks off the 'deploy' job, there would be a UI prompt for them to enter what environment they want (eg 'environment-12') and that gets passed to the appropriate task

[concourse-bot]

Hi there!

We use Pivotal Tracker to provide visibility into what our team is working on. A story for this issue has been automatically created.

The current status is as follows:

• [ ] #134606085 Trigger job with custom parameters

This comment, as well as the labels on the issue, will be automatically updated as the status in Tracker changes.

[k0s]

So I've thought about this somewhat (I'm not one of the concourse developers, for the record, just an avid user) and while I certainly want this ability, I can also see why its absence could be considered a good practice.

Ultimately variations in parameter space (e.g. setting a branch) is a different build dimension than what a (static [*]) pipeline does. A build and build artifacts is effectively, albeit intentionally, altered. Consider CI that does allow this (Jenkins, buildbot, etc). In general builds and artifacts go into the same pool when invoked with user-defined parameters as when triggered in "the usual way". This makes it difficult to automatically promote artifacts or introspect build history (at least in my opinion).

While I cannot discern your particular use-case, I would tend to go about it in a manner that:

• there is an artifact or hard-coded list of environments that require deployment to
• there is a single job
• two options:
  1. deployment can be made idempotent and quickly so; in this case just deploy to all environments to ensure concurrency
  2. alternately, you can check to see if an environment is up to date and only deploy to that environment

(Actually I may just have the twenty jobs and make it easy to recreate / update the pipeline as needed, just for easy visual audit.)

My apologies if your problem is more complex and I'm inferring too much. That's entirely possible. I would hope that the existing mechanisms of concourse could be leveraged to achieve what you want. And again, I don't speak for the concourse team.

---

[*] this assumes a static pipeline in concourse. Of course, concourse pipelines can be reconfigured which represents yet another dimension that I would love to see concourse account for.

[cameron-r]

Yup, I definitely agree that allowing users to put custom parameters into builds could facilitate a lot of not-so-great patterns. It is, however, the existing 'common' solution in other CI/CD tools for the problem I want to solve with Concourse.

Just addressing your suggested work-arounds:

Deploying to all environments: We don't want to deploy to multiple environments at the same time, we kind of want them to be isolated from each other for now (there are multiple teams working on different projects, and we don't want project A's changes to disrupt project B)

Write 20 jobs: 20 jobs is not ideal since maintaining them would be more difficult (either by hand, or by creating some custom script to generate the jobs). Even from a UI perspective, I don't want to find my one job out of 20 each time, or clicking through each of them when I only care about who deployed in the past 2 hours (or something).

Maybe there is a solution in having a specific option you can give to Resources that tells them to prompt the user for input if there's a Put in that step. I'm trying to think of something that solves my problem, but still constrains users enough to discourage using it for bad patterns.

[brainsik]

I have a similar use case I'm trying to solve. A client of mine uses Jenkins to deploy new development environments. Those environments are created via AWS CloudFormation stacks which take a username parameter that's used to set a DNS record pointing at the host which will be created. These are specifically deployed ad-hoc / on-demand so people can blow away and get new environments whenever they want / need them. There are quite a few people at this company, so a pipeline per person is not a great option.

One workaround that seems possible is to have a task with params and then use fly to run the task with the parameter set. This feels pretty hacky though as it's not tracked in a visible way. Nothing is in the UI and running fly builds just shows them as "one-off" without any other detail.

Perhaps a way to alleviate some of @k0s's concerns would be if these jobs could only exist as end nodes in the graph. Meaning, nothing else would triggered, output, etc. from them.

[holycheater]

Yeah. That would be nice. Because if I have for example a set of pull requests as resources. I build&test each pull-request in a job. And then I have another job where I optionally want to deploy specific version (specific pull-request) to some staging area.

[chendrix]

Have you tried using the pool-resource to model your environments?

Alternatively can't you use input_mapping to use the same task step but specify a different environment to deploy to?

The concourse team (and many many Cloud Foundry teams) uses the pool resource when choosing one of several different environments to deploy to.

[tmszdmsk]

We have a similar use case. We are considering moving from Jenkins to concourse but we use parametrised builds in dev related jobs (e.g. deploy this branch on this dev env). I think that having a resource that can be triggered from web ui/fly tool with a set of predefined parameters that will be stored on out as files would be great.

[chendrix]

Looks like something also desired in #903

[clarafu]

@cameron-r I'm also wondering if you can commit the environment name to a git resource and have the deploy job only triggered off that git resource and read the environment name from it. That way you can keep a history of which environments have been triggered and it will have a somewhat similar flow to a UI prompt.

[pieterza]

Any update on this? We also have this requirement.

[dadacafe]

+1 for this request. We also have a need for such feature.

[geofffranks]

Our use case for this is to provide build-time credentials tied specifically to a real person running the build, for auditing of who deployed an application to production. Having this feature would really help us continue to use concourse for our CI/CD, rather than be pushed into using Jenkins for our deployments.

[mustafaakin]

We have this requirement also and willing to contribute

[ringods]

One of the redesigns in progress is around Resources v2. Part of this proposal is resources spaces, originally tracked in concourse/concourse#1707. Spaces are the generic solution to support matrix builds.

I was thinking that with a proper design, manual input could be one of the different spaces which could be supported. Just tying one idea to another... 😃

[res0nat0r]

I'm essentially looking to do the same. Right now we use Jenkins parameterized builds to let our developers use our large Rake code to launch new environments like:

bundle exec rake "launch_environment[${COOKBOOK},${ENVIRONMENT},${ENV_NUM}]"

bundle exec rake "launch_environment[web,prod,101]"

I would like to be able to replicate this in concourse by passing the rake inputs via trigger-job, something like:

fly -t concourse trigger-job ops/launch_environment -v COOKBOOK=web -v ENVIRONMENT=prod -v ENV_NUM=101

[spmsaj]

curious, is this a possibility ( ability to trigger job and pass new variables as part of it?)

[LunaGao]

I need this also.

[jeffreylim17]

Any updates on this one?

[luwang-vmware]

Concourse is a brilliant CI and it has easy simple interface to create a pipeline for testing. I like it. In our daily work, I realize that supporting custom parameter in build triggering is very important for us. It could give developer/tester more flexibility. Supporting custom parameter with GUI or fly cli are both ok. Hope to see it soon.

[stale[bot]]

Beep boop! This issue has been idle for long enough that it's time to check in and see if it's still important.

If it is, what is blocking it? Would anyone be interested in submitting a PR or continuing the discussion to help move things forward?

If no activity is observed within the next week, this issue will be exterminated closed, in accordance with our stale issue process.

[geofffranks]

Still looking forward to this 🤞

[dfuentes77]

This is really needed

[dfuentes77]

Have you tried using the pool-resource to model your environments?

Alternatively can't you use input_mapping to use the same task step but specify a different environment to deploy to?

The concourse team (and many many Cloud Foundry teams) uses the pool resource when choosing one of several different environments to deploy to.

Can anyone show an example of this for the purposes of deploying to multiple environments?

[stale[bot]]

Beep boop! This issue has been idle for long enough that it's time to check in and see if it's still important.

If it is, what is blocking it? Would anyone be interested in submitting a PR or continuing the discussion to help move things forward?

If no activity is observed within the next week, this issue will be exterminated closed, in accordance with our stale issue process.

[freelock]

Hi,

Like other commenters, we would find this useful for provisioning development copies for individual developer use.

As a workaround, I've been thinking of using the Matrix Notification Resource for this -- right now it's mainly used to push messages into Matrix from Concourse from build jobs, but Matrix does allow storing state in rooms, and our bot already manages this state as well as triggers Concourse jobs. We could set it up so that Concourse could read state from a Matrix room, as a resource to load info inside a job -

However, we currently use the Matrix Notification resource only in a single Matrix room as a sort of "message bus" -- our jobs post results there, and our bot relays the data and messages to project-specific rooms. So it would be quite a bit simpler for us to have our bot pass a parameter to the Trigger Job it already does, than to add the Matrix resource to a bunch of rooms and have it read/write state directly...

[dfuentes77]

I wholeheartedly agree. The simplicity of just passing a parameter in the UI to a job and have it run, for individual and unique PR based deploys, would be extremely beneficial to us.

[geofffranks]

+1 again

[jamieklassen]

instanced pipelines is a pretty interesting abstraction that could, with the right UX, cover the use-cases described here pretty nicely.

[stale[bot]]

Beep boop! This issue has been idle for long enough that it's time to check in and see if it's still important.

If it is, what is blocking it? Would anyone be interested in submitting a PR or continuing the discussion to help move things forward?

If no activity is observed within the next week, this issue will be exterminated closed, in accordance with our stale issue process.

[geofffranks]

+1

Sent from my iPhone

On Jan 25, 2020, at 6:01 PM, stale[bot] notifications@github.com wrote:

 Beep boop! This issue has been idle for long enough that it's time to check in and see if it's still important.

If it is, what is blocking it? Would anyone be interested in submitting a PR or continuing the discussion to help move things forward?

If no activity is observed within the next week, this issue will be exterminated closed, in accordance with our stale issue process.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

[dfuentes77]

+1

[Xercoy]

+1

[mkrish004c]

+1

[v3nturetheworld]

+1

[dhilgaertner]

+1

[drewhemm]

A CI system that doesn't support parameterised builds!? We are currently running a POC for migrating to Concourse from Jenkins and I don't want to have to create n versions of a job to cater for n parameter permutations...

[drewhemm]

For additional context, a task runner can be used for many other things than compiling source code and packaging applications. There are many use cases for just doing x, and x needs to be doable in different environments...

[jamieklassen]

@drewhemm if you don't need the full auditability of "doing x" being linked to to an immutable history of versions (which you might not, if you're not compiling source or packaging applications), you could use fly execute. Especially true if the subtext of "just doing x" could meaningfully include "... as a one-off", which it might given your use of the more-generic term "task runner".

Concourse is pretty opinionated about being stateless and auditable, and it's tricky to satisfy these constraints while allowing parameterised builds in the way that most people are familiar with. However, feedback similar to yours has been flowing in for a long time, so it's a goal that is being steadily worked towards, as discussed pretty thoroughly in this blog post and this RFC.

Jenkins vs Concourse is always an interesting comparison, but based on your interrobang I get the sense that you're not doing it for 'basic research' purposes. I don't think Concourse has an explicit mandate to compete with Jenkins; the niche it fits is a bit more about people who are planning to implement a few strongly held opinions about automation. Those opinions tend not to be held quite as strongly in other CI systems, so Concourse winds up distinguishing itself in the CI use case.

[drewhemm]

Thanks @jamieklassen I'll definitely take a look at fly execute.

All we need really is the ability to run tasks, sometimes manually with parameters and record those parameters entered and the log of the job; this is for auditing of actions performed against an environment, primarily with Ansible. There are other things we will be doing Concourse, such as building AMIs and Docker images etc, that are probably closer to the standard use-cases.

So far this week, we have got the telia-oss github-pr-resource setup for validating our PRs, along with pipelines that use static inputs and we're loving it so far. It's quite different from Jenkins, but so far in good ways. Being able to update a pipeline using fly without having to first commit a Jenkinsfile and trigger a job you know will fail just so it can pick up the updated Jenkinsfile is awesome. We're about three to fours weeks into the POC project.

[pieterza]

A CI system that doesn't support parameterised builds!? We are currently running a POC for migrating to Concourse from Jenkins and I don't want to have to create n versions of a job to cater for n parameter permutations...

Do yourself a favor, and stay on Jenkins. We've gone full circle, as you'll see by my more than 2 year old comment on this very thread which I'm now finally unsubscribing from.

[vito]

If your workflow is inherently dependent on manual, human-provided parameters, then yes, you may want to use some other tool like Jenkins. User-provided input is antithetical to what Concourse is designed for, which is to be continuous, driven by a declarative configuration that runs in the background, triggered by external systems that act as the source of truth for all inputs.

Concourse is not designed to be an interactive workflow driver. It is not trying to be everything to everyone.

However, if you are able to model what you're trying to do in a way that can be automated and driven by an external source of truth (i.e. in a Concoursey way), then there are many RFCs in the works that can provide a path forward, all of which are outlined on the v10 roadmap, which was posted a year ago.

Namely, the following RFCs provide new and interesting ways to parameterize workloads in a pipeline:

• RFC #39 adds pipeline-level var sources, which are a bit like resources but for ((vars)).
• RFC #37 adds a 'Prototype' interface that can be used to implement arbitrary ways for var values to enter into a pipeline (from an external system)
• RFC #27 adds a get_var step which can trigger the build when the value for a var changes
• RFC #29 adds an across step which can run a step across all vars
• RFC #34 introduces "pipeline instances" which is a way of having a pipeline associated to a var key/value

Designing a system like Concourse takes time. It involves a ton of iteration based on detailed user feedback and case studies, and it requires a lot of second-guessing and redesigning in order to stay small enough to fit in your head. The road to v10 is paved with failed inventions. Providing feedback on RFCs is incredibly important for us to be able to move forward more quickly and with higher confidence.

I'm going to close this issue, as the feature it requests is not likely to be implemented as requested, and Issues are meant for fully validated backlog work (per this update). Thanks all for the feedback, and sorry this took so long to be closed - I left it open while I was working on a revision of RFC 27 that theorized on allowing arbitrary input, but I ultimately decided against it. Unfortunately I left this issue in limbo the whole time that I was stuck in analysis paralysis.

Based on the emoji reactions, there are clearly many people that wanted this sort of thing over the years, but in conversations I've had with other Concourse users I've learned that there are also many who don't want this feature implemented and would rather Concourse stick to its current ethos of being the "dot connector" and not the "dot".

The v10 roadmap will enable new and interesting patterns around passing parameters to builds and pipelines, and I think it would be wise to wait and see how that plays out first. I would more than welcome new Discussions around this in the future, especially as more of v10 becomes implemented.

[bukowa]

Concourse is designed for, which is to be continuous, driven by a declarative configuration that runs in the background, triggered by external systems that act as the source of truth for all inputs.

What if the truth (the vars) are declared in the pipeline file and we can choose from them before the pipeline runs?

[ghost]

This decision kills any hopes of using Concourse in place of Jenkins for us.

I appreciate standing firm on project objectives though in this instance it seems a little petulant and moralistic. The decision to not implement useful features such as triggers and parametrised jobs leaves this software in a no-mans land as far as Im concerned - neither a good fit for one thing nor another.

The concourse project may remain pure in its ambition but that means little when so many users have come and gone.