Support for destroying pipelines

[vito]

We should ultimately add delete to the resource interface for this, but until then it'd be nice if put supported destroying a set of pipelines. Probably from a file listing their names.

The main use case for this is automating Concourse pipeline lifecycles based on Git branches via https://github.com/vito/git-branches-resource - which detects new and removed branches.

[robdimsdale]

Totally agree. I'd much rather see the delete implemented first though, otherwise the API will break when switching from the files-style to the delete.

What's the roadmap for delete? I'm excited to see it implemented in other resources too.

[vito]

Just dragged it up.

[vito]

It'll take us some time to reason about delete, both in the resource semantics and in things like how it affects the UI (e.g. what happens when versions that were previously detected/produced become deleted).

[robdimsdale]

Ok. I'm happy to make a new binary under /opt/resource/delete so we can iterate on the API and the integration with concourse. Can you sketch out what you'd like the input (stdin) and output (stdout) from/to the ATC to look like? That will allow me to get started on this.

[miromode]

+1 Can't wait for delete functionality - any ideas on the roadmap for this?

[robdimsdale]

I have no idea where this falls on concourse's roadmap - @vito ?

[miromode]

For the time being, I just implemented a present: <bool> parameter for each pipeline (in our forked code of this repo). In retrospect, maybe I'm spending too much time in idempotent configuration management scripts.

- name: my-old-pipeline
  team: my-team
  present: false

Not sure if you'd be interested in implementing something similar for your codebase until the Concourse team picks this up, I'm positive you can come up with something better, but at least it works within the confines of the current put-based architecture.

[robdimsdale]

Your solution makes sense. However, I would prefer not to implement anything before concourse defines an API for resources to delete, because whatever interface we come up for this resource will be different than that, and this will result in a breaking change to this resource's interface. I'd rather not provide a feature we know we intended on removing.

[mzelinsky]

@vito are there any plans for adding resource delete functionality to concourse?

[mzelinsky]

@robdimsdale giving no progress in implementing delete functionality in resources, do you think it's possible to implement the workaround @miromode mentioned? I don't think this workaround will break in a year or two...

[robdimsdale]

I don't have a lot of bandwidth right now, so I'm open to PRs.

One design for this feature might look like adding a destroy boolean field to the pipeline configuration. This would make the config_file field optional (though one of either destroy or config_file must be provided). The naming of destroy mirrors the fly destroy-pipeline command.

I'm open to suggestions on whether this should error or succeed if the pipeline/team combination doesn't already exist. It appears that the fly CLI exits without error so I would suggest the same should be true here.

Under this proposal static configuration would look like the following:

jobs:
- name: set-my-pipelines
  plan:
  - put: my-pipelines
    params:
      pipelines:
      - name: my-pipeline
        team: team-1
        destroy: true

dynamic configuration would be unchanged but, following the existing convention, the contents of the files would be the same as the static configuration above i.e.:

---
jobs:
- name: set-my-pipelines
  plan:
  - put: my-pipelines
    params:
      pipelines_file: path/to/pipelines/file

@miromode @mzelinsky what do you think? Are either of you up for submitting a PR to add this?

[miromode]

@robdimsdale we already have a working and proven version (trivially different with present: false instead of destroy: true) - I'll see about getting a PR together in the next day or two.

[jtarchie]

@miromode was there any update to this?

[miromode]

@jtarchie I'm not sure - https://github.com/concourse/concourse-pipeline-resource/pull/16 got rejected by @chendrix so I assumed that was that. Since then I've written a new version of this same logic - that we use here internally - on top of the recent major code refactor. I can do a new PR for that one if this sort of workaround is still acceptable.

[chrishiestand]

Was there a reason given for the #16 rejection? I don't see it on the PR.

[vito]

@chrishiestand Sorry, really not sure why we closed it without giving a reason... Only reason I can think of was to wait for explicit support for delete actions in the resource interface, but that was a long way off at the time of closing it.

[joaosa]

This would be a really useful feature to have :)

[vito]

FWIW I think the v10 roadmap outlines something much better than this - notably, projects + instanced pipelines + archiving pipelines. In fact I'm hoping the v10 roadmap puts us on a path towards deprecating this resource altogether.

[eedwards-sk]

I know it's great to be optimistic about the new resources ideas and proposals, but it's still going to be a while until we see them.

Can we please look at https://github.com/concourse/concourse-pipeline-resource/pull/16 again?

This issue has been open for over 3 years -- I have the same use case as #16 -- what do we have to do to get it over the line?

[eedwards-sk]

Looks like #16 is pretty out of date. I've forked the repo, I'm porting it to 1.13 as a module, and will be seeing if I can get the delete PR ported to the current design.

[vito]

@eedwards-sk Sweet, thanks for picking this up! We're pretty reliant on community contributions with our own resources at the moment, despite them living in the concourse org. 🙁

[joaosa]

@eedwards-sk any updates? It'd be really great to have this :). Thanks!

[eedwards-sk]

@joaosa Sorry, I gave up on this after it became clear the repo isn't being actively maintained. I've since then switched to GitHub Actions to avoid all the problems with branch discovery in Concourse -- which was the only reason I needed this feature.