search

cloudfoundry / cf-deployment

The canonical open source deployment manifest for Cloud Foundry
Apache License 2.0
294 stars 306 forks source link

Questions about pipelines/cf-deployment.yml #73

Closed Amit-PivotalLabs closed 7 years ago

Amit-PivotalLabs commented 7 years ago
  1. Could domains like hermione.cf-app.com be parameterized, to make it easier to know that the string only needs to be changed in one place?
  2. Why do some tasks come from https://github.com/cloudfoundry/runtime-ci and some from https://github.com/cloudfoundry/cf-deployment-concourse-tasks, especially things like uploading stemcell?

/cc @fushewokunze-pivotal

cf-gitbot commented 7 years ago

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/140168053

The labels on this github issue will be updated when the story is started.

dsabeti commented 7 years ago

Hey @Amit-PivotalLabs.

I'm assuming you're talking about pipeline configuration from runtime-ci. It sounds like you'd like to re-use this pipeline configuration for your own concourse, but I don't think we've ever considered this yaml file to be part of a product interface. I'm hesitant to treat it like one since each team has pretty different needs, but it would be great if you could elaborate on what you need from the RelInt team -- we may be able to help some other way.

Here are the specific answers to your questions. They might help explain why I'd rather not productize this pipeline yamls:

  1. We could try to parameterize the pipeline, but we'd have to do it once for each domain that we use. If, in the future, we decide to add a 4th environment to our pipeline, we'd parameterize that, too, and anyone consuming pipelines/cf-deployment.yml would have to deploy the new environment, and fly set-pipeline with a new variable.
  2. cf-deployment-concourse-tasks is the repo where we try to keep tasks that other teams will likely use (things like bosh-deploy and upload-stemcell-from-cf-deployment). By and large, tasks in runtime-ci are tasks that we don't think other teams would benefit from. The runtime-ci task that you cited (for uploading stemcells) is used only for deploying the windows cell, which is something we actually probably should ship in cf-deployment-concourse-tasks. Let me know if there are any other tasks that you think would be of use more widely.
dsabeti commented 7 years ago

Hey @Amit-PivotalLabs, I'm going to close the issue, but feel free to re-open if you have more questions.

Amit-PivotalLabs commented 7 years ago

Don't think you should treat the pipelines as being products for others to consume, I agree with you there.

For your own purposes, DRYing up "hermione.cf-app.com" would save anyone on your team from having to know all the correct places to change that value. When trying to make a whole bunch of changes, it was easy to rely on concourse complaining that some variable is missing (which we'd get if it were a variable) and harder (marginally) to make sure we remembered to change that value everywhere. It's a small cognitive overhead in isolation, but when we were trying to make a whole bunch of changes to easily adapt it to our needs, it adds up a little. Overall the ability to take the pipeline and chop it up to suit our needs was very very smooth.

The other task worth sharing would be run-bosh-cleanup. Everyone's pipeline is eventually going to break because their director will eventually fill up, unless they run this.

dsabeti commented 7 years ago

I see. I think that might make sense. @anEXPer, any thoughts on this point about parameterizing non-credential values as a way to DRY up config?

Also, here's a story for running bosh cleanup: https://www.pivotaltracker.com/story/show/141382703

anEXPer commented 7 years ago

I'd sooner use anchors than variables for this. Our variables are private. Anything we put in them is hidden. I'd rather hide as little as possible about our pipelines. Also, it reduces dev load to have similar rules about vars for our pipes and our bosh manifests.

Honestly, I'd sooner use find and replace than anchors, too, unless this becomes painful for some reason.

I've seen great complication arise from unnecessary parameterization.