Open afeld opened 7 years ago
We have a similar issue, but splitting into teams and thus separate pipelines comes with overhead and also some serious downside ( AFAICS ):
a) extra team means extra vault handling just for that case - re-setting all values for every single team, over and over again ( docker reg creds, maven creds, nexus creds )
b) Beside that, a more serious issue is, that AFAICS passed:[]
can only be used in one pipeline, "external" pipeline cannot use that. So job to job is very different to pipeline to pipeline, and thats what you are suggesting. since extra team means extra pipeline
We are currently doing the following things:
on commit
or on base docker image
changed builds. So all instance builds. _release is for all tag based / semver file based releases, which are basically triggered by manually bumping the version in the file / creating a git tag. Release and Staging do usually not have any fan in/fan out dependencies. That for now seems to be a concept to map things for us, but simply put, people work around a simple classification issue which can be fixed so horribly easy, that it hurts even looking at it. Even by the most simple implementation, being able to assign tags to jobs/tasks/pipelines and then use a JQL based "search" which you can save and put into the navigation would make you make any scenario you like. Also those, where you actually do need different views for the same "things" depending on your current "hat" ( release hat / staging hat / backtracing hat).
The approach i mentioned above does has a downside you cannot simply fix using the UI, and thats simply the structure / size of the pipeline.xml. Since you define
We though about utilizing YAML based includes here, but go-yaml does yet not support those, only PHP yaml. So we would need to implement that ourselfs, so a pre-set-pipeline task to compile the pipeline.yml and push it to the server. That would be even better if set-pipeline
could consume the actual pipeline for std-in, so tooling could be like fly sp -t target --config-stdin -p foo < concourse-template compile ci/pipeline.yml.tpl
Short of Concourse having additional features to handle what I'm calling multi-object builds, having the large number of groups in a single pipeline is no longer working; for starters, the group names no longer wrap, so the ones off to the right are no longer clickable.
Since Concourse now supports teams (and https://github.com/18F/concourse-compliance-testing/pull/127 will recommend having a dedicated team for these scans), I suggest we instead generate one pipeline per site, rather than one group per site. There are tradeoffs (like not being able to easily pause all at once), but I think it's worthwhile.