EugenMayer
commented
6 years ago We have a similar issue, but splitting into teams and thus separate pipelines comes with overhead and also some serious downside ( AFAICS ):
a) extra team means extra vault handling just for that case - re-setting all values for every single team, over and over again ( docker reg creds, maven creds, nexus creds )
b) Beside that, a more serious issue is, that AFAICS passed:[] can only be used in one pipeline, "external" pipeline cannot use that. So job to job is very different to pipeline to pipeline, and thats what you are suggesting. since extra team means extra pipeline
We are currently doing the following things:
- departments like "infrastructure" are in a separate teams - the basically build resources for others to consume by trigger and do not have any cross-dependencies or fan-in scenarios
- We have always 2 pipelines for a "thing",
_staging and _release - the first one is for all on commitoron base docker imagechanged builds. So all instance builds. _release is for all tag based / semver file based releases, which are basically triggered by manually bumping the version in the file / creating a git tag. Release and Staging do usually not have any fan in/fan out dependencies. - groups are using to build different sub-things in a thing, be it like app-image (interpreter) builder image ( builder pattern ).. build the artifact using the builder and so on. Those things have a lot fan out/in stuff, so keeping them into one pipeline is very important
That for now seems to be a concept to map things for us, but simply put, people work around a simple classification issue which can be fixed so horribly easy, that it hurts even looking at it. Even by the most simple implementation, being able to assign tags to jobs/tasks/pipelines and then use a JQL based "search" which you can save and put into the navigation would make you make any scenario you like. Also those, where you actually do need different views for the same "things" depending on your current "hat" ( release hat / staging hat / backtracing hat).
The approach i mentioned above does has a downside you cannot simply fix using the UI, and thats simply the structure / size of the pipeline.xml. Since you define
- resources several times for different paths ( like ci/ .. /src ... docker ) and also multiplied by branches.. same goes for docker-image and other things, those resources are very very large
- jobs, even though completely unrelated in terms of order, are still defined in a sequence of a single file. Considered the beauty of concourse that order is defined implicit ( trigger ) it would make a lot of sense putting each job into jobs/
like tasks/ are already externalizede
We though about utilizing YAML based includes here, but go-yaml does yet not support those, only PHP yaml. So we would need to implement that ourselfs, so a pre-set-pipeline task to compile the pipeline.yml and push it to the server. That would be even better if set-pipeline could consume the actual pipeline for std-in, so tooling could be like fly sp -t target --config-stdin -p foo < concourse-template compile ci/pipeline.yml.tpl
Short of Concourse having additional features to handle what I'm calling multi-object builds, having the large number of groups in a single pipeline is no longer working; for starters, the group names no longer wrap, so the ones off to the right are no longer clickable.
Since Concourse now supports teams (and https://github.com/18F/concourse-compliance-testing/pull/127 will recommend having a dedicated team for these scans), I suggest we instead generate one pipeline per site, rather than one group per site. There are tradeoffs (like not being able to easily pause all at once), but I think it's worthwhile.