Allow using multiple templates per .drone.yml file

[andrii-kasparevych]

As a Dev/DevOps I want a more agile way to manage the code duplication across the pipelines using templates. As of now the templates are allowed to use only once per pipeline, meaning there's no way to define a template for one pipeline and have a repo's .drone.yml using several templates that would kick off different (or same) pipelines with different parameters. Example. I have an app that I want to build for 3 different platforms (e.g. Windows, iOS, Android). The way I do it without a template is:

---
kind: pipeline
name: build-windows
... 300 lines of pipeline definition
---
kind: pipeline
name: build-iOS
... 300 lines of pipeline definition
---
kind: pipeline
name: build-android
... 300 lines of pipeline definition
---
kind: pipeline
name: report-to-slack
depends_on:
  - build-windows
  - build-android
  - build-iOS

The build pipelines are identical with a small difference in some command line arguments, but quite large (300 lines). The report pipeline waits for the others to finish, gathers the results and reports to slack. It would great to leverage templates on the level of build pipeline here, so that the amount of code duplication reduces significantly. As a result the above would change to smth like this:

---
kind: template
name: build-windows
load: build.template.yaml
data:
  targetPlatform: Windows
---
kind: template
name: build-iOS
load: build.template.yaml
data:
  targetPlatform: iOS
---
kind: template
name: build-android
load: build.template.yaml
data:
  targetPlatform: Android
---
kind: pipeline
name: report-to-slack
depends_on:
  - build-windows
  - build-android
  - build-iOS

[MarieReRe]

Hey 👋🏻 @andrii-kasparevych

Thanks for submitting your proposal. We usually review PRs every Tuesday but I will keep you up to date if anything changes. 🎉

[bradrydzewski]

I do want to clarify that a template yaml can contain multiple pipelines, however, a .drone.yml can only contain a single template. I completely agree that we should support multiple templates in the drone.yml file. I do think the current implementation could support your use case, as an immediate workaround.

[AmitBaranes]

@bradrydzewski This is a mandatory feature and this workaround is a bit messy. any ETL ?

[andrii-kasparevych]

@bradrydzewski thanks for that, I'm aware of the current capabilities. More than that - I've also built a separate repository for storing templates so that they are also version-controlled somehow and built an automation for auto-deployment to the drone server. If anyone is interested in details, I'm happy to share.

But still I think the feature of using multiple templates per .drone.yml is a must. I can think of an alternative approach - allow having more than one .drone.yml file per repository - each could contain 1 or more pipelines and just 1 template. That could also work.

[tjbrockmeyer]

My first thought when using the templates was that they could be used in the same way as a pipeline - just put a few of them, one for each pipeline, and now you have a shorthand way of defining each. It took me quite a while to realize that the first template in my file was just replacing the whole file. Having the ability to have multiple templates per file sounds natural and is a good way to go.

The idea of having the templates versioned in the drone server also sounds quite good. I was resorting to a sort of naming convention to handle that part.

I had this idea as a temporary workaround. With a .drone.yml file like this one:

kind: template
load: my-template.yaml
data:
  pipeline_a:
    - input:
        name: my-pipelineA-1
        field1: abc123
        field2: xyz098
    - input:
        name: my-pipelineA-2
        field1: abc123
        field2: xyz098
  pipeline_b:
    - input:
        name: my-pipelineB-1
        field1: abc123
        field2: xyz098

You will be able to have a template file that looks like this:

{{- range .input.pipeline_a -}}
---
kind: pipeline
name: {{ .input.name }}
# ... more pipeline stuff
{{ end }}
{{- range .input.pipeline_b -}}
---
kind: pipeline
name: {{ .input.name }}
# ... more pipeline stuff
{{ end }}

Thus, sort of emulating the idea of having multiple usable templates. You can opt in to each pipeline defined in the template file by supplying it's name as a top-level key, and you can define multiple of each instance using different parameters because it is a list. Using a property name of input for each pipeline instance makes it easier to migrate to multiple templates per file at a later time, as those will require you to use {{ .input.field1 }} rather than just {{ .field1 }}.

This workaround does make it pretty impossible to do versioned changes, but I find it to be a decent solution.