Generate .drone.yml programmatically

[awly]

Feature Request

Consider writing a program to generate .drone.yml.

Motivation

.drone.yml has a lot of duplication and its diffs are hard to review (especially when sections get moved around). Use a real programming language (and not yaml/bash) to generate this config and get some code reuse. Signing could be triggered as part of it too, making the process more fool-proof.

Who's it for?

Teleport devs

[klizhentas]

@awly @webvictim if we are at this point, are we at the point of exceeding of what Drone was designed for?

[webvictim]

I think that the way we're using Drone is probably a bit more complex than the types of simple pipelines the authors had in mind. They certainly intended that for complex operations, you write a script and call that from Drone instead.

The actual core of Drone is very powerful, but the fact that its pipelines are YAML which doesn't have any concept of functions, includes or ways to reuse blocks is a complete pain. It's very easy to make mistakes when actually writing the pipelines too as everything is just repeated 60 times and all the statements blend into one.

Drone has varying degrees of support for jsonnet and Starlark for generating pipelines rather than just using straight YAML. I wrote a prototype Starlark script ages ago to generate the YAML. Starlark is reasonably good (better than jsonnet anyway) because it's a subset of Python 3, so it's quite easy to pick up and can be syntax highlighted/processed using standard Python tooling in an editor. .drone.yml was ~1800 lines when I wrote the Starlark protoype and the Starlark script came out to ~500 lines (although I don't think I quite managed to implement identical functionality). This would obviously get better the more copies of code we add, as it'd be a few lines to add a new pipeline versus copy/pasting 200+ lines.

I'd love to use something to generate the YAML as it will standardise the pipeline's steps and make it way easier to make sweeping changes. The question is what language to use - I'm not the world's best Go developer, but I can write reasonable Python. I agree it shouldn't be bash, and I don't really want to start writing scripts to change everything together or start shoehorning logic into the Makefile anyway.

In the short term, maybe I'll just try bringing the Starlark up to date and see what we end up with. At any rate the blocks I isolate should be reusable enough that we could drop them into any other language as well.

[awly]

Anything with loops, functions and some kind of data/type composition would work. There are certainly more people familiar with Go on the team than Starlark. I'm more than happy to help with a Go-based generator.

@klizhentas this feels like a simple enough problem to solve, definitely less effort than picking a new CI system again.

[webvictim]

That'd be awesome; I think with a good start I'd be able to pick up a Go-based generator pretty easily.

We'd have this problem with the majority of CI systems because they all use YAML and the longer a YAML file gets, the harder it is to work with. At least with a separate system we could generate and lint each pipeline completely separately (Drone currently refuses to lint our pipeline because it contains exec steps which run on darwin), then concatenate the file together at the end and get Drone to sign it. It would also give us the ability to have an external approval process for the Drone config and allow us to work around the constant merge conflicts that happen whenever we change .drone.yml in two different branches.

[klizhentas]

@awly @webvictim does Drone have API? May be we can use it's API in Golang instead of generating YAML files. Can we also compile Drone pipelines in Go using go modules?

[awly]

@klizhentas Drone has an API but it's not for sending dynamic build/test steps.

Can we also compile Drone pipelines in Go using go modules?

I don't understand what this means.

[bradrydzewski]

similar to jsonnet and starlark, you could create a conversion extension [1][2] that supports using the Go language to generate the configuration on the fly. There is a Go interpreter [3] that claims full compatibility with Go syntax and the Go standard library.

[1] https://docs.drone.io/extensions/conversion/ [2] https://github.com/drone/drone-convert-starlark/tree/v1.0.0 [3] https://github.com/traefik/yaegi

[webvictim]

https://github.com/gravitational/teleport/pull/5644 is a proof of concept.

[webvictim]

Solved in https://github.com/gravitational/teleport/pull/6071 for now.

[awly]

Not all pipelines have been migrated yet:

build-buildboxes
build-darwin-amd64
build-darwin-amd64-pkg
build-darwin-amd64-pkg-tsh
build-docker-images
build-ent-amis
build-oss-amis
promote-build
push-build-darwin-amd64
teleport-docker-cron
teleport-helm-cron
update-docs-webhook

[webvictim]

Good point!

[webvictim]

If we ever want other people on the team to be able to reliably make changes to Drone pipelines, we need to make time to finish this task by migrating everything to dronegen.

[zmb3]

Closing as not planned. We won't be investing much more effort into drone since we're moving off of it.