Automated testing of build pipeline.

[russjones]

Problem

Our build pipeline has no test coverage. We don't know if changes we're making to the build infrastructure will cause builds to fail or if we are building binaries that work.

Proposed Solution

For each tag that we want to promote to a release, we should:

1. Check if the tag or commit can be used to successfully build all binaries. If this step fails, it tell us we have a problem with our build infrastructure.
2. Even though the build infrastructure can build a binary, it does not mean the binary will actually run. We should put the binary through some sanity check to make sure it can actually run. If this step fails, it tells us if the binary was incorrectly built.

[webvictim]

Drone has some of this in already. When we cut a tag, Drone publishes all the binaries to a staging S3 bucket without making them available for public download. We can download those binaries and use them for sanity checks. When we've validated that they work and we're happy, we run the promotion step and they become available for download.

We currently also check that the Docker images can be built at the same time, but they're not yet pushed up to quay.io after build. We can potentially modify this process to push the built images to a different, non-production repo on quay.io and run sanity checks on them. Once we're happy with them, we can just retag the images and push them to the "production" repo as part of the promotion step.

[webvictim]

As part of #5292, Drone now builds all binaries when a commit is pushed to master or branch/*, which should increase confidence that a release built from master will always work.

[webvictim]

This can also be partly covered as part of #5533:

• [ ] look to integrate Drone’s linter directly into our generation code
• [ ] provide a way to automatically rewrite specific steps with a pull_request trigger so that they’re all tested when raising a PR, rather than having to push tags to test specific functionality
  • [ ] one issue here is secret visibility
  • [ ] an alternative method might be to rewrite kubernetes to docker and try to run the pipeline locally with drone exec