Webhook Proxy

[michaelsauter]

Currently, every component creates two Jenkins pipelines, dev and test, as OCP resources. In addition, every repo has two webhooks (actually four, but two of them are pointless), which trigger both pipelines for every commit. Inside the pipeline, Jenkins checks whether the pipeline is responsible for the commit (test=master,dev=*) and then continues.

This has several drawbacks:

• Both pipelines are always triggered and a check has to be made (which is slightly buggy, e.g. in case of a fast-forward merge)
• The Jenkinsfile is always grabbed from master instead of from the current branch, so changes to the Jenkinsfile cannot be tested inside a branch
• The dev pipeline is responsible for all sorts of branches, so its build status is all over the place
• Since there is only one dev pipeline, we cannot run builds for different branches in parallel

To improve this situation, we have build a "webhook proxy" internally. This proxy provides one endpoint accepting webhooks from BitBucket and forwards them to the corresponding Jenkins pipeline (which is determined based on the branch name). If there is no corresponding pipeline yet, it will be created on the fly. Once a branch is deleted or a pull request declined/merged, the corresponding Jenkins pipeline is deleted automatically. The webhook proxy is a Go application - a single, no-dependency binary, produced from one file using just the standard library.

This works very reliable so far (tested in two initiatives for about a month) and solves all the pain points mentioned above.

To include it in OpenDevStack, the following has to be changed:

• Currently we run on proxy in the cd OCP namespace. This works well, but it requires that the user inside the pod can create/delete resources in all other namespaces. This should not be the case, we would rather have a proxy running in each foo-cd project, and then the user only needs permissions to create/delete resources within its own project.
• The binary needs to be built from the source code. Currently, the build config is triggered locally via oc start-build webhook-proxy --from-dir . --follow --namespace cd, which copies a binary in the current working dir into the Docker image. While this works well, it requires Go to be installed on the local system which is not something we would want to impose on an ODS user. We either need to create a Jenkins Go slave to build it there, or we would need to build the binary within the OCP build (which is probably the easier option, but increases the size of the Docker image - currently that is based on a barebones alpine image).
• Remove the dev and test pipelines from the OCP templates (they are not needed, the master pipeline will be auto-created on the first push)
• Adapt the webhooks in BitBucket to point to the proxy.

FYI @tjaeschke @rattermeyer @clemensutschig @stitakis @gerardcl

[rattermeyer]

Actually, I like the idea of the webhook proxy. I would also vote for deploying the proxy into the foo-cd project, rather than the general cd project. Would it also be an option to use a s2i build in this case?

[michaelsauter]

Cool, thanks for feedback. I can look into whether s2i is an option. Also I have to check if the new OCP version allows multi-stage builds, which would also be handy in this case.

In addition to my proposal above, I realised that the webhook proxy could allow users to use a monorepo without loosing the capability to do individual pipelines/deployments. The basic idea would be to add a query param to the end of the webhook proxy url, e.g. ?dirs=frontend,backend and then the proxy would create/manage one pipeline per specified directory, pointing to the Jenkinsfile in them ... but yea, that would be something for later :)

[michaelsauter]

So s2i might be an option. However, the "official" golang s2i does not support Golang 1.11 on RHEL yet. So I opted for the simplest option, which is to just build the binary in the Docker image (which is now based on golang:1.11-alpineinsteas of alpine).

I'm now at a stage where the proxy image can be built in the cd namespace. In addition, we have a project-specific proxy running that is built that way and it works well. The next steps are:

1. ods-project-quickstarters: Add DC,SVC,Route definitions in the cd-jenkins-persistent.yml template.
2. ods-provisioning-app: Change the created webhooks to point to the proxy. Right now we create 4 webhooks in total. 2 for CI, and 2 for RShiny. I have no idea why we need the latter ones. Anyone ideas?
3. ods-project-quickstarters: Remove the pipelines (dev, test) from the component template (as pipelines are auto-created now).
4. ods-jenkins-shared-library: Remove the (buggy) code that tries to checkout the correct Git commit.

All in all, not too much work left :)

[michaelsauter]

Done!