Plan: Continuous perf testing

[miekg]

Continous performance testing for CoreDNS

Setup

drone.coredns.io (Packet machine) runs webhook behind a Caddy server. Caddy is used for Let's Encrypt TLS certs and only proxies.

Further more mbench is running to collect performance data (go test -bench=. output) and convert into prometheus metrics.

Prometheus is running locally, scrapping mbench.

Grafana is running locally displaying graphs from Prometheus.

Of all these things, only Caddy and mbench are not installed from the Debian repos.

Integration with Github

Webhook setup on github.com point to drone.io port.

Trigger a build from Github

On a PR add a comment: @mbench OK ..... Only the OK is needed. Only some people can trigger these.

Trigger locally

Send a REST request to another endpoint (blocks everything execpt ::1 and 127.0.0.1).

Master branch

A local cron job trigger the unauthenticated local endpoint every 30 minutes.

Future

• Each middleware should have more benchmarks
• Publish this data somewhere?

[miekg]

/cc @chrisohaver @johnbelamaric

[miekg]

Script to be used as webhook exe: bench.sh.txt

[johnbelamaric]

Sounds great. We were just discussing this yesterday. We were talking about having a webhook fire off a script that provisions a couple machines in packet.net, and run tests for proxy over the network. This will take a few things off of Chris's list, since the web hook server will be set up already.

The thinking was that at first it might be triggered manually (a comment is good), but eventually as we build out more CI for specific middleware (think kubernetes cluster, etc...), we would want to have filters that only run that CI on PRs that contain the specific middleware.

[miekg]

Ah, nice. I though about doing a multi machine setup - but that quickly gets complicated due to the synchronization issue for provisioning the machines, but maybe Packet has all kinds of stuff for this - dunno. If would be good to also export any metrics from this to a prometheus instance - to keep longer term metrics and have ease of sharing dashboards.

Re: triggering: yes sounds good. You're in favor of doing this automatically? But then you run into problems when people send you fake PRs with bitcoin mining code....?

Server setup should for drone` should be done after this weekend - this is all by hand, so no puppet/chef/ansible so it's all a bit meh.

[johnbelamaric]

Well we can set up our own K8s cluster and make it declarative :) At least the web hook setup stuff should be in a repo so we can add hooks easily, etc.

Maybe we make it auto only for PRs by existing contributors. That should prevent nastiness. Also we can run them in parallel even with multi-machine - we don't have so many concurrent PRs open. At least we can do that for tests that are just on 1-2 machines. What I mean is that the web hook triggers a provisioning of new machines. Only takes about 2 minutes according to Chris. For a large K8s cluster I think we would still want to keep it manually triggered.