Solidify benchmarking infrastructure and have reference numbers

[tli2]

Once we have a dedicated benchmark machines we should start formalizing a suite we want to run for new PRs, as well as numbers we think that would make sense. (TAS performance, DataTable throughput, and others)

[lmwnshn]

On a related note, per discussion with @pervazea I've been looking into Github apps, particularly the checks API. The goal is to have it report performance increase/regression on every Github PR.

[pervazea]

This comment is about item 1. Related issues, with respect to infrastructure, are 2-4.

1. Microbenchmarks
2. Macro benchmarks (e.g. OLTPBench)
3. Long running stress tests
4. Fuzz tests

The proposed flow, for a PR will be:

1. Travis build. If successful, triggers the next step
2. Jenkins build, which executes the following in parallel
   1. The "complete" set of unit tests. These will be run via Docker.
   2. Microbenchmarks. Run on dedicated benchmark system(s). Like the tests, they will report pass / fail (in addition to actual numbers).

Results from Travis, Jenkins tests and microbenchmarks should be available via Github, so we have a usable, consistent approach.

Implementation issues to be worked out:

• Travis to Jenkins trigger (in progress)
• Benchmark execution. What this (roughly) needs to do, is:
  1. Run the benchmark(s), saving results
  2. Compare result(s) against a baseline or against a moving average of past results (from the master branch), posting pass / fail designation and numbers back to the PR. Google benchmark has support for summarization and comparison of results. TBD where historical data is kept, possibly Ottertune's database.
• Benchmark execution and machine. Need to ensure benchmark results have low variability from run to run. This is expected to require a physical machine (not VM) with additional configuration to reduce variability. Expecting to run on Ubuntu 18.04 only, for near term. One of the MemSQL machines being considered for benchmark execution.

[tli2]

Blocked on benchmark machine