Spike one experiment test with dashboard

[lacasseio]

This issue will spike one build experiment test (performance) end-to-end. It will include the test, the results and dashboard generation. We are focusing on the process not the accuracy at this stage.

There should be around 5 distinct components to this spike:

1. Test project generation. We should consider using and improving the template that we already use in the functional tests. The template infrastructure should evolve toward build init templates, sample templates, guides templates, etc.
2. Performance test. We should consider adding any special/unopinionated requirement to our Gradle Runner Kit so the performance test writing can support JUnit or Spock testing framework (or even no framework).
3. Performance result. We need to avoid embedding any assertion in the result to open users to use any assertion library (i.e. Hamcrest, Truth, etc.).
4. Performance database. We need a way to collect performance results from all build experiments as well as between CI runs. The term database refers to the result organized in a logical way that can be queried and manipulated by the dashboard.
5. Performance dashboard. The dashboard will be a centralized view for all performance results. We need to consider at least two major clients: 1) Nokee developer to track the performance quality of the plugin and 2) Nokee users to inform how the performance compares to other build tools (or previous versions).

[lacasseio]

Here is the preliminary decision on each component for this spike:

1. Use a statically generated project committed to a Git repo, most likely using RemoteProject.
2. No decision yet. We need to look into how an experiment runner differs from a Gradle runner. We may look into something that wraps a Gradle runner or something similar given we will want the experiment runner to work with other build tools (make, Bazel, CMake, ninja, Xcodebuild, etc.). We don't want to tie too tightly to the Gradle profiler as other build tools are not exactly supported by the profiler. However, we do want to create a profile run when possible for our plugins.
3. Experiment results should be similar to what Gradle uses in their performance tests.
4. Use a Git repository. It's simple and easily gives access to all performance tests when generating the performance dashboards.
5. The performance dashboard will use our JBake plugin to bake a static website exposing all performance results.

[lacasseio]

So far, essentially finished the BuildExperimentRunner which can be used for both Gradle and Xcode. We can capture the data series from Gradle Profiler.

What we are missing is the reporting capability of the data outside the performance tests. There are several ways we can achieve this but none of objectively the best solution. We will most likely use two-stage reporting. First, we will generate an agnostic test result using a custom Test task implementation. We will most likely mirror JUnit XML test report, e.g. TEST-com.example.SomeTest.xml, with something like SCENARIO-com.example.SomePerfTest.xml. Then we will use another task to process those results into something our performance dashboard will like which should include Git commit ID, some URL back to commit and build, etc. Finally, the result will most likely be published in a Git repository to further processing which includes history performance comparison.

Our solution is similar to Gradle performance infrastructure. The major difference is leaning toward a single build experiment runner that has a pluggable executor for its backend. This means we can plug any other build system in the runner. We also ignored the cross-version/cross-build runners as they simply boil down to having multiple build experiment runners and collecting the result together. It's better this way as these cross runners would be too inflexible for our need, i.e. comparing Gradle core plugins with Nokee plugins or comparing Nokee plugin versions on the same Gradle version.

We developed our build experiment runner to feel like our current improved GradleRunner as well as to live side-by-side meaning other plugin authors should be able to benefit from this work. It will take some months to get to a point we offer the new runner and a performance testing plugin with the toolbox but we intent to move that code over there.