Automatic Airspeed Velocity benchmark results publish

trexfeathers commented 3 years ago

✨ Feature Request

Create an automated job to run asv gh-pages --rewrite, which uses the GitHub Pages infrastructure to create an ASV results dashboard (example).

Motivation

Easy visualisation of this project's performance over its commit history, in a shared standardised way, will aid in development and collaboration. It also improves the project's image by showing potential collaborators/users that coverage extends to routine performance monitoring.

NOTE: benchmarking is already part of the CI 'test harness', but this only produces numeric results and skips any particularly slow benchmarks.

More detail in here!

Expand for implementation ideas...

This was previously attempted but removed due to permissions/security concerns - see #88. It seems that a dedicated GitHub Action will probably be the right implementation - [here is a good example](https://github.com/tqdm/tqdm/blob/c1ec3b18cf7c0ff2c980c16346b87b9fea48aae8/.github/workflows/check.yml#L28-L63). Virtual machines that use shared resource - including those used for GHA, Cirrus and most/all other CI solutions - are vulnerable to variable performance, which is very unhelpful for performance testing! There are steps we should take to help compensate: * Publish results from a single continuous ASV run. * As opposed to the example above that forces ASV to recognise the machine as the same one each time, allowing just benchmarking new commits and appending their results to existing ones. * Avoids the following performance variability: * Variability between multiple GHA runs. * Variability on long time scales (e.g. GHA demand at different times of day). * Variability from GitHub periodically changing/upgrading their Actions architecture. * This will result in long run times. I initially suggest a full run+publish after every PR merge, but if this takes prohibitively long we can consider scheduling nightly/weekly/... runs. * Work with Airspeed Velocity's design to reduce the impact of noise. * ASV automatically performs repeats and applies statistics specifically with noise in mind. * [A variety of configurability](https://asv.readthedocs.io/en/stable/tuning.html#tuning-timing-measurements) is on offer to further compensate for within-run noise if it is still a problem. Some of these could also be useful for the existing CI benchmarking (see the [CLI commands](https://github.com/SciTools-incubator/iris-esmf-regrid/blob/2456f9b08c9a49cb6cd33555bc5938bdf0edb5be/noxfile.py#L339-L391) and the [benchmark scripts](https://github.com/SciTools-incubator/iris-esmf-regrid/tree/2456f9b08c9a49cb6cd33555bc5938bdf0edb5be/benchmarks)).