Performance Microbenchmark Infrastructure

[RyanCavanaugh]

(Ron to provide details on scenarios to support)

[rbuckton]

There are two kinds of microbenchmarks we would like to capture:

1. Benchmarking compiler code directly - For example, benchmarking the performance of our ts.filter implementation.
2. Benchmarking the compiler functionality given a snippet of TypeScript source - This would be similar to benchmarking a fourslash test.

I've been speaking with the C# team about the performance test infrastructure they use:

• https://github.com/dotnet/performance
• https://github.com/dotnet/performance/blob/master/docs/benchmarkdotnet.md

There are also a few open concerns to address:

• Benchmarks are very system dependent. Unlike a unit test, benchmark results can vary from system to system. Running benchmarks locally would require establishing a baseline per machine.
• Should microbenchmarks be run as part of the regular test suite (i.e. gulp runtests-parallel)? Microbenchmark performance can suffer if the system is busy doing other tasks (like running other tests), so they would need to be run separately from our other tests.
• Should we publish our microbenchmark performance results for CI test runs similar to our regular performance tests?
• Should we use benchmark for this, or adapt the ts-perf benchmarking functionality?
  • ts-perf is not published externally so we would have to consider publishing parts of this tool so that external contributors could also run microbenchmarks.
  • ts-perf already generates a benchmark JSON file that our current reporting infrastructure understands. If we chose to not use ts-perf we would likely still need to port this logic and keep it in sync.