bench.js: easy/fast way to answer the question 'did I regress?'

[nornagon]

Currently the benchmarks take upwards of 15 minutes to run. It's really hard to determine, for a given change, whether a performance regression was introduced.

Possible solutions:

1. only run each benchmark 3 times, instead of 9. This might lead to less stable numbers, but if it doesn't then it's a good solution.
2. define a 'fast' subset of benchmarks that's reasonably representative. You've been tracking the benchmarks over time, so it shouldn't be too hard to pick the few that have reliably indicated overall performance (least-squares analysis perhaps? :))
3. introduce a single large 'complete' benchmark. Even if it took 20s to run each time, that would only be 9*20s = 3 minutes total.

It would also be nice to be able to easily compute baseline numbers, e.g. by 'node bench.js --did-i-regress' performing a git clone into a temporary directory and running the benchmarks at origin/master, then running with your changes.

[nornagon]

(also, something that might make the numbers more stable: node --expose-gc bench.js, and call gc() between each run to invoke the collector deterministically.)

Edit: for reference, I just tried this, and I was unable to produce more predictable results by calling gc() manually. (method: I measured the standard deviation across 9 runs with and without manual gc() calls between run_benches. With the calls to gc() in, the standard deviation was not noticeably different to without them.)