Right now I'm taking the median of 251 runs, but I'm a bit unsatisfied with this because I have not confirmed that 251 is enough to eliminate unreasonable variance, nor does it tell me the standard deviation or any of those niceties. As pointed out in https://github.com/nolanlawson/optimize-js/issues/2#issuecomment-248308234, we should really have a more rigorous benchmark in place, especially as we start to potentially make changes to "improve" performance, which need to be proven to actually improve it.
I looked at benchmark.js but TBH couldn't figure out how to nicely integrate it with the current benchmark design; e.g. currently I request each script with a random query param to force the browser to re-parse and re-evaluate it, and the timings have to be inside the script itself, so it's very tricky to implement.
Right now I'm taking the median of 251 runs, but I'm a bit unsatisfied with this because I have not confirmed that 251 is enough to eliminate unreasonable variance, nor does it tell me the standard deviation or any of those niceties. As pointed out in https://github.com/nolanlawson/optimize-js/issues/2#issuecomment-248308234, we should really have a more rigorous benchmark in place, especially as we start to potentially make changes to "improve" performance, which need to be proven to actually improve it.
I looked at benchmark.js but TBH couldn't figure out how to nicely integrate it with the current benchmark design; e.g. currently I request each script with a random query param to force the browser to re-parse and re-evaluate it, and the timings have to be inside the script itself, so it's very tricky to implement.