search

LuaJIT / LuaJIT-test-cleanup

Cleanup Workspace for LuaJIT tests
46 stars 19 forks source link

Added bench/roulette.lua #9

Open lukego opened 7 years ago

lukego commented 7 years ago

This benchmark exhibits non-deterministic performance. See LuaJIT/LuaJIT#218.

I think that it is important to include tests that can demonstrate non-deterministic performance. The challenge I see is to make performance as consistent as possible while including as much entropy as possible on the inputs.

Paraphrasing Dijkstra:

I used to think that benchmarks have to be deterministic. Now I realize that deterministic benchmarks are only a special case of non-deterministic benchmarks, and not a particularly interesting one at that.

Requires a test harness that can account for variation. More on that to follow...

lukego commented 7 years ago

I pushed an update so that the random seed is not initialized in the benchmark script. This should be done separately by the test harness (if it is able to deal with randomness). For #10 the test harness is running the test N times and using a different but predictable random seed (1..N) each time. On the one hand this means that a full test run should be exposed to entropy, on the other hand it means that each individual test should be readily reproducible e.g. to reproduce the result of run 42 call math.randomseed(42) before running the script.