Make even random tests rerunnable to be able to debug rare failures

[axch]

This depends on #138. Given that, could instrument, say, get_ripl or statisticalTest to

• generate 32 bytes from the ambient PRNG
• store them
• seed the ripl with them
• print them out if the test fails
• somehow accept a seed for purposes of reproduction

[riastradh-probcomp]

Why don't we just use a fixed random seed (e.g., zero) for tests, and chain it when we iterate a test multiple times? Then it's always reproducible.

[axch]

You mean, reset the seed to 0 at the beginning of each distinct test? That's actually not a bad idea. I mildly prefer Jenkins to be exploring different initial seeds over its many runs of any one test, but reproducibility probably trumps that.

Speaking of which, it doesn't have to be 0, but can still be shared across all tests within a single test run, which the test suite can report at the end (and/or the beginning).

[riastradh-probcomp]

Zero is easier to remember and requires zero work to integrate it with our debugging capacity.

The point of chaining it when iterating a test multiple times is so that we can explore more of the space at the expense of time in long quality tests.

Although maybe in the latter case it would nonetheless be worthwhile to report the state before each new trial.

[axch]

138 being done, this is no longer blocked.

[axch]

@riastradh-probcomp recently wrote a related concept for Bayeslite, called, if I recall, @stochastic. The architecture is that the test function is expected to accept a seed argument, given which it is to be deterministic; the test framework (via the @stochastic annotation) will take care to run the test with various seeds, emit them in the event of failure, and permit the operator to reproduce a given seed run at will.

[riastradh-probcomp]

https://github.com/probcomp/bayeslite/blob/85dc511c5b9cb3683b3d347829bc22abff1a14f3/tests/stochastic.py

[axch]

@stochasticTest has been written and widely incorporated in August 2016; remaining task sufficient to close this ticket is to grep the test suite for get_ripl() (424 results as of this writing) and tag those tests @stochasticTest, or manually confirm that they are not, in fact, stochastic.