Unit testing: programs with heuristic algorithms: how?

[GoogleCodeExporter]

Hey dude,

This idea has been buzzing in my head all morning.

Heuristic algorithms: no two runs are alike: even with the same input, you
will get wildly different output, so you cannot diff a file and have a
pass or go mark.

What if you could map the text file output, to a bitmap? Should you not be
seeing clusters of pixels concentrated close enough? Maybe then, you could
compare bitmaps, or portions of them.

A complication of doing unit testing like this, could be that you need
more than a couple of cycles to perform the check…

PS: On top of things, hm ... Have a series of runs of such programs, in
order to have some graphing/correlation of the running time of such a
program with the input size, even if the theory says "the running time is
unpredictable". All things start and end. I am sure I could put a dot on a
graph for each run. This has not been done before, because it requires CPU
power. The same kind that is used for the actual experiments themselves.

Just something off the top of my head.

Original issue reported on code.google.com by geo...@marsel.is on 12 Jun 2013 at 10:38

[GoogleCodeExporter]

Suggestion: use automated testing over distributions

Original comment by geo...@marsel.is on 12 Jun 2013 at 11:59

[GoogleCodeExporter]

Stochastic Unit Tests

Probabilistic Unit Tests

http://www.urbansim.org/downloads/manual/dev-version/opus-userguide/node394.html

Stochastic Unit Tests

Testing stochastic systems presents challenges for the traditional unit testing 
framework, since in general either the tests are trivial, or else they will 
sometimes fail. A research project by our group developed a methodology to put 
stochastic unit tests on a firm statistical basis, and this is now used in 
writing such tests in Opus and UrbanSim. See Hana Ševcíková, Alan Borning, 
David Socha, and Wolf-Gideon Bleek, ``Automated Testing of Stochastic Systems: 
A Statistically Grounded Approach,'' in Proceedings of the International 
Symposium on Software Testing and Analysis, ACM, July 2006 (available from 
http://www.urbansim.org/papers/sevcikova-issta-2006.pdf).

A result of this is that, indeed, such stochastic unit tests may fail, even 
though the underlying code is correct. This formerly resulted in a periodic 
failure, which would go away when we ran the test again. To automate this 
process, the run_stochastic_test method in the StochasticTestCase class now 
includes a keyword parameter number_of_tries with a default value of 5, which 
simply tries again that number of times. Since introducing the number_of_tries 
parameter this issue has not come up

Paper mentioned attached

Original comment by geo...@marsel.is on 22 Jun 2013 at 12:16

Attachments:

• sevcikova-issta-2006.pdf