Fuzz testing

[kwmsmith]

For robustness, it would be good to have a separate test module that:

• is run with a random number of IPython engines (between 1 and 20 or so, inclusive)
• creates random distarrays with random sizes in each dimension (between 0 and 1e6, inclusive)
• with a random mix of undistributed, block distributed, and block cyclic distributed dimensions
• that tests ufuncs, reductions, and other fundamental stuff

We could test the edge cases in a separate test class as well -- size of zero in one or more dimensions, all dimensions undistributed, using a view with just one target /engine, etc.

[cowlicks]

We would have to have a way of reusing the random data. So we would need test suite return whatever the initialization vector was.

[markkness]

Would it be enough to save/return only the seed used to initialize the random number generator?

On Mon, Feb 3, 2014 at 12:26 PM, Blake Griffith notifications@github.comwrote:

We would have to have a way of reusing the random data. So we would need test suite return whatever the initialization vector was.

Reply to this email directly or view it on GitHubhttps://github.com/enthought/distarray/issues/95#issuecomment-33983892 .

[cowlicks]

@markkness Yes. That is what I meant random seed is a better word than initialization vector.

[rkern]

It should take a RandomState instance and use its methods to generate the random data. Don't use the convenience functions in numpy.random that use (and thus modify) the global state.

[bgrant]

John T. mentioned using the Python port of QuickCheck for this:

https://pypi.python.org/pypi/pytest-quickcheck/0.8

@kwmsmith : Is this still a possible task for @markkness ?

[kwmsmith]

I've never used py.test -- what's the consensus on it? Can we keep our current unittest-based test suite and have py.test run it with the fuzz testing turned on as an extra?

[bgrant]

@kwmsmith : It seems to be pretty widespread, though I haven't used it either. TravisCI installs it by default. pytest will run unittest testcases:

http://pytest.org/latest/goodpractises.html#python-test-discovery

[kwmsmith]

Putting on wishlist for now -- can move to milestone 0.3 if and when we get there.

[bgrant]

Possibly consider: https://github.com/DRMacIver/hypothesis