feature request: option to shuffle test sequencing

[mtmiller]

For all normal uses, tests should be executed in a deterministic order. But sometimes it may be helpful to shuffle the test order to expose dependencies between tests.

I've found this useful in the past, and done this manually by constructing ad hoc lists of files to test or by hacking the test driver to skip or reorder certain problematic tests by file name. At the time thought it would be really useful to run the whole test suite with a random shuffle applied to see what shakes out.

[apjanke]

Sounds good. I'll add it.

[cbm755]

+1, I sometimes do this when I have a very slow test set, then I start multiple copies with different seeds: that way I can wake up the next morning and have some probability of finding every test that failed.

Now that I've read that, I gather this is what interactive mode of test is supposed to toggle.

[mtmiller]

@cbm755 I was thinking more of inter-dependencies between tests, things like changing Octave persistent state or process state, loading a library that breaks something else, etc. But yes, tests that use random without setting their own seed will also be affected.

[apjanke]

Added -shuffle and -shuffle-seed options to runtests2_refactor and test2_refactor in https://github.com/apjanke/octave-testify/commit/e48e7f76d9bd637297722210f45ba5ec9ac0d827 and https://github.com/apjanke/octave-testify/commit/ae9034754e6d70a0684f4760fcbfd451d154ecb7. Closing as complete.