Add support for parametric testing

[ewu63]

First of all, thank you guys for developing testflo! We (the MDO lab) has recently switched to testflo for our internal tests, since it fits our needs perfectly---almost all of our tests require MPI support.

However, most of our tests are typically defined parametrically, and I was wondering if it's possible to have testflo support that, via something like pytest? Being able to specify a test matrix would be much better than manually listing out all the possible combinations. pytest also has tons of other features that would be useful for us.

Since we typically test everything both in serial and in parallel, it would also be nice if testflo has an easier way of doing that than than simply defining multiple tests, perhaps by allowing NUM_PROCS to be defined as a list?

Thanks again!

[JustinSGray]

We can certainly consider adding some functionality, but it would be helpful if you worked out the API you wanted. N_PROCS as a list is clear enough. But "pytest has lots of other useful features" is pretty vague...

What other kinds of test matrices do you want?

[ewu63]

I'm thinking of the decorator approach from either pytest or preferably from parameterized. I think both also allow matrix expansions. Right now we manually list out all the possible tests and give each a unique test name.

Do you know if testflo supports the subTest feature from unittest? That would be an okay approach for now I think. Although there are other good features of pytest, such as the ability to mark a test as xfail that would be good to have.

[naylor-b]

Does xfail mean expected failure? If so, testflo currently handles expected failures using the @unittest.expectedFailure decorator. It also runs tests that use parameterized, although we haven't used it to modify N_PROCS. Assuming that doesn't currently work, we could probably figure out a way to make it work.

[ewu63]

I had no idea that testflo supports parameterized, that's great. It's not very clear from the README what is supported beyond unittest testcases and MPI support via N_PROCS.

I also didn't know about the expectedFailure decorator from unittest, I should've read the unittest documentation better. Thanks for that.

[robfalck]

I think I'd like to see testflo support subTest, but give output on whether each subtest fails, possibly treating each subtest as a separate test that can succeed or fail in the final report.

For instance, I use subTest like this:

        for tx_class in (dm.Radau, dm.GaussLobatto):
            for control_type in ('control', 'polynomial_control'):
                for g in (9.80665, 1.62):
                    with self.subTest(msg=f'{tx_class.__name__} - {control_type} - g = {g}'):
                        ...

Inside pycharm, subtests are printed out with a success or failure message based on their

[ewu63]

Just to update that we have managed to get parameterized to work with MDO Lab tools. In particular, we have used parameterized classes to run the same test with multiple N_PROCS values by parameterizing that class attribute. So at this time there are no outstanding issues from our side.