Method to specify the order of execution for tests

[FeldrinH]

Testing Problem

I need some way to specify the order in which properties are tested. I have a lot of properties in one container that I want to order and group to make the final report easier to read and better structured. In my specific use case the readability of the test report is very important and the tests have a specific natural order so I would like to have them execute and be reported in this order.

PS: I know you can group tests using @Group, but that doesn't really solve my issue, because the groups and tests within groups still need to be ordered in a specific way for it to make sense.

Suggested Solution

Any way to specify the order in which property and example tests are executed would be fine. Ideally I would like something similar to JUnit's MethodOrderer where I can implement my own ordering algorithm.

Discussion

Obviously this is a fairly niche need that not many people have, but I am restricted by technical and design constraints from other places that make this highly desireable in my specific situation, so I hope there is some way to accommodate this use case in Jqwik.

PS: I appologize if there already is a way to set the order of execution for tests, but I could not find any note about this in the documentation.

[jlink]

Method ordering creates a lot of mess because it interferes with the canonical way of discovering and executing tests. Jupiter's MethodOrderer implementation shows how difficult this can become with nested tests and packages and other grouping mechanisms present. Thus, I'm really unwilling to go that route unless there're are really good reasons why a specific testing problem cannot be solved differently.

I have a lot of properties in one container that I want to order and group to make the final report easier to read and better structured

Could you expand on that? Especially what kind of reporting you use and what your ideal report would look like?

[FeldrinH]

Could you expand on that? Especially what kind of reporting you use and what your ideal report would look like?

I'll try, though there is more nuance and fiddlyness to our problem than what is presented in the following overview.

The reporting is done using a custom TestExecutionListener-based wrapper around a JUnit launcher that outputs the info to stdout in a format that is compatible with VPL (a Moodle plugin for automatically grading student-submitted code). The whole system is an educational tool that we use in some CS alogrithmics courses in my university to automatically give feedback on student submissions. At least one of the courses is fairly entry level, which is why we are concerned with making the feedback as readable and logically structured as possible (also, it would feel uncomfortable to have to explain to students that the reason our tests are in a seemingly random order is because a testing framework just doesn't support ordering tests, especially since a lot of them don't even know what a testing framework is yet). The order of tests is based partly on how examples are presented in the homework description and partly chosen to make examples progress from simplest to most complex. The order is specified manually (For JUnit Jupiter we have a custom method orderer that actually uses the source-code order of tests. This is admittedly a little dirty, but it makes writing test a lot faster).

Also, we have some overall feedback tests that need to run after all the other tests if and only if all the other tests passed. In JUnit Jupiter we have done this by enforcing a strict ordering on all tests, but it could probably be done with an @AfterAll/@AfterContainer annotation and some dirty tricks. I would prefer to avoid this, because doing testing in a lifecycle hook creates quite a few new problems, but I suppose it's better than having no way to do it at all.

[jlink]

The reporting is done using a custom TestExecutionListener-based wrapper around a JUnit launcher that outputs the info to stdout in a format that is compatible with VPL (a Moodle plugin for automatically grading student-submitted code).

Would it suffice to just reorder the report, e.g. using a table of contents from your course description? In this case, you could get rid of any engine-specific ordering mechanisms.

Also, we have some overall feedback tests that need to run after all the other tests if and only if all the other tests passed. In JUnit Jupiter we have done this by enforcing a strict ordering on all tests, but it could probably be done with an @AfterAll/@AfterContainer annotation and some dirty tricks. I would prefer to avoid this, because doing testing in a lifecycle hook creates quite a few new problems, but I suppose it's better than having no way to do it at all.

In jqwik you don't need any dirty tricks since a lifecycle hook can be registered to execute after all jqwik tests have been run.

[FeldrinH]

In jqwik you don't need any dirty tricks since a lifecycle hook can be registered to execute after all jqwik tests have been run.

The problematic part when using lifecycle hooks is that those tests that need to run after other tests are also fully-featured tests that need to report success and failure. The 'dirty tricks' I was referring to was somehow detecting the existence of lifecycle hooks and reporting them as if they were regular tests. (TestExecutionListener doesn't report successful lifecycle hooks and the failures are tied to the test container rather than a specific hook. We need to tie failures and successes to specific lifecycle hooks because we have some per-method metadata in the form of annotations that we need to access.)

Would it suffice to just reorder the report, e.g. using a table of contents from your course description?

Maybe. There are at least two drawbacks to this approach for us:

1. In general all existing tests (which are often reused or copied from) are written with the assumption of strict ordering and getting rid of that assumption may or may not be problematic. I'm not too sure on this because there are a lot of tests written by other people that I haven't reviewed. I have found at least some tests that (ab)use the ordering of tests to check info stored by previous tests.
2. If we reorder the tests after the fact then we can't produce output as the tests are running. This means that if the testing environment crashes halfway in (it's not common but some student submissions have managed it) then we won't get any results, which will make debugging harder.

[jlink]

@FeldrinH I'll put method ordering on the TODO list. No promises, though, that I'll get to it in the near future.