search

sageserpent-open / americium

Generation of test case data for Scala and Java, in the spirit of QuickCheck. When your test fails, it gives you a minimised failing test case and a way of reproducing the failure immediately.
MIT License
15 stars 1 forks source link

Seamless JUnit5 integration. #38

Closed sageserpent-open closed 2 years ago

sageserpent-open commented 2 years ago

This follows on from the train of work from issue #18, which had been repurposed as a bug-fix ticket.

Ongoing spike work started off the back of that ticket indicates that it is worth considering a tighter integration with JUnit5, whereby something akin to the ParameterizedTest annotation can be provided using the TestTemplate framework (also used by ParameterizedTest, which looks to be too specialised to be amenable to integration with Trials, not least because implementing inlined filtration won't play well with its notion of pulling from a stream of Arguments - it doesn't provide a context for the InlinedCaseFiltration to be available when the test instances are run.

What the spike is investigating right now is whether an InlinedCaseFiltration can be provided as a context by embedding an iterator over cases and the inlined case filtration within an extension that implements TestTemplateInvocationContextProvider and InvocationInterceptor...

If so, then the end result will be TrialsTest annotation that couples to an instance of Trials, supplying the test method with arguments that align with the cases yielded by the trials instance - so a simple trials will support a single-argument test, whereas one constructed by a single application of .and will support a test with two arguments, etc.

It is hoped that using InvocationInterceptor will allow inlined case filtration.

A goal for the future is to build on this ticket's deliverable functionality so that once a test case fails, further test cases will be generated using the existing shrinkage logic - so whereas ParameterizedTest always generates the same test cases for deterministic value sources, TrialsTest only does this when all test cases pass - if there are failures then the shrinkage kicks in and will vary depending on the first failing test case. This is desirable, as when we have a failure we simply want to work on a minimal test case, and as long as the mode of failure remains the same then we will have repeatable failing tests leading to the same minimal test case, which hopefully will be amenable to using IntelliJ to run it directly without recapitulating the test cases leading to it.

sageserpent-open commented 2 years ago

Owing to the restriction on Java annotations, TestTrials has to use a similar approach to ParameterizedTest in how it specifies its trials instances - namely to supply the names of the fields in the class housing the tests that are typed as Trials.

Now this leads to a situation where one would have to provide a field purely to allow a definition of a conjoined trials that is defined by the and combinator - and if several tests make use of varying combinations of the same basic trials instances, each combination would need its own field so that the corresponding @TestTrials would have something to name. This feels awkward - so again in the spirit of ParameterizedTest, TestTrials can take multiple names of trials fields and will do its own internal joining, rather than using a Trials built externally via and. The test method then takes the corresponding number of arguments.

It is possible that folk may wish to supply either a single such externally conjoined trials instance and have it unpicked to drive a test method with multiple arguments - or perhaps a single tuple argument should be supplied? What happens if both approaches are mixed and matched - do we look for tuple types in a multi-argument test method and supply them directly from the corresponding conjoined trials, and if we have a run of non-tupled arguments, do we supply them from a single conjoined trials and then carry on as usual with the rest of the arguments? This needs some thought ....

sageserpent-open commented 2 years ago

What exists right now is workable - where inlined filtration is used, JUnit interprets a rejected test case in the same way as if a call to assumeTrue etc failed - it marks the individual test as rejected.

Indeed, one can mix both the use of Trials.whenever and the Assumptions.assume* methods - both are picked up by the JUnit integration furnished by a @TestTrials annotation on a test method. The usual test before and after methods are also supported, as is selecting an individual test case in IntelliJ to run it.

sageserpent-open commented 2 years ago

Merged into master in commit 6921fe703f0e629df31ac9bafb7f6a4f27f1167a, released in 1.3.0 .