Provide shrinkage of failing cases in tests driven by @TestTrials

[sageserpent-open]

This follows on from #38 - right now the cases supplied to a test method decorated with @TestTrials are deterministic even when there are failing cases. What we want here is to have the first failing case initiate a shrinkage process that fits into the existing model of JUnit driving the test method, so the overall block of test runs furnished by JUnit culminate in a maximally shrunk failing case.

As shrinkage may turn up some cases that pass after it finds the maximally shrunk case, it would be useful to either report the maximally shrunk case - the cases are indexed and / or to replay that case so it always comes right at the end.

[sageserpent-open]

This now works insofar that JUnit5 will execute test templates for a test annotated with @TestTrials, and once a failing case is detected, the generation of further test templates is guided by shrinkage - so the last failing case is the maximally shrunk one.

Example here:

What is missing is that when one clicks on say, a specific test template execution in IntelliJ (or whatever) in order to just run that specific test template, then this will only work for the test templates generated up to and including the first failing execution. Any test template generated by shrinkage will not be available for direct re-execution this way - either some other test template will run with different arguments for the test, or the test won't run at all.

This is down to the model JUnit5 uses for re-execution of test templates - it simply regenerates the whole lot, and checks an associated UniqueId for each test template with one passed in by the runner that is re-executing some previous test template run. As TrialsExtension only yields repeatable test templates in the absence of test failures, this means that the regenerated test templates only match those from the previous run up to the point where shrinkage starts.

Grrr.

[sageserpent-open]

Two possibilities for getting out of this mess:

1. Copy an idea from Jqwik - it stores information about its analogue of a test template in a temporary database, so it is possible to tell it to run the maximally shrunk case - although this is done via an annotation on the test itself - as far as I can tell this isn't a case of pointing and clicking on a failing test case in IntelliJ etc. It is still a nice idea and could be shoehorned into JUnit5's test template model simply by storing all the recipes for the test templates in a database along with the overall outcome of running all the test templates; if there was a failure, the recipes are consulted to recreate exactly the sequence of test templates that lead to the desired one, along with the correct associated unique ids. Hokey, but workable.
2. Have TrialsExtension pluck a Java property that is the recipe for the maximally shrunk case - if it is present, only the test template for that case is generated - so one re-runs the entire test in the usual fashion, but passes in the required Java property - this is simple enough to do in IntelliJ, although it is not quite as seamless as just clicking on a failing test template to re-execute it.

[sageserpent-open]

There is a third possibility and that is to get right in front of JUnit5's test template machinery - not sure how one would do this - maybe write a custom JUnit5 engine? That's not going to happen right now...

[sageserpent-open]

After some thought, I think this should be spilt up.

1. To start with, test template instances in JUnit5 that are generated as part of the shrinking process are labelled as such, so the user is aware that these are somehow special and therefore will be less surprised when they can't simply be clicked on in IntelliJ to re-execute them. This doesn't add any new functionality, but it does help manage expectations.

2. Plugging in support for the first idea from before - cacheing the exact sequence of test cases from a previous run - turns out to be quite a tricky thing to implement deep down in TrialsImplementation, which at first sight looked like the best place. The major problem is the need for some key to associate a particular trials instance with the previous test run's cases - this has to be something that is defined across processes - we should expect to be able to reboot the computer and still reproduce the sequence of test template executions. Now, we could simply use the test itself as the key, rather than the trials instance, but TrialsImplementation is deliberately unaware of any test context - it just supplies test cases to a consumer either by pushing or as an iterator.

Furthermore, I'm not sure that folk using the lean and mean approach of running tests via the .supplyTo would even want this behaviour - the idea is to simply sit back and let the machinery hammer whatever test lambda is on the receiving end. Recall that the whole point of reproducing a previous run's shrinkage sequence is to support JUnit5 allowing a test template execution to be re-run in isolation, this isn't an option when using supplyTo.

So with that in mind, let's relegate any notion of reproducing previous shrinkage sequences to the JUnit5 support - which means that a) we can use the test itself as the key to fetch previous run data and b) we can report synthetic exceptions from TrialsTestExtension to fool TrialsImplementation into shrinking based on stored data that tracks how many test templates the extension has produced versus stored indices.

3. There should be a Java property that allows a recipe to override the generation of test cases, applicable in all cases via the TrialsImplementation. This is essentially just .withRecipe expressed via a property. It does mean that the current sprawling mess of JSON that existing recipes are expressed in need to be condensed down to something pithy enough to fit in a Java property. Now that opens up the possibility of storing recipes against a short-form key that can be used instead of the full recipe...

[sageserpent-open]

That Java property has a name: trials.recipeHash. It doesn't have to uniquely specify a test case, and it might be invalid or expired. This means that simple filtering of a sequence of streamed cases can be used to implement this...

[sageserpent-open]

So, after much huffing and puffing, the first and third ideas from above have been implemented. Test templates in JUnit that were generated as part of shrinkage have labels that reflect this - and the JVM property trials.recipeHash can be set to force the generation of just that corresponding test template (this works across the board at test case level, so holds for Scala users too).

There is another property too: trials.runDatabase that allows a choice of name for the database used to store recipes by their hashes. This database sits in the Java temporary directory - if no explicit name is chosen then a default is used.

[sageserpent-open]

Note shrinkage labelling and recipe hash in addition to the usual recipe JSON:

[sageserpent-open]

Setting the JVM property via -Dtrials.recipeHash=32b039ef1f828344710f330b0083ca03 yields this:

[sageserpent-open]

This has gone out in release 1.3.2, commit SHA: a834dbe5b33a02d087259a3efaf8199343d282d7