sageserpent-open / americium

Generation of test case data for Scala and Java, in the spirit of QuickCheck. When your test fails, it gives you a minimised failing test case and a way of reproducing the failure immediately.
MIT License
15 stars 1 forks source link

Make shrinkage generally wonderful. #33

Closed sageserpent-open closed 2 years ago

sageserpent-open commented 2 years ago
  1. Beat Jqwik at its own game. :smile:
  2. Be able to shrink the pathologically slow example that has been commented out in the shrinkage test in TrialsSpec in a reasonable time - this used to take around 20 minutes on my fancy MacBook, as opposed to 1 to 10 seconds for the other examples.
  3. Improve the quality of the shrinkage results in TrialsSpec.
sageserpent-open commented 2 years ago

This is a retrospective ticket, the work is already done. After much futile messing around with variations on multivariate shrinkage (again), I revisited an old idea from the start of working on Trials - namely to try using guided case generation with an element of value shrinkage thrown in - the code had got to the point where this became (comparatively) much easier to implement.

Kudos to the blog of David MacIver for jogging my memory back to reduction of an existing failed case, as opposed to simply generating cases within a more restricted search space, which is how Trials does its initial shrinkage.

There is now a 'panic mode' that is invoked when normal shrinkage fails to find any further failing cases; it takes the last failing case and tries variations in regenerating that case, adjusting the indices for any case factories to more 'minimal' values according to the factories' idea of the minimal case input. If this causes the decision steps for the case being built up to fundamentally diverge from those used to generated the last failing case, then it carries on the rest of the generation using the usual approach.

It turns out that this works a treat...

sageserpent-open commented 2 years ago

Results from TrialsSpec.it should "be shrunk to a simple case":

Provoking case for 'Has more than one text item whose converted values sum to more than 7.': Vector(13012, 37024) Provoking case for 'Has either four or five characters.': Vector(㿜, ঞ, 풍, 䮋) Provoking case for 'Has either four or five characters - variation.': Vector(⑕, 鄷, 灍, ) Provoking case for 'Has either four or five characters - variation with shrinkable character range.': Vector(q, q, q, q) Provoking case for 'Has either four or five characters - pathologically slow example.': Vector(䠘, ҿ, 쁂, 텡) Provoking case for 'Has more than one item and sums to more than 7.': Vector(0.0, 7.907022966713223) Provoking case for 'Has more than one item and sums to more than 7.': Vector(8, 0) Provoking case for 'Has more than one item and sums to more than 7.': Vector(8, 0) Provoking case for 'Has more than one item, no zeroes and sums to more than 7.': Vector(4, 4) Provoking case for 'Has more than one item, at least one zero and sums to more than 7.': Vector(8, 0) Provoking case for 'Has more than one item and sums to more than 7.': Vector(8, 0) Provoking case for 'Has more than one item, no zeroes and sums to more than 7.': Vector(3, 5) Provoking case for 'Has more than one item, at least one zero and sums to more than 7.': Vector(8, 0) Provoking case for 'Has more than 7 items.': Vector(0, 0, 0, 0, 0, 0, 0, 0) Provoking case for 'Has more than one item, at least one non-zero and sums to a multiple of 7.': Vector(0, 7) Provoking case for 'Has more than one item, no zeroes and sums to a multiple of 7.': Vector(1, -1) Provoking case for 'Has more than one item, at least one zero and sums to a multiple of 7.': Vector(0, 0) Provoking case for 'Has more than one item and sums to a positive multiple of 7.': Vector(0, 7) Provoking case for 'Has more than one item, no zeroes and sums to a positive multiple of 7.': Vector(3, 4) Provoking case for 'Has more than one item, at least one non-zero and sums to a positive multiple of 7.': Vector(0, 7) Provoking case for 'Flattened binary tree with more than one leaf that sums to a multiple of 19 greater than 19.': Vector(0, 38) Provoking case for 'Flattened binary tree with more than one leaf and no zeroes that sums to a multiple of 19 greater than 19.': Vector(-1, 39) Provoking case for 'Flattened binary tree with more than one leaf and at least one zero that sums to a multiple of 19 greater than 19.': Vector(38, 0) Provoking case for 'Has more than five items, at least one non-zero and sums to a multiple of 7.': Vector(1, -1, 0, 0, 0, 0) Provoking case for 'Has more than five items and sums to a multiple of 7 less than -7.': Vector(0, 0, -14, 0, 0, 0) Provoking case for 'Has more than two items and is not sorted in ascending order.': Vector(1, 0, 0) Provoking case for 'Has more than two items and no duplicates.': Vector(1, -1, 0) Provoking case for 'Flattened binary tree with more than two leaves whose odd-indexed leaves contain zeroes and even-indexed leaves contain non-zero values that sum to a multiple of 31 greater than 31.': Vector(46, 0, 16) Provoking case for 'List with more than two elements whose odd-indexed elements contain zeroes and even-indexed elements contain non-zero values that sum to a multiple of 31 greater than 31.': Vector(22, 0, 40)

sageserpent-open commented 2 years ago

Results from TrialsApiTests.copiedFromJqwik:

Trial exception with underlying cause: java.lang.AssertionError: Strings aren't allowed to be of length 4 or 5 characters in this test. Case: [aa,00] Reproduce with recipe: [ { "FactoryInputOf" : { "input" : 2 } }, { "FactoryInputOf" : { "input" : 97 } }, { "FactoryInputOf" : { "input" : 97 } }, { "FactoryInputOf" : { "input" : 2 } }, { "FactoryInputOf" : { "input" : 48 } }, { "FactoryInputOf" : { "input" : 48 } } ]

sageserpent-open commented 2 years ago

This has gone out in release 1.1.3, commit 0aa82efcf4997a570e55d61971dd46f2926b16f4.

Happy New Year!