SERG-Delft / andy

Andy assesses student's test code. It's used in CSE1110, TU Delft.
MIT License
78 stars 22 forks source link

Add config setting for max discard ration to 2 #250

Closed Zakrok09 closed 1 year ago

Zakrok09 commented 1 year ago

This PR adds another configuration step inside the jqwik config inside RunJUnitTestsStep that sets the maximum discard ratio to 2.

Additionally, this PR adds a new Solution fixture to check for discarding of a "correct" solution that uses Assume to filter out disproportionately big number of entries and changes the testMultiplePropertyTestsFailing method and relating fixture to test the scenario if a correct property with bad assumption filtering fails in the context of other failing property-based tests.

Closes #246

Zakrok09 commented 1 year ago

Hello again :wave: I added a new Solution fixture to check for discarding of a "correct" solution that uses Assume to filter out disproportionately big number of entries:

    @Property
    void testNoElementInWholeArrayWithAssumption(
            @ForAll @Size(value=10) List<@IntRange(min = -2100000000, max = 2100000000) Integer> numbers,
            @ForAll int valueToFind) {

        Assume.that(valueToFind > 2100000000 || valueToFind < -2100000000);

        int[] arr = convertListToArray(numbers);
        assertEquals(-1, ArrayUtils.indexOf(arr, valueToFind, 0));
    }

I chose the numbers 2100000000 to minimize the risk of flakiness in the test. It could be too unrealistic however.

Additionally, I added this same method to a fixture that runs multiple failing properties and asserted that it fails too, in order to test this scenario too.