search

pholser / junit-quickcheck

Property-based testing, JUnit-style
MIT License
954 stars 121 forks source link

Does Pitest work with the junit-quickcheck property test tool? #337

Open ricardosm opened 3 years ago

ricardosm commented 3 years ago

I am trying to use the junit-quickcheck property testing tool together with Pirtest 1.6.4. However, I am encountering a problem. Pitest doesn't seem to be able to find the tests in my test class.

The message "PIT >> INFO: Found 0 tests" is always returned.

I believe that this problem occurs because the junit-quickcheck tool requires @RunWith (JUnitQuickcheck.class).

Does anyone have a solution to this problem?

Below is my code, test code and pom.xml:

package sort_selection_sort;

public class SelectionSort {

    public int[] sort(int arr[]) {
        int n = arr.length;

        // One by one move boundary of unsorted subarray
        for (int i = 0; i < n - 1; i++) {
            // Find the minimum element in unsorted array
            int min_idx = i;
            for (int j = i + 1; j < n; j++)
                if (arr[j] < arr[min_idx])
                    min_idx = j;

            // Swap the found minimum element with the first
            // element
            int temp = arr[min_idx];
            arr[min_idx] = arr[i];
            arr[i] = temp;
        }

        return arr;
    }
}
package sort_selection_sort;

import static org.junit.Assert.assertTrue;

import java.util.Arrays;

import org.junit.runner.RunWith;

import com.pholser.junit.quickcheck.Property;
import com.pholser.junit.quickcheck.runner.JUnitQuickcheck;

@RunWith(JUnitQuickcheck.class)
public class SelectionSortPropertiesTest {

    @Property(trials = 10)
    public void property(int arr[]) {
        SelectionSort ob = new SelectionSort();
        int[] copyOfArr = Arrays.copyOf(arr, arr.length);
        Arrays.sort(copyOfArr);
        int[] sort = ob.sort(arr);

        assertTrue(Arrays.equals(copyOfArr, sort));
    }
}

    <build>
        <plugins>
            <plugin>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>3.8.1</version>
            </plugin>

            <!-- Configuration of Maven Surefire Plugin for support to jUnit 5 (Unit 
                tests) -->
            <plugin>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>2.22.2</version>
            </plugin>

            <!-- Configuration of Pitest and Pitest jUnit 5 plugin -->
            <plugin>
                <groupId>org.pitest</groupId>
                <artifactId>pitest-maven</artifactId>
                <version>1.6.4</version>

                <configuration>
                    <targetClasses>
                        <param>sort_selection_sort.*</param>
                    </targetClasses>
                    <targetTests>
                        <param>sort_selection_sort.*</param>
                    </targetTests>
                    <mutators>
                        <mutator>ALL</mutator>
                    </mutators>
                    <verbose>true</verbose>
                </configuration>
            </plugin>
        </plugins>
    </build>
    <!-- ... -->
    <dependencies>

        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>4.13</version>
            <scope>test</scope>
        </dependency>

        <dependency>
            <groupId>com.pholser</groupId>
            <artifactId>junit-quickcheck-core</artifactId>
            <version>1.0</version>
        </dependency>

        <dependency>
            <groupId>com.pholser</groupId>
            <artifactId>junit-quickcheck-generators</artifactId>
            <version>1.0</version>
        </dependency>

        <dependency>
            <groupId>com.pholser</groupId>
            <artifactId>junit-quickcheck-guava</artifactId>
            <version>1.0</version>
        </dependency>

        <dependency>
            <groupId>org.slf4j</groupId>
            <artifactId>slf4j-api</artifactId>
            <version>1.7.30</version>
        </dependency>
        <dependency>
            <groupId>org.slf4j</groupId>
            <artifactId>slf4j-simple</artifactId>
            <version>1.7.30</version>
        </dependency>
        <!--  -->
    </dependencies>
16:55:16 PIT >> FINE : Maximum available memory is 3612 mb
16:55:16 PIT >> FINE : MINION : Installing PIT agent
16:55:16 PIT >> INFO : Sending 2 test classes to minion
16:55:16 PIT >> INFO : Sent tests to minion
16:55:17 PIT >> INFO : MINION : 16:55:16 PIT >> FINE : Expecting 2 tests classes from parent
16:55:17 PIT >> INFO : MINION : 16:55:16 PIT >> FINE : Tests classes received
16:55:17 PIT >> INFO : MINION : 16:55:16 PIT >> INFO : Checking environment
16:55:17 PIT >> INFO : MINION : 16:55:17 PIT >> INFO : Found  0 tests
16:55:17 PIT >> INFO : MINION : 16:55:17 PIT >> INFO : Dependency analysis reduced number of potential tests by 0
16:55:17 PIT >> INFO : MINION : 16:55:17 PIT >> INFO : 0 tests received
16:55:17 PIT >> INFO : MINION : 16:55:17 PIT >> FINE : Running 0 units
16:55:17 PIT >> FINE : Coverage generator Minion exited ok
16:55:17 PIT >> INFO : Calculated coverage in 0 seconds.
16:55:17 PIT >> FINE : Used memory after coverage calculation 23 mb
16:55:17 PIT >> FINE : Free Memory after coverage calculation 219 mb
16:55:17 PIT >> FINE : Incremental analysis set 153 mutations to a status of NOT_STARTED
16:55:17 PIT >> INFO : Incremental analysis reduced number of mutations by 0
16:55:17 PIT >> INFO : Created  1 mutation test units
16:55:17 PIT >> FINE : Used memory before analysis start 47 mb
16:55:17 PIT >> FINE : Free Memory before analysis start 136 mb
16:55:17 PIT >> FINE : Running 1 units
16:55:17 PIT >> INFO : Completed in 0 seconds
================================================================================
- Mutators
================================================================================
> org.pitest.mutationtest.engine.gregor.mutators.rv.AOR3Mutator
>> Generated 2 Killed 0 (0%)
> KILLED 0 SURVIVED 0 TIMED_OUT 0 NON_VIABLE 0 
> MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0 
> NO_COVERAGE 2 
--------------------------------------------------------------------------------
> org.pitest.mutationtest.engine.gregor.mutators.rv.AOR2Mutator
>> Generated 2 Killed 0 (0%)
> KILLED 0 SURVIVED 0 TIMED_OUT 0 NON_VIABLE 0 
> MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0 
> NO_COVERAGE 2 
--------------------------------------------------------------------------------
> org.pitest.mutationtest.engine.gregor.mutators.rv.AOR1Mutator
>> Generated 2 Killed 0 (0%)
> KILLED 0 SURVIVED 0 TIMED_OUT 0 NON_VIABLE 0 
> MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0 
> NO_COVERAGE 2 
--------------------------------------------------------------------------------
> org.pitest.mutationtest.engine.gregor.mutators.rv.AOR4Mutator
>> Generated 2 Killed 0 (0%)
> KILLED 0 SURVIVED 0 TIMED_OUT 0 NON_VIABLE 0 
> MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0 
> NO_COVERAGE 2 
--------------------------------------------------------------------------------
> org.pitest.mutationtest.engine.gregor.mutators.RemoveConditionalMutator_ORDER_IF
>> Generated 3 Killed 0 (0%)
> KILLED 0 SURVIVED 0 TIMED_OUT 0 NON_VIABLE 0 
> MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0 
> NO_COVERAGE 3 
--------------------------------------------------------------------------------
> org.pitest.mutationtest.engine.gregor.mutators.MathMutator
>> Generated 2 Killed 0 (0%)
> KILLED 0 SURVIVED 0 TIMED_OUT 0 NON_VIABLE 0 
> MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0 
> NO_COVERAGE 2 
--------------------------------------------------------------------------------
> org.pitest.mutationtest.engine.gregor.mutators.rv.CRCR3Mutator
>> Generated 3 Killed 0 (0%)
> KILLED 0 SURVIVED 0 TIMED_OUT 0 NON_VIABLE 0 
> MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0 
> NO_COVERAGE 3 
--------------------------------------------------------------------------------
> org.pitest.mutationtest.engine.gregor.mutators.ConditionalsBoundaryMutator
>> Generated 3 Killed 0 (0%)
> KILLED 0 SURVIVED 0 TIMED_OUT 0 NON_VIABLE 0 
> MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0 
> NO_COVERAGE 3 
--------------------------------------------------------------------------------
> org.pitest.mutationtest.engine.gregor.mutators.IncrementsMutator
>> Generated 2 Killed 0 (0%)
> KILLED 0 SURVIVED 0 TIMED_OUT 0 NON_VIABLE 0 
> MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0 
> NO_COVERAGE 2 
--------------------------------------------------------------------------------
> org.pitest.mutationtest.engine.gregor.mutators.experimental.RemoveIncrementsMutator
>> Generated 1 Killed 0 (0%)
> KILLED 0 SURVIVED 0 TIMED_OUT 0 NON_VIABLE 0 
> MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0 
> NO_COVERAGE 1 
--------------------------------------------------------------------------------
> org.pitest.mutationtest.engine.gregor.mutators.rv.CRCR4Mutator
>> Generated 2 Killed 0 (0%)
> KILLED 0 SURVIVED 0 TIMED_OUT 0 NON_VIABLE 0 
> MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0 
> NO_COVERAGE 2 
--------------------------------------------------------------------------------
> org.pitest.mutationtest.engine.gregor.mutators.rv.CRCR2Mutator
>> Generated 2 Killed 0 (0%)
> KILLED 0 SURVIVED 0 TIMED_OUT 0 NON_VIABLE 0 
> MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0 
> NO_COVERAGE 2 
--------------------------------------------------------------------------------
> org.pitest.mutationtest.engine.gregor.mutators.NullReturnValsMutator
>> Generated 1 Killed 0 (0%)
> KILLED 0 SURVIVED 0 TIMED_OUT 0 NON_VIABLE 0 
> MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0 
> NO_COVERAGE 1 
--------------------------------------------------------------------------------
> org.pitest.mutationtest.engine.gregor.mutators.rv.AOD1Mutator
>> Generated 2 Killed 0 (0%)
> KILLED 0 SURVIVED 0 TIMED_OUT 0 NON_VIABLE 0 
> MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0 
> NO_COVERAGE 2 
--------------------------------------------------------------------------------
> org.pitest.mutationtest.engine.gregor.mutators.InlineConstantMutator
>> Generated 3 Killed 0 (0%)
> KILLED 0 SURVIVED 0 TIMED_OUT 0 NON_VIABLE 0 
> MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0 
> NO_COVERAGE 3 
--------------------------------------------------------------------------------
> org.pitest.mutationtest.engine.gregor.mutators.rv.AOD2Mutator
>> Generated 2 Killed 0 (0%)
> KILLED 0 SURVIVED 0 TIMED_OUT 0 NON_VIABLE 0 
> MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0 
> NO_COVERAGE 2 
--------------------------------------------------------------------------------
> org.pitest.mutationtest.engine.gregor.mutators.ReturnValsMutator
>> Generated 1 Killed 0 (0%)
> KILLED 0 SURVIVED 0 TIMED_OUT 0 NON_VIABLE 0 
> MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0 
> NO_COVERAGE 1 
--------------------------------------------------------------------------------
> org.pitest.mutationtest.engine.gregor.mutators.rv.CRCR6Mutator
>> Generated 3 Killed 0 (0%)
> KILLED 0 SURVIVED 0 TIMED_OUT 0 NON_VIABLE 0 
> MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0 
> NO_COVERAGE 3 
--------------------------------------------------------------------------------
> org.pitest.mutationtest.engine.gregor.mutators.rv.CRCR1Mutator
>> Generated 1 Killed 0 (0%)
> KILLED 0 SURVIVED 0 TIMED_OUT 0 NON_VIABLE 0 
> MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0 
> NO_COVERAGE 1 
--------------------------------------------------------------------------------
> org.pitest.mutationtest.engine.gregor.mutators.rv.CRCR5Mutator
>> Generated 3 Killed 0 (0%)
> KILLED 0 SURVIVED 0 TIMED_OUT 0 NON_VIABLE 0 
> MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0 
> NO_COVERAGE 3 
--------------------------------------------------------------------------------
> org.pitest.mutationtest.engine.gregor.mutators.rv.ABSMutator
>> Generated 18 Killed 0 (0%)
> KILLED 0 SURVIVED 0 TIMED_OUT 0 NON_VIABLE 0 
> MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0 
> NO_COVERAGE 18 
--------------------------------------------------------------------------------
> org.pitest.mutationtest.engine.gregor.mutators.RemoveConditionalMutator_ORDER_ELSE
>> Generated 3 Killed 0 (0%)
> KILLED 0 SURVIVED 0 TIMED_OUT 0 NON_VIABLE 0 
> MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0 
> NO_COVERAGE 3 
--------------------------------------------------------------------------------
> org.pitest.mutationtest.engine.gregor.mutators.rv.ROR3Mutator
>> Generated 3 Killed 0 (0%)
> KILLED 0 SURVIVED 0 TIMED_OUT 0 NON_VIABLE 0 
> MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0 
> NO_COVERAGE 3 
--------------------------------------------------------------------------------
> org.pitest.mutationtest.engine.gregor.mutators.rv.ROR5Mutator
>> Generated 3 Killed 0 (0%)
> KILLED 0 SURVIVED 0 TIMED_OUT 0 NON_VIABLE 0 
> MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0 
> NO_COVERAGE 3 
--------------------------------------------------------------------------------
> org.pitest.mutationtest.engine.gregor.mutators.rv.ROR1Mutator
>> Generated 3 Killed 0 (0%)
> KILLED 0 SURVIVED 0 TIMED_OUT 0 NON_VIABLE 0 
> MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0 
> NO_COVERAGE 3 
--------------------------------------------------------------------------------
> org.pitest.mutationtest.engine.gregor.mutators.rv.ROR2Mutator
>> Generated 3 Killed 0 (0%)
> KILLED 0 SURVIVED 0 TIMED_OUT 0 NON_VIABLE 0 
> MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0 
> NO_COVERAGE 3 
--------------------------------------------------------------------------------
> org.pitest.mutationtest.engine.gregor.mutators.rv.ROR4Mutator
>> Generated 3 Killed 0 (0%)
> KILLED 0 SURVIVED 0 TIMED_OUT 0 NON_VIABLE 0 
> MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0 
> NO_COVERAGE 3 
--------------------------------------------------------------------------------
> org.pitest.mutationtest.engine.gregor.mutators.rv.UOI1Mutator
>> Generated 18 Killed 0 (0%)
> KILLED 0 SURVIVED 0 TIMED_OUT 0 NON_VIABLE 0 
> MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0 
> NO_COVERAGE 18 
--------------------------------------------------------------------------------
> org.pitest.mutationtest.engine.gregor.mutators.NegateConditionalsMutator
>> Generated 3 Killed 0 (0%)
> KILLED 0 SURVIVED 0 TIMED_OUT 0 NON_VIABLE 0 
> MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0 
> NO_COVERAGE 3 
--------------------------------------------------------------------------------
> org.pitest.mutationtest.engine.gregor.mutators.rv.UOI3Mutator
>> Generated 18 Killed 0 (0%)
> KILLED 0 SURVIVED 0 TIMED_OUT 0 NON_VIABLE 0 
> MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0 
> NO_COVERAGE 18 
--------------------------------------------------------------------------------
> org.pitest.mutationtest.engine.gregor.mutators.rv.UOI2Mutator
>> Generated 18 Killed 0 (0%)
> KILLED 0 SURVIVED 0 TIMED_OUT 0 NON_VIABLE 0 
> MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0 
> NO_COVERAGE 18 
--------------------------------------------------------------------------------
> org.pitest.mutationtest.engine.gregor.mutators.rv.UOI4Mutator
>> Generated 18 Killed 0 (0%)
> KILLED 0 SURVIVED 0 TIMED_OUT 0 NON_VIABLE 0 
> MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0 
> NO_COVERAGE 18 
--------------------------------------------------------------------------------
================================================================================
- Timings
================================================================================
> scan classpath : < 1 second
> coverage and dependency analysis : < 1 second
> build mutation tests : < 1 second
> run mutation analysis : < 1 second
--------------------------------------------------------------------------------
> Total  : < 1 second
--------------------------------------------------------------------------------
================================================================================
- Statistics
================================================================================
>> Generated 153 mutations Killed 0 (0%)
>> Mutations with no coverage 153. Test strength 0%
>> Ran 0 tests (0 tests per mutation)
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  1.596 s
[INFO] Finished at: 2021-04-20T16:55:17-03:00
[INFO] ------------------------------------------------------------------------
pholser commented 3 years ago

@hcoles Any advice you can offer? TBH I haven't tried running junit-quickcheck tests under pitest.

tj-byun commented 3 years ago

Pitest + junit-quickcheck will be super cool..! Strong test generator & oracle meets strong test adequacy criterion. @ricardosm Did you get it to work yet, by any chance?