Pitest not working with Weka SimpleKMeans

[fenghuadong]

I am pretty familiar with Pitest, tried it on various programs before. While I was trying to test a package of Weka, something interesting came up, and I believe it could be a issue, I hope you can take a look when you have time.

This is the simple test that I wrote.

`package weka.clusterers;

import static org.junit.Assert.*; import java.io.IOException; import java.nio.file.Files; import java.nio.file.Paths; import org.junit.Test;

public class SimpleKMeansTest {

@Test
public void Test1() throws IOException {
    int whatwaynum = 1;
    String whatway = "1way";

    SimpleKMeansGen instance = new SimpleKMeansGen();
    instance.Setway(whatwaynum);
    instance.generateresult();

    byte[] encoded = Files.readAllBytes(Paths.get("./inputs/K-Means-" + whatway + ".csv"));
    String file = new String(encoded, "UTF-8");
    String[] lines = file.split("\r\n");

    for (int j = 0; j < lines.length; j++){

        byte[] expencoded = Files.readAllBytes(Paths.get("./outputs/K-Means-" + whatway + "/" + Integer.toString(j) + ".txt"));
        String expoutputstring = new String(expencoded, "UTF-8");

        byte[]actencoded = Files.readAllBytes(Paths.get("./testoutputs/" + Integer.toString(j) + ".txt"));
        String actoutputstring = new String(expencoded, "UTF-8");

        assertEquals(expoutputstring,actoutputstring);

    }
}

}`

What it does is basically run SimpleKMeans with the exact same configuration and data, and store the result in txts, and compare them, confirmed with mvn -Dtest=TestApp1 test, the test passes. But Pitest will say not all tests are green. Even when I change the assertion to assertEquals("test","test");, basically comparing two identical strings, it still won't work. But if I remove SimpleKMeansGen instance = new SimpleKMeansGen(); instance.Setway(whatwaynum); instance.generateresult(); Pitest will start working obviously. Because the test would be simply comparing two identical string without actually executing anything. Looks like something went wrong when executing these 3 lines before the program even reached the assertEquals()

`

org.pitest

    <artifactId>pitest-maven</artifactId>
    <version>1.1.10</version>
    <configuration>
        <timeoutConstant>10000</timeoutConstant>
        <threads>1</threads>
        <mutators>
            <mutator>NEGATE_CONDITIONALS</mutator>
            <mutator>MATH</mutator>
        </mutators>
        <targetClasses>
            <param>weka.clusterers.SimpleKMeans</param>
            <param>weka.clusterers.Canopy</param>
            <param>weka.clusterers.FarthestFirst</param>
            <param>weka.clusterers.RandomizableClusterer</param>
            <param>weka.clusterers.AbstractClusterer</param>
        </targetClasses>
        <targetTests>
            <param>weka.clusterers.SimpleKMeansTest</param>
        </targetTests>
    </configuration>
  </plugin>`

Command line output is also attached. command line output.txt

[hcoles]

Hi @fenghuadong - thanks for raising this.

If you can parcel this up as a project on github that I can clone to reproduce the issue I'll take a look.

[fenghuadong]

Hello Henry, thank you very much . https://github.com/fenghuadong/Weka-SimpleKMeans-Pitest @hcoles

[hcoles]

If I checkout that project and run

mvn test

weka.clusterers.SimpleKMeansTest.Test1 fails with a NullPointerException.

[fenghuadong]

Sorry. made some changes and forgot to reverse it before uploaded it. in the outputs/K-Means-1way/1.txt, change it to 0.txt

[hcoles]

Could you push whatever changes are required to the repo and confirm that mvn test runs green for you.

[fenghuadong]

Yes sir, the newest one is pushed.

[hcoles]

Most of tests rely on some system properties that are not being set in the pitest configuration - that isn't the problem for the SimpleKMeansTest though.

Something strange is happening in the Weka code. It's throwing

ava.lang.NoClassDefFoundError: build/classes/weka/core/EuclideanDistance (wrong name: weka/core/EuclideanDistance)

when pitest runs the test.

I don't have time to dig any further just now but clearly pitest is setting up the environment in a way that differs somehow from what the surefire plugin does. From error it looks like Weka is somehow reading a path name and using this to determine the name a class should be loaded under.

[fenghuadong]

Thank you very much for looking into this Mr. Coles!

[fenghuadong]

Hello @hcoles One quick question, I am still working on testing the weka tool, I have many test cases in a testset(lets say 100 test cases), they are configured as parameterized tests so I don't have to create individual tests for all of them. I see pitest generated around 800 mutants for the files that I specified. For example, if test case #1 was able to kill mutant #1 to #799, will test case #2 still be executed against mutants #1 to #799 and #800? Or only mutant #800 since #1 to #799 have all been killed by previous test case.

[hcoles]

@fenghuadong

Pitest loads a mutant then keeps running tests until one of them kills it. Then it loads another mutant and runs tests until it kills it.

So if a mutant is killed by a test case no further tests will be run against it.

[fenghuadong]

@hcoles Got it. Thank you sir!

[fenghuadong]

@hcoles Hello Mr. Coles, I was wondering does pitest provide progress information during the execution? I am running pitest on some long running programs with hundreds of test cases and mutants, the program has been running for a week, and I don't know whether it is half way through or not even remotely close to finish. Is there any way that I can check the progress? Such as the # of mutants been tested and the # of yet to be tested mutants?