Daikon DynComp IllegalArgumentException on a Defects4J Project

[CelloCorgi]

We see the issue #200 persist even with the latest version of Daikon and an example in Defects4J. We are creating a new Issue since #200 is closed.

Steps to reproduce:

1. Configuration:
   • Machine "Ubuntu 22.04.3 LTS" (Windows Subsystem for Linux 2),
   • JDK
   • openjdk version "1.8.0_392"
   • OpenJDK Runtime Environment (build 1.8.0_392-8u392-ga-1~22.04-b08)
   • OpenJDK 64-Bit Server VM (build 25.392-b08, mixed mode)
2. Setup Defects4J from here: [(https://github.com/rjust/defects4j)]
3. Checkout Math, bug 1

   defects4j checkout -p Math -v 1b -w Math1b
   cd Math1b
   Defects4j compile
   Defects4j test

4. Setup Daikon (5-8-18) and tested it on StackAr
5. Build the TestRunner class [https://github.com/boyang9602/APR_resources/blob/master/scripts/launcher/TestRunner.java] and build it
   • ~/defects4j$ javac -cp .:$DAIKONDIR/daikon.jar:./junit-4.13.2.jar:./hamcrest-core-1.3.jar:./Math1b/target/classes:./Math1b/target/test-classes TestRunner.java
   • ~/defects4j$ java -cp .:$DAIKONDIR/daikon.jar:./junit-4.13.2.jar:./hamcrest-core-1.3.jar:./Math1b/target/classes:./Math1b/target/test-classes daikon.DynComp --ppt-omit-pattern=org.junit --verbose TestRunner org.apache.commons.math3.linear.BlockFieldMatrixTest::testTranspose | tee diakon.DynComp.output.txt
   • Output:

     at org.junit.runners.model.TestClass.<init>(TestClass.java:48)
     at org.junit.runners.JUnit4.<init>(JUnit4.java:23)
     at org.junit.internal.builders.JUnit4Builder.runnerForClass(JUnit4Builder.java:10)
     at org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:70)
     at org.junit.internal.builders.AllDefaultPossibilitiesBuilder.runnerForClass(AllDefaultPossibilitiesBuilder.java:37)
     at org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:70)
     at org.junit.internal.requests.ClassRequest.createRunner(ClassRequest.java:28)
     at org.junit.internal.requests.MemoizingRequest.getRunner(MemoizingRequest.java:19)
     at org.junit.internal.requests.FilterRequest.getRunner(FilterRequest.java:36)
     at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
     at TestRunner.main(TestRunner.java:14)

The above issue shows up with both junit-4-13.2.jar as well as junit5-engine-5.0.0-ALPHA.jar. Any help is greatly appreciated!

[markro49]

I cannot reproduce on our standard linux boxes. I am trying Ubuntu 22.04.3 LTS" (Windows Subsystem for Linux 2) now.

[markro49]

I just realized that it is failing, but in a different way. I get no exceptions, but TestRunner outputs "FAILURE" instead of "SUCCESS". I'll have to look at this more but I'm curious why the results are different when I copied all you did. Could you please post the entire daikon.DynComp.output.txt file.

[CelloCorgi]

Sure! I have attached the file diakon.DynComp.output.txt

Thanks for your help with this!

[CelloCorgi]

If it is helpful for reproducing, here is our test runner as well:

import org.junit.runner.JUnitCore;
import org.junit.runner.Request;
import org.junit.runner.Result;
import org.junit.runner.notification.Failure;
 
 
public class TestRunner {
    public static void main(String... args) throws ClassNotFoundException {
        int retCode = 0;
        String resultMessage = "SUCCESS";
        for(String cm :  args) {
            String[] classAndMethod = cm.split("::");
            Request request = Request.method(Class.forName(classAndMethod[0]), classAndMethod[1]);
            Result result = new JUnitCore().run(request);
            if (!result.wasSuccessful()) {
                resultMessage = "FAILURE";
                for (Failure f : result.getFailures()) {
                    System.out.println(f.getTrace());
                }
            } else {
                resultMessage = "SUCCESS";
                }
            System.out.println(resultMessage);
        }
        System.exit(retCode);
    }
}

It has the stacktrace print, and also it returns SUCCESS when daikon.DynComp is removed

[markro49]

That explains the difference, your TestRunner is slightly different than the one you pointed to in the original post. Glad that's sorted.

[CelloCorgi]

Thanks for the repro! To clarify, the modification to TestRunner is needed only to understand the FAILURE cause with the stack trace, but the actual TestRunner still fails with FAILURE, instead of SUCCESS (when daikon.DynComp is absent).

[markro49]

The problem is using JUnitCore.run(request) instead of JUnitCore.main(args). run(request) takes a path through JUnit that we don't recognize as being part of JUnit so we generate the wrong instrumentation.
org.junit.runner.JUnitCore org.apache.commons.math3.linear.BlockFieldMatrixTest works fine - except it runs all the tests not just ::method I will look to see how hard 'request' is to support, but in the meantime you could modify the test to "@Ignore" all the tests you're not interested in and use JUnitCore instead of JunitRunner - bit of a pain, but it will work.

[CelloCorgi]

Thank you so much for your help on this. We look forward to a fix, but this gives us a path forward for now. Thanks again!

[markro49]

Unfortunately, there is no easy fix for this situation. The call to ClassForName on the tests class causes it to be loaded (and thus instrumented) before there is any reference to a JUnit class. Thus we do a standard instrumentation rather than the special JUnit instrumentation. I think a fix would require some sort of option to DynComp indicating this situation.

I'm not closing this issue as yet - I want to think about it some more.

[markro49]

I have figured out how to make this work. With today's merge of pull request 548, your original test case now runs to success.