GZoltar / gzoltar

GZoltar - Java Library for Automatic Debugging http://www.gzoltar.com
Other
77 stars 34 forks source link

ClassFormatError and NoSuchMethodError when using GZoltar to execute tests for JxPath #67

Open Instein98 opened 1 year ago

Instein98 commented 1 year ago

Context

When using GZoltar to execute tests for JxPath-1, most of the tests failed with unexpected errors, i.e., 201 ClassFormatError and 83 NoSuchMethodError.

Steps to Reproduce

  1. Create a script localize.sh with the following content. Please change the value of "work_dir" and "D4J_HOME" correspondingly.

    #!/bin/bash
    
    work_dir="/tmp/test"
    export D4J_HOME="/Users/xxx/defects4j/"
    rm -rf "$work_dir"; mkdir "$work_dir"
    
    export _JAVA_OPTIONS="-Xmx6144M -XX:MaxHeapSize=4096M"
    export MAVEN_OPTS="-Xmx1024M"
    export ANT_OPTS="-Xmx6144M -XX:MaxHeapSize=4096M"
    
    #
    # Get GZoltar
    #
    
    cd "$work_dir"
    if [ ! -d "$work_dir/gzoltar" ]; then
        git clone https://github.com/GZoltar/gzoltar.git
        cd "$work_dir/gzoltar"
        mvn clean package
    fi
    
    export GZOLTAR_AGENT_JAR="$work_dir/gzoltar/com.gzoltar.agent.rt/target/com.gzoltar.agent.rt-1.7.4-SNAPSHOT-all.jar"
    export GZOLTAR_CLI_JAR="$work_dir/gzoltar/com.gzoltar.cli/target/com.gzoltar.cli-1.7.4-SNAPSHOT-jar-with-dependencies.jar"
    
    # #
    # # Get D4J
    # #
    
    # cd "$work_dir"
    # git clone https://github.com/rjust/defects4j.git
    # cd "$work_dir/defects4j"
    # ./init.sh
    
    export TZ='America/Los_Angeles' # some D4J's requires this specific TimeZone
    export LC_ALL=en_US.UTF-8
    export LANG=en_US.UTF-8
    export LANGUAGE=en_US.UTF-8
    
    #
    # Checkout JxPath-1b, compile it, and get its metadata
    #
    
    PID="JxPath"
    BID="1"
    
    # Checkout
    cd "$work_dir"
    rm -rf "$PID-${BID}b"; "$D4J_HOME/framework/bin/defects4j" checkout -p "$PID" -v "${BID}b" -w "$PID-${BID}b"
    
    # Compile
    cd "$work_dir/$PID-${BID}b"
    "$D4J_HOME/framework/bin/defects4j" compile
    
    # Collect metadata
    cd "$work_dir/$PID-${BID}b"
    test_classpath=$($D4J_HOME/framework/bin/defects4j export -p cp.test)
    src_classes_dir=$($D4J_HOME/framework/bin/defects4j export -p dir.bin.classes)
    src_classes_dir="$work_dir/$PID-${BID}b/$src_classes_dir"
    test_classes_dir=$($D4J_HOME/framework/bin/defects4j export -p dir.bin.tests)
    test_classes_dir="$work_dir/$PID-${BID}b/$test_classes_dir"
    echo "$PID-${BID}b's classpath: $test_classpath" >&2
    echo "$PID-${BID}b's bin dir: $src_classes_dir" >&2
    echo "$PID-${BID}b's test bin dir: $test_classes_dir" >&2
    
    #
    # Collect unit tests to run GZoltar with
    #
    
    cd "$work_dir/$PID-${BID}b"
    unit_tests_file="$work_dir/$PID-${BID}b/unit_tests.txt"
    relevant_tests="*"  # Note, you might want to consider the set of relevant tests provided by D4J, i.e., $D4J_HOME/framework/projects/$PID/relevant_tests/$BID
    
    java -cp "$D4J_HOME/framework/projects/lib/junit-4.11.jar:$test_classpath:$test_classes_dir:$GZOLTAR_CLI_JAR" \
      com.gzoltar.cli.Main listTestMethods \
        "$test_classes_dir" \
        --outputFile "$unit_tests_file" \
        --includes "$relevant_tests"
    head "$unit_tests_file"
    
    #
    # Collect classes to perform fault localization on
    # Note: the `sed` commands below might not work on BSD-based distributions such as MacOS.
    #
    
    cd "$work_dir/$PID-${BID}b"
    
    loaded_classes_file="$D4J_HOME/framework/projects/$PID/loaded_classes/$BID.src"
    if [[ "$OSTYPE" == "darwin"* ]]; then
        normal_classes=$(cat "$loaded_classes_file" | gsed 's/$/:/' | gsed ':a;N;$!ba;s/\n//g')
        inner_classes=$(cat "$loaded_classes_file" | gsed 's/$/$*:/' | gsed ':a;N;$!ba;s/\n//g')
    else
        normal_classes=$(cat "$loaded_classes_file" | sed 's/$/:/' | sed ':a;N;$!ba;s/\n//g')
        inner_classes=$(cat "$loaded_classes_file" | sed 's/$/$*:/' | sed ':a;N;$!ba;s/\n//g')
    fi
    classes_to_debug="$normal_classes$inner_classes"
    echo "Likely faulty classes: $classes_to_debug" >&2
    
    #
    # Run GZoltar
    #
    
    cd "$work_dir/$PID-${BID}b"
    
    ser_file="$work_dir/$PID-${BID}b/gzoltar.ser"
    java -XX:MaxPermSize=4096M -javaagent:$GZOLTAR_AGENT_JAR=destfile=$ser_file,buildlocation=$src_classes_dir,includes=$classes_to_debug,excludes="",inclnolocationclasses=false,output="FILE" \
      -cp "$D4J_HOME/framework/projects/lib/junit-4.11.jar:$src_classes_dir:$test_classpath:$test_classes_dir:$GZOLTAR_CLI_JAR" \
      com.gzoltar.cli.Main runTestMethods \
        --testMethods "$unit_tests_file" \
        --collectCoverage
    
    #
    # Generate fault localization report
    #
    
    cd "$work_dir/$PID-${BID}b"
    
    java -cp "$D4J_HOME/framework/projects/lib/junit-4.11.jar:$src_classes_dir:$test_classpath:$GZOLTAR_CLI_JAR" \
        com.gzoltar.cli.Main faultLocalizationReport \
          --buildLocation "$src_classes_dir" \
          --granularity "line" \
          --inclPublicMethods \
          --inclStaticConstructors \
          --inclDeprecatedMethods \
          --dataFile "$ser_file" \
          --outputDirectory "$work_dir/$PID-${BID}b" \
          --family "sfl" \
          --formula "ochiai" \
          --metric "entropy" \
          --formatter "txt"
  2. Execute the script and redirect the output to a log file:
    bash localize.sh |& tee localize.log
  3. In the localize.log file, we can observe that there are 83 NoSuchMethodError and 201 ClassFormatError. However, JxPath-1b should only fail 2 tests with NullPointerException.

Expected behaviour

The test execution should only fail 2 tests with NullPointerException.

Environment (please complete the following information, if relevant):

I have reproduced the problem on both of my MacOS and Linux machines.

  1. MacOS 12.0.1 (21A559)
  2. Ubuntu 18.04.6 LTS
jose commented 1 year ago

Hi @Instein98,

Thanks for taking the time to report this issue, I will try to debug it on my end. But before I go ahead,

  1. Defects4J requires Java-8, what's the output of the following commands on your box?
javac -version
java -version
  1. After the checkout/compile step, could you please run
    cd "$work_dir/$PID-${BID}b"
    "$D4J_HOME/framework/bin/defects4j" test

    and let me know the content of the failing_tests file? Do you get the same content as in here?

-- Best, Jose

Instein98 commented 1 year ago

Thanks for your prompt reply! On my macOS machine:

% java -version 
openjdk version "1.8.0_292"
OpenJDK Runtime Environment (AdoptOpenJDK)(build 1.8.0_292-b10)
OpenJDK 64-Bit Server VM (AdoptOpenJDK)(build 25.292-b10, mixed mode)
% javac -version
javac 1.8.0_292

On my Linux (Ubuntu) machine:

$ javac -version
javac 1.8.0_362
$ java -version
openjdk version "1.8.0_362"
OpenJDK Runtime Environment (build 1.8.0_362-8u362-ga-0ubuntu1~18.04.1-b09)
OpenJDK 64-Bit Server VM (build 25.362-b09, mixed mode)

The output of running defects4j test in the /tmp/test/JxPath-1b/:

Running ant (compile.tests)................................................ OK
Running ant (run.dev.tests)................................................ OK
Failing tests: 2
  - org.apache.commons.jxpath.ri.model.dom.DOMModelTest::testGetNode
  - org.apache.commons.jxpath.ri.model.jdom.JDOMModelTest::testGetNode

Content in failing_tests file:

--- org.apache.commons.jxpath.ri.model.dom.DOMModelTest::testGetNode
java.lang.NullPointerException
    at org.apache.commons.jxpath.JXPathTestCase.assertXPathNodeType(JXPathTestCase.java:224)
    at org.apache.commons.jxpath.ri.model.dom.DOMModelTest.testGetNode(DOMModelTest.java:68)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at junit.framework.TestCase.runTest(TestCase.java:176)
    at junit.framework.TestCase.runBare(TestCase.java:141)
    at junit.framework.TestResult$1.protect(TestResult.java:122)
    at junit.framework.TestResult.runProtected(TestResult.java:142)
    at junit.framework.TestResult.run(TestResult.java:125)
    at junit.framework.TestCase.run(TestCase.java:129)
    at junit.framework.TestSuite.runTest(TestSuite.java:255)
    at junit.framework.TestSuite.run(TestSuite.java:250)
    at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:520)
    at org.apache.tools.ant.taskdefs.optional.junit.JUnitTask.executeInVM(JUnitTask.java:1484)
    at org.apache.tools.ant.taskdefs.optional.junit.JUnitTask.execute(JUnitTask.java:872)
    at org.apache.tools.ant.taskdefs.optional.junit.JUnitTask.executeOrQueue(JUnitTask.java:1972)
    at org.apache.tools.ant.taskdefs.optional.junit.JUnitTask.execute1(JUnitTask.java:824)
    at org.apache.tools.ant.taskdefs.optional.junit.JUnitTask.execute(JUnitTask.java:2277)
    at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:291)
    at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106)
    at org.apache.tools.ant.Task.perform(Task.java:348)
    at org.apache.tools.ant.Target.execute(Target.java:392)
    at org.apache.tools.ant.Target.performTasks(Target.java:413)
    at org.apache.tools.ant.Project.executeSortedTargets(Project.java:1399)
    at org.apache.tools.ant.Project.executeTarget(Project.java:1368)
    at org.apache.tools.ant.helper.DefaultExecutor.executeTargets(DefaultExecutor.java:41)
    at org.apache.tools.ant.Project.executeTargets(Project.java:1251)
    at org.apache.tools.ant.Main.runBuild(Main.java:811)
    at org.apache.tools.ant.Main.startAnt(Main.java:217)
    at org.apache.tools.ant.launch.Launcher.run(Launcher.java:280)
    at org.apache.tools.ant.launch.Launcher.main(Launcher.java:109)
--- org.apache.commons.jxpath.ri.model.jdom.JDOMModelTest::testGetNode
java.lang.NullPointerException
    at org.apache.commons.jxpath.JXPathTestCase.assertXPathNodeType(JXPathTestCase.java:224)
    at org.apache.commons.jxpath.ri.model.jdom.JDOMModelTest.testGetNode(JDOMModelTest.java:65)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at junit.framework.TestCase.runTest(TestCase.java:176)
    at junit.framework.TestCase.runBare(TestCase.java:141)
    at junit.framework.TestResult$1.protect(TestResult.java:122)
    at junit.framework.TestResult.runProtected(TestResult.java:142)
    at junit.framework.TestResult.run(TestResult.java:125)
    at junit.framework.TestCase.run(TestCase.java:129)
    at junit.framework.TestSuite.runTest(TestSuite.java:255)
    at junit.framework.TestSuite.run(TestSuite.java:250)
    at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:520)
    at org.apache.tools.ant.taskdefs.optional.junit.JUnitTask.executeInVM(JUnitTask.java:1484)
    at org.apache.tools.ant.taskdefs.optional.junit.JUnitTask.execute(JUnitTask.java:872)
    at org.apache.tools.ant.taskdefs.optional.junit.JUnitTask.executeOrQueue(JUnitTask.java:1972)
    at org.apache.tools.ant.taskdefs.optional.junit.JUnitTask.execute1(JUnitTask.java:824)
    at org.apache.tools.ant.taskdefs.optional.junit.JUnitTask.execute(JUnitTask.java:2277)
    at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:291)
    at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106)
    at org.apache.tools.ant.Task.perform(Task.java:348)
    at org.apache.tools.ant.Target.execute(Target.java:392)
    at org.apache.tools.ant.Target.performTasks(Target.java:413)
    at org.apache.tools.ant.Project.executeSortedTargets(Project.java:1399)
    at org.apache.tools.ant.Project.executeTarget(Project.java:1368)
    at org.apache.tools.ant.helper.DefaultExecutor.executeTargets(DefaultExecutor.java:41)
    at org.apache.tools.ant.Project.executeTargets(Project.java:1251)
    at org.apache.tools.ant.Main.runBuild(Main.java:811)
    at org.apache.tools.ant.Main.startAnt(Main.java:217)
    at org.apache.tools.ant.launch.Launcher.run(Launcher.java:280)
    at org.apache.tools.ant.launch.Launcher.main(Launcher.java:109)

Please let me know if you need more information.

jose commented 1 year ago

Hi @Instein98,

This issue has been confirmed and reproduced on my end and, unfortunately, there are also other unexpected failing tests on Defects4J v2.0.0 (see #71 and #72. I'm not sure when I will have the time to debug and fix those.

-- Best, Jose