org.junit.runner.manipulation.Filter#initializationError when running GZoltar on Closure

[IntHelloWorld]

Hi, sorry to border you. I have met a problem when testing Closure_27 in defects4j benchmark with junit-4.12.jar, something wrong happened when testing some of the testing methods, as showed below: <<<< com.google.debugging.sourcemap.SourceMapConsumerV1Test#testInvalidMappingArrayFailure,PASS,14944622, com.google.debugging.sourcemap.SourceMapConsumerV1Test#testMultipleMappingFragments,PASS,37103646, org.junit.runner.manipulation.Filter#initializationError,FAIL,1353876,java.lang.Exception: No tests found matching Method initializationError(com.google.debugging.sourcemap.SourceMapGeneratorV3Test) from org.junit.internal.requests.ClassRequest@7843cd84 at org.junit.internal.requests.FilterRequest.getRunner(FilterRequest.java:40) at org.junit.runner.JUnitCore.run(JUnitCore.java:115) at com.gzoltar.internal.core.test.junit.JUnitTestTask.call(JUnitTestTask.java:63) at com.gzoltar.internal.core.test.junit.JUnitTestTask.call(JUnitTestTask.java:27) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) com.google.debugging.sourcemap.Base64VLQTest#testBase64VLQSelectedValues1,PASS,3342966, com.google.debugging.sourcemap.Base64VLQTest#testBase64VLQSelectedValues2,PASS,1494154, <<<< the whole sfl_tests.csv file link is here: test.txt

I wonder if you have ever met this problem and know how to resolve this fault?

[jose]

Hi @IntHelloWorld,

It seems GZoltar identified and executed test cases that are not from the project under test, e.g., org.junit.runner.manipulation.Filter. That's a JUnit test from the JUnit project, not from the Closure project.

Could you please describe how are you executing GZoltar on Closure-27?

-- Best, Jose

[IntHelloWorld]

Hi @jose

Thanks for your fast response! I executed GZoltar on Closure-27 in this way: (1) I checkout Closure-27 down and then compile it by defects4j2.0.0 (2) I executed GZoltar on it through two .sh script (i changed them into .txt to show here) runGZoltar.txt pathUtil.txt

-- Best, CUE

[jose]

Hi @IntHelloWorld,

I failed to reproduce this issue with your scripts mostly due to missing .jar files (e.g., gzoltar/com.gzoltar.core/target/dependency/junit-4.12.jar, gzoltar/com.gzoltar.core/target/dependency/hamcrest-core-1.3.jar, gzoltar/com.gzoltar.core/target/dependency/easymock-4.2.jar, etc) and hardcoded paths.

Nevertheless here are the commands to successfully run the latest version of GZoltar (i.e., commit 9482dc5) on Closure-27 (Defects4J v2.0.0 aka commit 01f13c6):

#
# Setup
#

$ javac -version
javac 1.8.0_212

$ mvn --version
Apache Maven 3.6.1 (d66c9c0b3152b2e69ee9bac180bb8fcc8e6af555; 2019-04-04T20:00:29+01:00)

$ ant -version
Apache Ant(TM) version 1.9.13 compiled on July 10 2018

$ work_dir="/tmp/issue_30"
$ rm -rf "$work_dir"; mkdir "$work_dir"

#
# Get GZoltar
#

$ cd "$work_dir"
$ git clone https://github.com/GZoltar/gzoltar.git
$ cd "$work_dir/gzoltar"
$ git checkout 9482dc5f94a0f2a7488bda08c5d33d689da0a900
$ mvn clean package

$ export GZOLTAR_AGENT_JAR="$work_dir/gzoltar/com.gzoltar.agent.rt/target/com.gzoltar.agent.rt-1.7.3-SNAPSHOT-all.jar"
$ export GZOLTAR_CLI_JAR="$work_dir/gzoltar/com.gzoltar.cli/target/com.gzoltar.cli-1.7.3-SNAPSHOT-jar-with-dependencies.jar"

#
# Get D4J
#

$ cd "$work_dir"
$ git clone https://github.com/rjust/defects4j.git
$ cd "$work_dir/defects4j"
$ git checkout 01f13c69425fb8a0db290d12b1d48da1641bf6a9
$ ./init.sh

$ export D4J_HOME="$work_dir/defects4j"
$ export TZ='America/Los_Angeles' # some D4J's requires this specific TimeZone
$ export LC_ALL=en_US.UTF-8
$ export LANG=en_US.UTF-8
$ export LANGUAGE=en_US.UTF-8

#
# Checkout Closure-27, compile it, and get its metadata
#

$ PID="Closure"
$ BID="27"

# Checkout
$ cd "$work_dir"
$ rm -rf "$PID-${BID}b"; "$D4J_HOME/framework/bin/defects4j" checkout -p "$PID" -v "${BID}b" -w "$PID-${BID}b"

# Compile
$ cd "$work_dir/$PID-${BID}b"
$ "$D4J_HOME/framework/bin/defects4j" compile

# Collect metadata
$ cd "$work_dir/$PID-${BID}b"
$ test_classpath=$($D4J_HOME/framework/bin/defects4j export -p cp.test)
$ src_classes_dir=$($D4J_HOME/framework/bin/defects4j export -p dir.bin.classes)
$ src_classes_dir="$work_dir/$PID-${BID}b/$src_classes_dir"
$ test_classes_dir=$($D4J_HOME/framework/bin/defects4j export -p dir.bin.tests)
$ test_classes_dir="$work_dir/$PID-${BID}b/$test_classes_dir"
$ echo "$PID-${BID}b's classpath: $test_classpath" >&2
$ echo "$PID-${BID}b's bin dir: $src_classes_dir" >&2
$ echo "$PID-${BID}b's test bin dir: $test_classes_dir" >&2

#
# Collect unit tests to run GZoltar with
#

$ cd "$work_dir/$PID-${BID}b"
$ unit_tests_file="$work_dir/$PID-${BID}b/unit_tests.txt"
$ relevant_tests="*"  # Note, you might want to consider the set of relevant tests provided by D4J, i.e., $D4J_HOME/framework/projects/$PID/relevant_tests/$BID

$ java -cp "$test_classpath:$test_classes_dir:$D4J_HOME/framework/projects/lib/junit-4.11.jar:$GZOLTAR_CLI_JAR" \
  com.gzoltar.cli.Main listTestMethods \
    "$test_classes_dir" \
    --outputFile "$unit_tests_file" \
    --includes "$relevant_tests"
$ head "$unit_tests_file"

#
# Collect classes to perform fault localization on
#

$ cd "$work_dir/$PID-${BID}b"

$ loaded_classes_file="$D4J_HOME/framework/projects/$PID/loaded_classes/$BID.src"
$ normal_classes=$(cat "$loaded_classes_file" | sed 's/$/:/' | sed ':a;N;$!ba;s/\n//g')
$ inner_classes=$(cat "$loaded_classes_file" | sed 's/$/$*:/' | sed ':a;N;$!ba;s/\n//g')
$ classes_to_debug="$normal_classes$inner_classes"
$ echo "Likely faulty classes: $classes_to_debug" >&2

#
# Run GZoltar
#

$ cd "$work_dir/$PID-${BID}b"

$ ser_file="$work_dir/$PID-${BID}b/gzoltar.ser"
$ export _JAVA_OPTIONS="-Xmx6144M -XX:MaxHeapSize=4096M"
$ java -XX:MaxPermSize=4096M -javaagent:$GZOLTAR_AGENT_JAR=destfile=$ser_file,buildlocation=$src_classes_dir,includes=$classes_to_debug,excludes="",inclnolocationclasses=false,output="FILE" \
  -cp "$src_classes_dir:$D4J_HOME/framework/projects/lib/junit-4.11.jar:$test_classpath:$GZOLTAR_CLI_JAR" \
  com.gzoltar.cli.Main runTestMethods \
    --testMethods "$unit_tests_file" \
    --collectCoverage

#
# Generate fault localization report
#

$ cd "$work_dir/$PID-${BID}b"

$ java -cp "$src_classes_dir:$D4J_HOME/framework/projects/lib/junit-4.11.jar:$test_classpath:$GZOLTAR_CLI_JAR" \
    com.gzoltar.cli.Main faultLocalizationReport \
      --buildLocation "$src_classes_dir" \
      --granularity "line" \
      --inclPublicMethods \
      --inclStaticConstructors \
      --inclDeprecatedMethods \
      --dataFile "$ser_file" \
      --outputDirectory "$work_dir/$PID-${BID}b" \
      --family "sfl" \
      --formula "ochiai" \
      --metric "entropy" \
      --formatter "txt"

Once you execute this step-by-step, you can then check whether GZoltar reported the expected set of failing tests, check the list of most suspicious statements, get a statement coverage matrix ($work_dir/$PID-${BID}b/sfl/txt/matrix.txt) where each row represents the coverage of a test and its outcome (+ pass, - fail) and each column represents a statement in the program under test.

Does GZoltar report the expected set of failing tests? Yes.

As reported in here, officially, for Closure-27, there are 3 failing tests:

com.google.javascript.rhino.IRTest::testIssue727_1
com.google.javascript.rhino.IRTest::testIssue727_2
com.google.javascript.rhino.IRTest::testIssue727_3

And GZoltar does report the exact same set of failing tests:

$ grep ",FAIL," "$work_dir/$PID-${BID}b/sfl/txt/tests.csv" | cut -f1 -d','
com.google.javascript.rhino.IRTest#testIssue727_1
com.google.javascript.rhino.IRTest#testIssue727_2
com.google.javascript.rhino.IRTest#testIssue727_3

Most suspicious statements

$ head "$work_dir/$PID-${BID}b/sfl/txt/ochiai.ranking.csv"
name;suspiciousness_value
com.google.javascript.rhino$IR#tryCatch(com.google.javascript.rhino.Node,com.google.javascript.rhino.Node):231;0.8164965809277261
com.google.javascript.rhino$IR#tryCatch(com.google.javascript.rhino.Node,com.google.javascript.rhino.Node):232;0.8164965809277261
com.google.javascript.rhino$IR#tryCatch(com.google.javascript.rhino.Node,com.google.javascript.rhino.Node):233;0.8164965809277261
com.google.javascript.rhino$IR#catchNode(com.google.javascript.rhino.Node,com.google.javascript.rhino.Node):246;0.8164965809277261
com.google.javascript.rhino$IR#catchNode(com.google.javascript.rhino.Node,com.google.javascript.rhino.Node):247;0.8164965809277261
com.google.javascript.rhino$IR#catchNode(com.google.javascript.rhino.Node,com.google.javascript.rhino.Node):248;0.8164965809277261
com.google.javascript.rhino$IR#mayBeStatement(com.google.javascript.rhino.Node):501;0.8164965809277261
com.google.javascript.rhino$IR#tryFinally(com.google.javascript.rhino.Node,com.google.javascript.rhino.Node):224;0.5773502691896258
com.google.javascript.rhino$IR#tryCatchFinally(com.google.javascript.rhino.Node,com.google.javascript.rhino.Node,com.google.javascript.rhino.Node):239;0.5773502691896258

-- Best, Jose

[IntHelloWorld]

Hi @jose

Thanks a lot for your response! I will try the new version to see if everthing is going right.

Best, CUE

[QuanjunZhang]

Hi @jose Problem: I failed to run the script in some bugs (e.g., Closure-90 and later) from defecst4j. For example, when I ran Closure-90, it would return the follwing error message:

Setting: The Gzoltar and Defects4j are latest version and java version is jdk1.8.0_212.

Log: The execution logs of Closure-89(success) and 90(fail) are presented. Closure-89.log Closure-90.log

-- Best, Echo