Maven goal fl-report with -Dgzoltar.granularity=basicblock generates the same matrix.txt as the line granularity level

[Ojda22]

Context

I'm trying to generate matrices both for line and basic block coverage. More specifically, I want to obtain which test covered both lines and basic blocks. However, it seems that matrix.txt files for both granularity levels are the same.

Steps to Reproduce

1. I'm running mgzoltar maven plugin goal
2. Target project is https://github.com/jhy/jsoup.git
3. Executed for both commits 0f7e0cc373aced32629f5321c9521f81d8248647 , 895367092ec5567f97a7cd1ba2ce616507cc0978
4. In target project pom, I added

   <plugin>
       <groupId>org.apache.maven.plugins</groupId>
       <artifactId>maven-surefire-plugin</artifactId>
       <version>2.20</version>
       <configuration>
         <properties>
           <property>
             <name>listener</name>
             <value>com.gzoltar.internal.core.listeners.JUnitListener</value>
           </property>
         </properties>
       </configuration>
     </plugin>

5. After I cloned gzoltar locally, I did packaging (mvn package)

GZoltar arguments

1. I'm running the prepare-agent goal with mvn com.gzoltar:com.gzoltar.maven:1.7.2:prepare-agent test
2. I'm running the fl-report goal with mvn com.gzoltar:com.gzoltar.maven:1.7.2:fl-report -Dgzoltar.granularity=basicblock

Expected behaviour

I'm expecting to obtain spectra.csv files with different content (lines, basicblocks) After 2 separate runs with different granularity, I obtain the same output files (both matrix.txt, spectra.csv)

Environment (please complete the following information, if relevant):

• OS: MacOS

Additional comments

Please help

[jose]

Thanks @Ojda22 for reporting this!

I've just pushed a fix and here is a step-by-step on how to get coverage/fault-localization report at basic-block level.

Get latest version of GZoltar

mkdir /tmp/gz_issue_34
cd /tmp/gz_issue_34

git clone https://github.com/GZoltar/gzoltar.git gzoltar
cd /tmp/gz_issue_34/gzoltar
git checkout 065aabfa9f78e55d61ce66b2c96f8af2cb0d1421
mvn clean install

Get project under test

cd /tmp/gz_issue_34
git clone https://github.com/jhy/jsoup.git jsoup
cd /tmp/gz_issue_34/jsoup

git checkout 0f7e0cc373aced32629f5321c9521f81d8248647
mvn clean test # Tests run: 653, Failures: 0, Errors: 0, Skipped: 11

No failing test

As none of the test cases is failing, I change one to fail. Otherwise all components will have a 0% likelihood of being faulty.

diff --git a/src/test/java/org/jsoup/helper/StringUtilTest.java b/src/test/java/org/jsoup/helper/StringUtilTest.java
index 5af8c238..d1035443 100644
--- a/src/test/java/org/jsoup/helper/StringUtilTest.java
+++ b/src/test/java/org/jsoup/helper/StringUtilTest.java
@@ -38,7 +38,7 @@ public class StringUtilTest {
         assertTrue(StringUtil.isBlank(null));
         assertTrue(StringUtil.isBlank(""));
         assertTrue(StringUtil.isBlank("      "));
-        assertTrue(StringUtil.isBlank("   \r\n  "));
+        assertFalse(StringUtil.isBlank("   \r\n  "));

         assertFalse(StringUtil.isBlank("hello"));
         assertFalse(StringUtil.isBlank("   hello   "));

pom.xml

And I also adapted the pom.xml as follows

diff --git a/pom.xml b/pom.xml
index 4061cf22..b5d02576 100644
--- a/pom.xml
+++ b/pom.xml
@@ -150,6 +150,22 @@
         <artifactId>maven-release-plugin</artifactId>
         <version>2.5.3</version>
       </plugin>
+
+      <plugin>
+        <groupId>org.apache.maven.plugins</groupId>
+        <artifactId>maven-surefire-plugin</artifactId>
+        <version>2.20</version>
+        <configuration>
+          <testFailureIgnore>true</testFailureIgnore>
+          <properties>
+            <property>
+              <name>listener</name>
+              <value>com.gzoltar.internal.core.listeners.JUnitListener</value>
+            </property>
+          </properties>
+        </configuration>
+      </plugin>
+
     </plugins>
     <resources>
       <resource>

# Run project's test suite and collect coverage
mvn clean com.gzoltar:com.gzoltar.maven:1.7.3-SNAPSHOT:prepare-agent test

# Run fault-localization at line level
mvn com.gzoltar:com.gzoltar.maven:1.7.3-SNAPSHOT:fl-report -Dgzoltar.granularity=line
mv target/site/gzoltar gzoltar-line

# Run fault-localization at basicblock level
mvn com.gzoltar:com.gzoltar.maven:1.7.3-SNAPSHOT:fl-report -Dgzoltar.granularity=basicblock
mv target/site/gzoltar gzoltar-basicblock

Then using a tool like meld, e.g.,

meld gzoltar-line/sfl/txt/ochiai.ranking.csv gzoltar-basicblock/sfl/txt/ochiai.ranking.csv

you can easily see that the ranking at basicblock level is different/smaller.

PS: Feel free to report (i.e., open a new issue) if you find any inconsistency while using the basicblock granularity level.

-- Best, Jose

[Ojda22]

Many thanks @jose, it works like a charm now