-measureCoverage returns incorrect coverage values with Test Suites generated by EvoSuite

[LeandroMoura3]

Similar to this issue: https://github.com/EvoSuite/evosuite/issues/19

This is the EvoSuite gen:

Going to analyze the coverage criteria
Coverage analysis for criterion LINE
Coverage of criterion LINE: 100%
Total number of goals: 33
Number of covered goals: 33
Coverage analysis for criterion BRANCH
Coverage of criterion BRANCH: 100%
Total number of goals: 21
Number of covered goals: 21
Coverage analysis for criterion EXCEPTION
Coverage of criterion EXCEPTION: 100%
Total number of goals: 9
Number of covered goals: 9
Coverage analysis for criterion WEAKMUTATION
Coverage of criterion WEAKMUTATION: 95%
Total number of goals: 58
Number of covered goals: 55
Coverage analysis for criterion OUTPUT
Coverage of criterion OUTPUT: 100%
Total number of goals: 9
Number of covered goals: 9
Coverage analysis for criterion METHOD
Coverage of criterion METHOD: 100%
Total number of goals: 6
Number of covered goals: 6
Coverage analysis for criterion METHODNOEXCEPTION
Coverage of criterion METHODNOEXCEPTION: 100%
Total number of goals: 6
Number of covered goals: 6
Coverage analysis for criterion CBRANCH
Coverage of criterion CBRANCH: 100%
Total number of goals: 21
Number of covered goals: 21

This is the measureCoverage:

Coverage of criterion LINE: 100%
Number of covered goals: 33 / 33
Coverage of criterion BRANCH: 95%
Number of covered goals: 20 / 21
Coverage of criterion EXCEPTION: 100% (no goals)
Coverage of criterion WEAKMUTATION: 89%
Number of covered goals: 54 / 61
Coverage of criterion OUTPUT: 0%
Number of covered goals: 0 / 9
Coverage of criterion METHOD: 0%
Number of covered goals: 0 / 6
Coverage of criterion METHODNOEXCEPTION: 0%
Number of covered goals: 0 / 6
Coverage of criterion CBRANCH: 95%
Number of covered goals: 20 / 21
Total number of covered goals: 127 / 157
Total coverage: 81%

I am using evosuite-1.1.0 on win10.

[VoglSebastian]

Hi can you provide more information on how to reproduce this issue, e.g. used commands and class under test?

[LeandroMoura3]

Of course.

To gen the test suit, where arg1 is the class name and the other variable are paths. java -jar "%evosuitepath%\evosuite-1.1.0.jar" -class %arg1% -projectCP %projectCP% -Dsearch_budget=60 -Dshow_progress=false -Dwrite_covered_goals_file=true -Doutput_variables=TARGET_CLASS,Coverage,Size,Length,LineCoverage,BranchCoverage,MethodCoverage,OutputCoverage,WeakMutationScore,CBranchCoverage,ExceptionCoverage,Total_Goals,Covered_Goals,Lines,Covered_Lines,Total_Branches,Covered_Branches,Total_Methods,Covered_Methods,MutationScore,Generations,Total_Time,criterion > evosgen_%arg1%.txt 2> evosgen_%arg1%_error.txt

That give me the first report below.

Then, I compile the CUT, and so mensure the coverage with this: java -jar "%evosuitepath%\evosuite-1.1.0.jar" -class %arg1% -projectCP %projectCP%";.\evosuite-tests" -measureCoverage -Dcoverage_matrix=true > evoscoverage_%arg1%.txt 2> evoscoverage_%arg1%_error.txt

That give me the second report below. The matrix evosuite-report is wrong too.

Here is the class under test:

import java.util.HashMap;
import java.util.Map;
import java.lang.Math;

public class Bank {
    private final Map<String, Integer> accounts = new HashMap<String, Integer>();

    private void checkAccount(String name) throws Exception {
        if (!accounts.containsKey(name)) {
            throw new Exception("Account not found");
        }
    }

    private void checkValue(Integer value) throws Exception {
        if (value <= 0.0) {
            throw new Exception("Negative or zero value");
        }
    }

    public int balance(String name) throws Exception {
        checkAccount(name);
        return accounts.get(name);
    }

    public boolean createAccount(String name) throws Exception {
        if (name.equals("")) {
            throw new Exception("Empty name");
        } else if (accounts.containsKey(name)) {
            return false;
        } else {
            accounts.put(name, 0);
            return true;
        }
    }

    public boolean closeAccount(String name) throws Exception {
        checkAccount(name);
        if (accounts.get(name) == 0) {
            accounts.remove(name);
            return true;
        } else {
            return false;
        }
    }

    public boolean withdrawOrLoan(String name, Integer value) throws Exception {
        checkAccount(name);
        checkValue(value);
        if (Math.subtractExact(accounts.get(name), value) < 0.0) {
            final int newBalance = accounts.values().stream().reduce(0, Math::addExact) - value;
            if (newBalance / accounts.size() < 1000000) {
                return false;
            }
        }
        accounts.replace(name, Math.subtractExact(accounts.get(name), value));
        return true;
    }

    public void deposit(String name, Integer value) throws Exception {
        checkAccount(name);
        checkValue(value);
        accounts.replace(name, Math.addExact(accounts.get(name), value));
    }
}

[VoglSebastian]

I had a brief look into this, but don't know how to fix this: The measureCoverage execution mode has no access to the internal representation of a test case used by the test generation and uses dummy test cases and sets directly the execution trace with the assumption that this trace is enough to measure the coverage. Unfortunately, the Output,Method and MethodNoException criteria rely on the internal representation of the test case.

In CoverageAnalysis#analyzeCoverageCriterion https://github.com/EvoSuite/evosuite/blob/master/client/src/main/java/org/evosuite/junit/CoverageAnalysis.java#L458 the test case is set to a dummy TestCase and the ExecutionTrace is directly set.

The 3 above mentioned criteria then assume that an empty test case was executed and nothing is covered.