EvoSuite / evosuite

EvoSuite - automated generation of JUnit test suites for Java classes
http://www.evosuite.org
GNU Lesser General Public License v3.0
832 stars 341 forks source link

Covered goals greater than the number of goals when 100% coverage is reached #367

Open PdedP opened 3 years ago

PdedP commented 3 years ago

Context

Similar to #347, there seems to be a bug in the covered goals count when 100% coverage is reached. It happens with different classes under test. As a result, statistic data is not written.

I am using the last .jar file (downloaded on 7th, June). I have also tried by building the code locally with: mvn clean compile / mvn package -DskipTests=true.

Steps to Reproduce

  1. For example, we can use the ATM.java class that can be downloaded from EvoSuite's tutorial
wget http://evosuite.org/files/tutorial/Tutorial_Experiments.zip
unzip Tutorial_Experiments.zip
  1. Run EvoSuite (see below)

I attach the zip file with the necessary files to reproduce the bug. ATM_Example.zip

EvoSuite Arguments

java -jar evosuite-1.1.0.jar -class tutorial.ATM -projectCP target/classes:target/dependency/commons-collections-3.2.2.jar

Current Result

* EvoSuite 1.1.0
* Going to generate test cases for class: tutorial.ATM
* Starting Client-0
* Connecting to master process on port 6204
* Analyzing classpath: 
  - target/classes
  - target/dependency/commons-collections-3.2.2.jar
* Finished analyzing classpath
* Generating tests for class tutorial.ATM
* Test criteria:
  - Line Coverage
  - Branch Coverage
  - Exception
  - Mutation testing (weak)
  - Method-Output Coverage
  - Top-Level Method Coverage
  - No-Exception Top-Level Method Coverage
  - Context Branch Coverage
* Total number of test goals for DYNAMOSA: 119
* Using seed 1625221421920
* Starting evolution
[Progress:>                             0%] [Cov:>                                  0%]* Initial Number of Goals in DynaMOSA = 29 / 119
[Progress:==============================100%] [Cov:================================>  94%]
* Search finished after 61s and 427 generations, 374690 statements, best individual has fitness: 4.0
* Minimizing test suite
* Going to analyze the coverage criteria
* Coverage analysis for criterion LINE
* Coverage of criterion LINE: 100%
* Total number of goals: 31
* Number of covered goals: 31
* Coverage analysis for criterion BRANCH
* Coverage of criterion BRANCH: 100%
* Total number of goals: 20
* Number of covered goals: 20
* Coverage analysis for criterion EXCEPTION
* * Coverage of criterion EXCEPTION: 100%
* Total number of goals: 4
* Number of covered goals: 4
* Coverage analysis for criterion WEAKMUTATION
* Coverage of criterion WEAKMUTATION: 97%
* Total number of goals: 36
* Number of covered goals: 35
* Coverage analysis for criterion OUTPUT
* Coverage of criterion OUTPUT: 100% (no goals)
* Coverage analysis for criterion METHOD
* Coverage of criterion METHOD: 100%
* Total number of goals: 6
* Number of covered goals: 6
* Coverage analysis for criterion METHODNOEXCEPTION
* Coverage of criterion METHODNOEXCEPTION: 100%
* Total number of goals: 6
* Number of covered goals: 6
* Coverage analysis for criterion CBRANCH
* Coverage of criterion CBRANCH: 100%
* Total number of goals: 20
* Number of covered goals: 20
* Generated 16 tests with total length 114
* Resulting test suite's coverage: 94% (average coverage for all fitness functions)
* Generating assertions
* Resulting test suite's mutation score: 58%
* Compiling and checking tests
* Writing tests to file
* Writing JUnit test case 'ATM_ESTest' to evosuite-tests
* Done!

* Computation finished 
[MASTER] 12:24:58.774 [main] ERROR RuntimeVariable - Obtained invalid goal count: covered 122 out of 119
[MASTER] 12:24:58.775 [main] ERROR SearchStatistics - Not going to write down statistics data, as some data is invalid
[MASTER] 12:24:58.877 [main] ERROR TestGeneration - failed to write statistics data

Expected result

As it can be seen, the number of covered goals is 122, but the actual number of goals is 119. Statistic data is not written. Note also that the sum of the goals of each criterion is not equal to 119 but 123.

Additional info

I have tried with different classes to understand what was going on:

[MASTER] 12:49:42.857 [main] ERROR RuntimeVariable - Obtained invalid goal count: covered 101 out of 100
[MASTER] 12:49:42.859 [main] ERROR SearchStatistics - Not going to write down statistics data, as some data is invalid
[MASTER] 12:49:42.964 [main] ERROR TestGeneration - failed to write statistics data

The rest of the times (when not all goals are covered) statistics are written.

Thanks in advance.

VoglSebastian commented 3 years ago

Hi, the logs show that you are using version 1.1.0. As already mentioned in a comment on the issue you referenced, this bug should be fixed in the current master branch.

I think using the current master branch (or waiting for the next release) will solve this issue.