search

EvoSuite / evosuite

EvoSuite - automated generation of JUnit test suites for Java classes
http://www.evosuite.org
GNU Lesser General Public License v3.0
832 stars 341 forks source link

Invalid goal count error #296

Closed agb94 closed 4 years ago

agb94 commented 4 years ago

Context

Please provide below a detailed introduction to the issue itself, and describe what you were doing when the issue happened. Or, what do you want to achieve?

Hi! Thanks for developing and providing this tool.

I've encountered invalid goal count error while generating a test suite for Lang-1b in Defects4J benchmark. The progress shows

* Total number of goals: 356
* Number of covered goals: 342

, but it ends up producing the following errors.

Obtained invalid goal count: covered 357 out of 356

I don't why the # covered goals are inconsistent.

Steps to Reproduce

Please break down here below all the needed steps to reproduce the issue. [If possible, please upload an example of the project you are generating tests for.]

  1. Install defects4j v2
  2. defects4j checkout -p Lang -v 1b -w /tmp/lang-1b && defects4j compile
  3. java -cp /home/defects4j/framework/lib/test_generation/generation/evosuite-current.jar org.evosuite.EvoSuite -class org.apache.commons.lang3.math.NumberUtils -projectCP ./target/classes -seed 4001 -Dsearch_budget=50 -Dassertion_timeout=50 -Dtest_dir=./evosuite -Dcriterion line -Dstopping_condition=MaxTime -Dshow_progress=true -Djunit_check=false -Dfilter_assertions=false -Dtest_comments=true -Dcoverage=true -Dcoverage_matrix=true -Dreport_dir=evosuite-report/ -mem 1500

EvoSuite Arguments

Please provide the whole EvoSuite command you executed (if relevant)

java -cp /home/defects4j/framework/lib/test_generation/generation/evosuite-current.jar org.evosuite.EvoSuite -class org.apache.commons.lang3.math.NumberUtils -projectCP ./target/classes -seed 4001 -Dsearch_budget=50 -Dassertion_timeout=50 -Dtest_dir=./evosuite -Dcriterion line -Dstopping_condition=MaxTime -Dshow_progress=true -Djunit_check=false -Dfilter_assertions=false -Dtest_comments=true -Dcoverage=true -Dcoverage_matrix=true -Dreport_dir=evosuite-report/ -mem 1500

Current Result

Please describe here below the current result you got (if relevant) [if relevant, include a screenshot]

root@51209fc60c16:/tmp/lang-1b# java -cp /home/defects4j/framework/lib/test_generation/generation/evosuite-current.jar org.evosuite.EvoSuite     -class org.apache.commons.lang3.math.NumberUtils     -projectCP ./target/classes -Dsearch_budget=50   -seed=4001  -Dassertion_timeout=50     -Dtest_dir=./evosuite  -Dcriterion line -Dstopping_condition=MaxTime -Dshow_progress=true -Djunit_check=false -Dfilter_assertions=false -Dtest_comments=true -Dcoverage=true -Dcoverage_matrix=true -Dreport_dir=/home/evosuite-report/ -mem 1500 
* EvoSuite 1.0.6
* Going to generate test cases for class: org.apache.commons.lang3.math.NumberUtils
* Starting client
* Connecting to master process on port 10068
* Analyzing classpath: 
  - ./target/classes
* Finished analyzing classpath
* Generating tests for class org.apache.commons.lang3.math.NumberUtils
* Test criterion:
  - Line Coverage
* Setting up search algorithm for whole suite generation
[Progress:>                             0%] [Cov:>                                  0%]* Total number of test goals: 356
* Using seed 4001
* Starting evolution
[Progress:==============================100%] [Cov:=================================> 96%]
* Search finished after 51s and 271 generations, 111233 statements, best individual has fitness: 39.79659088760756
* Minimizing test suite
* Going to analyze the coverage criteria
* Coverage analysis for criterion LINE
* Coverage of criterion LINE: 96%
* Total number of goals: 356
* Number of covered goals: 342
* Generated 91 tests with total length 129
* Resulting test suite's coverage: 96%
* Generating assertions
* Resulting test suite's mutation score: 61%
* Writing JUnit test case 'NumberUtils_ESTest' to ./evosuite
* Done!

* Computation finished
[MASTER] 17:31:00.966 [main] ERROR RuntimeVariable - Obtained invalid goal count: covered 357 out of 356
[MASTER] 17:31:00.967 [main] ERROR SearchStatistics - Not going to write down statistics data, as some data is invalid
[MASTER] 17:31:01.067 [main] ERROR TestGeneration - failed to write statistics data

Expected result

Please describe here below what should be the expected behaviour (if relevant)

to produce the statistics file and the coverage matrix

Additional info

Please add any information of interest here below

apanichella commented 4 years ago

Hi, I just ran the five times (last version of Defects4j from git) but with no issue found (see log messages below). @agb94 can you check with the latest version of Defects4j? 

root@c74385632598:/tmp/lang_1_buggy# java -cp /home/defects4j/framework/lib/test_generation/generation/evosuite-current.jar org.evosuite.EvoSuite -class org.apache.commons.lang3.math.NumberUtils -projectCP ./target/classes -seed 4001 -Dsearch_budget=50 -Dassertion_timeout=50 -Dtest_dir=./evosuite -Dcriterion line -Dstopping_condition=MaxTime -Dshow_progress=true -Djunit_check=false -Dfilter_assertions=false -Dtest_comments=true -Dcoverage=true -Dcoverage_matrix=true -Dreport_dir=evosuite-report/ -mem 1500
* EvoSuite 1.0.6
* Going to generate test cases for class: org.apache.commons.lang3.math.NumberUtils
* Starting client
* Connecting to master process on port 10068
* Analyzing classpath: 
  - ./target/classes
* Finished analyzing classpath
* Generating tests for class org.apache.commons.lang3.math.NumberUtils
* Test criterion:
  - Line Coverage
* Setting up search algorithm for whole suite generation
* Total number of test goals: 356
[Progress:>                             0%] [Cov:>                                  0%]* Using seed 4001
* Starting evolution
[Progress:==============================100%] [Cov:=================================> 95%]
* Search finished after 52s and 209 generations, 85486 statements, best individual has fitness: 42.69123374475042
* Minimizing test suite
* Going to analyze the coverage criteria
* Coverage analysis for criterion LINE
* Coverage of criterion LINE: 96%
* Total number of goals: 356
* Number of covered goals: 341
* Generated 90 tests with total length 128
* Resulting test suite's coverage: 96%
* Generating assertions
* Resulting test suite's mutation score: 61%
* Writing JUnit test case 'NumberUtils_ESTest' to ./evosuite
* Done!

* Computation finished
agb94 commented 4 years ago

Hmm. it's weird. I've been using the latest version of defects4j (eaebff11cf6bdd723afbc0328b57cb7df2d106cc), and never changed anything of it. If you cannot reproduce this bug on your env, you may use the docker image agb94/d4j-v2. Btw, thanks for your answer! @apanichella 

# in docker
# docker run -dt --name d4j-v2 agb94/d4j-v2:latest
# docker exec -it d4j-v2 bash
defects4j checkout -p Lang -v 1b -w /tmp/lang_1_buggy
cd /tmp/lang_1_buggy
defects4j compile
java -cp /root/defects4j/framework/lib/test_generation/generation/evosuite-current.jar org.evosuite.EvoSuite -class org.apache.commons.lang3.math.NumberUtils -projectCP ./target/classes -seed 4001 -Dsearch_budget=50 -Dassertion_timeout=50 -Dtest_dir=./evosuite -Dcriterion line -Dstopping_condition=MaxTime -Dshow_progress=true -Djunit_check=false -Dfilter_assertions=false -Dtest_comments=true -Dcoverage=true -Dcoverage_matrix=true -Dreport_dir=evosuite-report/ -mem 1500

current output:

root@83d5e7f77c4e:~# cd defects4j/; git log -1; export PATH=$PATH:$PWD/framework/bin
commit eaebff11cf6bdd723afbc0328b57cb7df2d106cc
Author: Greg4cr <greg@greggay.com>
Date:   Wed May 6 10:24:07 2020 +0200

    Adds flaky test to failing_tests files for Jsoup (partial fix to #329)
root@83d5e7f77c4e:~/defects4j# defects4j checkout -p Lang -v 1b -w /tmp/lang_1_buggy
Checking out 687b2e62 to /tmp/lang_1_buggy................................. OK
Init local repository...................................................... OK
Tag post-fix revision...................................................... OK
Excluding broken/flaky tests............................................... OK
Excluding broken/flaky tests............................................... OK
Excluding broken/flaky tests............................................... OK
Initialize fixed program version........................................... OK
Apply patch................................................................ OK
Initialize buggy program version........................................... OK
Diff 687b2e62:2c454a4c..................................................... OK
Apply patch................................................................ OK
Tag pre-fix revision....................................................... OK
Check out program version: Lang-1b......................................... OK
root@83d5e7f77c4e:~/defects4j# cd /tmp/lang_1_buggy/
root@83d5e7f77c4e:/tmp/lang_1_buggy# defects4j compile
Running ant (compile)...................................................... OK
Running ant (compile.tests)................................................ OK
root@83d5e7f77c4e:/tmp/lang_1_buggy# java -cp /root/defects4j/framework/lib/test_generation/generation/evosuite-current.jar org.evosuite.EvoSuite -class org.apache.commons.lang3.math.NumberUtils -projectCP ./target/classes -seed 4001 -Dsearch_budget=50 -Dassertion_timeout=50 -Dtest_dir=./evosuite -Dcriterion line -Dstopping_condition=MaxTime -Dshow_progress=true -Djunit_check=false -Dfilter_assertions=false -Dtest_comments=true -Dcoverage=true -Dcoverage_matrix=true -Dreport_dir=evosuite-report/ -mem 1500
* EvoSuite 1.0.6
* Going to generate test cases for class: org.apache.commons.lang3.math.NumberUtils
* Starting client
* Connecting to master process on port 10068
* Analyzing classpath: 
  - ./target/classes
* Finished analyzing classpath
* Generating tests for class org.apache.commons.lang3.math.NumberUtils
* Test criterion:
  - Line Coverage
* Setting up search algorithm for whole suite generation
[Progress:>                             0%] [Cov:>                                  0%]* Total number of test goals: 356
* Using seed 4001
* Starting evolution
[Progress:==============================100%] [Cov:=================================> 96%]
* Search finished after 51s and 227 generations, 92394 statements, best individual has fitness: 43.4816577213586
* Minimizing test suite
* Going to analyze the coverage criteria
* Coverage analysis for criterion LINE
* Coverage of criterion LINE: 96%
* Total number of goals: 356
* Number of covered goals: 342
* Generated 91 tests with total length 129
* Resulting test suite's coverage: 96%
* Generating assertions
* Resulting test suite's mutation score: 62%
* Writing JUnit test case 'NumberUtils_ESTest' to ./evosuite
* Done!

* Computation finished
[MASTER] 08:59:00.285 [main] ERROR RuntimeVariable - Obtained invalid goal count: covered 357 out of 356
[MASTER] 08:59:00.289 [main] ERROR SearchStatistics - Not going to write down statistics data, as some data is invalid
[MASTER] 08:59:00.395 [main] ERROR TestGeneration - failed to write statistics data
apanichella commented 4 years ago

@agb94 I already used a docker container for the test. I run it 20 times, and the problem happens very rarely 1/20 runs. However, I checked with the latest version of EvoSuite (v1.0.7 / master branch), while Defects4j uses the version 1.0.6. The errors should be fixed in EvoSuite v1.0.7 as I could not replicate the error with the latest version.

I used the following commands to use the latest version of EvoSuite: cd /home git clone https://github.com/EvoSuite/evosuite.git cd /home/evosuite mvn package -DskipTests=true

Then, we can run EvoSuite on the same example as follows:

cd /tmp/lang_1_buggy

java -cp /home/evosuite/master/target/evosuite-master-1.0.7-SNAPSHOT.jar org.evosuite.EvoSuite -class org.apache.commons.lang3.math.NumberUtils -projectCP ./target/classes -seed 4001 -Dsearch_budget=50 -Dassertion_timeout=50 -Dtest_dir=./evosuite -Dcriterion=branch -Dstopping_condition=MaxTime -Dshow_progress=true -Djunit_check=false -Dfilter_assertions=false -Dtest_comments=true -Dcoverage=true -Dcoverage_matrix=true -Dreport_dir=evosuite-report/ -mem 1500 -generateMOSuite -Dalgorithm=DynaMOSA

Notice that in the latest command I used DynaMOSA as the search algorithm. In my trials, the coverage is higher with this configuration.