MutationScore not not available when assertion_strategy=ALL

[baptisteroziere]

Context

I am trying to generate some pairs of (method/class, high-quality tests). I'm trying to generate tests for many methods or classes and to use some metrics to select the ones for which the tests were of high quality. For quality, I mostly want to make it difficult to write functions that are semantically incorrect and still pass the tests (I'm not really concerned with test verbosity). I am trying to maximize the mutation score as it seems to be a pretty good proxy for my "quality" metric and I'd like to keep all the assertions since useless assertions are not a problem for me. As I was playing with the parameters described in the tutorial, I encountered an issue when simultaneously setting the assertion strategy to all and asking for the MutationScore in the output.

Steps to Reproduce

Please break down here below all the needed steps to reproduce the issue. [If possible, please upload an example of the project you are generating tests for.]

1. Using the Stack class described in the tutorial in ./tutorial/Stack.java
2. Compile with javac: javac tutorial/Stack.java
3. Run evosuite: java -jar evosuite-1.1.0.jar -class tutorial.Stack -Doutput_variables=configuration_id,TARGET_CLASS,criterion,Total_Branches,Covered_Branches,BranchCoverage,Lines,Covered_Lines,LineCoverage,MethodCoverage,Size,Length,MutationScore -projectCP=. -Dminimize=false -Dsearch_budget=20 -Dassertion_strategy=ALL

EvoSuite Arguments

java -jar evosuite-1.1.0.jar -class tutorial.Stack  -Doutput_variables=configuration_id,TARGET_CLASS,criterion,Total_Branches,Covered_Branches,BranchCoverage,Lines,Covered_Lines,LineCoverage,MethodCoverage,Size,Length,MutationScore  -projectCP=.  -Dminimize=false  -Dsearch_budget=20 -Dassertion_strategy=ALL

Current Result

evosuite successfully generates tests and writes them to file but the MutationScore variable is missing and it fails to generate an output. Here is the end of the terminal output:

* Generated 11 tests with total length 66
* Resulting test suite's coverage: 93% (average coverage for all fitness functions)
* Generating assertions
* Compiling and checking tests
* Writing tests to file
* Writing JUnit test case 'Stack_ESTest' to evosuite-tests
* Done!

* Computation finished
[MASTER] 14:21:45.997 [main] ERROR SearchStatistics - No obtained value for output variable: MutationScore
[MASTER] 14:21:48.000 [main] ERROR SearchStatistics - No obtained value for output variable: MutationScore
[MASTER] 14:21:49.001 [main] ERROR SearchStatistics - No obtained value for output variable: MutationScore
[MASTER] 14:21:50.001 [main] ERROR SearchStatistics - No obtained value for output variable: MutationScore
[MASTER] 14:21:51.002 [main] ERROR SearchStatistics - No obtained value for output variable: MutationScore
[MASTER] 14:21:52.002 [main] ERROR SearchStatistics - No obtained value for output variable: MutationScore
[MASTER] 14:21:52.002 [main] ERROR SearchStatistics - Not going to write down statistics data, as some are missing
[MASTER] 14:21:52.103 [main] ERROR TestGeneration - failed to write statistics data

Expected result

I expected evosuite to also compute the strong mutation score and write it in the stats file.

Additional information

I downloaded the evosuite jar today using this link https://github.com/EvoSuite/evosuite/releases/download/v1.1.0/evosuite-1.1.0.jar

[jose]

Hi @brozi,

I believe you must inform EvoSuite to compute mutation score of the generated suite, e.g.,

-Danalysis_criteria="LINE,BRANCH,EXCEPTION,WEAKMUTATION,OUTPUT,METHOD,METHODNOEXCEPTION,CBRANCH,STRONGMUTATION"

By default, EvoSuite only computes coverage/mutation for the enabled criterion.

-- Best, Jose

[baptisteroziere]

Thank you @jose , it works !