search

EvoSuite / evosuite

EvoSuite - automated generation of JUnit test suites for Java classes
http://www.evosuite.org
GNU Lesser General Public License v3.0
832 stars 341 forks source link

[Release 1.1.0] Unsupported type of fitness function: class org.evosuite.coverage.line.LineCoverageSuiteFitness #362

Closed victorgveloso closed 3 years ago

victorgveloso commented 3 years ago

Context

EvoSuite 1.1 fails to generate tests for a simple java project.

I've tried four alternative solutions:

  1. Java 8 + Evosuite release jar;
  2. Java 11 + Evosuite release jar;
  3. Docker with evosuite/evosuite:1.1.0-java-8 image;
  4. Docker with evosuite/evosuite:1.1.0-java-11 image;

Steps to Reproduce

  1. Fetching the example project: git clone https://github.com/victorgveloso/RingBufferedQueue .

  2. Fetching evosuite release jar: wget https://github.com/EvoSuite/evosuite/releases/download/v1.1.0/evosuite-1.1.0.jar

  3. Generating tests: Solution 1: /usr/lib/jvm/java-8-openjdk/jre/bin/java -jar evosuite-1.1.0.jar -prefix br.ufmg.aserg.victorveloso.queue -projectCP target/classes

    Solution 2: /usr/lib/jvm/java-11-openjdk/jre/bin/java -jar evosuite-1.1.0.jar -prefix br.ufmg.aserg.victorveloso.queue -projectCP target/classes

    Solution 3: docker run -it -u ${UID} -v ${PWD}:/evosuite evosuite/evosuite:1.1.0-java-8 -prefix br.ufmg.aserg.victorveloso.queue -projectCP target/classes

    Solution 4: docker run -it -u ${UID} -v ${PWD}:/evosuite evosuite/evosuite:1.1.0-java-11 -prefix br.ufmg.aserg.victorveloso.queue -projectCP target/classes

EvoSuite Arguments

-prefix br.ufmg.aserg.victorveloso.queue -projectCP target/classes

Current Result

* EvoSuite 1.1.1-SNAPSHOT
* Analyzing classpath (generating inheritance tree)
  - target/classes
* Found 1 matching classes for prefix br.ufmg.aserg.victorveloso.queue
* Current class: br.ufmg.aserg.victorveloso.queue.Queue
* Going to generate test cases for class: br.ufmg.aserg.victorveloso.queue.Queue
* Starting Client-0
* Connecting to master process on port 3475
* Analyzing classpath:
* Inheritance tree loaded from /tmp/ES_inheritancetree4731698684683663700.xml.gz
* Finished analyzing classpath
* Generating tests for class br.ufmg.aserg.victorveloso.queue.Queue
* Test criteria:
  - Line Coverage
  - Branch Coverage
  - Exception
  - Mutation testing (weak)
  - Method-Output Coverage
  - Top-Level Method Coverage
  - No-Exception Top-Level Method Coverage
  - Context Branch Coverage
* Setting up search algorithm for whole suite generation
Originally, MOSA was implemented with a 'RANK_CROWD_DISTANCE_TOURNAMENT' selection function. You may want to consider using it.
* Computation finished
[MASTER] 07:29:16.774 [logback-2] ERROR ClientNodeImpl - Error when generating tests for: br.ufmg.aserg.victorveloso.queue.Queue with seed 1623310155841. Configuration id : null
java.lang.IllegalArgumentException: Unsupported type of fitness function: class org.evosuite.coverage.line.LineCoverageSuiteFitness
    at org.evosuite.ga.metaheuristics.TestSuiteAdapter.mapFitnessFunctionToTestCaseLevel(TestSuiteAdapter.java:627)
    at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
    at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
    at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
    at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
    at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
    at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
    at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:566)
    at org.evosuite.ga.metaheuristics.TestSuiteAdapter.addFitnessFunctions(TestSuiteAdapter.java:634)
    at org.evosuite.strategy.WholeTestSuiteStrategy.generateTests(WholeTestSuiteStrategy.java:67)
    at org.evosuite.TestSuiteGenerator.generateTests(TestSuiteGenerator.java:630)
    at org.evosuite.TestSuiteGenerator.generateTestSuite(TestSuiteGenerator.java:205)
    at org.evosuite.rmi.service.ClientNodeImpl.lambda$startNewSearch$0(ClientNodeImpl.java:150)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
[MASTER] 07:29:16.878 [main] ERROR SearchStatistics - No statistics has been saved because EvoSuite failed to generate any test case
[MASTER] 07:29:16.979 [main] ERROR TestGeneration - failed to write statistics data

Expected result

EvoSuite should generate tests correctly. I've used EvoSuite previous release (1.0.6) through maven (configuring the pom.xml file) and it works flawlessly (see the output below).

[INFO] Scanning for projects...
[INFO]
[INFO] ------------------< br.ufv.aserg.victorveloso:Queue >-------------------
[INFO] Building Queue 1.0-SNAPSHOT
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] >>> evosuite-maven-plugin:1.0.6:generate (default-cli) > compile @ Queue >>>
[INFO]
[INFO] --- jacoco-maven-plugin:0.8.2:prepare-agent (default) @ Queue ---
[INFO] argLine set to -javaagent:/home/victor/.m2/repository/org/jacoco/org.jacoco.agent/0.8.2/org.jacoco.agent-0.8.2-runtime.jar=destfile=/home/victor/Universidade/Master/TestGeneration/ToolEvaluation/EvoSuite/RingBufferedQueue/target/jacoco.exec
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ Queue ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /home/victor/Universidade/Master/TestGeneration/ToolEvaluation/EvoSuite/RingBufferedQueue/src/main/resources
[INFO]
[INFO] --- maven-compiler-plugin:3.8.1:compile (default-compile) @ Queue ---
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] <<< evosuite-maven-plugin:1.0.6:generate (default-cli) < compile @ Queue <<<
[INFO]
[INFO]
[INFO] --- evosuite-maven-plugin:1.0.6:generate (default-cli) @ Queue ---
[INFO] Going to generate tests with EvoSuite
[INFO] Total memory: 800mb
[INFO] Time per class: 2 minutes
[INFO] Number of used cores: 1
[INFO] Target: /home/victor/Universidade/Master/TestGeneration/ToolEvaluation/EvoSuite/RingBufferedQueue/target/classes
[INFO] Basedir: /home/victor/Universidade/Master/TestGeneration/ToolEvaluation/EvoSuite/RingBufferedQueue
[INFO] Started spawn process manager on port 36395
[INFO] SLF4J: Class path contains multiple SLF4J bindings.
[INFO] SLF4J: Found binding in [jar:file:/home/victor/.m2/repository/org/evosuite/evosuite-master/1.0.6/evosuite-master-1.0.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
[INFO] SLF4J: Found binding in [jar:file:/home/victor/.m2/repository/ch/qos/logback/logback-classic/1.1.3/logback-classic-1.1.3.jar!/org/slf4j/impl/StaticLoggerBinder.class]
[INFO] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
[INFO] SLF4J: Actual binding is of type [ch.qos.logback.classic.util.ContextSelectorStaticBinder]
[INFO] * EvoSuite 1.0.6
[INFO] Registered remote process from /127.0.0.1:46250
[INFO] Going to execute 1 jobs
[INFO] Estimated completion time: 2 minutes, by 2021-06-10T04:36:08.972
[INFO] Going to start job for: br.ufmg.aserg.victorveloso.queue.Queue. Expected to end in 120 seconds, by 2021-06-10T04:36:08.985
[INFO] Registered remote process from /127.0.0.1:46252
[INFO] Registered remote process from /127.0.0.1:46254
[INFO] Completed job. Left: 0
[INFO] * Updating database to br.ufmg.aserg.victorveloso.queue.Queue
[INFO] === CTG run results ===
[INFO] Removed test suites: 0
[INFO] New test suites: 1
[INFO] Stopping spawn process manager
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  01:17 min
[INFO] Finished at: 2021-06-10T04:35:25-03:00
[INFO] ------------------------------------------------------------------------

Additional info

  1. While EvoSuite 1.0.6 works for my example project, it doesn't support Java 9+ and I would like to generate tests for Java 9+ projects.
  2. Running EvoSuite 1.0.6 generates four logs std_(err|out)_(CLIENT|MASTER).log:
    • std_err_MASTER.log is empty
    • std_out_MASTER.log has four lines that describes configuration and ends with Computation finished
    • std_err_CLIENT.log has 493 lines, all of them are similar to the following template: [MASTER] 05:03:32.675 [TEST_EXECUTION_THREAD_0] ERROR AbstractStatement - Error assigning value of type Queue[] defined at statement Queue<String>[] queueArray0 = (Queue<String>[]) Array.newInstance(Queue.class, 5), assignment statement: queueArray0[0] = (Queue<String>) queueArray0; SUT=br.ufmg.aserg.victorveloso.queue.Queue
      • The only two differences are
        1. Array.newInstance(Queue.class, #), being # any number from 1 to 19
        2. the timestamp
    • std_out_CLIENT.log seems to be the expected result if I were using the command line jar:
    • Starting client
    • Connecting to master process on port 16006
    • Analyzing classpath:
    • /home/victor/Universidade/Master/TestGeneration/ToolEvaluation/EvoSuite/RingBufferedQueue/target/classes
    • /home/victor/.m2/repository/org/junit/jupiter/junit-jupiter/5.6.2/junit-jupiter-5.6.2.jar
    • /home/victor/.m2/repository/org/junit/jupiter/junit-jupiter-api/5.6.2/junit-jupiter-api-5.6.2.jar
    • /home/victor/.m2/repository/org/apiguardian/apiguardian-api/1.1.0/apiguardian-api-1.1.0.jar
    • /home/victor/.m2/repository/org/opentest4j/opentest4j/1.2.0/opentest4j-1.2.0.jar
    • /home/victor/.m2/repository/org/junit/platform/junit-platform-commons/1.6.2/junit-platform-commons-1.6.2.jar
    • /home/victor/.m2/repository/org/junit/jupiter/junit-jupiter-params/5.6.2/junit-jupiter-params-5.6.2.jar
    • /home/victor/.m2/repository/org/junit/jupiter/junit-jupiter-engine/5.6.2/junit-jupiter-engine-5.6.2.jar
    • /home/victor/.m2/repository/org/junit/platform/junit-platform-engine/1.6.2/junit-platform-engine-1.6.2.jar
    • /home/victor/.m2/repository/org/assertj/assertj-core/3.11.1/assertj-core-3.11.1.jar
    • Finished analyzing classpath
    • Generating tests for class br.ufmg.aserg.victorveloso.queue.Queue
    • Test criteria:
    • Line Coverage
    • Branch Coverage
    • Exception
    • Mutation testing (weak)
    • Method-Output Coverage
    • Top-Level Method Coverage
    • No-Exception Top-Level Method Coverage
    • Context Branch Coverage
    • Setting up search algorithm for whole suite generation [MASTER] 05:02:26.729 [pool-2-thread-1] WARN TestSuiteSerialization - Cannot load tests because file does not exist: /home/victor/Universidade/Master/TestGeneration/ToolEvaluation/EvoSuite/RingBufferedQueue/.evosuite/evosuite-seeds/br.ufmg.aserg.victorveloso.queue.Queue.seed
    • Loaded 0 tests from /home/victor/Universidade/Master/TestGeneration/ToolEvaluation/EvoSuite/RingBufferedQueue/.evosuite/evosuite-seeds/br.ufmg.aserg.victorveloso.queue.Queue.seed
    • Total number of test goals:
    • Line 55
    • Branch 36
    • Exception 0
    • MutationFactory 163
    • Output 30
    • Method 14
    • MethodNoException 14
    • CBranchFitnessFactory 36
    • Using seed 1623312145763
    • Starting evolution
    • Search finished after 66s and 658 generations, 259939 statements, best individual has fitness: 10.1
    • Writing sequences to pool
    • Minimizing test suite
    • Going to analyze the coverage criteria
    • Coverage analysis for criterion LINE
    • Coverage of criterion LINE: 100%
    • Total number of goals: 55
    • Number of covered goals: 55
    • Coverage analysis for criterion BRANCH
    • Coverage of criterion BRANCH: 100%
    • Total number of goals: 36
    • Number of covered goals: 36
    • Coverage analysis for criterion EXCEPTION
    • Coverage of criterion EXCEPTION: 100%
    • Total number of goals: 9
    • Number of covered goals: 9
    • Coverage analysis for criterion WEAKMUTATION
    • Coverage of criterion WEAKMUTATION: 98%
    • Total number of goals: 163
    • Number of covered goals: 159
    • Coverage analysis for criterion OUTPUT
    • Coverage of criterion OUTPUT: 73%
    • Total number of goals: 30
    • Number of covered goals: 22
    • Coverage analysis for criterion METHOD
    • Coverage of criterion METHOD: 100%
    • Total number of goals: 14
    • Number of covered goals: 14
    • Coverage analysis for criterion METHODNOEXCEPTION
    • Coverage of criterion METHODNOEXCEPTION: 100%
    • Total number of goals: 14
    • Number of covered goals: 14
    • Coverage analysis for criterion CBRANCH
    • Coverage of criterion CBRANCH: 100%
    • Total number of goals: 36
    • Number of covered goals: 36
    • Generated 29 tests with total length 127
    • Resulting test suite's coverage: 96% (average coverage for all fitness functions)
    • Generating assertions
    • Resulting test suite's mutation score: 88%
    • Compiling and checking tests
    • Writing JUnit test case 'Queue_ESTest' to /home/victor/Universidade/Master/TestGeneration/ToolEvaluation/EvoSuite/RingBufferedQueue/.evosuite/tmp_2021_06_10_05_02_24/tests
    • Done!
gofraser commented 3 years ago

Thanks for reporting, this should be fixed as of 62621a30d4d58d3ac62158447fd593b4f90692bc.