EvoSuite / evosuite

EvoSuite - automated generation of JUnit test suites for Java classes
http://www.evosuite.org
GNU Lesser General Public License v3.0
832 stars 341 forks source link

Corner case issue with the Instrumentation of subclasses #288

Open emaiannone opened 4 years ago

emaiannone commented 4 years ago

Context

Hello, I'm extending EvoSuite in order to add an additional coverage criterion. Without going into deep details, I can say that a single coverage goal is identified by a triple (targetClass, targetMethod, targetLine). My approach does not generate tests for targetClass, but for any classes that is able to reach targetClass, directly or indirectly.
So a common scenario is:
CUT -> ... -> TargetClass

Most of the cases, instrumentation works fine: I can fetch all bytecode lines I need, I can extract CFGs and CDGs in order to build by coverage goals... but I think I found a particular case where EvoSuite has some problem (or maybe it's me that I don't properly set a secret Property).

Briefly:

  1. I have a CUT A that should cover a target of class B. B is not called directly by A, in fact A passes though B2, one of its subclass.
  2. B2 does nothing in particular: it just inherit targetMethod(), without modifyng it;
  3. A calls the targetMethod() of B2 (that should directly reference targetMethod() of B);
  4. For unknown reason, the instrumentation does not correctly record the fact that targetMethod() of B is called.

I try to summarize some of my console prints

When I run the above scenario, this is what comes out:

WARN  org.evosuite.assertion.CheapPurityAnalyzer - org.example.hierarchy.BadChild was not found in the inheritance tree. Using DEFAULT value for cheap-purity analysis

It's just a simple warning and it always happen when I have a hierarchy in my classpath. The above example if problematic, but there are other instances where I use hierarchies without problems with the said message still present (later I will provide two examples of hierarchies).

I am able to successfully fetch other things (the X says that everything is okay, a missing X means that it is not okay):

Whenever I try to get the CDG I get this error:

ERROR org.evosuite.rmi.service.ClientNodeImpl - Error when generating tests for: com.examples.with.different.packagename.myapp.MyCUTCallsJARDirectlyHierarchyV2 with seed 2. Configuration id : null
java.lang.IllegalStateException: expect GraphPool to know CDG for every method for which an instruction is known
    at org.evosuite.graphs.cfg.BasicBlock.getCDG(BasicBlock.java:146) ~[classes/:na]
    at org.evosuite.graphs.cfg.BasicBlock.getControlDependencies(BasicBlock.java:191) ~[classes/:na]
    at org.evosuite.graphs.cfg.BytecodeInstruction.getControlDependencies(BytecodeInstruction.java:452) ~[classes/:na]
    at org.evosuite.coverage.vulnerability.VulnerabilityFitnessFactory.getAllControlDependencies(VulnerabilityFitnessFactory.java:103) ~[classes/:na]
    at org.evosuite.coverage.vulnerability.VulnerabilityFitnessFactory.getCoverageGoals(VulnerabilityFitnessFactory.java:73) ~[classes/:na]
    at org.evosuite.coverage.vulnerability.VulnerabilitySuiteFitness.<init>(VulnerabilitySuiteFitness.java:22) ~[classes/:na]
    at org.evosuite.coverage.FitnessFunctions.getFitnessFunction(FitnessFunctions.java:138) ~[classes/:na]
    at org.evosuite.strategy.TestGenerationStrategy.getFitnessFunctions(TestGenerationStrategy.java:87) ~[classes/:na]
    at org.evosuite.strategy.WholeTestSuiteStrategy.generateTests(WholeTestSuiteStrategy.java:68) ~[classes/:na]
    at org.evosuite.TestSuiteGenerator.generateTests(TestSuiteGenerator.java:681) ~[classes/:na]
    at org.evosuite.TestSuiteGenerator.generateTestSuite(TestSuiteGenerator.java:245) ~[classes/:na]
    at org.evosuite.rmi.service.ClientNodeImpl$1.run(ClientNodeImpl.java:157) ~[classes/:na]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_242]
    at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_242]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_242]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_242]
    at java.lang.Thread.run(Thread.java:748) [na:1.8.0_242]
ERROR org.evosuite.statistics.SearchStatistics - No statistics has been saved because EvoSuite failed to generate any test case
ERROR org.evosuite.executionmode.TestGeneration - failed to write statistics data

Steps to Reproduce

I have provided some example projects for this issue. Check here: https://github.com/emaiannone/exploit-generation/tree/fitness_experiments/examples.

Here you can find all steps (in README) to launch some tests on five artificial projects both with the JAR and JUnit tests

Current Result

As said in the README, you should see an error for the last example (Example 4-Version 2). You should a similar stacktrace of the one above.

Expected result

I expect to see something similar to Example 4-Version 1, but it's not happening :(

Additional info

If you need more clarification just write in the comments :)

Thank you!