Prioritize Symbolic Engine executions

[sofurihafe]

Description

Sometimes we need to turn off Symbolic Engine and stop the test generation (and do not collect this execution) in case of mocking taking place when generating parameterized tests. However, Fuzzer can still add its executions which may lead to unwanted executions being processed later. The suggestion is to prioritize Symbolic Engine executions in minimization process.

Details

Run parameterized test generation for vertexSum method from Recursion test set in plugin. Currently, generated test will fail. With #845 changes applied it will lead to empty test class because we cut off all Fuzzer executions for the reason described above.

[dtim]

It seems that this issue requires further discussion. Minimization algorithm can't guarantee that no "bad" (for the parametrized test generator) fuzzed executions are added. The only way to completely guarantee it is to exclude all fuzzed executions, but it will drop coverage, and anyway this can be done at the execution generation phase, so there is no need to modify the minimizer.

There are several distinct problems indeed.

• Fuzzer can generate flaky tests (failing by design) because it does not mock things like java.util.Random methods. This problem is not specific to parametrized tests and requires a systematic design involving the fuzzer itself, the concrete executor, and probably the symbolic engine.
• There is some issue with the generation of parametrized tests for fuzzed executions. Maybe this issue can be fixed without changing the minimizer behavior or turning off the fuzzer, maybe not -- we need to investigate it.

[CaelmBleidd]

@EgorkaKulikov @sofurihafe do we need to fix this issue? We discussed it with @sergeypospelov and @dtim, looks like minimization should not know anything about fuzzer and symbolic execution, therefore, it is better to close this task