Evosuite produces a syntactically incorrect test suite in some conditions

Context

When running EvoSuite v. 1.1.0 on the PDFBOX_8 benchmark of the SBST 2020 competition, EvoSuite generates a test suite that is syntactically incorrect when run with a specific combination of command line parameters.

Steps to Reproduce

See the attached file pdfbox-2.0.18.zip. To reproduce the issue uncompress the file to a directory named pdfbox-2.0.18 and run Evosuite to generate tests for the class org.apache.pdfbox.pdmodel.font.FileSystemFontProvider with -Dno_runtime_dependency and -Dvirtual_fs=false and a short search budget (60 seconds in my example). In 100% of the runs Evosuites generates a test suite with a single test case with a variable executor that is used but never declared.

EvoSuite Arguments

java -Xmx4G -jar evosuite-1.1.0.jar -class org.apache.pdfbox.pdmodel.font.FileSystemFontProvider -mem 2048 -DCP=pdfbox-2.0.18/pdfbox-2.0.18.jar:pdfbox-2.0.18/commons-logging-1.2.jar:pdfbox-2.0.18/bcmail-jdk15on-1.60.jar:pdfbox-2.0.18/fontbox-2.0.18.jar:pdfbox-2.0.18/junit-4.12.jar:pdfbox-2.0.18/jai-imageio-jpeg2000-1.3.0.jar:pdfbox-2.0.18/jai-imageio-core-1.4.0.jar:pdfbox-2.0.18/jbig2-imageio-3.0.3.jar:pdfbox-2.0.18/hamcrest-core-1.3.jar:pdfbox-2.0.18/diffutils-1.3.0.jar:pdfbox-2.0.18/bcprov-jdk15on-1.60.jar:pdfbox-2.0.18/bcpkix-jdk15on-1.60.jar -Dsearch_budget=60 -Dno_runtime_dependency -Dvirtual_fs=false

Current Result

This is an example of the test suite that is produced:

package org.apache.pdfbox.pdmodel.font;

import ...

public class FileSystemFontProvider_ESTest {

  @Test(timeout = 4000)
  public void test0()  throws Throwable  {
    Future<?> future = executor.submit(new Runnable(){ 
            @Override public void run() { 
          FontCache fontCache0 = new FontCache();
          FileSystemFontProvider fileSystemFontProvider0 = new FileSystemFontProvider(fontCache0);
      } 
    });
    future.get(4000, TimeUnit.MILLISECONDS);
  }
}

The variable executor in method test0 is used but never declared.

Expected result

It is unclear why the execution of the block of code

FontCache fontCache0 = new FontCache();
FileSystemFontProvider fileSystemFontProvider0 = new FileSystemFontProvider(fontCache0);

in test0 should be delegated to an executor with a timeout of 4 sec, which by the way is already the timeout of the JUnit test. I would expect that test0 would directly be:

  @Test(timeout = 4000)
  public void test0()  throws Throwable  {
      FontCache fontCache0 = new FontCache();
      FileSystemFontProvider fileSystemFontProvider0 = new FileSystemFontProvider(fontCache0);
  }

as Evosuite does if we remove the -Dvirtual_fs=false option (by the way, in this case it also produces many more tests). If there is any reason why the code must be run by an executor, this must be properly declared and initialized.

Additional info

The full benchmark can be found at https://github.com/JUnitContest/junitcontest. My environment is a mac with macOS Big Sur 11.0.1, so it is possible that Evosuite is garbled by some restrictions that Big Sur imposes on file system access (the class under test's constructor looks for the available fonts in the system).

EvoSuite delegates test cases that have a denied Permission during Execution to an Executor.

The class TestSuiteWriter assumes, that in any case the executor is defined and initialized when a permission was denied: https://github.com/EvoSuite/evosuite/blob/d0753ce04489d8ecf7125e16ab3c7ebea3483c66/client/src/main/java/org/evosuite/junit/writer/TestSuiteWriter.java#L659

There are two cases, where the executor may be declared, but none can be reached, if the flag -Dno_runtime_dependencies is absent:

1.) Generate theBeforeAndAfter methods of the scaffolding class: https://github.com/EvoSuite/evosuite/blob/d0753ce04489d8ecf7125e16ab3c7ebea3483c66/client/src/main/java/org/evosuite/junit/writer/TestSuiteWriter.java#L329 2.) Extend the Scaffolding class: https://github.com/EvoSuite/evosuite/blob/d0753ce04489d8ecf7125e16ab3c7ebea3483c66/client/src/main/java/org/evosuite/junit/writer/TestSuiteWriter.java#L541

We either have to remove the delegation to the executor, if the no_runitme_dependencies flag is set or add a proper declaration, if the flag is set.

EvoSuite / evosuite