EvoSuite fails on classes with many constants

PYangDizzle commented 4 years ago

Context

Please provide below a detailed introduction to the issue itself, and describe what you were doing when the issue happened. Or, what do you want to achieve? I tried to run a default EvoSuite on a class with 102 static final String fields and some utility methods. EvoSuite fails. I tried it on a different class with 12 String constants and 9 int constants. EvoiSuite succeeds.

Steps to Reproduce

Please break down here below all the needed steps to reproduce the issue. [If possible, please upload an example of the project you are generating tests for.]

java -jar evosuite-1.0.6.jar -class MyClass -projectCP MyCP

Current Result

Please describe here below the current result you got (if relevant) [if relevant, include a screenshot]

It threw ClassNotFoundException. I enabled trace logging and saw this line.

[MASTER] 10:55:38.684 [logback-1] DEBUG CFGMethodAdapter - NOT Creating CFG of myPackage.MyClass.<clinit>()V: true, false, true, true
[MASTER] 10:55:38.686 [logback-1] INFO  InstrumentingClassLoader - Error while loading class: java.lang.RuntimeException: Method code too large!
* Error while initializing target class: myPackage.MyClass
[MASTER] 10:55:38.705 [logback-2] ERROR TestSuiteGenerator - Problem for myPackage.MyClass. Full stack:
java.lang.ClassNotFoundException: myPackage.MyClass
    at java.lang.Class.forNameImpl(Native Method) ~[na:2.9 (07-31-2018)]
    at java.lang.Class.forName(Class.java:415) ~[na:2.9 (07-31-2018)]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:90) ~[na:1.8.0]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55) ~[na:1.8.0]
    at java.lang.reflect.Method.invoke(Method.java:508) ~[na:1.8.0]
    at org.evosuite.testcase.statements.MethodStatement$1.execute(MethodStatement.java:257) ~[evosuite-1.0.6.jar:1.0.6]
[MASTER] 10:55:38.698 [logback-1] DEBUG TestRunnable - Exception thrown in statement: Class<?> class0 = Class.forName(string0, boolean0, classLoader0); - java.lang.ClassNotFoundException - myPackage.MyClass
[MASTER] 10:55:38.698 [logback-1] DEBUG TestRunnable - java.lang.Class.forNameImpl(Native Method)

Additional info

Please add any information of interest here below

Maybe the method can be broken down to smaller pieces? (i.e. -> clinit0(), clinit1(), etc.) Also, I can't find an easy way to skip some classes other than physically removing them.

apanichella commented 4 years ago

Hi, 'ClassNotFoundException' means that either there are missing dependencies in the classpath (-classPath) or that the path of the class under test (-class) is not correct.

From the text, I would argue it is the second. If you unzip the jar file, do you find the target class in the specified path?

PYangDizzle commented 4 years ago

It's a /bin folder and the class file is in there. If you see the trace log, there is a message that says the method size is too large. Behavior-wise, the class is loaded and the string constants are visited, then Class.forName fails. Probably because the instrumentation created a method (clinit, static initializer?) that's too large

On Mon, Mar 16, 2020, 12:10 PM Annibale Panichella notifications@github.com wrote:

Hi, 'ClassNotFoundException' means that either there are missing dependencies in the classpath or that the path of the class under test is not correct.

From the text, I would argue it is the second. If you unzip the jar file, do you find the target class in the specified path?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/EvoSuite/evosuite/issues/285#issuecomment-599656722, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADX32ED7LK5QK6L6FPOASKLRHZMRDANCNFSM4LMN446A .

PYangDizzle commented 4 years ago

I think the logs shows that the class was actually loaded.

default log

C:\Users\pyang\My\AE\EvoSuite\Result>C:\My\jdk\bin\java -jar C:\pylib\EvoSuite\evosuite-1.0.6.jar -class myPackage.Application -projectCP "C:\Users\pyang\My\AE\EvoSuite\my Common\bin"
* EvoSuite 1.0.6
* Going to generate test cases for class: myPackage.Application
* Starting client
* Connecting to master process on port 11196
* Analyzing classpath:
* Error while initializing target class: myPackage.Application
[MASTER] 18:15:10.968 [logback-2] ERROR TestSuiteGenerator - Problem for myPackage.Application. Full stack:
java.lang.ClassNotFoundException: myPackage.Application
        at java.lang.Class.forNameImpl(Native Method) ~[na:2.9 (07-31-2018)]
        at java.lang.Class.forName(Class.java:415) ~[na:2.9 (07-31-2018)]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:90) ~[na:1.8.0]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55) ~[na:1.8.0]
        at java.lang.reflect.Method.invoke(Method.java:508) ~[na:1.8.0]
        at org.evosuite.testcase.statements.MethodStatement$1.execute(MethodStatement.java:257) ~[evosuite-1.0.6.jar:1.0.6]
        at org.evosuite.testcase.statements.AbstractStatement.exceptionHandler(AbstractStatement.java:169) ~[evosuite-1.0.6.jar:1.0.6]
        at org.evosuite.testcase.statements.MethodStatement.execute(MethodStatement.java:220) ~[evosuite-1.0.6.jar:1.0.6]
        at org.evosuite.testcase.execution.TestRunnable.executeStatements(TestRunnable.java:307) ~[evosuite-1.0.6.jar:1.0.6]
        at org.evosuite.testcase.execution.TestRunnable.call(TestRunnable.java:213) ~[evosuite-1.0.6.jar:1.0.6]
        at org.evosuite.testcase.execution.TestRunnable.call(TestRunnable.java:55) ~[evosuite-1.0.6.jar:1.0.6]
        at java.util.concurrent.FutureTask.run(FutureTask.java:277) [na:1.8.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1160) [na:1.8.0]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [na:1.8.0]
        at java.lang.Thread.run(Thread.java:811) [na:2.9 (07-31-2018)]
* Computation finished

part of trace log

[MASTER] 18:05:49.025 [logback-1] DEBUG CFGMethodAdapter - Method has no branches: myPackage.Application.getContextPathsForExternalApplications()Ljava/util/Collection;
[MASTER] 18:05:49.025 [logback-1] DEBUG CFGMethodAdapter - Counting: myPackage.Application.getContextPathsForExternalApplications()Ljava/util/Collection;
[MASTER] 18:05:49.029 [logback-1] INFO  CFGClassAdapter - Analyzing CFG of myPackage/Application
[MASTER] 18:05:49.029 [logback-1] INFO  CFGMethodAdapter - Method has EvoSuite annotation: Lorg/evosuite/runtime/annotation/EvoSuiteExclude;
[MASTER] 18:05:49.029 [logback-1] DEBUG CFGMethodAdapter - Creating CFG of myPackage.Application.hashCode()I
[MASTER] 18:05:49.029 [logback-1] DEBUG CFGMethodAdapter - NOT Creating CFG of myPackage.Application.hashCode()I: true, false, true, true
[MASTER] 18:05:49.030 [logback-1] INFO  EndOfClassInitializerVisitor - Creating <clinit> in class myPackage/Application
[MASTER] 18:05:49.030 [logback-1] INFO  CFGClassAdapter - Analyzing CFG of myPackage/Application
[MASTER] 18:05:49.030 [logback-1] INFO  CreateClassResetClassAdapter - Found static initializer in class myPackage/Application
[MASTER] 18:05:49.031 [logback-1] DEBUG CFGMethodAdapter - Creating CFG of myPackage.Application.<clinit>()V
[MASTER] 18:05:49.031 [logback-1] DEBUG CFGMethodAdapter - NOT Creating CFG of myPackage.Application.<clinit>()V: true, false, true, true
[MASTER] 18:05:49.033 [logback-1] INFO  InstrumentingClassLoader - Error while loading class: java.lang.RuntimeException: Method code too large!
* Error while initializing target class: myPackage.Application
[MASTER] 18:05:49.040 [logback-1] DEBUG AbstractStatement - Exception thrown in method forName: {}
java.lang.ClassNotFoundException: myPackage.Application
        at java.lang.Class.forNameImpl(Native Method) ~[na:2.9 (07-31-2018)]
        at java.lang.Class.forName(Class.java:415) ~[na:2.9 (07-31-2018)]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:90) ~[na:1.8.0]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55) ~[na:1.8.0]
        at java.lang.reflect.Method.invoke(Method.java:508) ~[na:1.8.0]
        at org.evosuite.testcase.statements.MethodStatement$1.execute(MethodStatement.java:257) ~[evosuite-1.0.6.jar:1.0.6]
        at org.evosuite.testcase.statements.AbstractStatement.exceptionHandler(AbstractStatement.java:169) ~[evosuite-1.0.6.jar:1.0.6]
        at org.evosuite.testcase.statements.MethodStatement.execute(MethodStatement.java:220) ~[evosuite-1.0.6.jar:1.0.6]
        at org.evosuite.testcase.execution.TestRunnable.executeStatements(TestRunnable.java:307) [evosuite-1.0.6.jar:1.0.6]
        at org.evosuite.testcase.execution.TestRunnable.call(TestRunnable.java:213) [evosuite-1.0.6.jar:1.0.6]
        at org.evosuite.testcase.execution.TestRunnable.call(TestRunnable.java:55) [evosuite-1.0.6.jar:1.0.6]
        at java.util.concurrent.FutureTask.run(FutureTask.java:277) [na:1.8.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1160) [na:1.8.0]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [na:1.8.0]
        at java.lang.Thread.run(Thread.java:811) [na:2.9 (07-31-2018)]

jose commented 4 years ago

Hi @PYangDizzle,

The problem here is that the code injected by EvoSuite's made the class under test exceeded the bytecode limit. A possible workaround to address this issue is to use fewer criteria. By default, EvoSuite tries to optimize the following criteria: Line, Branch, Exception, WeakMutation, Output, Method, MethodNoException, and CBranch. You could try to enable one by one to identify which one is causing the issue. (If I've to guess, I would say WeakMutation is the one that is causing problems, so -Dcriterion=LINE:BRANCH:EXCEPTION:OUTPUT:METHOD:METHODNOEXCEPTION:CBRANCH should work).

I hope this helps.

-- Best, Jose

EvoSuite / evosuite