UnitTestBot generates four tests for a method. All four tests are documents as successful, but three of the four tests actually fail during initialization of the class that contains the method under test.
To Reproduce
Steps to reproduce the behavior:
Check out https://github.com/suqun/JavaSecurity. For the issue reported here, the current master head revision of this repository was suqun/JavaSecurity@89d6203c8345f49f835fb01af94591eee7957b8a, but it is not clear whether this issue is sensitive to this specific revision.
Add the following to pom.xml to explicitly select Java 1.8:
Open this project in IntelliJ IDEA and link its pom.xml, thereby making this a Maven project.
In the IntelliJ IDEA Project Settings, select a Java 1.8 SDK.
Use the UnitTestBot IntelliJ IDEA plugin to generate tests for the com.larry.security.signature.dsa.DSA.jdkDSA() method. Leave all UnitTestBot settings at their defaults, other than selecting this one method to generate tests for.
Expected behavior
The jdkDSA method under test clearly was not designed with testing in mind. It takes no arguments and returns no value, so extensive mocking would be needed in order to test this method at all. Without mocks and/or output capture, the best we could hope for is a test that verifies that the method returns when called, without throwing an exception.
That being said, we expect that comments associated with the generated tests should correctly reflect which will pass and which will fail.
Actual behavior
UnitTestBot generated a test class with four test methods:
The last, fuzzer-generated test is simply a direct call to the method under test. This test passes, and is all one could expect without mocking.
The first three tests, though, all fail with NullPointerExceptions when run. This result disagrees with the expectations set by the generated comments, all of which predict that the tests will pass. For example, all three of these failing tests are enclosed in a region bounded by:
///region SYMBOLIC EXECUTION: SUCCESSFUL EXECUTIONS for method jdkDSA()
...
///endregion
These uncaught NullPointerExceptions may seem surprising, since the entire body of the method under test is enclosed within a try block that catches any Exception. However, careful examination of the following stack traces reveals that the NullPointerExceptions arise immediately before each DSATest test method calls DSA.jdkDSA(). The problem seems to be triggered when loading and initializing the DSA class.
Failing stack trace when running `testJdkDSA_CatchException`
```
java.lang.NullPointerException
at java.security.cert.CertificateFactory.getInstance(CertificateFactory.java:185)
at sun.security.pkcs.PKCS7.parseSignedData(PKCS7.java:300)
at sun.security.pkcs.PKCS7.parse(PKCS7.java:186)
at sun.security.pkcs.PKCS7.parse(PKCS7.java:154)
at sun.security.pkcs.PKCS7.(PKCS7.java:136)
at sun.security.util.SignatureFileVerifier.(SignatureFileVerifier.java:125)
at java.util.jar.JarVerifier.processEntry(JarVerifier.java:310)
at java.util.jar.JarVerifier.update(JarVerifier.java:239)
at java.util.jar.JarFile.initializeVerifier(JarFile.java:392)
at java.util.jar.JarFile.ensureInitialization(JarFile.java:633)
at java.util.jar.JavaUtilJarAccessImpl.ensureInitialization(JavaUtilJarAccessImpl.java:69)
at sun.misc.URLClassPath$JarLoader$2.getManifest(URLClassPath.java:993)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:456)
at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
at com.larry.security.signature.dsa.DSATest.testJdkDSA_CatchException(DSATest.java:36)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:725)
at org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
at org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
at org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:149)
at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:140)
at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:84)
at org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(ExecutableInvoker.java:115)
at org.junit.jupiter.engine.execution.ExecutableInvoker.lambda$invoke$0(ExecutableInvoker.java:105)
at org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
at org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
at org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
at org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
at org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:104)
at org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:98)
at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$7(TestMethodTestDescriptor.java:214)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:210)
at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:135)
at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:66)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:151)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
at java.util.ArrayList.forEach(ArrayList.java:1259)
at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
at java.util.ArrayList.forEach(ArrayList.java:1259)
at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:35)
at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor.execute(HierarchicalTestExecutor.java:57)
at org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute(HierarchicalTestEngine.java:54)
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:107)
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:88)
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.lambda$execute$0(EngineExecutionOrchestrator.java:54)
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.withInterceptedStreams(EngineExecutionOrchestrator.java:67)
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:52)
at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:114)
at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:86)
at org.junit.platform.launcher.core.DefaultLauncherSession$DelegatingLauncher.execute(DefaultLauncherSession.java:86)
at org.junit.platform.launcher.core.SessionPerRequestLauncher.execute(SessionPerRequestLauncher.java:53)
at com.intellij.junit5.JUnit5IdeaTestRunner.startRunnerWithArgs(JUnit5IdeaTestRunner.java:57)
at com.intellij.rt.junit.IdeaTestRunner$Repeater$1.execute(IdeaTestRunner.java:38)
at com.intellij.rt.execution.junit.TestsRepeater.repeat(TestsRepeater.java:11)
at com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:35)
at com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:232)
at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:55)
```
Failing stack trace when running `testJdkDSA_CatchException_1`
```
java.lang.NullPointerException
at java.security.cert.CertificateFactory.getInstance(CertificateFactory.java:185)
at sun.security.pkcs.PKCS7.parseSignedData(PKCS7.java:300)
at sun.security.pkcs.PKCS7.parse(PKCS7.java:186)
at sun.security.pkcs.PKCS7.parse(PKCS7.java:154)
at sun.security.pkcs.PKCS7.(PKCS7.java:136)
at sun.security.util.SignatureFileVerifier.(SignatureFileVerifier.java:125)
at java.util.jar.JarVerifier.processEntry(JarVerifier.java:310)
at java.util.jar.JarVerifier.update(JarVerifier.java:239)
at java.util.jar.JarFile.initializeVerifier(JarFile.java:392)
at java.util.jar.JarFile.ensureInitialization(JarFile.java:633)
at java.util.jar.JavaUtilJarAccessImpl.ensureInitialization(JavaUtilJarAccessImpl.java:69)
at sun.misc.URLClassPath$JarLoader$2.getManifest(URLClassPath.java:993)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:456)
at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
at com.larry.security.signature.dsa.DSATest.testJdkDSA_CatchException_1(DSATest.java:68)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:725)
at org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
at org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
at org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:149)
at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:140)
at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:84)
at org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(ExecutableInvoker.java:115)
at org.junit.jupiter.engine.execution.ExecutableInvoker.lambda$invoke$0(ExecutableInvoker.java:105)
at org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
at org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
at org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
at org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
at org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:104)
at org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:98)
at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$7(TestMethodTestDescriptor.java:214)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:210)
at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:135)
at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:66)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:151)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
at java.util.ArrayList.forEach(ArrayList.java:1259)
at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
at java.util.ArrayList.forEach(ArrayList.java:1259)
at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:35)
at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor.execute(HierarchicalTestExecutor.java:57)
at org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute(HierarchicalTestEngine.java:54)
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:107)
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:88)
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.lambda$execute$0(EngineExecutionOrchestrator.java:54)
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.withInterceptedStreams(EngineExecutionOrchestrator.java:67)
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:52)
at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:114)
at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:86)
at org.junit.platform.launcher.core.DefaultLauncherSession$DelegatingLauncher.execute(DefaultLauncherSession.java:86)
at org.junit.platform.launcher.core.SessionPerRequestLauncher.execute(SessionPerRequestLauncher.java:53)
at com.intellij.junit5.JUnit5IdeaTestRunner.startRunnerWithArgs(JUnit5IdeaTestRunner.java:57)
at com.intellij.rt.junit.IdeaTestRunner$Repeater$1.execute(IdeaTestRunner.java:38)
at com.intellij.rt.execution.junit.TestsRepeater.repeat(TestsRepeater.java:11)
at com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:35)
at com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:232)
at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:55)
```
Failing stack trace when running `testJdkDSA_CatchException_2`
```
java.lang.NullPointerException
at java.security.cert.CertificateFactory.getInstance(CertificateFactory.java:185)
at sun.security.pkcs.PKCS7.parseSignedData(PKCS7.java:300)
at sun.security.pkcs.PKCS7.parse(PKCS7.java:186)
at sun.security.pkcs.PKCS7.parse(PKCS7.java:154)
at sun.security.pkcs.PKCS7.(PKCS7.java:136)
at sun.security.util.SignatureFileVerifier.(SignatureFileVerifier.java:125)
at java.util.jar.JarVerifier.processEntry(JarVerifier.java:310)
at java.util.jar.JarVerifier.update(JarVerifier.java:239)
at java.util.jar.JarFile.initializeVerifier(JarFile.java:392)
at java.util.jar.JarFile.ensureInitialization(JarFile.java:633)
at java.util.jar.JavaUtilJarAccessImpl.ensureInitialization(JavaUtilJarAccessImpl.java:69)
at sun.misc.URLClassPath$JarLoader$2.getManifest(URLClassPath.java:993)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:456)
at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
at com.larry.security.signature.dsa.DSATest.testJdkDSA_CatchException_2(DSATest.java:103)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:725)
at org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
at org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
at org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:149)
at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:140)
at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:84)
at org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(ExecutableInvoker.java:115)
at org.junit.jupiter.engine.execution.ExecutableInvoker.lambda$invoke$0(ExecutableInvoker.java:105)
at org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
at org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
at org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
at org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
at org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:104)
at org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:98)
at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$7(TestMethodTestDescriptor.java:214)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:210)
at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:135)
at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:66)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:151)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
at java.util.ArrayList.forEach(ArrayList.java:1259)
at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
at java.util.ArrayList.forEach(ArrayList.java:1259)
at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:35)
at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor.execute(HierarchicalTestExecutor.java:57)
at org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute(HierarchicalTestEngine.java:54)
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:107)
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:88)
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.lambda$execute$0(EngineExecutionOrchestrator.java:54)
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.withInterceptedStreams(EngineExecutionOrchestrator.java:67)
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:52)
at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:114)
at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:86)
at org.junit.platform.launcher.core.DefaultLauncherSession$DelegatingLauncher.execute(DefaultLauncherSession.java:86)
at org.junit.platform.launcher.core.SessionPerRequestLauncher.execute(SessionPerRequestLauncher.java:53)
at com.intellij.junit5.JUnit5IdeaTestRunner.startRunnerWithArgs(JUnit5IdeaTestRunner.java:57)
at com.intellij.rt.junit.IdeaTestRunner$Repeater$1.execute(IdeaTestRunner.java:38)
at com.intellij.rt.execution.junit.TestsRepeater.repeat(TestsRepeater.java:11)
at com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:35)
at com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:232)
at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:55)
```
Environment
IntelliJ IDEA 2023.2.6 (Ultimate Edition)
JetBrains UnitTestBot plugin version 2023.10
All UnitTestBot plugin settings were left at their default values.
Description
UnitTestBot generates four tests for a method. All four tests are documents as successful, but three of the four tests actually fail during initialization of the class that contains the method under test.
To Reproduce
Steps to reproduce the behavior:
Check out https://github.com/suqun/JavaSecurity. For the issue reported here, the current
master
head revision of this repository was suqun/JavaSecurity@89d6203c8345f49f835fb01af94591eee7957b8a, but it is not clear whether this issue is sensitive to this specific revision.Add the following to
pom.xml
to explicitly select Java 1.8:Open this project in IntelliJ IDEA and link its
pom.xml
, thereby making this a Maven project.In the IntelliJ IDEA Project Settings, select a Java 1.8 SDK.
Use the UnitTestBot IntelliJ IDEA plugin to generate tests for the
com.larry.security.signature.dsa.DSA.jdkDSA()
method. Leave all UnitTestBot settings at their defaults, other than selecting this one method to generate tests for.Expected behavior
The
jdkDSA
method under test clearly was not designed with testing in mind. It takes no arguments and returns no value, so extensive mocking would be needed in order to test this method at all. Without mocks and/or output capture, the best we could hope for is a test that verifies that the method returns when called, without throwing an exception.That being said, we expect that comments associated with the generated tests should correctly reflect which will pass and which will fail.
Actual behavior
UnitTestBot generated a test class with four test methods:
Generated `DSATest.java`
```java package com.larry.security.signature.dsa; import org.junit.jupiter.api.DisplayName; import org.junit.jupiter.api.Test; import org.mockito.MockedStatic; import sun.security.jca.GetInstance; import sun.security.jca.GetInstance.Instance; import java.util.ArrayList; import static org.mockito.ArgumentMatchers.any; import static org.mockito.Mockito.mockStatic; import static org.utbot.runtime.utils.java.UtUtils.createInstance; import static org.utbot.runtime.utils.java.UtUtils.setField; public final class DSATest { ///region Test suites for executable com.larry.security.signature.dsa.DSA.jdkDSA ///region SYMBOLIC EXECUTION: SUCCESSFUL EXECUTIONS for method jdkDSA() /** * @utbot.classUnderTest {@link DSA} * @utbot.methodUnderTest {@link DSA#jdkDSA()} * @utbot.invokes {@link Exception#printStackTrace()} * @utbot.caughtException {@code Exception e} */ @Test @DisplayName("jdkDSA: Catch (Exception e) -> ExceptionPrintStackTrace") public void testJdkDSA_CatchException() { MockedStatic mockedStatic = null; try { mockedStatic = mockStatic(GetInstance.class); ArrayList arrayList = new ArrayList(); (mockedStatic.when(() -> GetInstance.getServices(any(String.class), any(String.class)))).thenReturn(arrayList); DSA.jdkDSA(); } finally { mockedStatic.close(); } } /** * @utbot.classUnderTest {@link DSA} * @utbot.methodUnderTest {@link DSA#jdkDSA()} * @utbot.invokes {@link Exception#printStackTrace()} * @utbot.caughtException {@code Exception e} */ @Test @DisplayName("jdkDSA: Catch (Exception e) -> ExceptionPrintStackTrace") public void testJdkDSA_CatchException_1() { MockedStatic mockedStatic = null; try { mockedStatic = mockStatic(GetInstance.class); ArrayList arrayList = new ArrayList(); Object object = new Object(); arrayList.add(object); arrayList.add(null); arrayList.add(null); arrayList.add(null); arrayList.add(null); arrayList.add(null); arrayList.add(null); arrayList.add(null); arrayList.add(null); arrayList.add(null); (mockedStatic.when(() -> GetInstance.getServices(any(String.class), any(String.class)))).thenReturn(arrayList); DSA.jdkDSA(); } finally { mockedStatic.close(); } } /** * @utbot.classUnderTest {@link DSA} * @utbot.methodUnderTest {@link DSA#jdkDSA()} * @utbot.invokes {@link Exception#printStackTrace()} * @utbot.caughtException {@code Exception e} */ @Test @DisplayName("jdkDSA: -> Catch (Exception e)") public void testJdkDSA_CatchException_2() throws Exception { MockedStatic mockedStatic = null; try { mockedStatic = mockStatic(GetInstance.class); ArrayList arrayList = new ArrayList(); arrayList.add(null); arrayList.add(null); arrayList.add(null); arrayList.add(null); arrayList.add(null); arrayList.add(null); arrayList.add(null); arrayList.add(null); arrayList.add(null); arrayList.add(null); (mockedStatic.when(() -> GetInstance.getServices(any(String.class), any(String.class)))).thenReturn(arrayList); Instance instance = ((Instance) createInstance("sun.security.jca.GetInstance$Instance")); byte[] impl = {}; setField(instance, "sun.security.jca.GetInstance$Instance", "impl", impl); (mockedStatic.when(() -> GetInstance.getInstance(any(), any()))).thenReturn(instance); DSA.jdkDSA(); } finally { mockedStatic.close(); } } ///endregion ///region FUZZER: SUCCESSFUL EXECUTIONS for method jdkDSA() /** * @utbot.classUnderTest {@link DSA} * @utbot.methodUnderTest {@link DSA#jdkDSA()} */ @Test @DisplayName("jdkDSA: ") public void testJdkDSA() { DSA.jdkDSA(); } ///endregion ///endregion } ```The last, fuzzer-generated test is simply a direct call to the method under test. This test passes, and is all one could expect without mocking.
The first three tests, though, all fail with
NullPointerException
s when run. This result disagrees with the expectations set by the generated comments, all of which predict that the tests will pass. For example, all three of these failing tests are enclosed in a region bounded by:These uncaught
NullPointerException
s may seem surprising, since the entire body of the method under test is enclosed within atry
block that catches anyException
. However, careful examination of the following stack traces reveals that theNullPointerException
s arise immediately before eachDSATest
test method callsDSA.jdkDSA()
. The problem seems to be triggered when loading and initializing theDSA
class.Failing stack trace when running `testJdkDSA_CatchException`
``` java.lang.NullPointerException at java.security.cert.CertificateFactory.getInstance(CertificateFactory.java:185) at sun.security.pkcs.PKCS7.parseSignedData(PKCS7.java:300) at sun.security.pkcs.PKCS7.parse(PKCS7.java:186) at sun.security.pkcs.PKCS7.parse(PKCS7.java:154) at sun.security.pkcs.PKCS7.Failing stack trace when running `testJdkDSA_CatchException_1`
``` java.lang.NullPointerException at java.security.cert.CertificateFactory.getInstance(CertificateFactory.java:185) at sun.security.pkcs.PKCS7.parseSignedData(PKCS7.java:300) at sun.security.pkcs.PKCS7.parse(PKCS7.java:186) at sun.security.pkcs.PKCS7.parse(PKCS7.java:154) at sun.security.pkcs.PKCS7.Failing stack trace when running `testJdkDSA_CatchException_2`
``` java.lang.NullPointerException at java.security.cert.CertificateFactory.getInstance(CertificateFactory.java:185) at sun.security.pkcs.PKCS7.parseSignedData(PKCS7.java:300) at sun.security.pkcs.PKCS7.parse(PKCS7.java:186) at sun.security.pkcs.PKCS7.parse(PKCS7.java:154) at sun.security.pkcs.PKCS7.Environment