search

microsoft / vscode-java-test

Run and debug Java test cases in Visual Studio Code.
https://marketplace.visualstudio.com/items?itemName=vscjava.vscode-java-test
Other
293 stars 126 forks source link

[Request] Hide the JUnit Callstack from the stacktrace #1281

Closed leogott closed 2 years ago

leogott commented 3 years ago

There are (few, but there are) cases in which I'd like to take a look at the callstack for a failed (as opposed errored) test. Mainly when using assertThrows. However I would like to have it shortened by default so that in the simplest case every line starting with at org.junit is replaced with the ... X more lines just like the end is shortened right now.

I'd be satisfied with a setting like "shorten lines matching this regex from the stack trace".

For example when assertThrows fails, because instead of ExpectedException I got DifferentException, I'd love the output to look like

org.opentest4j.AssertionFailedError: Unexpected exception type thrown ==> expected: <org.example.ExpectedException> but was: <org.example.DifferentException>
    at org.junit.jupiter.api.AssertThrows.assertThrows(AssertThrows.java:65)
    ... 70 more lines
Caused by: org.example.DifferentException: Message
    at x
    at y
    at z
jdneo commented 3 years ago

@leogott Could you give an example of what's the original stack trace looks like?

In the example you gave, why we just hide the first part of the stack trace but leave the second one as it is?

leogott commented 3 years ago

For a small example I wrote (junit5 assert throws expects A but receives B caused by C), I get this unabbreviated result.

Every line beginning with at org.junit.** (for example) is part of the testing implementation and I don't care about the junit internals, because I'm confident my own code is the likely cause of the failed test.

I don't care about the callstack of the AssertionFailedError (beyond that it was in line 25 of AppTest.java) , because it was thrown by the assertThrows method.

I do however care about the Cause (the exception of wrong type), and where it originated.

So in this case I'm getting the following information:

%TESTC  1 v2
%TSTTREE2,org.example.AppTest,true,1,false,1,AppTest,,[engine:junit-jupiter]/[class:org.example.AppTest]
%TSTTREE3,someFeature_stackTrace(org.example.AppTest),false,1,false,2,someFeature_stackTrace(),,[engine:junit-jupiter]/[class:org.example.AppTest]/[method:someFeature_stackTrace()]
%TESTS  3,someFeature_stackTrace(org.example.AppTest)
%FAILED 3,someFeature_stackTrace(org.example.AppTest)
%TRACES 
org.opentest4j.AssertionFailedError: Unexpected exception type thrown ==> expected: <java.io.IOException> but was: <java.lang.IllegalArgumentException>
        at org.junit.jupiter.api.AssertThrows.assertThrows(AssertThrows.java:65)
        at org.junit.jupiter.api.AssertThrows.assertThrows(AssertThrows.java:37)
        at org.junit.jupiter.api.Assertions.assertThrows(Assertions.java:3007)
        at org.example.AppTest.someFeature_stackTrace(AppTest.java:25)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:688)
        at org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
        at org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
        at org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:149)
        at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:140)
        at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:84)
        at org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(ExecutableInvoker.java:115)
        at org.junit.jupiter.engine.execution.ExecutableInvoker.lambda$invoke$0(ExecutableInvoker.java:105)
        at org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
        at org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
        at org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
        at org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
        at org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:104)
        at org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:98)
        at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$6(TestMethodTestDescriptor.java:210)
        at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
        at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:206)
        at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:131)
        at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:65)
        at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:139)
        at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
        at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:129)
        at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
        at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:127)
        at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
        at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:126)
        at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:84)
        at java.util.ArrayList.forEach(ArrayList.java:1259)
        at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:38)
        at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:143)
        at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
        at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:129)
        at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
        at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:127)
        at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
        at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:126)
        at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:84)
        at java.util.ArrayList.forEach(ArrayList.java:1259)
        at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:38)
        at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:143)
        at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
        at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:129)
        at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
        at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:127)
        at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
        at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:126)
        at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:84)
        at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:32)
        at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor.execute(HierarchicalTestExecutor.java:57)
        at org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute(HierarchicalTestEngine.java:51)
        at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:108)
        at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:88)
        at org.junit.platform.launcher.core.EngineExecutionOrchestrator.lambda$execute$0(EngineExecutionOrchestrator.java:54)
        at org.junit.platform.launcher.core.EngineExecutionOrchestrator.withInterceptedStreams(EngineExecutionOrchestrator.java:67)
        at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:52)
        at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:96)
        at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:84)
        at org.eclipse.jdt.internal.junit5.runner.JUnit5TestReference.run(JUnit5TestReference.java:98)
        at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:40)
        at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:529)
        at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:756)
        at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:452)
        at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:210)
Caused by: java.lang.IllegalArgumentException: Argument must not be null
        at org.example.App.someFeatureImTesting(App.java:20)
        at org.example.AppTest.lambda$0(AppTest.java:26)
        at org.junit.jupiter.api.AssertThrows.assertThrows(AssertThrows.java:55)
        ... 70 more
Caused by: java.lang.NullPointerException
        at org.example.App.someFeatureImTesting(App.java:17)
        ... 72 more
%TRACEE 
%TESTE  3,someFeature_stackTrace(org.example.AppTest)
%RUNTIME175

> Test run finished at 21.8.2021, 01:23:53 <

But this is what I care about:

TEST: someFeature_stackTrace(org.example.AppTest)
RESULT: failed
TRACE:
org.opentest4j.AssertionFailedError: Unexpected exception type thrown ==> expected: <java.io.IOException> but was: <java.lang.IllegalArgumentException>
        ... 3 more
        at org.example.AppTest.someFeature_stackTrace(AppTest.java:25)
        ... X more
Caused by: java.lang.IllegalArgumentException: Argument must not be null
        at org.example.App.someFeatureImTesting(App.java:20)
        at org.example.AppTest.lambda$0(AppTest.java:26)
        ... 71 more
Caused by: java.lang.NullPointerException
        at org.example.App.someFeatureImTesting(App.java:17)
        ... 72 more

Would it be possible to at least cut the callstack for the AssertionFailed Exception that fails the Test?

jdneo commented 3 years ago

I see, so that is to only reserve the stack trace which belongs to the source?

leogott commented 3 years ago

Oh you're correct. I didn't look at it that way, but it absolutely is just the parts of the callstack within the source at test.

Yes, I'd like to see a filtered view or something that cuts away the callstack that doesn't belong to the source.

jdneo commented 2 years ago

Get a very similar feedback from a customer interview. He hope the stacktrace can be hidden by clicking some buttons like show more, show less

jdneo commented 2 years ago

Hi @leogott,

I pushed a change to filter out those 'internal' traces. Currently we don't have a setting to control it. But we can definitely add a setting for that if users require.

You can try the new behavior in the next pre-release of the extension (should be available tomorrow).

image

Let me know if you have any feedback, thanks!

jdneo commented 2 years ago

To verify this issue

// cc @testforstephen

testforstephen commented 2 years ago

The latest implementation works well. One minor suggestion is to hide the highlighting two call stacks as well, since they doesn't provide any valid info for diagnose. image