Notifications about test executions

[mickaelistria]

One very common case of launching/debugging is test execution. I believe this activity would integrate very well in the Debug Adapter Protocol (instead of making it yet another protocol).

I suggest the Debug Adapter Protocol should specify the typical notifications that more JUnit, TestNG, NUnit and so on have been using: willRunTests, testRunning, testComplete[SUCCESS, FAILURE, ERROR], allTestComplete... So a Debug Adapter could optionally have a dedicated action to run a test according to the json configuration, and the Debug Adapter would behave as usual and only send a few more notifications for the IDE to report progress.

[gorkem]

@tsmaeder @fbricon This is relevant

[weinand]

As an author of some debug adapters I'd like to better understand how my DA implementation could actually generate those events.

For instance, lots of testing frameworks exist for node.js. How can they interact with the DA to send those events? Output scraping? Injecting some VS Code/DA specific emitter into the testing frameworks (if they support this)?

For me as an debug adapter author it does not look attractive (or even possible) to support all possible testing frameworks by emitting the correct DAP events.

For me testing support looks like a feature that is orthogonal to debugging: it should be possible to combine the support for JavaScript testing frameworks with all JavaScript debuggers (debug adapters). Requiring that the testing support lives inside the adapters results in duplication and complicates debug adapters unnecessarily.

[mickaelistria]

How can they interact with the DA to send those events?

As I imagine it, I see that testing is just another way of launching I imagine that additionally to "launch" or "attach" value for the request tag in launch.json there could be some kind of launchTest or just test . It would be up to the Debug Adapter to support such new launch type or not, not a responsibility of the protocol itself. The testing framework is a blackbox for the protocol, just like the debugger itself or the execution command. Now, let's imagine a Debug Adapter that does manage to launch tests, what is the simplest way for such debug adapter to the IDE? I have the impression that this DebugAdapterProtocol is the best place for it. While there can be specificites to a test framework or another, just like there are specificities about debuggers, but I have the impression that a only a few events are needed to cover the vast majority of use-cases for testing. So it seems relatively easy to just define a few notificiations in the protocol to report some test results, and I think since the Debug Adapter Protocol is also about execution, it should include support for testing which isn't more than one flavor of executing.

HTH

[hbenl]

I agree with @weinand that testing and debugging are orthogonal features. Yes, both involve launching, but only as a means to an end. The core functionality, however, is completely separate. Besides, @weinand is right that in order to capture the events from the testing framework, specialized code needs to be written for each supported testing framework. If this responsibility was pushed onto the debug adapters, a lot of code would need to be duplicated between them. By the way, I am currently working on a "Test Explorer UI" extension, which will offer an API to other (testing framework specific) extensions which they can use to send the test events. I guess that could be what you're looking for.

[weinand]

"Imagining" a debug adapter that launches tests is simple since all DAs can do this already today and they are used for that already.

But the unanswered question is: how can a debug adapter generate those "test events"? A DA runs the test framework like any other program on the given runtime. The runtime is not aware of the fact that tests are running. The DA might be aware of the fact that the program is a test suite but it does not know about what tests are running and whether they are failing or not.

Since we are not designing test frameworks here, we do not have the freedom to decide how they work and how they should communicate to the DA.

@mickaelistria we have to support existing test frameworks and I would like to understand how you would imagine that a DA could get the necessary information from the runtime in order to send out the test events.

In the existing DAP all protocol elements correspond to more or less standard requests and events available from existing debuggers/runtimes. I'm not aware that test frameworks provide standard events at all.

[weinand]

@hbenl your "Test Explorer" and the "testing framework specific extensions" sound exactly like what I was envisioning as a promising approach.

[mickaelistria]

all DAs can do this already today and they are used for that already.

Ok. And how do such DA report test results in VSCode? Is there a documented protocol to achieve that that can be used by other IDEs and other languages?

But the unanswered question is: how can a debug adapter generate those "test events"?

In Java, it could be the DA which contains the launcher, when running tests, does set a JUnit listener on the running app and receives JUnit events. Then it could transform those events in some "standardized" test report events as defined in the DA protocol and send those events on the same stream as regular execution or debug info.

[weinand]

Ok. And how do such DA report test results in VSCode? Is there a documented protocol to achieve that that can be used by other IDEs and other languages?

As mentioned before: a DA does not know that it runs tests. So the test results are reported as configured by the user in the launch config, e.g. sent to stdout.

VS Code (or the DAP for that matter) does not know anything about "test results" and since test frameworks (and their output reports) are not standardised in any way and VS Code has no generic test result UI, we do not have protocol for that.

In Java, it could be the DA which contains the launcher, when running tests, does set a JUnit listener on the running app and receives JUnit events.

If I understand JUnit Listeners correctly, then this is just a Java class that needs to be hooked up with a JUnit Runner (https://memorynotfound.com/add-junit-listener-example/). This custom JUnit Listener could use any mechanism to emit "test events" which the DA needs to pick up.

I don't know how easy it is in Java to bind (link?) a custom JUnit Listener to an existing JUnit test dynamically as part of the launch but maybe it is even possible to have a compile/build step as part of the launch.

The next question is how the custom JUnit Listener would actually communicate with the DA and make it emit the "test events" described above.

A simple approach is to have the JUnit Listener send test events to stdout. The problem with this approach is that the DA has no access to stdout of the debuggee (the test suite) if the debuggee runs in the VS Code integrated terminal or in an external terminal.

Another approach is to let the JUnit Listener directly communicate with the DA via a socket or named pipe. This requires that the DA "serves" this socket or named pipe.

Even if the last approach looks feasible for those test frameworks that support "injectable listeners", it confirms @hbenl's observation that testing and debugging are orthogonal: test events are not emitted by the runtime but by the test framework and they are communicated via a side channel and not via debugger protocol.

[mickaelistria]

As mentioned before: a DA does not know that it runs tests. So the test results are reported as configured by the user in the launch config, e.g. sent to stdout. VS Code (or the DAP for that matter) does not know anything about "test results" and since test frameworks (and their output reports) are not standardised in any way and VS Code has no generic test result UI, we do not have protocol for that.

Ok, so there is currently nothing IDE/Language agnostic to report test progress and result. That's IMO a missing piece in this new era of dev tools. I thought (and still thinks) that the DAP could easily handles that by considering test execution as just a way of running/debugging and handle additional messages.

I don't know how easy it is in Java to bind (link?) a custom JUnit Listener to an existing JUnit test dynamically as part of the launch

It is very easy to attach a listener as a parameter of the launch. For example, if the DA is invoked with junitTests goal, the code that will be invoked can simply place the JUnit listener on the test process and receive results from JUnit to emit them over whatever more standard format.

The next question is how the custom JUnit Listener would actually communicate with the DA and make it emit the "test events" described above.

It's only a matter in the JUnit listener to implement in testFailure for instance an operation that sends a message to the IDE over a pre-defined format. Concretely, using the DAP library that we have for Java, it could be something like

 public void testFailure(Failure failure) throws Exception {
  IDebugProtocolServer server = getDebugServer(); // the current DebugAdapter obviously has an instance of the debug server.
  server.notifyTestResult(new TestResult(TestResult.FAILURE, failure.getMessage(), ...); // whatever is useful to show user
}

I imagine other languages (C#, TypeScript, Python...) and test runner do allow similar approach.

A simple approach is to have the JUnit Listener send test events to stdout.

What if the application or test does use stdout for other reason, and what if the debug adapter also uses stdin/out for other reason. One the key criteria of success of LSP/DAP are that they allow to stream data over whichever channel. It would seem important to keep that advantage to report tests in an agnostic way.

test events are not emitted by the runtime but by the test framework and they are communicated via a side channel and not via debugger protocol.

As I genuinely and pragmatically envision it, the test runner is just an API of the language and running tests are just the same as running an application, instead that instead of calling the typical entry-point of the app, it calls the typical entry-point of the test framework.

[mickaelistria]

I see the protocol already has support for multiple kinds of OutputEvent, mostly assuming stdio and console based interaction. I believe reports about test exections could fit as OutputEvent or dedicated TestEvents.

[hbenl]

See also https://github.com/Microsoft/vscode/issues/9505#issuecomment-390492089

[weinand]

@hbenl your extension nicely shows how supporting testing frameworks is orthogonal to debug adapters. So you can combine a TestAdapter with different debuggers.

Since we have no plans to add test execution support to DAP, I'm closing this issue now.

[mickaelistria]

@hbenl: Seems like you found a good approach. Can you please elaborate about how for example I could add support for JUnit report in the Java Debug Adapter and receive language/IDE agnostic notifications?

[hbenl]

@mickaelistria I'm not sure I understand what you're trying to do. If you want to write an extension providing JUnit test support using my approach, then you have to create an implementation of the TestAdapter interface. How that implementation works (and in particular, how it interfaces with Java) is up to you, there are some JUnit extensions for VS Code out there, so you could have a look at how they do it. This is (almost) completely independent of the Java Debug Adapter. The only connection to that would be in your implementation of TestAdapter.debug(), which would start the Java Debug Adapter using vscode.debug.startDebugging() with the appropriate type for the Java Debug Adapter in the DebugConfiguration.

[mickaelistria]

Ok thanks. So the test adapter would be responsible of starting the debug adapter. How would the client be notified that a debug adapter is started and expected to be used and know how to connect to it?

-- Mickael Istria Eclipse IDE https://www.eclipse.org/downloads/eclipse-packages/ developer, for Red Hat Developers https://developers.redhat.com/

[weinand]

the client would be notified in the same way as when not using a debug adapter, i.e. when just running the tests and not debugging them.

(but actually I do not really understand what "notifying a client" really means...)

[mickaelistria]

the client would be notified in the same way as when not using a debug adapter, i.e. when just running the tests and not debugging them.

So test reports remains a specific integration between client IDE and the test framework? Like one would still have to write a specific JUnit integration for VSCode and another one for Eclipse IDE, a specific Mocha integration for VSCode and another one for Eclipse IDE...? If so, then I think it's not really a progress compared to the state of art.

I do not really understand what "notifying a client" really means...

By that, I mean that when a test suite is running, the test framework can while executing tests, send reports to any listener about test being run, test complete, test failed... Those are notifications from the test framework to the client IDE. What I'm dreaming of are standard notification mechanism for tests, which would be language independent and IDE-independent (like the a form of "OutputEvent" in the protocol) so one would only have to integrate with those notifications and could hook any kind of test framework.

But I think I start to understand the missing brick to the DAP for it to cover my use-case. In Eclipse IDE, a run and a debug execution are very similar things. You configure both more or less the same way, just the debug has a few extra settings. A run configuration is more like a debug configuration that just disable debug notifications. It doesn't seem like it's the case for this protocol, and that the DAP doesn't cover a plain execution scenario. Am I right with that? If I'm right, is there a more abstract "Launch Adapter" mechanism built in VSCode that can cover scheduling and I/O for both run or debug? Or are both run and debug implemented with very different ways of launching?

[weinand]

I envision a Test Adapter Protocol (TAP) that is independent from (orthogonal to) IDE, test framework, and underlying execution environment (run/debug). A JUnit Test Adapter based on TAP could be used for any IDE if the IDE supports TAP.

Thanks for your explanation of "notifying a client". If I understand this right then "notifying a client" is just a fundamental part of TAP.

Run vs. debug: VS Code adds a predefined "noDebug" attribute to a debug configuration in order to "run" a program instead of "debugging" it. So it uses a similar model as Eclipse (btw, I'm one of the original authors of the Eclipse debugger). But since "noDebug" is optional, many debug adapters do not bother implementing it because they see no need for it. One reason is that VS Code users are CLI oriented and run their program from the command line via some CLI tool, e.g. something like "npm start". So they do not want to create a debug configuration just for running their program. "Debugging" is not just a variant of "running" for them.

Yes, VS Code has a "Start Without Debugging" command, but this is neither well known, nor frequently used (and it falls back to "Start Debugging" anyway if the debug extension doesn't honour the "noDebug" attribute).

[mickaelistria]

I feel the need to repeat same question as earlier: if we make a distinct DAP (that cares about launching and debugging) and TAP (that cares about launching and test reports), and if I'm writing a client of the TAP, how would the Test Adapter notify the IDE that debug is available under the DAP and send the necessary information for client to connect? It's not really orthogonal to me. If we want to properly support the "Debug tests" scenario, either TAP has to inform IDE about debugging being available through DAP (so TAP references DAP), or the other way round (so DAP references TAP).

In your explanation, to cover the "debug tests" scenario, a Test Adapter would be expected to build on top of the DAP launch/debug features and would add some extra-messages for test execution? Interaction with the test for debug would happen over DAP, while test results would happen over TAP? And the client would have to configure both connection to debug adapter and test adapter? Additionally to those same concerns above, isn't it more complex than having the DAP enabling some extensibility to add test related messages?

I really think test reports and debug are as orthogonal as console logging and debug in the general case. Both share the responsibility of launching, and by not trying to factorize those in the DAP, there is the risk of ending up with multiple ways of launching things where having everyone relying on the DAP would be better. The type of OutputEvents should be extensible to allow arbitrary kind of messages and not only logging. The TAP would become an extension to DAP with extra messages, a given Test Adapter would be a regular Debug Adapter sending some extra messages, and clients like VSCode or Eclipse IDE could just add listener to those extra messages.

Tests are IMHO, or at least we'd all love them to be, as important as console in execution. I think they could really be a first citizen of a debug story.

[weinand]

a TAP shouldn't/cannot assume the existence of DAP because:

• an IDE might not support DAP,
• an IDE might use DAP only for debugging but not for execution.

DAP is really an implementation detail and TAP should avoid being coupled to DAP.

The debug API available for TAP in VS Code should be vscode.debug.startDebugging() or the Task API or just via the command line.

You said:

Interaction with the test for debug would happen over DAP

Why? How? If I ran tests via "npm test", DAP is not used but I still want to run my tests and see the results in the IDE.

Folding TAP into DAP: how should this work? Someone ships a Java debug extension for VS Code and someone else wrote a Testing Framework and now needs to "subclass" the Java Debug Extension for supporting the new test framework via TAP? We'll end up with lots of copies of the Java debug extension...

Tests are IMHO, or at least we'd all love them to be, as important as console in execution. I think they could really be a first citizen of a debug story.

IMO tests are so important that they deserve to be first-class citizen of their own story: a test story.

[mickaelistria]

If I ran tests via "npm test", DAP is not used but I still want to run my tests and see the results in the IDE.

A Test Adapter could run without using DAP. That said, at this stage of the domain, all IDEs that have support for Test Adapter also have support for DAP. I think trying to force them to be independent is not really necessary at this stage. Wouldn't it be interesting, for VSCode as well, to promote the DAP as the reference way of launching anything?

If you run "npm test", how does it happen in VSCode to have this execution recognized as a Test Adapter for instance?

But still, this doesn't seem to enable a good "debug test execution" story in the IDE that both show test reports in dedicated view, and allow debug.

Folding TAP into DAP:

how should this work? Someone ships a Java debug extension for VS Code and someone else wrote a Testing Framework and now needs to "subclass" the Java Debug Extension for supporting the new test framework via TAP?

Yes. Concretely, in that case:

• The Java Debug extension is already here
• The JUnit extension would build on top of the debug extension (debug is a requirement) and use the Debug Adapter for launching
• Creating a JUnit launch would start the tests as a regular Java app (with the right settings and so on) and the test framework notificiations done by JUnit would be caught by the JUnit Launch, turned into TAP messages, and sent back to the client over the same stream as the other debug or console informations.

By simplified example.

• Original Java command line for regular entry point: java -cp project/class/path EntryPoint
• Command for Debug: java -Xdebug ...-cp project/class/path MyProjectEntryPoint. The Java Debug Adapter already does something like this internally and translates debug info from/to DAP. Either it uses JVM interaction to interact with runtime process, or it uses a different "DebugEntryPoint" which is internal code to the Java DAP support to bootstrap listeners.
• Command for tests: java -cp project/class/path org.junit.Launcher -Dsettings=.... A Java Test Adapter would run this command and either use a subclassed Launcher to stream to TAP or use JUnit listener facility with "-Dlisteners=..." setting to catch events and forward them over TAP
• Command line to Debug Test combine both java -Xdebug -cp project/class/path org.junit.Launcher -Dsettings=...

The thing is that if TAP and DAP are independent, it's hard for one to reuse the other and to generate the command-line variations easily, while the potential for reuse is huge. As the main difference between a regular run and a test run is going to be a few Test notifications (and not tons of different ones), it seems way easier to achieve this combinations by allowing or recommending DAP and TAP to work together, use the same channels, or even to be the same thing. Adapters could support the variations they want, as it's already the case.

Now, from a client perspective, I still miss the answer to "if I start a test which support test adapter, and this is in debug mode and uses the Debug Adapter, how can I discover and connect to the debug adapter without code specific to the Test Adapter". If we make both use the same channel for connection, then the answer is "once you have one, you have both". If we don't then we need an extra strategy for a Test Adapter to announce to client it's debugging. Or we keep both separate, but we need in test adapter a way to announce it's debugging over Debug Adapter (making a relationship between both).

We'll end up with lots of copies of the Java debug extension...

Why would you need lot of copies? Only one instance of the Java debug extension, and the JUnit extension would just extend it with its specific code, the TestNG extension with its specific code... to just stream test events over the same channel as debug info.

To be clear, if you can give me a good example of how I can debug test execution in VSCode and get test notifications in the dedicated view over TAP, and ability to debug them over DAP, without writing any code specific to this Test Adapter, then I may give you a break ;)

[weinand]

if you can give me a good example of how I can debug test execution in VSCode and get test notifications in the dedicated view over TAP

In your regular Java debug configuration JSON just add an org.junit.TAPAdapter argument so that effectively the following is executed:

java -cp project/class/path org.junit.TAPAdapter -Dsettings=...`

The org.junit.TAPAdapter uses TAP to talk to the VS Code TAP Test View (independently from DAP). It might need another argument or env var to know how to communicate back to VS Code.

If you do not want to use a VS Code launch config because you don't want to debug, then just type the following in VS Code's integrated terminal:

java -cp project/class/path org.junit.TAPAdapter -Dsettings=...`

If you like to run this from Maven:

mvn -Dtest=...

... you get the idea.

[mickaelistria]

Thanks for your answer, it really helps.

In your regular Java debug configuration JSON just add an org.junit.TAPAdapter argument so that effectively the following is executed:

So in this case, it's the Debug Adapter that is aware of the Test Adapter Protocol and sets up extra connection to TAP. Sounds ok to me (but shows that while they can be orthogonal in definition, in practice, having the debug test story requires one adapter to be aware of the other one, and I believe it will become mandatory for serious adapters as the debug tests is a critical use-case).

The org.junit.TAPAdapter uses TAP to talk to the VS Code TAP Test View

s/VS Code TAP Test View/whatever TAP client/ I guess.

It might need another argument or env var to know how to communicate back to VS Code.

What could this argument be? A path to a file, a socket, a TCP port, any of those...? I believe wd need something standard-ish for most clients to be able to consume the DAP+TAP combo. Somethink like at least specifying the argument name and possible values so we don't have multiple debug adapters using different approaches, making it impossible for various clients to implement the various possibilities.

[weinand]

No, the Debug Adapter is not aware of the Test Adapter Protocol. But the "module" org.junit.TAPAdapter is. I was just using your example from above:

Command for tests: java -cp project/class/path org.junit.Launcher -Dsettings=.... A Java Test Adapter would run this command and either use a subclassed Launcher to stream to TAP or use JUnit listener facility with "-Dlisteners=..." setting to catch events and forward them over TAP

I've just replaced the "org.junit.Launcher" by a "org.junit.TAPAdapter". I do not really know what a "org.junit.Launcher" does, but my assumption was that there is a mechanism to configure JUnit's test report by plugging in some code. If this assumption is correct, then it should be possible to redirect the test report into TAP.

I believe wd need something standard-ish for most clients to be able to consume the DAP+TAP combo.

Again, there is no requirement for or connection to DAP. The passed in module org.junit.TAPAdapter knows nothing about DAP because it must work in any situation. And when running tests from Maven or from the command line, there is no DAP involved either.

And another thing: the DAP spec has a launch request but it does not define any attributes (because they are debugger/runtime specific). See https://github.com/Microsoft/vscode-debugadapter-node/blob/e1b45efc60d2f58dff4cf5c8771b213fecc524e3/protocol/src/debugProtocol.ts#L355-L362

The only interesting attribute is "noDebug" and nothing else. So the launching aspect of DAP is not something TAP could profit from.