Closed connor4312 closed 3 years ago
๐ I'm the maintainer of the Ruby Test Adapter extension and just wanted to drop in and give a bit of context on the pain points I've had with the Test Adapter API. The Adapter API works well for the most part, but the biggest problem I've had with it is getting it work well with large codebases.
I developed the Ruby adapter with my Rails app vglist in mind, and at the time it had around 300-400 tests in total (it now has 700-800). My adapter worked fine with that, it took a few seconds (maybe 5-10) to reload all the test suite data whenever a change was made to the test code. However, even 5-10 seconds is annoying to wait for when you just want to run your tests right after you modify them. And now I work at a company with a Rails app that has over 12,000 RSpec tests, so my extension now takes about 45 seconds to reload the tests whenever I change them (and it'd be even longer if I didn't have a pretty well-spec'd laptop). I don't ever really use my test adapter with that codebase because it slows me down far too much.
The problem is that the API doesn't support modifying the existing set of tests, only replacing all of them at once. Which requires RSpec to generate information for every test in the codebase whenever it reloads the tests.
I've considered a workaround where I'd cache the information in a tempfile or something, and have rspec only load information about changed files, and then modify the cached tree and send it up to the explorer UI, but that's complex and I haven't gotten around to actually doing it.
There are other problems - which I'm not really sure how to solve - with being unable to run the tests while it's loading the test suite because it doesn't handle the tests that pass while it's loading (the data can theoretically get out-of-sync if you, say, change a bunch of things between reloads, but >99% of the tests in the codebase will be exactly the same before and after any given change I make to the suite). So it's maybe worth considering that when designing the invalidation logic.
There are also problems with being able to view test logs to figure out issues when tests fail, but that's more of a problem with how the RSpec framework works than with anything the Test Adapter API does.
The current API Sketch looks quite good from what I'm seeing. I especially like the use of a cancellation token, which if I understand correctly seems like a much cleaner solution than the Adapter API where you just blindly attempt to kill any relevant running processes. There are some things missing there (e.g. how do I handle the case where the suite is unloadable because the user has created an invalid test suite via some syntax error?), but overall it looks like it takes a lot of the good decisions from the Adapter extension API.
Anyway, I hope this makes some sense. I understand it's probably not a super cohesive comment since I wrote it on-the-fly, but I've been hoping something like this would be introduced into the core of VS Code eventually, so I wanted to make sure I got my two cents in :)
Thank you for that feedback, it's super helpful!
Scoping tests and handling large codebases is definitely something we want to tackle from the outset. Like all features we build, we will be dogfooding it in the VS Code repo, so it is a priority to be able to handle that. As you saw in the sketch, I think the collection approach solves the pain points around having to replace all tests.
The idea of running tests during load is interesting. The base case of running all tests is pretty simple from the API perspective -- we would just runTests
without any filters and let the extension run as many as it can. So long as it's able to reconcile the collection into the right state running concurrent discovery and execution, everything should be fine.
One scenario that isn't covered is running a subset of tests as discovery is happening (e.g. if I wanted to run all tests with the word "foo" in them) without starting a new test run. This would take some finessing and will not be supported by all adapters, for instance Go could never support it without changes to its toolchain, so is probably not worth designing for...
There are some things missing there (e.g. how do I handle the case where the suite is unloadable because the user has created an invalid test suite via some syntax error?),
I think the right thing to do would be for the test extension to emit a diagnostic for that using our existing API. I've added it to the open questions as well though for discussion.
@connor4312 generally speaking I'd want to give the user an error notification so they know something has broken and their suite failed to load, rather than creating a diagnostic. But as long as the extension can choose what to do if that occurs, I suppose the Test API in VS Code doesn't need to care about that.
When I say diagnostic I mean the "Diagnostic" in the VS Code API that appears in the problems view (and can also have an error squiggle in the editor)
The downside there is that if they have a syntax error, having the problem be duplicated by both the language server and test extension will be some ugly clutter.
The reason I wouldn't use a diagnostic is that 1) I personally rarely ever look at diagnostics/errors in the diagnostics view because the signal to noise ratio in my experience is so high, and 2) there are situations where a diagnostic wouldn't really make sense, for example if the user's tests fail to load because their local Postgres server isn't running or because they forgot to install their Ruby gems so the RSpec gem isn't even available.
Some thoughts from my side (I'm the author of the Test Explorer UI extension):
TestProvider
authors. This is obviously necessary if you want to support only detecting tests in the currently opened files (which I had never thought about) but even without that there seem to be some scenarios where TestAdapter
authors struggle with this part of the Test Explorer APITestAdapter
s to the new world could be done with a bridge extension (or built into Test Explorer itself) which registers all the tests coming from TestAdapter
s with the new TestProvider
APITestProvider
should run in the Extension Host: I'm not worried about expensive work being done in the Extension Host process here: the most expensive part is usually running the tests and TestProvider
s should always do that in a child process anywayTestAdapter
authors would also like to share their thoughts: @Raagh @kavod-io @Gwenio @Testy @marcellourbani @numaru @bneumann @Florin-Popescu @matepek @fredericbonnet @drleq @dampsoft @betwo @zcoinofficial @vshaxe @kondratyev-nv @recca0120 @DEVSENSE @swellaby @prash-wghats @Derivitec @Bochlin @maziacThanks for the mention @hbenl. Looks good to me, I also agree with @connorshea about the notificantion in case something is broken. I had that a lot and having a possibility to tell the users what went wrong might be good.
One thing that I was missing, and I am not sure if it possible, is to match my tests with the source files. My adapter runs C code and I can't simply grep all files for the function names. I could do that if the user registered the toolchain somehow but that would require a lot of setup for the user to make prior to testing. So it would be nice to somehow get cross information from the language server and check which files in my project are actually test files.
Not sure if that is meant by:
In a golden scenario, invalidation of tests would be done by a language server which can intelligently determine specific tests that should be invalidated when a file or a file dependency changes. Maybe this is still handled by an event on the TestProvider, but if we take a "Test Protocol" approach then coordination will be harder.
One more thought: the central difference between tests and suites in Test Explorer is that the state of a suite is always computed from the states of its children (with one exception: a suite can be marked as errored). TestProvider
doesn't distinguish between tests and suites but it should (optionally) provide the capability to let the state of a test be computed from the states of its children. Perhaps this could be flagged by another TestRunState
(e.g. INHERIT
). Of course it could be left to TestProvider
authors to update the parents but this can become a slightly hairy issue if you want to avoid potential performance problems, so it would be nice if VS Code provided that.
Hello I'm the author of C++ TestMate
The Test Adapter API makes the distinction between suites and tests, we do not. They have almost identical capabilities, and in at least one scenario the 'suites' are more like tests and the leaf 'tests' cannot be run individually.
Praise the initiative. The feature will be useful for Catch2 too. (related issue)
invalidation and auto-run
I think only language server isn't enough. We need the flexibility to retire tests in case of external dependencies and file changes.
A request of my user: I don't see an "output window" concept on the current API but I assume that it is unavoidable. (Some test frameworks can provide useful output which cannot be associated to any tests.) The person expressed the need to have the output window updated before the test itself finishes. I believe it is a reasonable request. It seems to me that multiple TestState with "Running" can be a solution. But I don't see how the output window fits into the current API.
About test state propagation: For my extension if a test fails all the ascendants has to be failed too. Even a node has a "Passed" state if one of the descendant's has a "Failed" or "Errored" state than has to be propagated. Some precedence also seems necessary like: "Running" over "Errored" over "Failed" over "Skipped" over "Unset".
But this raises another question. Scenario: B and C under A. Test B reports failure and test B still running. What should be the state of A? According to previous precedence it should be "Running" but users might would find it useful to see that some tests part of that "Suite"/node has already failed. I have some ideas about it but all of the sacrifices the simplicity and clearness of the API.
Question: The integrated test explorer what extra features will be available for VSCode users? I mean we have a working system now thanks to @hbenl. What is strongest the motivation of the integration?
Thanks for the mention @hbenl
In my abapfs extension I only know I have a test after running it, a discover function would take weeks to run.
Proposed API looks ok as long as I can fire onDidChangeTest any time
/cc @connectdotz who has been doing great work on vscode-jest /cc @captbaritone who I had a conversation exactly about this a year ago
Thank you for the feedback and the tags!
Point taken around the error notifications. This week is endgame for our September release, but we'll continue discussion and update this issue + the sketch next week.
TestProvider doesn't distinguish between tests and suites but it should (optionally) provide the capability to let the state of a test be computed from the states of its children
But this raises another question. Scenario: B and C under A. Test B reports failure and test B still running. What should be the state of A? According to previous precedence it should be "Running" but users might would find it useful to see that some tests part of that "Suite"/node has already failed. I have some ideas about it but all of the sacrifices the simplicity and clearness of the API.
@hbenl You bring up a good point that, in the current design, state is never inherited. Generally for tests we want the opposite -- that state is always inherited -- but thinking about this more it's only applicable within a tree UI. Perhaps if the state is unset, then we say UI implementations should do the right thing automatically. For example, a test explorer would show parents as failed if any of their children failed, or running if any of their children are running. But a status bar item that shows failing/passing tests would only count those who had a non-unset status.
I think this approach would also make @matepek's scenario nice from an API point of view; the UI can be arbitrarily smart.
One thing that I was missing, and I am not sure if it possible, is to match my tests with the source files. My adapter runs C code and I can't simply grep all files for the function names.
@bneumann I think this problem is mostly out of scope. We will provide hooks in VS Code to build tests, but won't add any special new discovery mechanisms.
I think only language server isn't enough. We need the flexibility to retire tests in case of external dependencies and file changes.
@matepek Definitely, the language server won't be the only way (we aren't going to require test providers to also integrate their language server) but we should not block that path.
I don't see an "output window" concept on the current API but I assume that it is unavoidable. (Some test frameworks can provide useful output which cannot be associated to any tests.) The person expressed the need to have the output window updated before the test itself finishes. I believe it is a reasonable request. It seems to me that multiple TestState with "Running" can be a solution. But I don't see how the output window fits into the current API.
@matepek Yes, test output and diffing is one particular area we want to improve. Thank you for mentioning the streaming scenario, I'll make sure we look into that and will tag you with the outcome/proposal.
What is strongest the motivation of the integration?
We actually are not set yet on making this integrated. Our roadmap outlines the purpose here (which I should add to the original thread for clarity):
Investigate how VS Code can improve the testing support. Several extensions are already providing testing support, explore what APIs/UIs could be added to improve these testing extensions and the test running experience.
Some benefits of being built-in, mentioned briefly in the table in the original issue, are:
engines
field) and guarenteesAt the moment we're sketching an extension host API to provider an understanding of the 'ideal world' API. This may become a built in API, it could be a protocol, or we could end up working with Holger to apply some principles the test explorer UI.
hi, I work with @orta in vscode-jest. This is an interesting thread, have a few questions:
First, the high-level question: I am wondering what is the main feature from the end-users' perspective? Is it a new build-in sidebar view listing all tests that can be run/debug via standard UI? For test extensions, does it boil down to using the build-in "Test Explorer" instead of their own sidebar views?
The proposed API only seems to cover the Diagnostic
(the PROBLEMS tab), our extension also heavily uses Decoration
for inline visual indications and assistance in the editor, as well as OUTPUT
for detailed test output and notifications for error/help-tips (as others also mentioned). If a consistent interface is the goal, then maybe all these UI extensions should be included as well?
In jest, a test is identified by its name(label) and the hierarchical context (the parent block). It is hard to replace a single test because its name or line location can all change when users are developing their tests. Therefore we always end up reconstruct the test tree for the given file on each run. The proposed API seems to take the "bottom-up" approach, where each TestItem
points to its parent. To find all tests by a given parent will require traversing all the items. This will be quite expensive for our use case, but if we are the only test framework prefer top-down, we could just build our own internal structure to track them... is this not an issue for other test frameworks/extensions?
Given there is no update
operation for TestCollection, I assume TestItem is immutable? But the TestItem
interface didn't have readonly
so not sure... How does one update a test status for example? delete + add? or mutate the TestItem + firing onDidChangeTest? You are right that it is kind of redundant if you already observe the collection and yet still require the change event...
In multi-root workspaces? Does the explorer display the current folder only? Will users see the full workspace (summary of each folder)?
A stretch goal: In addition to standard UI, it would be great if this can also address our users' biggest pain point, which is often the test setup, env configuration, etc... how about something like a "test config", similar to a debug config, that Provider can provide snippet or even wizard to help set up in launch.json?
hey all, I maintain the Jest Extension for the @hbenl's Test Explorer UI. Here is a bit of a brain-dump, some of which might be useful:
I agree with @connectdotz comments about the structure of Jest tests being very hierarchical and top-down. I maintain internal state in my extension as a tree and map from the internal structure to the Test UI structure when I need to emit events as the internal structure is highly correlated to the output. IMHO the test collection as a mutable list-like collection and mutable test objects would be more difficult to translate to/from than what I currently do. Now it may be that the Test UI extension can live on as an intermediary extension taking care of this translation on our behalf (amongst other tasks e.g. multiple workspaces). But I think it would be worth taking a little time to think about how the test information will be consumed by UI extensions as noted in the todo. Maybe leave this API in draft until the UI API has been fleshed out. It doesn't make sense to flatten it, pass it to VS Code to pass it to a UI extension where it is transformed back into tree form again. Other extension authors should comment on what their preferred format is.
The proposed API describes how the TestProvider
can be controlled by VS Code - through the runTests
and addWorkspaceTests
methods. However, at least in the context of the extension I maintain, it doesn't always make sense to obey that command. In the case of the addWorkspaceTests
method, the extension may have already discovered all of the tests before that command is invoked making it a no-op. Also because my extension is currently monitoring the file system for changes and running tests as it sees fit, what if the user wants to initiate a test run through the runTests
method? Should the internal test run be cancelled in favour of the users choice? Is it Ok to plainly disobey the request?
TestProvider
has a state, e.g. Parsing/Discovering, Running Tests, Idle etc, potentially with a corresponding event emitter for status changes. This could also serve to notify VS Code of events such as failure to load/run tests etc.TestItem
. The Test Explorer UI allows this but I can't see how you can find the source file with the proposed API.Test Messages:
message
and a tooltip
which would live at the equivalent level of the TestState
. However, I don't have any suggestions on how to improve things, while remaining UI agnostic. I use the message
to display detailed stack traces, I can't remember if I use the tooltip
or not.TestMessage
s if it is possible to provide messages that are not errors (info, warning etc).Also maybe a category for testing could be useful.
Currently I'm using Other
.
Hello again everyone ๐ with our housekeeping wrapped up, I'm back on this for the remainder of the year. My goal by early Jan is to have an API to parity of what is show in the sketch and what the test explorer provides working, as well as a branch or fork of the test explorer that works with the new API.
First, the high-level question: I am wondering what is the main feature from the end-users' perspective? Is it a new build-in sidebar view listing all tests that can be run/debug via standard UI? For test extensions, does it boil down to using the build-in "Test Explorer" instead of their own sidebar views? -- @connectdotz
The proposed API only seems to cover the Diagnostic(the PROBLEMS tab), our extension also heavily uses Decoration for inline visual indications and assistance in the editor, as well as OUTPUT for detailed test output and notifications for error/help-tips (as others also mentioned). If a consistent interface is the goal, then maybe all these UI extensions should be included as well? -- @connectdotz
The sketch has TestMessage
s in the TestState
, which are optionally associated with a location
. Locations contain a URI and a range, which allows us to show decorations
In jest, a test is identified by its name(label) and the hierarchical context (the parent block). It is hard to replace a single test because its name or line location can all change when users are developing their tests... -- @connectdotz
I agree -- @rossknudsen
The model for this was as you mention somewhat challenging. We have prior art in the extension API for lazily providing trees of hierarchical data via the TreeDataProvider<T>
. In principle we could use this or something similar for tests as well. However, the TreeDataProvider is built to provide information gradually about its data, while many cases for tests (e.g. showing how many tests passed/failed) require the entire test tree to be materialized.
So, we could go for a more traditional recursive data structure, e.g type TestItem = { children: TestItem[]; ... }
. One benefit of this approach is that it puts the onDidChangeTest
event in less of a "weird spot" and more similar to onDidChangeTreeData
; the event can fire when a TestItem changes or when its array of children updates. Then we have a testRoot
element, which can be generally be hidden if there's only a single test provider, or shown in the edge case that there's multiple (e.g. if I'm developing a Go server and a client with Jest tests.) I like this approach more than the current collection-based approach, cc @sandy081
diff --git a/src/vs/vscode.proposed.d.ts b/src/vs/vscode.proposed.d.ts
index 98bada7b79..ca87f5d17b 100644
--- a/src/vs/vscode.proposed.d.ts
+++ b/src/vs/vscode.proposed.d.ts
@@ -2202,14 +2202,20 @@ declare module 'vscode' {
* via `addWorkspaceTests`.
*/
export interface TestProvider<T extends TestItem> {
- readonly collection: TestCollection<T>;
+ /**
+ * Root node for tests. The `testRoot` instance must not be replaced over
+ * the lifespan of the TestProvider, since you will need to reference it
+ * in `onDidChangeTest` when a test is added or removed.
+ */
+ readonly testRoot: T;
/**
* An event that fires when an existing test in the collection changes.
- * This can be a result of a state change in a test run, for instance.
+ * This can be a result of a state change in a test run or its properties
+ * update.
*
- * @todo VS Code already 'observes' the test collection, it's awkward
- * to have a separate event to notify changes
+ * This should also be fired when a test gains or loses a direct child
+ * from its `children` array.
*/
readonly onDidChangeTest: Event<T>;
@@ -2241,10 +2247,10 @@ declare module 'vscode' {
*/
export interface TestRunOptions<T extends TestItem> {
/**
- * Array of specific tests to run. If not given, then the provider should
- * run all tests.
+ * Array of specific tests to run. The {@link TestProvider.testRoot} may
+ * be provided as an indication to run all tests.
*/
- tests?: T[];
+ tests: T[];
/**
* Whether or not tests in this run should be debugged.
@@ -2287,9 +2293,9 @@ declare module 'vscode' {
location: Location;
/**
- * Parent item of this test.
+ * Optional list of nested tests for this item.
*/
- parent?: TestItem;
+ children?: TestItem[];
/**
* Test run state. Will generally be {@link TestRunState.Unset} by
Given there is no update operation for TestCollection, I assume TestItem is immutable? But the TestItem interface didn't have readonly so not sure... How does one update a test status for example? delete + add? or mutate the TestItem + firing onDidChangeTest? You are right that it is kind of redundant if you already observe the collection and yet still require the change event... -- @connectdotz
In the sketch as well as in the above, they would be mutable and the extension should fire onDidChangeTest
when they update. I like immutability, but that's not the paradigm most of our extension api takes ๐
A stretch goal: In addition to standard UI, it would be great if this can also address our users' biggest pain point, which is often the test setup, env configuration, etc... how about something like a "test config", similar to a debug config, that Provider can provide snippet or even wizard to help set up in launch.json?
That's an interesting idea. Currently test extensions generally hook into your existing test setup. I have not thought about it from the perspective of the test extension actually being the entrypoint to testing, but as tests become built-in this can happen in the same way we've been pushing the VS Code (particularly JS) debugger to be the entrypoint to debugging...
I will add that to the todos above, but will be something for after this month.
Maybe leave this API in draft until the UI API has been fleshed out. -- @rossknudsen
Yes, all this will remain in vscode.proposed.d.ts land for some time as we get feedback from the community!
In the case of the addWorkspaceTests method, the extension may have already discovered all of the tests before that command is invoked making it a no-op. -- @rossknudsen
I think this is fine. addWorkspaceTests
is a hint to the test runner that it should discover tests because there's now "something" that wants to see them. It is there to avoid test extensions having to go out and discover tests on startup, since startup time and performance is something we try to optimize for. If the tests were previously enumerated and they are already in the collection, it's fine to no-op.
Also because my extension is currently monitoring the file system for changes and running tests as it sees fit, what if the user wants to initiate a test run through the runTests method? Should the internal test run be cancelled in favour of the users choice? -- @rossknudsen
In the design thus far there's not particular requirement around serialization or lack thereof of test runs; I think this is a good thing. So it's generally up to you as the test extension. But as a user I would generally expect that the test states eventually reflect whatever my most recent run.
I think one missing feature would be the ability to open the corresponding source from a TestItem. The Test Explorer UI allows this but I can't see how you can find the source file with the proposed API. -- @rossknudsen
The location
on the test item provides the file URI and source code range where the test can be found.
Perhaps the TestProvider has a state, e.g. Parsing/Discovering, Running Tests, Idle etc, potentially with a corresponding event emitter for status changes. This could also serve to notify VS Code of events such as failure to load/run tests etc. -- @rossknudsen
This is something I will be adding within the next couple days. I don't know what it would look like yet.
Test Messages -- @rossknudsen
Thanks for the feedback. More in the next couple days; this is an area for improvement/expansion.
Some more updates building out the API.
These are my current thoughts. TestItems have a TestState. The TestState is defined as immutable. This lets us (and consuming UI) use instance equality checks when determining whether there was a change made to the state, instead of trying to do a deep or complex diff on every change.
As @rossknudsen suggested I added a severity
, copied from Diagnostic messages. Additionally there's an expectedOutput
and actualOutput
which can be shown in a diff editor/peek view. Finally the message is optionally a MarkdownString.
I really still like the idea of having a test tree as @connectdotz and @rossknudsen. However I start to struggle with it when it comes to document versus workspace tests.
Having a document/in-editor view of test metadata requires that their locations be able to be reconciled with unsaved changes, which are not visible in a filesystem-based workspace watcher. (OIt's for certain that not all test extensions will go through the trouble of implementing this, but doing so should be possible.) This means we either have their weird dual state in a single tree of tests, or we have two different trees. There's also the issue that TestItems may not necessarily have associated Locations, so in a single tree it's hard to tell what tests in the hierarchy are relevant for a given TextDocument anyway.
So this is the direction I'm thinking -- the providers return a "hierarchy" object on which events are fired:
This can be mirrored in the "test observers" which other extensions can call to receive a stream of tests:
In this sketch the provider result and observer just have an event that fires when tests are discovered, not granular "Parsing/Discovering, Running Tests, Idle" states, but this may be sufficient.
Talking with Sandeep this morning, we agree that the "Hierarchy" approach is worth exploring. So I've built the API it as well as a simple sample extension
Implementation at: https://github.com/microsoft/vscode-extension-samples/blob/test-provider-sample/test-provider-sample/src/testProvider.ts
Notes:
As we go into the holidays, don't expect too many updates. Recent discussion:
๐ Hi, I have some questions about the proposal:
addWorkspaceTests()
, is it able to lazy-load the tests in the explorer? (like add one level of the direct children when the user expand a node in the test explorer, this is especially useful when in a large code base having thousands of tests there)Thanks
Thanks for the questions ๐
About the method addWorkspaceTests(), is it able to lazy-load the tests in the explorer? (like add one level of the direct children when the user expand a node in the test explorer
Not at the moment. Such lazy loading would work for the explorer, but the moment we want to do something like show the percentage of passing or failed tests, we would need to enumerate all tests. So I'm not convinced that's something to design around.
the test config, will the API consider to generate and persist it somewhere? (for example, if the user wants to re-run the tests, is it the extension's responsible to store the launch configuration or the API will have the capability to handle that?)
I really want to avoid having a special VS Code test config. Most test runners have existing configuration, e.g. .mocharc
files, config in setup.py, etc.
I'm not sure specifically if you mean just "rerun the tests I just ran" or "rerun failed tests". In both cases the state for this shouldn't be stored in a config file; this would be noise that you wouldn't want to check in to source control.
runState
of tests instances.TBH I havenโt read the threat thoroughly just wanted to mention one use case which would be nice to be covered by the new API: Catch2 C++ test framework support BDD style. And the implementation comes with limitation: One can define โsub-test-caseโ in a test-case but that sub-test-case become available/listed only when the test-case is running/were run. This means that a leaf/test-case has to be able to became a branch/suite and sub-test-cases has to be added at runtime or shortly after the run with the corresponding states.
A couple of months ago the proposed API design might were able to support this. After that I lost tracking. Thanks for considering the use case.
Other thing we have to deal with is the activation event. For user-user perspective they usually prefer lazy init/listing but some actually the opposite. For them now they can init the extension with an โonOpenโ-task.
Bests,
Thank you for your consideration ๐
I've created an experimental extension that translates from the TestAdapter API to the TestProvider API, in other words: when this extension is installed, all TestAdapters will also show up as TestProviders. You can try it out here.
I have a few comments regarding the TestProvider API:
TestInfo.tooltip
) and when hovering over an error decoration (TestDecoration.hover
). In both cases a one-line message can be set (TestInfo.description
and TestDecoration.message
) with more details provided in the tooltip. The TestProvider API currently has only TestItem.description
and TestMessage.message
and it's not clear how they will be used in the UI.TestProvider.createWorkspaceTestHierarchy()
to return undefined
, or to have an empty TestHierarchy (currently it always contains at least one TestItem).If / When the Test Explorer implemented in core supports all the features that my Test Explorer UI extension supports, my extension could be replaced by this converter extension.
That's awesome, thank you!
One more thought about manually restarting the discovery: we'll also need to give visual feedback to the user that the test discovery is running - currently this is only possible for the initial discovery. Maybe something like TestHierarchy.onTestDiscovery: vscode.Event<'started' | 'finished'>
instead of TestHierarchy.onDidDiscoverInitialTests: vscode.Event<void>
.
Hey! Thanks for the effort in integrating this into the VSCode. I think @hbenl UI extension is amazing and it definitely a great starting point for building a common API. Overall, the current state of the proposed API looks good to me. Couple of things I'd like to ask/mention.
TestRunOptions
contains the collection of tests: T[]
. This is similar to the Test Explorer UI. However, I was always wondering, does some real-world scenario exist for the case of multiple tests to be run (not a single test/suite)?TextDocument
interface (which used for createDocumentTestHierarchy
for example), but does it always have a valid URI/fileName? I'm asking because it kind of does not make sense to discover tests that are not saved, or saved outside of current workspace...id
property on tests. I don't see such property in TestItem
of the proposed API. Should each test adapter extend this interface with its own specific properties?I was always wondering, does some real-world scenario exist for the case of multiple tests to be run (not a single test/suite)?
Actions like "rerun failed tests" will request subsets of tests to be run. Actually today I added the option to group by status in the explorer; clicking "Arrays" here would rerun only the two failed tests and not the other passed tests.
I'm not really familiar with TextDocument interface (which used for createDocumentTestHierarchy for example), but does it always have a valid URI/fileName? I'm asking because it kind of does not make sense to discover tests that are not saved, or saved outside of current workspace...
It does have a URI, textDocument.uri
. However, a purpose here is in fact to include untitled/unsaved changes. For example, Go tests are structured and very easy to identify by scanning the source code of a single file, so they can implement createDocumentTestHierarchy
which will allow vscode to show code lenses and structure even for unsaved changes.
If this is something a test extension doesn't want to deal with, you can now return undefined from createDocumentTestHierarchy
if you see one of these files ๐ (or not implement it at all)
Also, I will (though this is not yet done) have the action of running tests save all file changes, like we do when debug sessions are started.
Regarding file watchers and refresh/restart test discovery. It seems that every test adapter will need to do pretty much the same - watch for all test files and trigger onDidChangeTest on file save. It's not a huge or complex part, but maybe it is worth considering having that in the VSCode testing. This is also connected to autorun or save-before-run features if they will be considered in the future.
The perspective that the current API takes is that the test provider is the sole controller of the state and information in tests. I think this is a convenient property that makes reasoning easier, but it means that there's not a way to have 'out of bounds' signalling to invalidate tests. It'd be easy to have something that just sets the "state" back to Unset on tests if their file change, but I'm not convinced that something that's beneficial to do. (and the file watcher api is not hard to implement)
With regard to auto run: that will come soon :) I think that will just be implemented by having VS Code issue runTests
whenever a test or tests is created in or enters the Unset state.
One of the important things from Test Explorer UI is id property on tests. I don't see such property in TestItem of the proposed API. Should each test adapter extend this interface with its own specific properties?
Yea, I expect extensions to extend the test items with additional data and info. E.g. in the sample test provider I have separate classes for the root, file, heading, and test cases which implement the TestProvider.
Internally in the API we use object identity to differentiate and identify tests.
just make sure to consider accessibility of the test explorer, now or in the future before the feature leaves preview. None of the current solutions is accessible enough, for example as a screenreader user I cannot see which tests failed in the test tree.
Actions like "rerun failed tests" will request subsets of tests to be run. Actually today I added the option to group by status in the explorer; clicking "Arrays" here would rerun only the two failed tests and not the other passed tests.
Got it, that's a good example, thank you!
For example, Go tests are structured and very easy to identify by scanning the source code of a single file, so they can implement
createDocumentTestHierarchy
which will allow vscode to show code lenses and structure even for unsaved changes.
Yeah, I think in general tests can be discovered for other languages as well. But I still think it won't work properly in most cases (as it will require module imports, and it will depend on the location of the test file). Also, from what I saw in other test adapters most use external tools (test runners) to discover and run tests, which will require a file to be present in the filesystem. So I can only imagine something like doc-tests or simple tests to build small reproducible examples.
Don't get me wrong, I'm not opposing having a createDocumentTestHierarchy method as test adapter may not implement it. Just curious whether this is even a valuable feature that will have any real-world use cases.
With regard to auto run: that will come soon :) I think that will just be implemented by having VS Code issue
runTests
whenever a test or tests is created in or enters the Unset state.
How it'll interact with save-before-run then? I mean as one part (save-before-run) being on the VSCode side and the other (run on save) being on the adapter side, won't it trigger multiple concurrent test discoveries/executions?
Thanks for the answers! Maybe I missed it but is there a way to enable the current version of the UI and API to try integrating with it?
I was always wondering, does some real-world scenario exist for the case of multiple tests to be run (not a single test/suite)?
You can select multiple tests in the Test Explorer UI (using Ctrl-Click or Shift-Click) and then run them, I use that quite often. Some background: I have 2 projects where running all tests can take several minutes, so I often want to (re)run only some of them. Actually one of these projects (the Firefox debug adapter) was my original motivation for writing the Test Explorer UI: when running all tests is not an ideal option anymore you need a UI to be able to quickly select the tests that you want to run. There's another Test Explorer feature that was also motivated by slow tests: the "retired" test states (shown with faint icons in the UI) - when rerunning all tests takes a long time, you really want to be able to see the state that a test got on its last test run, even if that state is outdated. But you also want to see that the state is outdated - hence the "retired" states. @connor4312 I hope you will implement this in the new Test Explorer UI, it doesn't need to be reflected in the TestProvider API (it wasn't reflected in the TestAdapter API either), since this is just part of the UI state.
It seems that every test adapter will need to do pretty much the same - watch for all test files and trigger onDidChangeTest on file save. It's not a huge or complex part, but maybe it is worth considering having that in the VSCode testing.
I agree with @connor4312 that this should be left to the TestProvider because very often there are some other files to be watched for changes (like configuration files for the testing framework) as well as the extension's VSCode settings and only the TestProvider can know the full set of things to be watched for changes.
About createDocumentTestHierarchy
: I think it depends on the testing framework whether that makes sense. For many (probably most) testing frameworks only createWorkspaceTestHierarchy
makes sense, but I can imagine that there are some frameworks where createDocumentTestHierarchy
can be very useful.
Maybe I missed it but is there a way to enable the current version of the UI and API to try integrating with it?
It's enabled in current insider builds, to interact with the API you just need to enable proposed APIs. My experimental converter extension may be a good example.
@connor4312 Thank you for your reply
but the moment we want to do something like show the percentage of passing or failed tests, we would need to enumerate all tests. So I'm not convinced that's something to design around.
For some test framework like JUnit 5 in Java, it's using annotation to mark a method is a test method, and it supports meta-annotation. In such case, in order to accurately find out all the tests, we need to parse each Java files to AST and do some comparison with the annotation binding for each method. This is a time-consuming operation if the project has a large code base (parsing a large number of java files to AST). Thus:
This is what I'm concerning right now.
Thanks for the feedback. Going more into scenarios, though, I do not think it's practical to have levels be async. For example:
getTestAtPosition(p: Position
method that would let us expand the tree to that node.provideTestsMatching(query: string)
method...and other scenarios we decide to add in the future may require additionally methods, which becomes increasingly burdensome to implement. Particularly, implementing provideTestsMatching
would almost certainly require workspace tests to be discovered anyway, and if other methods are also called the extension would need to handle deduplicating the tests and their watchers.
Update on implementation. I continue to iterate on the selfhost test provider which I and several team members are using for VS Code.
The test explorer is enhanced with a text filter. It can also display tests using a list or a tree, grouping by status or displaying a single tree by filename
Building the tree in vscode where we have many thousands of tests is somewhat laggy right now. It (and other vscode tree views) will be faster once https://github.com/microsoft/vscode/pull/114237 is merged.
Clicking on a failed test will show a peek view if there's a message with expectedOutput and actualOutput present:
On the right hand side, I'll add a tree view similar to the references peek which will show all failed tests.
I'm still mulling over how to reconcile test failures and diagnostics (those error squiggles in your editor). Diagnostics work well and have keybindings people are used to, but they aren't dismissable and I've received feedback from people wanting to separate their code errors from their unit test errors. Maybe this will be done just by adding new capabilities to diagnostics. (cc @sandy081)
Finally, test can currently only be run by the test runner who contributed them. I received some feedback that there may be a desire for test runners to be contributed independently of test providers. Feedback: please let me know if you have thoughts on this or, better, scenarios where you think this would be useful.
I have updated my converter extension to return undefined
from createWorkspaceTestHierarchy()
instead of a dummy TestProvider - thanks for the API update!
Then I realized that we still need the ability to have an empty TestHierarchy. Imagine the following scenario:
TestProvider.createWorkspaceTestHierarchy()
. Since there are no tests in the workspace folder, the TestProvider returns undefined
.Now there is no way for the TestProvider to send these tests to VS Code. Of course the TestProvider could have returned a TestHierarchy in step 2, but then you'd always see an empty TestHierarchy from that TestProvider (if you have several TestProvider extensions installed, this can quickly become annoying).
Thank you so much for the prototype and the discussion above! Such feature is sorely needed, and I think this is the right approach to tackle it at an API level. I work on test runners for codebases with millions of files, and there are a few concerns due to size that I wanted to highlight.
For repos where users often open a workspace containing tens of thousands of nested subfolders, it is often impossible to enumerate all tests at once. Enabling lazy loading will allow test adapters to partially scan only parts of the repo that are already active in the editor, while analyzing additional folders only when needed/expanded.
For a repository with thousands of projects, an adapter will take forever (additionally wasting valuable CPU cycles) computing results for projects that will never be used, but it can provide a list of top-level projects (test parent nodes) in a couple of seconds by simply doing a shallow disk scan. This goes hand in hand with the VS Code philosophy of always optimizing for the UI responsiveness.
This does not mean dropping support for eager loading. This will definitely save many round trips for cases where the results are already available at the adapter side.
Will the API ever support customizations to the test run, like specifying how many tests to run in parallel? In a workspace with multiple test adapters active (node, python, etc...), how will a โRun Allโ/โRerun Allโ action work? would it invoke all of them at once? how would they manage to play nice and not attempt monopoly on precious CPU resources?
I wonder if we already thought about the hierarchy in case of deeply nested tests. Do we want to support custom icons (or perhaps another visual indicator) for parent-level nodes? Visually interpreting the tree is much easier when you can visually see a clear separation of (projects > modules > files > unit tests).
Without this, deeply nested trees would look like an infinite sea of green checkmarks and red crosses, and quickly looking up what exactly failed (and identifying a common root cause) in a tree with various collapsing/expanding children becomes much harder.
Can we provide custom actions on tree items (failures) and their inline failure viewer on the code editor itself? Is there a better supported way? this would be needed in cases like:
When tests run and report results, they return metadata (line number, specific assertion strings) that are extracted from the document when the test started. If the user edits the file before test is done (a common practice to run existing tests while authoring new ones), the metadata would be invalid, or can be displayed in inaccurate positions. So the editor would need to remember which version of the document the test was running at, or at least indicate that the results are outdated.
The search UI has a great feature to click a little button (X) to hide a search result. I suggest thinking about supporting this here as well. Users can quickly exclude an irrelevant/slow/flaky test from their local set as they are editing code. Like search, that preference will be reset when the tree updates.
[P0] Importance of lazy loading tests
@OmarTawfik Echo with this! ๐
Thanks for the feedback, Omar. I'll continue to think on lazy loading trees and review your suggestions more thoroughly next week.
Running multiple adapters together
At the moment it will run all of them in parallel. A user/workspace setting would be a nice way to configure this.
Hierarchy of the test tree
This is a good idea. We could make a reasonably easy to use by allowing an extra icon
property on test items, which if present would be shown alongside the status (with the status icon sort of like how I show a breakpoint icon here). Then you don't have to worry about providing 6x icons for all the states.
Code actions on failure
Definitely a good extension point. Currently in the proposed API, the notion of "test results" are not exposed, but I just added them in core this week so that is mostly wiring.
Related: dealing with results is pushing me towards having (probably optional) test IDs. I wanted to avoid these and just use object identity, and this works for most cases, but consider that you have a serialized result from a test run and no active test observers. If you go to a document and we ask for tests in that file, today we have no way to know how to correlate the test results with reported tests in the file.
Default behavior could be forming the ID from the 'path' of labels to each item. This should be a good default for most runners, and seems to be what a lot of you do today with the test explorer ui.
Hiding unwanted tests
Good idea. Would also require test IDs in order to be persistent ๐
Friday eye candy: moving to decorations for test results. I will probably have glyph margin icons on instead of code lenses by default, with an option to toggle each on/off. I will also have an extra property on test messages to allow a separate message to be shown in the inline decoration, instead of its full (possibly detailed, multiline) message.
Note that the icons will be slightly prettier with https://github.com/microsoft/vscode-codicons/issues/34
I have a question related to TestHierarchy.onDidChangeTest:
/**
* An event that fires when an existing test under the `root` changes.
* This can be a result of a state change in a test run, a property update,
* or an update to its children. Changes made to tests will not be visible
* to {@link TestObserver} instances until this event is fired.
*
* This will signal a change recursively to all children of the given node.
* For example, firing the event with the {@link testRoot} will refresh
* all tests.
*/
readonly onDidChangeTest: Event<T>;
As far as I understand I should fire this every time when I change some value/state-value on the test, right? So for example let's say I have the following hierarchy:
- root
- s1
- s1t1
- s1t2
- s2
- s2t1
- s2t2
User asks to run the s1t1
and s1t2
.
Now I have to do the following:
1) onDidChangeTest to modify s1t1
to running
2) onDidChangeTest to modify s1t2
to running
3) onDidChangeTest to modify s1
to running
4) onDidChangeTest to modify s1t1
to passed
5) onDidChangeTest to modify s1t2
to passed
6) onDidChangeTest to modify s1
to passed
While 1 and 2 seems redundant due to the recursive update of 3, I have to be aware of the structure to make that decision which is pricy.
After 5 I have to update the s1
again thats why I call 6 but 6 updates s1t1
and s1t2
again which seems redundant.
From running perspective the update mechanism sounds more useful if we do update ancestors instead of children.
If I'm not mistaken the problem here is the children
. The observer cannot know that a child is removed or added or modified. And because of this you need this contract:
This will signal a change recursively to all children of the given node.
Because the UI or other subscriber has to be notified and updated. This can cause a lot of extra unnecessary computation.
Unless we send some extra information with attached to the event. Let's consider the following API:
interface TestItemChange<T extends TestItem> {
item: T;
parentRecursively?: boolean;
childrenRecursively?: boolean;
}
...
onDidChangeTest: Event<TestItemChange<T>>
Pros:
onDidChangeTest
on ancestors (vscode will).Cons:
For finer control a more detailed version can be considered:
interface TestItemChange<T extends TestItem> {
item: T;
parentRecursively?: boolean;
childrenAddedOrRemoved?: boolean; // This does not imply that one of the existing child has changed.
childrenModified?: boolean; // This implies that at least 1 existing child has (may) changed.
}
It's not just interesting from extension developer side but maybe from the vscode side too. I'm not familiar with the vscode's code but if we can save thousands of node updates with this that could make a difference.
/**
* An event that fires when an existing test in the collection changes, or
* null if a top-level test was added or removed. When fired, the consumer
* should check the test item and all its children for changes.
*/
readonly onDidChangeTest: Event<TestChangeEvent>;
TestChangeEvent.added
and TestChangeEvent.removed
can be easily computed so the "more detailed version" might be useless. Though it worth to mention that this logic don't care about the order of the elements of the children, right?
The updated
is a bit unclear for me. How will vscode tell that which test's were updated? If I send and event about s1
will all it's descendant part of it or vscode will do a compare and only the actually changed ones will be reported? Sound also computation-heavy. (I guess no.)
Okay, I'm getting confused. Is this TestChangeEvent
will be send on the aggregation of multiple TestHierarchy.onDidChangeTest
? If not, I don't see the current point of the TestChangeEvent.updated
being an array. In that case I don't see the point of TestObserver.onDidChangeTest
: When fired, the consumer should check the test item and all its children for changes.
.
One thing for sure: Without TestProvider developer providing the childrenRecursively?: boolean;
information a lot of extra computation/event will happen.
I can imagine a TestState.startedAt?: Date
property. (It might can be calculated at construction time).
With that the UI can show that for how long the test is in that state. It's mostly interesting in case running tests I assume.
[P0] Importance of lazy loading tests
+1
Just an idea:
children?: Thenable<TestItem[]>
I would have
private _children: undefined | TestItem[];
public get children() {
if(!this._children) this.initChildren();
return this._children!;
}
private initChildren(): Promise<TestItem[]> { this._children = ... }
Okay this is not good enough, if it's a leaf we wanna know without triggering the init. :S
hasChildren?:boolean
.?
Also TestRunState.Loading = 7
.
About streaming output:
I can see that with multiple onDidChangeTest(sametest)
running events (TestRunState.Running
) with more and more TestMessage
it is possible to imitate the streaming experience. But.. updating the state: TestState;
of the TestItem will be costly because it is not an interface but a class. What is the reason TestState
being a class instead of an interface? (Okay I have an idea, like easily checking if it changed or not.)
Second thought: UI has to copy the content anyway but in that case at least the logic is there and not in my code so I can do a push
on TestState.messages
. (after readonly-ness has been removed)
TestMessage.message
can be a multiline string right? I'm not convinced that this will serve the purpose well.
For example a typical GoogleTest output:
[ RUN ] TestCas3.test2
some logline
other log line
cpp/gtest/gtest2.cpp:29: Failure
Value of: 1 == 2
Actual: false
Expected: true
more loglines
[ FAILED ] TestCas3.test2 (4007 ms)
It might happen that I wanna send 3 events: first:
new TestState(TestRunState.Running, [
{message: "some logline\nother log line"}
])
then:
new TestState(TestRunState.Running, [
{message: "some logline\nother log line"},
{
message: "Value of: 1 == 2\nActual: false\nExpected: true",
severity: TestMessageSeverity.Error,
expectedOutput: "true",
actualOutput: "false",
location: ...
}
])
and at last when it is finished:
new TestState(TestRunState.Failed, [
{message: "some logline\nother log line"},
{
message: "Value of: 1 == 2\nActual: false\nExpected: true",
severity: TestMessageSeverity.Error,
expectedOutput: "true",
actualOutput: "false",
location: ...
},
{message: "more loglines"},
])
Maybe this 3 event will fire in only 3 seconds. Note: Back to my previous comment: if there is a child the current API could imply that is might updated too which it probably wouldn't be the case. Performance again..
I've realized that I can edit the array:
const testMessage = [];
const ts = new TestState(TestRunState.Failed,testMessage);
testMessage.push({message: "some logline\nother log line"});
works just fine. I still don't get the point that class not being an interface and having readonly. I thought maybe from the side of the vscode developer to prevent the modification of the user's structure but for that an interface is just fine.
Now I have to do the following:
I do not recommend changing the state of s1
. I believe I mentioned this somewhere up in the thread, but I realized that the state of these suites is aesthetic in the explorer and can be computed based off their child state instead of having every extension implement 'inheritance'. So this is done automatically. You only need to update s1t1/2 when their corresponding states change.
This will signal a change recursively to all children of the given node.
I worded this badly in the documents. This only checks for added or removed children -- which we do quickly and efficiently by keeping a Set
of the last seen children. We don't check existing children for updates or recurse further down (since adding and removing nodes will result in entirely new or entirely removed subtrees)
Child modifications require that those tests be individually fired in the event emitter.
What is the reason TestState being a class instead of an interface? (Okay I have an idea, like easily checking if it changed or not.)
Indeed :)
We could make that an interface, but then diffing logic becomes more complex and costly. Right now we can keep a shallow-clone of the last-seen object to compare against when a change comes.
TestMessage.message can be a multiline string right?
Yes. I'm not sure what what you've shown the purpose that it doesn't fulfill.
I've realized that I can edit the array... works just fine
This is surprising, I might have some diffing logic that's picking up changes incorrectly. It's not intentional that this works.
the state of these suites is aesthetic in the explorer and can be computed based off their child state
Ah, okay, nice.
Since then I had the chance to do some migration and my extension has a prototype version working with the proposed-api. I've noticed a couple of things.
State inheritance (the first sentence of this comment) is not perfect: If there is a child with TestRunState.Skipped
then the parent will inherit that. The only scenario when I would like a parent to be in that state when all its children are skipped.
I've realized that I can edit the array... works just fine
This is surprising, I might have some diffing logic that's picking up changes incorrectly. It's not intentional that this works.
I take that back.
Hello!
I developed the Unity Framework for C Test Explorer and just wanted to give my opinion here.
When I say diagnostic I mean the "Diagnostic" in the VS Code API that appears in the problems view (and can also have an error squiggle in the editor)
The downside there is that if they have a syntax error, having the problem be duplicated by both the language server and test extension will be some ugly clutter.
How should errors loading tests be handled? Emit diagnostics or have some test-specific code?
I'm still mulling over how to reconcile test failures and diagnostics (those error squiggles in your editor). Diagnostics work well and have keybindings people are used to, but they aren't dismissable and I've received feedback from people wanting to separate their code errors from their unit test errors. Maybe this will be done just by adding new capabilities to diagnostics. (cc @sandy081)
I personally use the Problems view a lot while coding and it would be very nice to have the test results show up there.
I would like to propose that TestMessageSeverity
(and I also guess DiagnosticSeverity) should be extended with a value of type Test to have a distinct view for test failures due to actual assertions. Tests failing to build/run, syntax errors etc. would still show up as Errors, Warnings etc.
It could look something like this, with squiggles of a different type (and color) in the Editor to differentiate from other Diagnostic types:
This way test extensions would provide failed results in a distinct manner to the Problems view and the Editor. It would be easier to distinguish failed tests from actual compiler warnings, syntax errors etc since (to me at least) these have a different severity.
Also it could allow for functionality like if test_foo_1()
(test code) fails, extensions could have it trace back to the function foo()
(code under test) and point with an URI to that as well. Something similar to the errors in the screenshot which mention 2 URIs:
> test_foo_1() failed. Expected and actual... -> testSource.c [x, y]
- URI of test_foo_1()
Function under test foo() -> productionSource.c [x, y]
- URI of foo()
This would be optional for extensions which want to implement this, since it probably is expensive depending on language or test framework.
Also with regards to Diagnostics clutter (duplicated by both language server and test extension), I think that the extension could simply choose not to report syntax/compilation errors at all, since the language server already does that. Or it could parse existing Diagnostics via vscode.languages.getDiagnostics()
and not duplicate those which are already present from the language server, but report other errors. But I think this choice should rather belong to the test extension, cause it also depends on language or test framework. For instance in one of my projects there are different compiler versions used for unit tests and production code, so some errors will be identical, some will only be specific to the unit tests.
Anyways thank you so much for this initiative! Looking forward to seeing where it goes.
About retired states.
This basically a property which is perpendicular to the TestRunState
.
If the binary which contains the tests has changed then all the test results are retired/old/expired/invalid.
Me personally really liked this feature of the "Test Explorer" by @hbenl.
It's almost like setting the TestRunState to Unset
back but still indicating what was the previous result.
Please consider the support of it ๐
Just a summary of my wish/suggestion-list. (Hit the corresponding emojis if you have the same ^^)
TestRunState.Loading
๐Thanks! This thread is getting long and winding so I will open discussions on separate issues (linked).
API updates that people may be interested in in https://github.com/microsoft/vscode/issues/115101
@connor4312 can you please tell me if this has a timeline? when do we think this can be available in Stable?
One scenario that isn't covered is running a subset of tests as discovery is happening (e.g. if I wanted to run all tests with the word "foo" in them) without starting a new test run. This would take some finessing and will not be supported by all adapters, for instance Go could never support it without changes to its toolchain, so is probably not worth designing for...
@connor4312 Can you clarify your statement? I'm not sure how you would run a subset of tests without starting a new test run in any situation.
I am the developer of the Go Test Explorer. My extension discovers tests by executing the symbol provider for all *_test.go
files and marking func Test*
declarations as tests. It is not challenging to filter out tests containing "foo". On a lower level, tests containing "foo" can be executed by running go test -run 'foo'
.
I'm interested in community input on this issue! https://github.com/microsoft/vscode/issues/115102#issuecomment-778490513
can you please tell me if this has a timeline? when do we think this can be available in Stable?
We generally do not have set timelines for VS Code features, but I would estimate sometime this spring.
Can you clarify your statement? I'm not sure how you would run a subset of tests without starting a new test run in any situation
Say for example the user asked to run all tests with "foo" in the name. In the UI, this would be done by filtering tests in the explorer view and then hitting "run" for them. This causes VS Code to call runTests
with every foo-containing test we know about at that instant -- if discovery is still ongoing, those will not be included in that initial call. I was thinking whether there would be some streaming way that VS Code could, as tests were being discovered, say "also run this test in your current run". Go, at least in the interface that the go run
CLI provides, would not support that append mechanism.
This is lots of extra machinery and API for an edge case that would be painful or impossible for many test runners to support, so I don't want to do it.
I suggest either: A) partially or completely disable running tests while discovery is happening; B) retain the previous test set until discovery is complete and if the user triggers a filtered test run before discovery is complete, evaluate the filter against that previous test set; or C) allow the user to configure this behavior.
Down the line, an optional capability could be added, allowing test adapters to handle the filtering themselves. For Go, I could run go test -json -run '<filter>' ./...
, parse the result, and notify observers of any tests discovered in the process.
State of the World
Testing support in VS Code has been a feature request for a long time. The VS Code community has build excellent extensions around testing, for example:
Each implementation of testing presents a different set of features, UI, and idiomaticity. Because there is no sanctioned approach to tests in VS Code, extension developers tend to make bespoke implementations, as we've seen in the Python and Java language extensions. Ideally, like in debugging, a VS Code user would have just about the same experience as they work between projects and languages.
VS Code's Approach
The Test Explorer UI presents the best point of inspiration for us, as there are many existing extensions built on its API: it's capable and proven. Regardless of the direction we take in VS Code, we should have a way for its Test Adapters to be upgraded to the new world.
Wallaby is an excellent extension, but it's tailored and purpose-built to JavaScript, and includes functionality which is not readily portable to other languages. While it is a good source for inspiration, we're not aiming to encompass Wallaby's feature set in the extension points we provide, at least not yet.
We're prototyping an API in the extension host, but there are a number of approaches we can take:
+ Easier to manage state
+ Clear way to build 'official' test extensions
+ Could be theoretically shared with VS and other editors
+ Unclear whether there's significant functionality we'd want that's not already possible in exthost api
- Less friendly, additional complexity than TS APIs for extension authors
- Less clear there's an official pathway for test extensions
API Design
The following is a working draft of an API design. It should not be considered final, or anything close to final. This post will be edited as it evolves.
Changes versus the Test Adapter API
As mentioned, the test adapter API and this one provide a similar end user experience. Here are the notable changes we made:
The test adapter API does not distinguish between watching a workspace and watching a file. In some cases, there is an existing process that reads workspace tests (such as a language server in Java) or it's not much more expensive to get workspace tests than file tests (such as mocha, perhaps). However, some cases, like Go, providing tests for a single file can be done very cheaply and efficiently without needing to involve the workspace.
In this API we expect the
TestProvider
to, after activation, always provide tests for the visible text editors, and we only request tests for the entire workspace when required (i.e. when the UI needs to enumerate them).We have modeled the test state more closely after the existing
DiagnosticCollection
, where the Test Adapter API uses only events to enumerate tests and does not have a central collection.The Test Adapter API makes the distinction between suites and tests, we do not. They have almost identical capabilities, and in at least one scenario the 'suites' are more like tests and the leaf 'tests' cannot be run individually.
We use object identity rather than ID for referencing tests. This is in line with other items in the VS Code API, including Diagnostics.
Ideas and Open Questions
See the
testing
label for current work, questions, and problems.API
See the current working proposal in https://github.com/microsoft/vscode/blob/master/src/vs/vscode.proposed.d.ts (ctrl+f for 107467)