Closed connor4312 closed 5 months ago
yep, that's placeholder pending https://github.com/microsoft/vscode-codicons/issues/205, which should happen this iteration
I finished my work on test coverage this iteration and now consider it feature complete -- pending the missing icon. Here's a small demo video I made for the vscode team to show it (in the context of our own unit tests)
https://github.com/microsoft/vscode/assets/2230985/5ecf8e5b-3e27-4ac4-84d8-269b3744b79e
Kudos @connor4312! The feature is estimated to be released in early March as planned before, right?
BTW, the gutter decoration won't be seen under high contrast themes - not sure if this is by design or a bug.
The feature is estimated to be released in early March as planned before, right?
Yes
BTW, the gutter decoration won't be seen under high contrast themes - not sure if this is by design or a bug.
Good point, will fix
When hovering on the gutter decoration, there will be a pop-up:
Some questions about it:
<empty statement>
mean?Toggle Line Coverage
, nothing happens, is that supposed to turn off the gutter decoration?This is because the sample test provider at the moment only reports Positions and not Ranges for its coverage data.
Today in #202676 I made a change that coverage data that only reports a Position will 'cover' either the entire line or until the next reported un/covered position, whichever comes first. So in tomorrow Insiders' you will see better behavior.
So in tomorrow Insiders' you will see better behavior.
Thank you!
At the moment it seems that there is currently no mechanisms for reporting and showing "skipped" lines (e.g. code which can for whatever reason not be reached by the testing system). Is it possible to also integrate this into the API in such a way, that these lines could be shown with a different color?
The way to do that at the moment is just not having any data for a skipped range.
Do you have tools / test systems that report this data? This wasn't something I saw in reviewing common coverage formats and I'm interested to see how it's used/represented.
We currently use gcovr for our C projects. The JSON output (as well as the HTML output) both support also skipped/excluded lines (see https://gcovr.com/en/stable/output/json.html).
With the Test Coverage API coming up, I can also think of creating extensions which can provide the results for static code analysis - so in terms of ESLint for example you could report per line whether there are faults, everything is fine or whether a line has been ignored. The same could also apply for ts-ignore ... Also the SARIF standard can give some more insights: Chapter 3.27.23 describes the mechanism for excluding several rules. I found the definition here. An example for a suppression can be seen here.
Thanks! I don't anticipate having that for the initial API slated for finalization next month, but I've captured that in an issue for a followup addition.
Sounds good. Thank you!
I primarily used https://github.com/ryanluker/vscode-coverage-gutters, excited to see vscode having it built-in now
One API change arising from Python coverage: executionCount: number
has been changed to executed: boolean | number
https://github.com/microsoft/vscode/pull/204004
Further change: we renamed FunctionCoverage
to DeclarationCoverage
(https://github.com/microsoft/vscode/pull/204667) to better represent the fact that other types of data are represented by nominal "function" coverage. There's a back-compat shim in place for this iteration
fyi @jdneo
I'm trying to test this but getting the following error:
await this.finished Error: Extension 'socketry.sus-vscode' CANNOT use API proposal: testCoverage. Its package.json#enabledApiProposals-property declares: but NOT testCoverage. The missing proposal MUST be added and you must start in extension development mode or use the following command line switch: --enable-proposed-api socketry.sus-vscode at E (/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/api/node/extensionHostProcess.js:138:51177) at Object.set coverageProvider [as coverageProvider] (/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/api/node/extensionHostProcess.js:152:8287)
Any advice about what I should do?
@ioquatix have you followed the steps given at https://code.visualstudio.com/api/advanced-topics/using-proposed-api yet?
@gjsjohnmurray I thought so. But I'm not using the insiders release, I'm using vscode v1.86 - does that matter?
I was under the impression I would be able to test this out using the development environment:
Basically, by clicking this button:
Oh, I figured it out, I needed to add the highlighted argument:
Hi, just to update folks, we bumped this API to target finalization next iteration, rather than the this one (release next week.) This is a sizable API and there were several other exciting APIs entering proposed and finalization that took up our team's bandwidth.
@connor4312 appreciate the hard work regardless, thanks!
I think I ran into a bug, trying to use the coverage API to integrate coverage data loaded from a file. I am trying to follow the instructions which were previously provided up-thread
I am creating two test runs. Those test runs have no test cases attached, they only contain a name, some output (appendOutput
) and the coverage data.
I see two coverage entries in the test result list, and I can switch between them. Also, I see the output from the 2nd test run.
However, I do not see the test run names "/my/coverage.lcov" and "/my/second_coverage.lcov" anywhere. Also, there seems to be no way to see inspect the appendOutput
output from the first run.
I guess the "Close test coverage"/"View test coverage" should be grouped under a test run? Even if there are no tests and only coverage data inside this test run? Also, it would be good to be able to set the overall test run into a "success"/"failed" state, even without listing explicit sub-tests
@connor4312 Will the API change https://github.com/microsoft/vscode/pull/207512 affect the coverage API stable release ETA? Do we still target on March release now?
We're still targeting the March release, this API will be finalized in the next few hours.
@vogelsgesang I'll check that out, I've opened a new issue for tracking.
It would have been nice if I (like in Rider) could right-click a covered line and see all the tests that ran through that line, and then run a particular one.
This is not supported from this API, as it mostly brings total coverage across all tests.
It would have been nice if I (like in Rider) could right-click a covered line and see all the tests that ran through that line, and then run a particular one.
This is not supported from this API, as it mostly brings total coverage across all tests.
Tracked in https://github.com/microsoft/vscode/issues/212196
📣 2024-03 update: test coverage support has been finalized. You can find the finished, usable version of the API in
vscode.d.ts
and a guide in our docs.Original discussion has been preserved below.
Notes researching of existing formats:
This results in the following API. The
TestCoverageProvider
is given as an optional provider on the managed TestRun object. This is similar to theSourceControl
interface in vscode.d.ts. The provider is then pretty standard; it'll only be examined when the run finishes, so all its data is static.It has a method to get general coverage information for all files involved in the run, and a method to provide detailed coverage for a URI.
I refer to the atomic unit of coverage as "statement coverage". The examined test formats only provide line-based coverage, but statement coverage is what these are actually trying convey approximate and most tooling (e.g. istanbul/nyc) is technically capable of per-statement rather than per-line coverage. Therefore, it's called statement coverage, but it can be used as line coverage.
The API is physically large due to the nature of data being provided, but is conceptually pretty simple (at least compared to the Test Controller API!). I don't like the method names on the TestCoverageProvider, but have not thought of better ones yet.