Test Coverage API

[connor4312]

📣 2024-03 update: test coverage support has been finalized. You can find the finished, usable version of the API in vscode.d.ts and a guide in our docs.

Original discussion has been preserved below.

---

Notes researching of existing formats:

## Clover - Contains timestamp and total lines of code, files, classes, coverage numbers (see below) - Also _can_ include complexity information - Lines are covered as XML simply `` - Conditionals are represented as ``. Either truecount/falsecount being 0 indicates incomplete branch coverage. > Clover uses these measurements to produce a Total Coverage Percentage for each class, file, package and for the project as a whole. The Total Coverage Percentage allows entities to be ranked in reports. The Total Coverage Percentage (TPC) is calculated as follows: > ``` > TPC = (BT + BF + SC + MC)/(2*B + S + M) * 100% > > where > > BT - branches that evaluated to "true" at least once > BF - branches that evaluated to "false" at least once > SC - statements covered > MC - methods entered > > B - total number of branches > S - total number of statements > M - total number of methods > ``` ## gcov Format info on the bottom of: http://ltp.sourceforge.net/coverage/lcov/geninfo.1.php - Organized per test, which are associated with one or more files. Indicates functions and how many times each function was called. - Lines have number of times they were hit, `DA:,[,]` - Has similar branch coverage, but has indexed branches instead of true/false. A branch with 0 taken is uncovered. - `BRDA:,,,`, declared multiple times for line, one for each branch - No timestamp ## cobertura - General coverage information/count - Number of times each line was hit. - Has conditions in a slightly less-strict way, `` - Normal lines look like `` - Has timestamp and metadata - Has method call count (everything is organized in classes)

This results in the following API. The TestCoverageProvider is given as an optional provider on the managed TestRun object. This is similar to the SourceControl interface in vscode.d.ts. The provider is then pretty standard; it'll only be examined when the run finishes, so all its data is static.

It has a method to get general coverage information for all files involved in the run, and a method to provide detailed coverage for a URI.

I refer to the atomic unit of coverage as "statement coverage". The examined test formats only provide line-based coverage, but statement coverage is what these are actually trying convey approximate and most tooling (e.g. istanbul/nyc) is technically capable of per-statement rather than per-line coverage. Therefore, it's called statement coverage, but it can be used as line coverage.

The API is physically large due to the nature of data being provided, but is conceptually pretty simple (at least compared to the Test Controller API!). I don't like the method names on the TestCoverageProvider, but have not thought of better ones yet.

declare module 'vscode' {

    // https://github.com/microsoft/vscode/issues/123713

    export interface TestRun {
        /**
         * Test coverage provider for this result. An extension can defer setting
         * this until after a run is complete and coverage is available.
         */
        coverageProvider?: TestCoverageProvider;
        // ...
    }

    /**
     * Provides information about test coverage for a test result.
     * Methods on the provider will not be called until the test run is complete
     */
    export interface TestCoverageProvider<T extends FileCoverage = FileCoverage> {
        /**
         * Returns coverage information for all files involved in the test run.
         * @param token A cancellation token.
         * @return Coverage metadata for all files involved in the test.
         */
        // @API - pass something into the provide method:
        // (1) have TestController#coverageProvider: TestCoverageProvider
        // (2) pass TestRun into this method
        provideFileCoverage(token: CancellationToken): ProviderResult<T[]>;

        /**
         * Give a FileCoverage to fill in more data, namely {@link FileCoverage.detailedCoverage}.
         * The editor will only resolve a FileCoverage once, and only if detailedCoverage
         * is undefined.
         *
         * @param coverage A coverage object obtained from {@link provideFileCoverage}
         * @param token A cancellation token.
         * @return The resolved file coverage, or a thenable that resolves to one. It
         * is OK to return the given `coverage`. When no result is returned, the
         * given `coverage` will be used.
         */
        resolveFileCoverage?(coverage: T, token: CancellationToken): ProviderResult<T>;
    }

    /**
     * A class that contains information about a covered resource. A count can
     * be give for lines, branches, and functions in a file.
     */
    export class CoveredCount {
        /**
         * Number of items covered in the file.
         */
        covered: number;
        /**
         * Total number of covered items in the file.
         */
        total: number;

        /**
         * @param covered Value for {@link CovereredCount.covered}
         * @param total Value for {@link CovereredCount.total}
         */
        constructor(covered: number, total: number);
    }

    /**
     * Contains coverage metadata for a file.
     */
    export class FileCoverage {
        /**
         * File URI.
         */
        readonly uri: Uri;

        /**
         * Statement coverage information. If the reporter does not provide statement
         * coverage information, this can instead be used to represent line coverage.
         */
        statementCoverage: CoveredCount;

        /**
         * Branch coverage information.
         */
        branchCoverage?: CoveredCount;

        /**
         * Function coverage information.
         */
        functionCoverage?: CoveredCount;

        /**
         * Detailed, per-statement coverage. If this is undefined, the editor will
         * call {@link TestCoverageProvider.resolveFileCoverage} when necessary.
         */
        detailedCoverage?: DetailedCoverage[];

        /**
         * Creates a {@link FileCoverage} instance with counts filled in from
         * the coverage details.
         * @param uri Covered file URI
         * @param detailed Detailed coverage information
         */
        static fromDetails(uri: Uri, details: readonly DetailedCoverage[]): FileCoverage;

        /**
         * @param uri Covered file URI
         * @param statementCoverage Statement coverage information. If the reporter
         * does not provide statement coverage information, this can instead be
         * used to represent line coverage.
         * @param branchCoverage Branch coverage information
         * @param functionCoverage Function coverage information
         */
        constructor(
            uri: Uri,
            statementCoverage: CoveredCount,
            branchCoverage?: CoveredCount,
            functionCoverage?: CoveredCount,
        );
    }

    // @API are StatementCoverage and BranchCoverage etc really needed
    // or is a generic type with a kind-property enough

    /**
     * Contains coverage information for a single statement or line.
     */
    export class StatementCoverage {
        /**
         * The number of times this statement was executed, or a boolean indicating
         * whether it was executed if the exact count is unknown. If zero or false,
         * the statement will be marked as un-covered.
         */
        executed: number | boolean;

        /**
         * Statement location.
         */
        location: Position | Range;

        /**
         * Coverage from branches of this line or statement. If it's not a
         * conditional, this will be empty.
         */
        branches: BranchCoverage[];

        /**
         * @param location The statement position.
         * @param executed The number of times this statement was executed, or a
         * boolean indicating  whether it was executed if the exact count is
         * unknown. If zero or false, the statement will be marked as un-covered.
         * @param branches Coverage from branches of this line.  If it's not a
         * conditional, this should be omitted.
         */
        constructor(executed: number | boolean, location: Position | Range, branches?: BranchCoverage[]);
    }

    /**
     * Contains coverage information for a branch of a {@link StatementCoverage}.
     */
    export class BranchCoverage {
        /**
         * The number of times this branch was executed, or a boolean indicating
         * whether it was executed if the exact count is unknown. If zero or false,
         * the branch will be marked as un-covered.
         */
        executed: number | boolean;

        /**
         * Branch location.
         */
        location?: Position | Range;

        /**
         * Label for the branch, used in the context of "the ${label} branch was
         * not taken," for example.
         */
        label?: string;

        /**
         * @param executed The number of times this branch was executed, or a
         * boolean indicating  whether it was executed if the exact count is
         * unknown. If zero or false, the branch will be marked as un-covered.
         * @param location The branch position.
         */
        constructor(executed: number | boolean, location?: Position | Range, label?: string);
    }

    /**
     * Contains coverage information for a function or method.
     */
    export class FunctionCoverage {
        /**
         * Name of the function or method.
         */
        name: string;

        /**
         * The number of times this function was executed, or a boolean indicating
         * whether it was executed if the exact count is unknown. If zero or false,
         * the function will be marked as un-covered.
         */
        executed: number | boolean;

        /**
         * Function location.
         */
        location: Position | Range;

        /**
         * @param executed The number of times this function was executed, or a
         * boolean indicating  whether it was executed if the exact count is
         * unknown. If zero or false, the function will be marked as un-covered.
         * @param location The function position.
         */
        constructor(name: string, executed: number | boolean, location: Position | Range);
    }

    export type DetailedCoverage = StatementCoverage | FunctionCoverage;

}

[connor4312]

yep, that's placeholder pending https://github.com/microsoft/vscode-codicons/issues/205, which should happen this iteration

[connor4312]

I finished my work on test coverage this iteration and now consider it feature complete -- pending the missing icon. Here's a small demo video I made for the vscode team to show it (in the context of our own unit tests)

https://github.com/microsoft/vscode/assets/2230985/5ecf8e5b-3e27-4ac4-84d8-269b3744b79e

[jdneo]

Kudos @connor4312! The feature is estimated to be released in early March as planned before, right?

[jdneo]

BTW, the gutter decoration won't be seen under high contrast themes - not sure if this is by design or a bug.

[connor4312]

The feature is estimated to be released in early March as planned before, right?

Yes

BTW, the gutter decoration won't be seen under high contrast themes - not sure if this is by design or a bug.

Good point, will fix

[jdneo]

When hovering on the gutter decoration, there will be a pop-up:

Some questions about it:

1. What does <empty statement> mean?
2. When I click Toggle Line Coverage, nothing happens, is that supposed to turn off the gutter decoration?

[connor4312]

This is because the sample test provider at the moment only reports Positions and not Ranges for its coverage data.

Today in #202676 I made a change that coverage data that only reports a Position will 'cover' either the entire line or until the next reported un/covered position, whichever comes first. So in tomorrow Insiders' you will see better behavior.

[jdneo]

So in tomorrow Insiders' you will see better behavior.

Thank you!

[bigbug]

At the moment it seems that there is currently no mechanisms for reporting and showing "skipped" lines (e.g. code which can for whatever reason not be reached by the testing system). Is it possible to also integrate this into the API in such a way, that these lines could be shown with a different color?

[connor4312]

The way to do that at the moment is just not having any data for a skipped range.

Do you have tools / test systems that report this data? This wasn't something I saw in reviewing common coverage formats and I'm interested to see how it's used/represented.

[bigbug]

We currently use gcovr for our C projects. The JSON output (as well as the HTML output) both support also skipped/excluded lines (see https://gcovr.com/en/stable/output/json.html).

With the Test Coverage API coming up, I can also think of creating extensions which can provide the results for static code analysis - so in terms of ESLint for example you could report per line whether there are faults, everything is fine or whether a line has been ignored. The same could also apply for ts-ignore ... Also the SARIF standard can give some more insights: Chapter 3.27.23 describes the mechanism for excluding several rules. I found the definition here. An example for a suppression can be seen here.

[connor4312]

Thanks! I don't anticipate having that for the initial API slated for finalization next month, but I've captured that in an issue for a followup addition.

[bigbug]

Sounds good. Thank you!

[aszenz]

I primarily used https://github.com/ryanluker/vscode-coverage-gutters, excited to see vscode having it built-in now

[connor4312]

One API change arising from Python coverage: executionCount: number has been changed to executed: boolean | number https://github.com/microsoft/vscode/pull/204004

[connor4312]

Further change: we renamed FunctionCoverage to DeclarationCoverage (https://github.com/microsoft/vscode/pull/204667) to better represent the fact that other types of data are represented by nominal "function" coverage. There's a back-compat shim in place for this iteration

fyi @jdneo

[ioquatix]

I'm trying to test this but getting the following error:

await this.finished Error: Extension 'socketry.sus-vscode' CANNOT use API proposal: testCoverage. Its package.json#enabledApiProposals-property declares: but NOT testCoverage. The missing proposal MUST be added and you must start in extension development mode or use the following command line switch: --enable-proposed-api socketry.sus-vscode at E (/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/api/node/extensionHostProcess.js:138:51177) at Object.set coverageProvider [as coverageProvider] (/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/api/node/extensionHostProcess.js:152:8287)

Any advice about what I should do?

[gjsjohnmurray]

@ioquatix have you followed the steps given at https://code.visualstudio.com/api/advanced-topics/using-proposed-api yet?

[ioquatix]

@gjsjohnmurray I thought so. But I'm not using the insiders release, I'm using vscode v1.86 - does that matter?

I was under the impression I would be able to test this out using the development environment:

Basically, by clicking this button:

[ioquatix]

Oh, I figured it out, I needed to add the highlighted argument:

[connor4312]

Hi, just to update folks, we bumped this API to target finalization next iteration, rather than the this one (release next week.) This is a sizable API and there were several other exciting APIs entering proposed and finalization that took up our team's bandwidth.

[JustinGrote]

@connor4312 appreciate the hard work regardless, thanks!

[vogelsgesang]

I think I ran into a bug, trying to use the coverage API to integrate coverage data loaded from a file. I am trying to follow the instructions which were previously provided up-thread

Scenario

I am creating two test runs. Those test runs have no test cases attached, they only contain a name, some output (appendOutput) and the coverage data.

Minimal repro

```typescript import * as vscode from "vscode"; class BazelCoverageProvider implements vscode.TestCoverageProvider { provideFileCoverage( token: vscode.CancellationToken, ): vscode.ProviderResult { const detailedCoverage = [ new vscode.DeclarationCoverage( "test_func", 12, new vscode.Position(1, 12), ), new vscode.DeclarationCoverage( "test_func2", 0, new vscode.Position(10, 0), ), new vscode.StatementCoverage(true, new vscode.Position(15, 3)), new vscode.StatementCoverage(false, new vscode.Position(16, 0)), ]; return [ vscode.FileCoverage.fromDetails( vscode.Uri.file("/Users/avogelsgesang/hyper/hyper-db/hyper/ir/IR.cpp"), detailedCoverage, ), ]; } // Already fully resolved. Noting to resolve resolveFileCoverage?( coverage: vscode.FileCoverage, token: vscode.CancellationToken, ): vscode.ProviderResult { return coverage; } } export async function activate(context: vscode.ExtensionContext) { // Create the test controller const testController = vscode.tests.createTestController( "bazel-coverage", "Bazel Coverage", ); context.subscriptions.push(testController); // Provide coverage info 1 const run = testController.createTestRun( new vscode.TestRunRequest(), "/my/coverage.lcov", false, ); run.appendOutput("Information about where we loaded the coverage from..."); run.coverageProvider = new BazelCoverageProvider(); run.end(); // Provide coverage info 2 const run2 = testController.createTestRun( new vscode.TestRunRequest(), "/my/second_coverage.lcov", false, ); run2.appendOutput("Some other information..."); run2.coverageProvider = new BazelCoverageProvider(); run2.end(); } ```

Observed behavior

I see two coverage entries in the test result list, and I can switch between them. Also, I see the output from the 2nd test run.

However, I do not see the test run names "/my/coverage.lcov" and "/my/second_coverage.lcov" anywhere. Also, there seems to be no way to see inspect the appendOutput output from the first run.

Expected behavior

I guess the "Close test coverage"/"View test coverage" should be grouped under a test run? Even if there are no tests and only coverage data inside this test run? Also, it would be good to be able to set the overall test run into a "success"/"failed" state, even without listing explicit sub-tests

[jdneo]

@connor4312 Will the API change https://github.com/microsoft/vscode/pull/207512 affect the coverage API stable release ETA? Do we still target on March release now?

[connor4312]

We're still targeting the March release, this API will be finalized in the next few hours.

@vogelsgesang I'll check that out, I've opened a new issue for tracking.

[ffMathy]

It would have been nice if I (like in Rider) could right-click a covered line and see all the tests that ran through that line, and then run a particular one.

This is not supported from this API, as it mostly brings total coverage across all tests.

[connor4312]

It would have been nice if I (like in Rider) could right-click a covered line and see all the tests that ran through that line, and then run a particular one.

This is not supported from this API, as it mostly brings total coverage across all tests.

Tracked in https://github.com/microsoft/vscode/issues/212196