command tests and normal tests will produce duplicate and wrong code coverage results

[rH4rtinger]

When you have tests that execute commands and that test methods, you will have duplicate and wrong code coverage results.

Setup project

• yo code for example project
• update @vscode/test-cli to a higher version (in my case 0.0.9)
• in extension.ts: add a function

  export function add(a: number, b: number) {
  return a + b;
  }

• in extension.test.ts

  
  import * as assert from "assert";
  import { add } from "../extension";

suite("Extension Test Suite", () => { test("add", () => { assert.strictEqual(3, add(1, 2)); }); });

* create a new file `command.test.ts`:
```typescript
suite("command Test Suite", () => {
  test("hello world", async () => {
    await vscode.commands.executeCommand("dummy.helloWorld"); // use the dummy command from the extension
  });
});

Add a coverage script, e.g. npm run test -- --coverage

Bug

When I am executing the coverage, then I am getting two results per file.

This is not only in the text result, but also in lcov, html, json, ...

  Extension Test Suite
    ✔ add
  command Test Suite
    ✔ hello world
  2 passing (75ms)
[main 2024-04-05T07:31:49.592Z] Extension host with pid 24176 exited with code: 0, signal: unknown.
Exit code:   0
Done

--------------|---------|----------|---------|---------|-------------------
File          | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s 
--------------|---------|----------|---------|---------|-------------------
All files     |   65.51 |      100 |      50 |   65.51 | 
 extension.ts |   41.37 |      100 |   33.33 |   41.37 | 7-22,25
 extension.ts |   89.65 |      100 |   66.66 |   89.65 | 27-29
--------------|---------|----------|---------|---------|-------------------

Expected

I would expect to only have one result per file. In my opinion, it should not matter, if I have called a command or tested a method itself.

Workaround

To correct my wrong lcov files, I did the following:

• Create two .vscode-test.mjs files, and changed the files to only have one test file in it.
• added two test configs and set the config via --config parameter
• created two coverage executions (for each test config), also changed to --coverage-output to two different spots
• manually combined with https://github.com/mweibel/lcov-result-merger

However, this is not the way I want to use, as I wanted every coverage result combined.

[connor4312]

This duplicates https://github.com/microsoft/vscode-test-cli/issues/49