`vscode-test` does not include coverage of lines executed via `vscode.commands.executeCommand`

[leep-frog]

I have two kinds of tests in my extension's test suite:

1. Tests that directly call the code in my extensions (e.g. assert.Equals(helperFunc(...), expectedOutput))
2. Tests that call vscode.commands.executeCommand(<command defined in my extension>)

Both tests are passing, but coverage reports (generated via c8) only pick up the lines executed by the first type of test.

Steps to reproduce

Setup the project

1. Create a new project with yo code
2. Install c8 for coverage npm install --save-dev c8
3. Add the follwoing coverage script in package.json

     "test": "vscode-test",
     "coverage": "c8  --reporter lcov npm run test"

4. Create tests that directly call logic defined in the extension
5. Create tests that call vscode.commands.executeCommand(...) in the extension
6. Verify coverage report picks up code run for the former, but not for the latter

Expected behavior

Expected lines executed via vscode.commands.executeCommand(...) to be picked up by c8 coverage report

Actual behavior

Coverage is not detected when lines are executed via vscode.commands.executeCommand(...), but are detected when run directly by the test (e.g. directly calling helper functions).

[connor4312]

When you run c8 on the command line, you're getting coverage for the node process that's the CLI, not the extension host in VS Code.

Please just use the --coverage option in the test CLI, and let us know if there's anything you need to do with that that you can't.

[leep-frog]

Can you please provide documentation explaining the coverage option? When running vscode-test --help, there is no mention of it? And I can't find any info in this repo's documentation either.

[connor4312]

Please update to the latest version of the CLI for that https://code.visualstudio.com/updates/v1_87#_test-coverage-in-extensions

[leep-frog]

Awesome, thanks for the info!

If anyone else is curious about coverage reports, the script configuration is as follows:

"test": "vscode-test --coverage --coverageReporter lcov",