[Testing] Coverage API Issues: Monorepo & Accumulation

[connectdotz]

Type: Bug

As we integrate our extension, vscode-jest, with the newly available Testing Coverage API, we've identified some issues that notably impact usability. Below, I provide some video demonstrations from our development environment, followed by descriptions of each issue.

Issue 1: Incomplete Coverage Trigger in Monorepo Projects

In monorepo projects with multiple testControllers, initiating a top-level coverage run does not trigger all registered controllers. The coverage run is activated only for the first controller.

https://github.com/microsoft/vscode/assets/891093/1438add2-8a33-4f18-ad99-4b0a919d4a60

video description: In our setup with 'jest snapshot' and 'jest ts' controllers, initiating coverage from the top-level menu only triggered the "jest snapshot" controller's runHandler.

Issue 2: Non-Accumulative Coverage Display

The test coverage display currently does not accumulate data across test runs. This is a concern because generating coverage information usually requires significant resources, and users could execute partial coverage runs. Without cumulative coverage, historical data is not preserved, which can reduce the feature's usability. Ideally, test coverage could mirror the behavior of test run status, which consistently updates and maintains the latest information, regardless of the test run sequence.

https://github.com/microsoft/vscode/assets/891093/d6b0101d-fbb1-4020-9640-1a9eb227e435

video description:

1. Run coverage for "jest snapshot" => coverage info correctly displayed as expected.
2. Run coverage for "jest ts" => The "jest snapshot" coverage was replaced by the "jest ts" coverage in the panel. desired: both coverage info should be visible in the panel.
3. Run a regular test in "jest snapshot" => The coverage panel disappeared. desired: The coverage panel should still be visible.
4. Run "Test: Open Coverage" command didn't bring back the latest coverage panel either. desired: The command should bring back the latest coverage panel

VS Code version: Code 1.88.1 (e170252f762678dec6ca2cc69aba1570769a5d39, 2024-04-10T17:42:52.765Z) OS version: Darwin x64 23.4.0 Modes:

System Info

|Item|Value| |---|---| |CPUs|Intel(R) Core(TM) i7-8700B CPU @ 3.20GHz (12 x 3200)| |GPU Status|2d_canvas: enabled
canvas_oop_rasterization: disabled_off
direct_rendering_display_compositor: disabled_off_ok
gpu_compositing: enabled
multiple_raster_threads: enabled_on
opengl: enabled_on
rasterization: enabled
raw_draw: disabled_off_ok
skia_graphite: disabled_off
video_decode: enabled
video_encode: enabled
webgl: enabled
webgl2: enabled
webgpu: enabled| |Load (avg)|2, 2, 2| |Memory (System)|32.00GB (1.90GB free)| |Process Argv|| |Screen Reader|no| |VM|0%|

Extensions (48)

Extension|Author (truncated)|Version ---|---|--- better-comments|aar|3.0.2 vscode-color-pick|ada|0.6.2 emojisense|bie|0.10.0 vscode-markdownlint|Dav|0.54.0 vscode-eslint|dba|2.4.4 xml|Dot|2.5.1 gitlens|eam|14.9.0 prettier-vscode|esb|10.4.0 vscode-commands|fab|2.0.2 flow-for-vscode|flo|2.2.1 shell-format|fox|7.2.5 html-preview-vscode|geo|0.2.5 copilot|Git|1.180.821 copilot-chat|Git|0.14.1 vscode-pull-request-github|Git|0.86.1 go|gol|0.41.2 svg|joc|1.5.3 vscode-insertdatestring|jsy|2.3.1 vscode-colorize|kam|0.11.1 json-lines-viewer|leh|0.0.4 git-graph|mhu|1.30.0 vscode-docker|ms-|1.29.0 debugpy|ms-|2024.4.0 isort|ms-|2023.10.1 python|ms-|2024.4.1 vscode-pylance|ms-|2024.4.1 jupyter|ms-|2024.3.1 jupyter-keymap|ms-|1.1.2 jupyter-renderers|ms-|1.0.17 vscode-jupyter-cell-tags|ms-|0.1.9 vscode-jupyter-slideshow|ms-|0.1.6 remote-containers|ms-|0.354.0 wordcount|ms-|0.1.0 vscode-jest|Ort|6.3.0 material-icon-theme|PKi|4.34.0 java|red|1.29.0 markdown-preview-enhanced|shd|0.8.13 code-spell-checker|str|3.0.1 pdf|tom|1.2.2 intellicode-api-usage-examples|Vis|0.2.8 vscodeintellicode|Vis|1.3.1 vscode-java-debug|vsc|0.57.0 vscode-java-dependency|vsc|0.23.6 vscode-java-pack|vsc|0.26.0 vscode-java-test|vsc|0.41.0 vscode-maven|vsc|0.44.0 vim|vsc|1.27.2 markdown-all-in-one|yzh|3.6.2 (1 theme extensions excluded)

[connor4312]

Non-Accumulative Coverage Display

I'm not sure this is safe to do. For test items we can do that since the state of a test item is inherently tied to its existence in the tree. That's not true for coverage: I could have a test run that generates coverage for a certain file, and if I move/rename/delete that file, the old coverage for the file would be "stuck" unless I cleared all previous coverage data. This could also confusingly influence coverage statistics if a user runs their entire test suite with coverage and their coverage numbers are influenced by data for files that no longer exist.

[connectdotz]

@connor4312, thank you for your insights on this matter. Here are a few thoughts in response:

I could have a test run that generates coverage for a certain file, and if I move/rename/delete that file, the old coverage for the file would be "stuck" unless I cleared all previous coverage data.

Considering that each FileCoverage corresponds to a source file, it seems feasible for the provider—or perhaps even the Test Explorer itself—to manage these through life-cycle events. For instance, coverage could be automatically removed or reset when files are moved, renamed, or deleted, assuming there's an API to communicate such changes back to VSCode.

This could also confusingly influence coverage statistics if a user runs their entire test suite with coverage and their coverage numbers are influenced by data for files that no longer exist.

Allowing providers to opt-in to maintaining FileCoverage integrity as described above could mitigate this concern. Additionally, it might be beneficial to implement a "reset" function or possibly use a collection like TestItemCollection to manage the coverage tree comprehensively.

At a minimum, I believe the coverage panel should maintain separate coverage information for each controller. Initiating a coverage run in one controller should not result in the loss of coverage data from another controller.

Thank you once again for considering these suggestions. I look forward to your thoughts on these potential improvements.