"Run tests at cursor" takes exponentially longer with more pytest tests

[farmio]

Type: Bug

When using Python pytest to parametrize tests, the test explorer takes exponentially longer, while CPU is at ~100%, the more tests are created using parametize when using "Run tests at cursor" to start the tests (starting from the "Run tests" button in test explorer works fine).

Here is a minimal example:

import pytest

@pytest.mark.parametrize("value", range(10))
def test_test(value):
    assert True

With range(10) my system (Apple M1 Pro 16 GB) takes ~1 second (measured by hand with stopwatch). With range(100) it takes ~8 seconds With range(150) it takes ~50 seconds

I'd expect it to take roughly as long as running it from test explorer or pytest <file> from command line - which is about 0.1 seconds (<1 second for range(1500)).

VS Code version: Code 1.76.2 (Universal) (ee2b180d582a7f601fa6ecfdad8d9fd269ab1884, 2023-03-14T17:54:09.061Z) OS version: Darwin arm64 22.3.0 Modes: Sandboxed: No

System Info

|Item|Value| |---|---| |CPUs|Apple M1 Pro (10 x 24)| |GPU Status|2d_canvas: enabled
canvas_oop_rasterization: disabled_off
direct_rendering_display_compositor: disabled_off_ok
gpu_compositing: enabled
metal: disabled_off
multiple_raster_threads: enabled_on
opengl: enabled_on
rasterization: enabled
raw_draw: disabled_off_ok
skia_renderer: enabled_on
video_decode: enabled
video_encode: enabled
vulkan: disabled_off
webgl: enabled
webgl2: enabled
webgpu: disabled_off| |Load (avg)|5, 43, 74| |Memory (System)|16.00GB (0.07GB free)| |Process Argv|--crash-reporter-id 3f533b57-abc7-4565-8781-b3146cb3a6b8| |Screen Reader|no| |VM|0%|

Extensions (33)

Extension|Author (truncated)|Version ---|---|--- vscode-gcode-syntax|app|0.7.7 better-toml|bun|0.3.2 vscode-markdownlint|Dav|0.49.0 vscode-eslint|dba|2.4.0 EditorConfig|Edi|0.16.4 elm-ls-vscode|elm|2.6.0 json-tools|eri|1.0.2 prettier-vscode|esb|9.10.4 copilot|Git|1.78.9758 vscode-test-explorer|hbe|2.21.1 flux|inf|1.0.4 elixir-ls|Jak|0.13.0 vscode-edit-csv|jan|0.7.4 vscode-home-assistant|kee|1.35.0 vscode-sshfs|Kel|1.26.0 vscode-docker|ms-|1.24.0 vscode-language-pack-de|MS-|1.76.2023030809 flake8|ms-|2023.4.0 isort|ms-|2022.8.0 python|ms-|2023.4.1 vscode-pylance|ms-|2023.3.40 jupyter|ms-|2023.2.1200692131 jupyter-keymap|ms-|1.1.0 jupyter-renderers|ms-|1.0.15 vscode-jupyter-cell-tags|ms-|0.1.8 vscode-jupyter-slideshow|ms-|0.1.5 remote-containers|ms-|0.288.0 test-adapter-converter|ms-|0.1.7 vscode-typescript-tslint-plugin|ms-|1.3.4 vscode-commons|red|0.0.6 vscode-yaml|red|1.12.2 rust-analyzer|rus|0.3.1451 markdown-all-in-one|yzh|3.5.1 (6 theme extensions excluded)

A/B Experiments

``` vsliv368:30146709 vsreu685:30147344 python383:30185418 vspor879:30202332 vspor708:30202333 vspor363:30204092 vswsl492cf:30256860 vslsvsres303:30308271 vserr242cf:30382550 pythontb:30283811 vsjup518:30340749 pythonptprofiler:30281270 vshan820:30294714 vstes263:30335439 vscoreces:30445986 pythondataviewer:30285071 vscod805:30301674 binariesv615:30325510 bridge0708:30335490 bridge0723:30353136 cmake_vspar411:30581797 vsaa593cf:30376535 pythonvs932:30410667 cppdebug:30492333 vsclangdc:30486549 c4g48928:30535728 dsvsc012cf:30540253 pynewext54:30695312 azure-dev_surveyone:30548225 vsccc:30610678 nodejswelcome1:30587005 3biah626:30602489 pyind779:30671433 f6dab269:30613381 pythonsymbol12:30671437 2i9eh265:30646982 6233i204:30672705 vsccsb:30677849 pythonb192:30669360 functionswalk:30687959 pythonms35:30701012 ```

[vscodenpa]

Thanks for creating this issue! It looks like you may be using an old version of VS Code, the latest stable release is 1.77.0. Please try upgrading to the latest version and checking whether this issue remains.

Happy Coding!

[farmio]

I just did the update to 1.77.0 and the issue persists.

Version: 1.77.0 (Universal) Commit: 7f329fe6c66b0f86ae1574c2911b681ad5a45d63 Date: 2023-03-29T09:57:11.797Z Electron: 19.1.11 Chromium: 102.0.5005.196 Node.js: 16.14.2 V8: 10.2.154.26-electron.0 OS: Darwin arm64 22.3.0 Sandboxed: No

[connor4312]

This takes a while for me, but I think this is on the Python side. Looking at the Output window, they run one process per parameterized "variant" (not sure of the correct term.) So that's 150 processes for range(150). VS Code core is fine.

[connor4312]

oh, actually, this is because we "include" all of them individually. Hmm. We could probably optimize that in VS Code. I'll put in a change there.

[connor4312]

@eleanorjboyd I've added code for this in the linked PR, however it seems that parameterized tests lack a test.range. Is there a reason for this?

With the change I made, if the parameterize test parent item has a range that's equal to its children, then we'll coalesce the child tests.

[eleanorjboyd]

@connor4312 what do you mean by test.range? Is that the line number on the tests? I think the rewrite of testing in python I am working on should fix this but want to clarify what exactly would be needed from my end before confirming. Thanks!

[connor4312]

The TestItem interface has a range on it. That's not set for the main item with the test_test label, so vscode doesn't see that as something it can run at the cursor, and instead runs all its children individually

[eleanorjboyd]

ok gotcha, setting this as a bug for me to fix but I will wait until we release our new rewrite to work on it as the testing rewrite will effect how it is implemented.

[bw4sz]

This seems really critical to me, I can see a 10x speed slowdown in using vscode versus running pytest. Is there a workaround? x-dist?

[eleanorjboyd]

Hi @connor4312, wanted to follow up here sorry its been a minute since we spoke on this. I am attempting to implement and I am still seeing the parameterized children test items get passed as part of the runTests request: TestRunRequest. I am trying it with this example:

@pytest.mark.parametrize("number", [1, 2, 3, 4, 5, 6, 7, 8, 9, 10])
def test_numbers(number):
    assert number > 0

which produces 10 parameterized tests and then the function node that is the parent of those but is runnable would be test_numbers. I checked and test_numbers has the same range as the children items (the 10 parameterized tests) but both the 10 parameterized tests + the test_numbers node show in the test run request.

Are there any things other than range which are checked to determine if the children should be coalesce? And am I right assuming that the children being combined means I should expect one test run node for the parent test_numbers and not 11?

Thanks!

here you can see the 11 items in the request, the 9th being an example of a parameterized test and the 10th being the parent of the parameterized test

[connor4312]

Which version of the Python extension are you using? On the current prerelease I'm seeing the same behavior that the parent node of parameterized tests don't have an associated range here

You can also check this by seeing if the "go to test" button in the Test Explorer view works for a given test

[eleanorjboyd]

This was on a branch I made with edits to add a range for the parent item - Ill give the go to test button a look to confirm and if I am able to verify ill send over a link to the branch. Thanks!

[eleanorjboyd]

@connor4312 here's a video of what I am seeing, the issue being that all 4 (the 3 parameterized tests and single main test) are being included in the request. Here is a link to the branch: https://github.com/eleanorjboyd/vscode-python/tree/21151-param-gutter

https://github.com/microsoft/vscode-python/assets/26030610/1902b01c-b485-4619-be89-ad59c8730b04

[connor4312]

Over a call we noticed that Eleanor was testing the gutter click action where this logic hadn't been implemented. I made a PR on VS Code (linked) to do this. It's a little more complicated, but also generic.