Running multiple test configurations gives wrong results

[eliericha]

Hello,

I've encountered the following issue on my project and managed to reproduce it on the hello world test CLI sample project in this fork.

To reproduce, I changed the hello world project to define 2 test suites intended to run in two different workspaces.

After doing that, running each test alone from the UI works, i.e. the tests pass. But using the Run All Tests command or UI button makes the tests fail.

Since there are 2 configurations, we find 2 test runs in the UI, both named Unnamed Task. Here are screenshots of the outputs of each run.

After some investigation and debugging, here's what I found.

1. The extension is not selecting the subset of files belonging to a configuration when running it. We can see that in the command line of each test run where both test suite files were passed using --run even though we are running under one configuration with --label N.

   >  vscode-test
   --label 0
   --reporter /home/richa/ancr/src/vscode-extension-samples/helloworld-test-cli-sample/node_modules/@vscode/test-cli/out/fullJsonStreamReporter.cjs
   --run /home/richa/ancr/src/vscode-extension-samples/helloworld-test-cli-sample/out/test/suite2/extension.test.js
   --run /home/richa/ancr/src/vscode-extension-samples/helloworld-test-cli-sample/out/test/suite1/extension.test.js
   ...
   
   >  vscode-test 
   --label 1 
   --reporter /home/richa/ancr/src/vscode-extension-samples/helloworld-test-cli-sample/node_modules/@vscode/test-cli/out/fullJsonStreamReporter.cjs 
   --run /home/richa/ancr/src/vscode-extension-samples/helloworld-test-cli-sample/out/test/suite2/extension.test.js 
   --run /home/richa/ancr/src/vscode-extension-samples/helloworld-test-cli-sample/out/test/suite1/extension.test.js

   This makes all tests run in both configurations when the intention is that each config should run only the files matching its files patterns.

2. VS Code is triggering the execution of both configurations in parallel. That's why the second config fails with the following output:

 >  vscode-test --label 1 --reporter /home/richa/ancr/src/vscode-extension-samples/helloworld-test-cli-sample/node_modules/@vscode/test-cli/out/fullJsonStreamReporter.cjs --run /home/richa/ancr/src/vscode-extension-samples/helloworld-test-cli-sample/out/test/suite2/extension.test.js --run /home/richa/ancr/src/vscode-extension-samples/helloworld-test-cli-sample/out/test/suite1/extension.test.js
Found existing install in /home/richa/ancr/src/vscode-extension-samples/helloworld-test-cli-sample/.vscode-test/vscode-linux-x64-1.87.2. Skipping download
[0318/171559.102180:ERROR:file_io_posix.cc(152)] open /home/richa/ancr/src/vscode-extension-samples/helloworld-test-cli-sample/.vscode-test/user-data/Crashpad/pending/d9d1c5f0-571e-4dfa-ae15-5684858a9060.lock: File exists (17)
[main 2024-03-18T17:15:59.342Z] Running extension tests from the command line is currently only supported if no other instance of Code is running.
[main 2024-03-18T17:15:59.344Z] Error: Running extension tests from the command line is currently only supported if no other instance of Code is running.
    at Ee.e (/home/richa/ancr/src/vscode-extension-samples/helloworld-test-cli-sample/.vscode-test/vscode-linux-x64-1.87.2/resources/app/out/vs/code/electron-main/main.js:110:4461)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async /home/richa/ancr/src/vscode-extension-samples/helloworld-test-cli-sample/.vscode-test/vscode-linux-x64-1.87.2/resources/app/out/vs/code/electron-main/main.js:110:1230
    at async Ee.a (/home/richa/ancr/src/vscode-extension-samples/helloworld-test-cli-sample/.vscode-test/vscode-linux-x64-1.87.2/resources/app/out/vs/code/electron-main/main.js:110:1117)
Exit code:   1
Failed
Error: Test process exited with code 1

IMO (1) should be fixed in the extension. The launch of a config should use --run only for files included in that config. I will submit a tentative fix for review.

As for (2), I figured out that it stems from the fact that both concurrent launches are using the same user-data directory. I was able to work around it by using different user-data dirs as follows:

export default defineConfig([
    {
        files: 'out/test/suite1/**/*.test.js',
        workspaceFolder: 'ws-suite1',
        launchArgs: ['--user-data-dir', 'userdata1'],
    },
    {
        files: 'out/test/suite2/**/*.test.js',
        workspaceFolder: 'ws-suite2',
        launchArgs: ['--user-data-dir', 'userdata2'],
    },
]);

This is likely going to be an issue for any project with multiple test configurations. Should the extension do something internally to use different --user-data-dir automatically in the presence of multiple configurations?

The updated version is also available at the fork.

[eliericha]

I can confirm that this now works over my original use case. Thanks!