[Bug]: [VS Plugin] Tests run multiple times after a .env change causes a reload.

[dsmeltz]

Version

1.49.0

Steps to reproduce

This is regarding running tests with the Playwright Test for VS Code plugin. (v1.1.12)

Cause a test reload, by changing the .env file. (I just hit return, backspace and saved.)

Expected behavior

I would expect the tests to only run one time.

Actual behavior

The tests, that are run via the plugin, will run multiple times. It's usually two times, but I've had it repeat the tests three times.

Additional context

No response

Environment

System:
    OS: Windows 10 10.0.20348
    CPU: (48) x64 Intel(R) Xeon(R) Silver 4310 CPU @ 2.10GHz
    Memory: 1.44 TB / 1.50 TB
  Binaries:
    Node: 20.14.0 - C:\Program Files\nodejs\node.EXE
    npm: 10.9.0 - C:\Program Files\nodejs\npm.CMD
  IDEs:
    VSCode: 1.95.3 - C:\Users\dsmeltz\AppData\Local\Programs\Microsoft VS Code\bin\code.CMD
  Languages:
    Bash: 5.1.16 - C:\Windows\system32\bash.EXE
  npmPackages:
    @playwright/test: ^1.49.0 => 1.49.0

[dgozman]

@dsmeltz Could you please share a sample repository that shows the problem? How is your .env file used in the project? What triggers the reload upon .env change? Please be as specific as possible.

[dsmeltz]

Unfortunately no public repo.

I do have a single project that is dependent on a setup project.

The .env file is being used like this:

let x = process.env.BASE_URL;

Nothing out of the ordinary. It is being used in some tests.

Triggering the reload is as simple as saving the .env file or the playwright.config.ts file. So, the issue doesn't seem to be solely dependent on the .env file. The .env file is being used in the playwright.config.ts file.

The reload I'm referring to is the reload of the Test Explorer pane in the plugin.

After more investigation, I found the following: I can make one to several (around three) changes to the config file and sometimes may only get one test run. But if I make an additional change to the config, I get two test runs. (The change to the config I'm making is hitting Enter three times at the end of the file and then saving. VS Code removes the extra lines before it actually saves the file.) If after the two test runs issue is exhibited, I click the Reload icon on the Test Explorer tab and that clears the faulty condition. But if another change is made to the config, the two test runs issue is reexhibited.

The Test Results window has a pane on the right-hand side that shows the test names as they're run. If I'm in the bad condition, only the project names are displayed and not the actual test names. Again, if I click the Reload button on the Test Explorer tab, the test names will start to be displayed again.

The two projects I'm using are named Fed Admin and setup. Where you see those displayed is when the plugin is in the bad state. After hitting Reload, you can see that the test names, authenticate and Verify Home Page, are then displayed.

This is the Test Results window showing the tests being run twice in succession. (Also note the project names and not the test names in the other pane.)

[dgozman]

@dsmeltz Unfortunately, I cannot reproduce this issue. I guess we'll need an exact repro with your files and the exact steps performed in VSCode, to be able to make progress.

Just in case, perhaps you get multiple projects checked in the "Playwright" testing sidebar, and so tests run multiple times? Take a look at this issue explaining the config/project management, it might help.

[dsmeltz]

I do have multiple projects checked (two), but that's the only way to get the setup project to run when using the plugin.

After unchecking my setup project, I tried the steps above and the one test I ran, executed two times.

Also, when trying to reproduce, try putting extra lines in the middle of the config file, so VS Code doesn't strip them out.

[dgozman]

@dsmeltz Thank you for the explanation. Unfortunately, I still cannot reproduce. I have to request a repro that would contain all the files necessary to reproduce the issue. There must be something different between our setups, and I won't be able to just guess it. Please share a reproducible.

[dsmeltz]

I was able to reproduce the issue with just the default installation of Playwright.

1. Install Playwright
2. Install dotenv
3. Create .env file with a BASE_URL setting
4. Import dotenv into playwright.config.ts and invoke the config() method
5. Add that setting to the chromium projects use section as the baseUrl. baseUrl: process.env.BASE_URL
6. Execute the tests (Should run one time.)
7. Add some blank lines to the middle of the playwright.config.ts file.
8. Execute tests - see multiple runs

[dsmeltz]

I tried to reproduce this on another machine and couldn't do it. So I'm closing this.