[Regression]: Tests not recognized in Test Explorer. No Play Green Buttons Next To Test Names.

[ahmad-elkomey]

Last Good Version

None

First Bad Version

v1.1.7

Steps to reproduce

1. Run npm init playwright@latest --yes -- --quiet --browser=chromium --browser=firefox --browser=webkit --gha --install-deps playwright-intro (Same if you do it through Playwright VS Code extension)
2. Open VS Code in playwright-intro directory.
3. Switch to Testing.

Generated sources are uploaded here: https://github.com/ahmad-elkomey/err-report-playwright-regression/tree/main

Expected behavior

Project and tests are expected to be recognized and there should be green play buttons next to every test name.

Actual behavior

Project and tests are not recognized/found. No green play buttons.

Additional context

No response

Environment

System:
   OS: Linux komey-dell 5.15.0-118-generic #128-Ubuntu SMP Fri Jul 5 09:28:59 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
   CPU: x86_64. 12 CPUs.Intel(R) Core(TM) i7-8850H CPU @ 2.60GHz

Binaries:
- npm 10.8.2
- node v22.6.0
- playwright 1.46.0
- VS Code 1.92.1 eaa41d57266683296de7d118f574d0c2652e1fc4 x64
- Playwright VS Code Extension 1.1.7

[pavelfeldman]

try the reload button? you probably opened vscode before installing the dependencies, so Playwright was not there.

[ahmad-elkomy]

Oh I tried all kind of reloads. I tried "Developer: Reload Window", "TypeScript: Reload Project", and hitting the reload button in the title bar of "Test Explorer". Nothing worked. The latter results in this notification popping up: Running the suggested command says I'm up to date:

$ npm i --save-dev @playwright/test

up to date, audited 6 packages in 907ms

found 0 vulnerabilities

Did I miss something?

[pavelfeldman]

Sounds like you are running those in different environments. Sounds like your vscode is set up with alternative shell / chroot or something like that.

[ahmad-elkomey]

Why do you think so? What's next to progress with investigating the issue? I'm thinking maybe there some logs we can look at to confirm your theory.

[pavelfeldman]

The next step would be to obtain a repro that we would be able to execute locally. This sounds like a big breakage and given that this is the only report, it seems to be specific to your configuration. Without being able to repro it, we can't do much. If it was hitting multiple users, we would go further to suggest logging techniques and would even include more logging to try and locate the issue. But as long as this is one report with hundreds of thousands of active users, I'd suggest narrowing it down and sending us a repro.

[ecoms-ye]

I also found this issue in one of my local project (no problem before as well as my other projects) Tests not recognized in Test Explorer but can be run in cli.

[pavelfeldman]

I also found this issue in one of my local project (no problem before as well as my other projects) Tests not recognized in Test Explorer but can be run in cli.

Could you file a separate issue with a repro? I assume that since other projects work on your system, the problem is specific to the project rather than the environment.

[AlexDorha]

I have the same issue no tests are displayed anymore in Extension tab Everything is working fine from cmd, the run, debug, parameters I have latests playwright and vscode version ![Uploading Screenshot 2024-08-16 180428.png…]()

[pavelfeldman]

@AlexDorha any chance for the repro? Could you follow the BUG issue template and file something we can act on?

[AlexDorha]

@AlexDorha any chance for the repro? Could you follow the BUG issue template and file something we can act on?

Thank you for the reply I will leave a link here with the problem https://github.com/microsoft/playwright/issues/32199

[kevswanberg]

I'm also experiencing this issue.

Version: 1.92.2 (Universal)
Commit: fee1edb8d6d72a0ddff41e5f71a671c23ed924b9
Date: 2024-08-14T17:29:30.058Z
Electron: 30.1.2
ElectronBuildId: 9870757
Chromium: 124.0.6367.243
Node.js: 20.14.0
V8: 12.4.254.20-electron.0
OS: Darwin arm64 21.6.0

Due to some change in my playwright.config.ts working on figuring out what

[pavelfeldman]

To everyone experiencing this issue, it would be helpful to work your way up from a minimal working example (npm init playwright seed project) to the state where it breaks on your system. Thank you for your help!

[kevswanberg]

Thanks for looking at it Pavel, for me this happens if there is anything that throws an exception in playwright.config.ts

An example of the exact thing breaking for me is here

https://github.com/kevswanberg/vs_playwright_config_issue/blob/main/playwright.config.ts#L3

Any exception causes this though.

[pavelfeldman]

Thanks for looking at it Pavel, for me this happens if there is anything that throws an exception in playwright.config.ts

In this case CLI should not work as well though, so I wonder if that is a different issue. You probably see this error in VS Code when you open the config file there too.

We don't yell at the users when their config fails to parse because we interpret any file of the form playwright.*.config.[jt]s as Playwright config, while it could be just some stray user's file. We probably could be stricter and assume it is config - that way we can surface this error sooner.

[ahmad-elkomy]

I thank you, Pavel, for being active responder!

To everyone experiencing this issue, it would be helpful to work your way up from a minimal working example (npm init playwright seed project) to the state where it breaks on your system. Thank you for your help!

I assume this is what I did and posted in my original message. I ran the command: npm init playwright@latest --yes -- --quiet --browser=chromium --browser=firefox --browser=webkit --gha --install-deps playwright-intro (Same if you do it through Playwright VS Code extension).

But for the sake of trying to narrowing it down, I ran this npm init playwright seed project and I got the same minimal set of files which I already shared in my repo here: https://github.com/ahmad-elkomey/err-report-playwright-regression/tree/main. (Also posted in my original message). I diff'ed the two directories. The only difference is that seed is working with playwright 1.46.1 while the original one is using 1.46.0. I opened VS Code there by navigating to seed directory and ran code .. Still, tests are not recognized. The whole project isn't.

I understand that you're not planning to exert effort on the matter unless there are numerous reports against the same issue. But if you can point me where I can find the logs (or some troubleshooting tips/ideas) so that maybe this will help me troubleshoot the problem and possibly get it fixed.

[AlexDorha]

To everyone experiencing this issue, it would be helpful to work your way up from a minimal working example (npm init playwright seed project) to the state where it breaks on your system. Thank you for your help!

I have tried this one and it was not working I have made a new playwright project from scratch, using init command and it was not working...

[AlexDorha]

What is strange is the fact that the extension is not able to see the project's browsers anymore There was a checkbox with the browser name, now is not there I have tried to add a new browser and nothing was displayed also

[ahmad-elkomy]

I see the following lines get printed every 0.5 sec. It's flooding the Output window.

2024-08-17 17:27:34.750 [trace] ExtHostCommands#executeCommand _testing.getExplorerSelection
2024-08-17 17:27:35.001 [trace] ExtHostCommands#executeCommand testing.getExplorerSelection

I disabled the extension, the log lines aren't printed after. Enabling the extension back again, the logs get printed again.

I also saw this line after opening a new window:

2024-08-17 17:09:31.476 [info] ExtensionService#_doActivateExtension ms-playwright.playwright, startup: true, activationEvent: 'workspaceContains:**/*playwright*.config.{ts,js,mjs}'

And I found this Unexpected end of JSON input error in logs/20240817T150006/window3/renderer.log. What JSON is it referring to? :thinking:

      1 2024-08-17 15:04:37.027 [info] Started local extension host with pid 253426.
      2 2024-08-17 15:04:39.742 [info] [perf] Render performance baseline is 36ms
      3 2024-08-17 15:04:41.308 [info] Started local extension host with pid 253518.
      4 2024-08-17 15:04:43.357 [error] [Extension Host] [Playwright Test]: Unexpected end of JSON input
      5 2024-08-17 15:04:45.073 [info] [perf] Render performance baseline is 30ms
      6 2024-08-17 15:21:33.033 [error] [Extension Host] [Playwright Test]: Unexpected end of JSON input
      7 2024-08-17 15:47:51.281 [warning] SettingsEditor2: Settings not included in settingsLayout.ts: issueReporter.experimental.auxWindow
      8 2024-08-17 16:04:40.392 [info] Started local extension host with pid 317506.
      9 2024-08-17 16:04:43.496 [info] [perf] Render performance baseline is 29ms
     10 2024-08-17 16:04:58.567 [info] Started local extension host with pid 317734.
     11 2024-08-17 16:05:00.876 [error] [Extension Host] [Playwright Test]: Unexpected end of JSON input
     12 2024-08-17 16:05:01.402 [info] [perf] Render performance baseline is 48ms
     13 2024-08-17 16:05:09.671 [info] Started local extension host with pid 318160.
     14 2024-08-17 16:05:12.410 [info] [perf] Render performance baseline is 38ms
     15 2024-08-17 16:06:06.482 [info] Started local extension host with pid 318668.
     16 2024-08-17 16:06:08.697 [error] [Extension Host] [Playwright Test]: Unexpected end of JSON input
     17 2024-08-17 16:06:09.710 [info] [perf] Render performance baseline is 30ms

[mxschmitt]

The easiest might be to build the vscode extension yourself to debug this issue further where it throws, setting breakpoints here might be a good start, maybe exceptions get logged in the debugging window.

Thats how you can debug the vscode extension itself when you have playwright-vscode repository open:

General steps:

• https://github.com/microsoft/playwright-vscode
• npm ci
• npm run build
• and then inside vscode you can press the launch button

If you need any help, feel free to ping us here. Thanks for digging!

(This "Unexpected end of JSON input" looks indeed unexpected).

[ahmad-elkomy]

Thanks, Max! This is what I needed.

This is the first time to debug a VS Code extension (actually, this is the first time to use VS Code in debugging at all), and I'm encountering a bit of a problem here and I could use your assistance.

I cloned the repo, added two log statements (2 & 3 in the image below), added two breakpoints (4 & 5 in the image below), and reloaded the window with disabling all extensions (1 in the image below). I then ran the two commands npm ci and npm run build.

I then hit "Run Extension in Playwright folder", which opened a new VS Code window. I hit "Open Folder" button and open playwright test which I have a problem with. This is the folder that has the initiated playwright test and I reported this issue for.

So far so good. Now the problem is that the debugger uses files from out directory and not src. This result in the breakpoints not being reachable and the stack trace is pointing to way too far in line width. As you can see in the image below, the breakpoints are dimmed (1) as unbound. And the stack trace (2 through 5) is pointing to character position, if you take the last line in the stack trace for instance, at 21632.

How can I set VS Code to use source files instead of out build files? I tried to follow VS Code tutorial to do Hello World extension. However, I cannot even run the first command npx --package yo --package generator-code -- yo code successfully; as it's complaining about deprecated libraries and then exists! :cry:

Side Note: I also wanted to share that I confirmed all json files under playwright directory are well formed. I ran this command find -name "*.json" | while read -r json_file; do printf "json file: %s. %s.\n" "${json_file}" "$(cat "${json_file}" | jq > /dev/null 2>&1 && echo "Looks good" || echo "Something seems off")"; done

[mxschmitt]

I cloned the repo, added two log statements (2 & 3 in the image below), added two breakpoints (4 & 5 in the image below), and reloaded the window with disabling all extensions (1 in the image below). I then ran the two commands npm ci and npm run build.

Do npm run watch instead and things should work.

How can I set VS Code to use source files instead of out build files? I tried to follow VS Code tutorial to do Hello World extension. However, I cannot even run the first command npx --package yo --package generator-code -- yo code successfully; as it's complaining about deprecated libraries and then exists! 😢

No need to do that.

---

The screenshot you shared looks great! If you do npm run watch it should yield to good line numbers.

[ahmad-elkomy]

Fantastic! It did indeed put things in place.

Now, I initially found where the problem is. I'm trying to figure if env being empty is OK, and what's JSON is expected to be stored in pwtInfor. It's throwing the exception here at JSON.parse(pwtInfo).

[AlexDorha]

I think is a problem related to how extension works with new vs code versions I do not think that we have to make debug to extension to see where is the problem..

[AlexDorha]

LIke I said before I have made a fresh install on a new PC for VSCode I have used the init command from playwright to generate the template project I have installed the extension and it was not working How is this still a problem if I have made the setup on a new PC? :)

[mxschmitt]

@ahmad-elkomy great work! I recommend to put a breakpoint here to see what value Node.js has: https://github.com/microsoft/playwright-vscode/blob/655097bd3e295b9894ebe94b190c5aefc4660efb/src/utils.ts#L144

env being empty is ok, since these are just extra env vars, we spread them with the parent env here: https://github.com/microsoft/playwright-vscode/blob/655097bd3e295b9894ebe94b190c5aefc4660efb/src/utils.ts#L96

---

@AlexDorha please file a separate issue with more details and follow the bug template. Node.js/VSCode version etc are crucial for us to understand it better. We have thousands of users for whom its working - so most likely its a misconfiguration on your system of e.g. Node.js. More than happy to debug it further tho!

[ahmad-elkomy]

@mxschmitt, it's the correct path to node executable.

[mxschmitt]

I see, there might be some incompatibility between our extension and Node.js installed via Snap. There are also known VSCode issues, that if you have VSCode installed via Snap WebKit doesn't work. We recommend installing Node.js via nvm instead. Looking at https://github.com/microsoft/playwright/issues/31558.

We are unfortunately not very experienced with Snap - maybe @AlexDorha is also using snap.

[AlexDorha]

I see, there might be some incompatibility between our extension and Node.js installed via Snap. There are also known VSCode issues, that if you have VSCode installed via Snap WebKit doesn't work. We recommend installing Node.js via nvm instead. Looking at #31558.

We are unfortunately not very experienced with Snap - maybe @AlexDorha is also using snap.

I have raised a bug and the repo here https://github.com/microsoft/playwright/issues/32221