[Feature] "Headless" CLI Watch Mode

[RichiCoder1]

Playwright recently introduced a watch mode via a new --ui flag. While awesome, it doesn't quite meet some other use cases for a "Watch Mode". Namely continuous background validation of tests where a UI might be intrusive, but you still want red/green when test srcs are updated. At that point you might jump in the UI to debug, but by default you're mostly just interested in continuous background validation.

Playwright is also increasingly being used for unit tests or integration tests, where a full browser step through might not be relevant.

[pavelfeldman]

Try setting environment variable PWTEST_WATCH=1 and running your tests. It'll engage a different version of the watch mode that should look more like the traditional jest one. Tell us what you think about it and whether it worked for you. We'll collect upvotes here to see if we want to make it official --watch.

[lucasassisrosa]

While we all waiting on this there's also a way to run a watch mode if you're on CI, using the git staged files. You can pull the files changed in git and feed to Playwright CLI to run only the specs you want. Here's an example with Github Actions:

steps:
  - name: Checkout
    uses: actions/checkout@v3

  - name: Setup NodeJS
    uses: actions/setup-node@v3
    with:
      node-version: 16

  - name: Install dependencies
    run: npm install

  - name: Install latest playwright browsers
    run: npx playwright install --with-deps

  - name: Get changed specs in the src folder
    id: changed-files-folder
    uses: tj-actions/changed-files@v35
    with:
      files: folder/**

  - name: Test 'e2e' if folder files changed
    if: steps.changed-files-folder.outputs.any_changed == 'true'
    run: npx playwright test ${{ steps.changed-files-folder.outputs.all_changed_files }}

at the end, the above would issue a command like

npx playwright test example.spec.ts second.spec.ts

[jfgreffier]

I really love PWTEST_WATCH=1 with component testing, it makes the experience more enjoyable than UI Mode, especially if you attempt to do TDD. Also the possibility to see the actual browser with the real component (instead of Trace) is a plus compared to the UI Mode.

[naveedausaf]

This only watches test files. Especially with component testing which are basically unit tests and follow short "Red light - Green light - Refactor" loops, it would be very useful if the CLI watched source files also the way Jest's VS Code extension does.

[pavelfeldman]

This only watches test files. Especially with component testing which are basically unit tests and follow short "Red light - Green light - Refactor" loops, it would be very useful if the CLI watched source files also the way Jest's VS Code extension does.

That's what it does.

[nausaf]

UPDATE: Now using onchange with -k option instead of chokidar. If there's a file change while tests are already running in response to a previous change, this would kill running tests and restart. chokidar on the other hand would launch tests another time in parallel.

This is the new test-playwright:watch script:

"test-playwright:testcommand": "(playwright test quiz.spec.ts || true) && echo Test run report available at http://localhost:6007",

"test-playwright:watch": "concurrently --names \"REPORT,TEST\" --hide REPORT --prefix-colors \"blue,magenta\" \"http-server playwright-report -s --port 6007\" \"onchange -i -k tests/**/*.spec.ts pages/* components/**/* -- npm run test-playwright:testcommand\"",

Hi @pavelfeldman,

I am pretty certain that watch mode of playwright test only monitors test files. I couldn't trigger it by making changes to any of my code files (React components) and saving the files. I also couldn't find any command line option to make it watch non-test code.

Another thing it doesn't do is create or serve HTML report that can be really useful when there are failures.

I am currently using a workaround using chokidar, playwright test, http-server etc. (given below). It would be great if the watch mode built into playwright test could incorporate these two features.

My current watch mode setup is as follows: I have a script named test-playwright:watch in package.json to do both (i.e. watch for changes to both test and code-under-test files, and to serve HTML report). It uses another two scripts and all three are shown below:

"test-playwright-watch-testcommand": "(playwright test quiz.spec.ts || true) && echo Test run report available at http://localhost:6007",

"test-playwright-watch-watchcommand": "chokidar tests/**/*.spec.ts pages/* components/**/* -c \"npm run test-playwright-watch-testcommand\"",

"test-playwright:watch": "concurrently -k -s first -n \"REPORT,TEST\" -c \"magenta,blue\" \"http-server playwright-report -s --port 6007\"  \"run-s --silent -c test-playwright-watch-testcommand test-playwright-watch-watchcommand\"",

Since every run of playwright test opens up a new HTTP server on the playwright-report directory (although by default the report is only served on failure), a lot of these would end up running at the same time (each on a different, probabaly randomly assinged, port but on the same underlying folder playwright-report) after tests have run enough time due to file changes detected by chokidar. Therefore, I configure the HTML reporter in playwright.config.ts to generate the HTML report but not to serve it:

reporter: [['html', { open: 'never' }]],

and instead launch http-server on the report directory only once at the time the script is launched. I do this by running the actual watch/test command in parallel with http-server using concurrently (see definition of test-playwright:watch above).

Then at the end of every run of playwright-test that had started on detection of a file change, the URL of the report is printed so you can just Ctrl+Click it to look at the report in a browser window:

To get the above scripts to work, I think the following NPM packages would need to be installed (in addition to playwright of course):

npm install -D concurrently http-server chokidar npm-run-all

Finally, for anyone using VS Code who wants test-playwright:watch script to start running automatically whenever the project folder is opened, they can define a task in .vscode/tasks.json like this (full tasks.json shown so you can just copy and paste it in a blank .vscode/tasks.json if you don't have one already). Note for this to work you would also need to configure webServer in playwright.config.ts to launch your app whenever tests start running. This is shown after tasks.json below.

{
  // See https://go.microsoft.com/fwlink/?LinkId=733558
  // for the documentation about the tasks.json format
  "version": "2.0.0",
  "tasks": [
    {
      "label": "Playwright watch",
      "type": "shell", //"process" does not work as value
      "command": "npm run test-playwright:watch",
      "args": [],
      "isBackground": true, //this makes it keep running in the background without interrupting what you're doing
      "runOptions": {
        "runOn": "folderOpen" //start this task when folder is opened in VS Code
      },
      "presentation": {
        "panel": "dedicated"
      },
      "problemMatcher": {
        "fileLocation": ["relative", "${workspaceFolder}/tests"],
        "severity": "error", //probabaly redundant as it is the default
        "pattern": {
          "regexp": "^\\[TEST\\]\\s*\\[(.*)\\]\\s\\S\\s(.*):(\\d+):(\\d+).*$",
          "file": 2,
          "line": 3,
          "column": 4,
          "message": 1

        },
        "background": {
          "activeOnStart": false,
          //I have verified that if either of the following regexes is wrong,
          //you don't get the tick on the terminal to indicate once
          //a test run has completed (even though our watch task carries on running)
          "beginsPattern": "^\\[TEST\\]\\s*Running \\d* tests using \\d* workers\\s*$",
          //"beginsPattern": "^\\s*Running \\d* tests using \\d* workers\\s*$",
          //USE THE FOLLOWING endsPattern IF REVERTING TO CHOKIDAR TO WATCH
          "endsPattern": "^\\[TEST\\]\\s*To open last HTML report run:\\s*$"
          //"endsPattern": "^Waiting for file changes\\. Press enter to run tests, q to quit or h for more options\\.\\s*"
        }
      }
    }
  ]
}

In playwright.config.ts, configure webServer sort of like this:

webServer: {
    command: 'npm run dev',
    url: 'http://127.0.0.1:3000',
    reuseExistingServer: !process.env.CI,
    ignoreHTTPSErrors: true,
  },

With the three patterns in problemMatcher in the task (regex in pattern block, beginsPatter and endsPattern), the VS Code background task provides a nice indication of its status in the header of the terminal window in which it is running:

If there are failures, you can see indications in a few different places:

• terminal window label (shown above)

• The terminal itself (if in view), where you can see the failing tests listed and Ctrl+Click URL of the test run report to open it in a browser (also shown above)

• In Problems pane, from where you can also navigate directly to the the test that failed:

  

[aslushnikov]

Everybody: are you still using this mode? How do you feel about us removing this experiment? Do you feel like it's adding enough value atop of chokidar?

Let us know what you think!

[naveedausaf]

Dear Andrey,

I don’t think it’s adding any value as the UX is not there.

A watch mode should be supplemented with visual indicators of failure and launch on opening the folder in the editor of the sort you get when you run the watch command from a VS Code task in .vscode/tasks.json.

Also, the mode doesn’t generate or serve Playwright’s marvellous HTML report. I had to stitch that together with concurrently, http-server and plain “npm playwright test –reporter=html”.

See my comment below if you haven't already; this I think is a fairly complete watch mode with a good UX:

https://github.com/microsoft/playwright/issues/21960#issuecomment-1654901071

One place where you could add lots of value I think is if you could provide a command in Command Pallette to generate both the watch command in package.json and an accompanying task in .vscode/tasks.json. Writing the whole thing for the first time can be real pain because:

• some patterns that regex101 excepts are not matched by VS Code. Playwright watch mode generates a line of text with a chevron in it that VS Code can't match. Figuring this out can be a pain.

• In all my watch modes (Storybook, accessibility tests and Playwright tests) I have to echo out a beginning line of text and an ending line of text so that the problemMatcher in VS Code task can know when a tests run started and when it ended (crucial as the watch task itself runs perpetually). You could instead use echo these out to match you VS Code task's regex patterns.

• There are lots of nuances of VS Code tasks that take a lot of trial and error to figure out the first time you write a watch mode.

Bests,

Naveed

Sent from Mail https://go.microsoft.com/fwlink/?LinkId=550986 for Windows

From: Andrey Lushnikov @.> Sent: 31 July 2023 19:05 To: microsoft/playwright @.> Cc: naveedausaf @.>; Comment @.> Subject: Re: [microsoft/playwright] [Feature] "Headless" CLI Watch Mode (Issue #21960)

Everybody: are you still using this mode? How do you feel about us removing this experiment? Do you feel like it's adding enough value atop of chokidar?

Let us know what you think!

— Reply to this email directly, view it on GitHub https://github.com/microsoft/playwright/issues/21960#issuecomment-1658899791, or unsubscribe https://github.com/notifications/unsubscribe-auth/AE3REKWEFACZPJJUAQISNWLXS7XWVANCNFSM6AAAAAAWGZ4ERE . You are receiving this because you commented.Message ID: @.***>

[naveedausaf]

On second thoughts, it would be great if Playwright extension could just provide a command that could launch a similar watch mode in a separate terminal window with similar UX (indications of failure, successful completion and running tests in terminal header).

The folders to watch in addition to tests could be configured in playwright.config.ts.

It really is very useful to have it. Basically, iPlaywright is so fast and I have enough cores that as I am done with git add and git commit, I see a tick to say that all my tests have passed. I can then do git push:

Thanks,

Naveed

On Tue, Aug 1, 2023 at 1:33 AM Naveed Ausaf @.***> wrote:

Dear Andrey,

I don’t think it’s adding any value as the UX is not there.

A watch mode should be supplemented with visual indicators of failure and launch on opening the folder in the editor of the sort you get when you run the watch command from a VS Code task in .vscode/tasks.json.

Also, the mode doesn’t generate or serve Playwright’s marvellous HTML report. I had to stitch that together with concurrently, http-server and plain “npm playwright test –reporter=html”.

See my comment below if you haven't already; this I think is a fairly complete watch mode with a good UX:

https://github.com/microsoft/playwright/issues/21960#issuecomment-1654901071

One place where you could add lots of value I think is if you could provide a command in Command Pallette to generate both the watch command in package.json and an accompanying task in .vscode/tasks.json. Writing the whole thing for the first time can be real pain because:

• some patterns that regex101 excepts are not matched by VS Code. Playwright watch mode generates a line of text with a chevron in it that VS Code can't match. Figuring this out can be a pain.

• In all my watch modes (Storybook, accessibility tests and Playwright tests) I have to echo out a beginning line of text and an ending line of text so that the problemMatcher in VS Code task can know when a tests run started and when it ended (crucial as the watch task itself runs perpetually). You could instead use echo these out to match you VS Code task's regex patterns.

• There are lots of nuances of VS Code tasks that take a lot of trial and error to figure out the first time you write a watch mode.

Bests,

Naveed

Sent from Mail https://go.microsoft.com/fwlink/?LinkId=550986 for Windows

From: Andrey Lushnikov @.> Sent: 31 July 2023 19:05 To: microsoft/playwright @.> Cc: naveedausaf @.>; Comment @.> Subject: Re: [microsoft/playwright] [Feature] "Headless" CLI Watch Mode (Issue #21960)

Everybody: are you still using this mode? How do you feel about us removing this experiment? Do you feel like it's adding enough value atop of chokidar?

Let us know what you think!

— Reply to this email directly, view it on GitHub https://github.com/microsoft/playwright/issues/21960#issuecomment-1658899791, or unsubscribe https://github.com/notifications/unsubscribe-auth/AE3REKWEFACZPJJUAQISNWLXS7XWVANCNFSM6AAAAAAWGZ4ERE . You are receiving this because you commented.Message ID: @.***>

[bingocaller]

@aslushnikov I like this mode.

Even though I mostly use UI Mode, I still regularly use PWTEST_WATCH=1 on its own, without chokidar or anything, and it works like a charm for my purposes!

Some of our team members even prefer PWTEST_WATCH=1 over UI Mode and I suspect they would be sad to see this experiment removed.

[jfgreffier]

@aslushnikov I use this mode for component testing, with and without showing the browser. Also, without browser, the experience is similar to Jest's --watch which feels confortable

[naveedausaf]

@aslushnikov I use this mode for component testing, with and without showing the browser. Also, without browser, the experience is similar to Jest's --watch which feels confortable

I do think it's a good idea to have a console watch mode like Jest.

But Jest watches code files also (not just tests files like Playwright console watch mode does). This is a good idea in Playwright, especially for component tests.

Also, once you enable watch mode, playwright test does not generate html report even if you say --reporter=html which is a shortcoming. Everyone else here seems to also be using visual test runner (I use uiMode which is great) which I think is perfect for targeted running of a test while you're working on that test. Therefore the value of a watch mode I think is to:

• always be silently churning away in the background testing all your code, i.e. running all your tests for regressions, not just the targeted test you might be working on at any given point in time (for which we have uiMode etc.)
• Let you know conspicuously but unobtrusively when there are failures. Jest VS Code extension does it by showing a red rectangle in status bar which works very well. In my solution above, I make the terminal windows header turn red which works equally well (you might not have that terminal window in view at the time but its title/header bar is always visible). But it is when there are failures that you want to see Playwright's rich HTML report with trace, video, screenshots etc. This does not apply to Jest but, in my view, very much does to Playwright as the HTML report is so brilliant and so viewing it is the natural next step when you see a failures reported by what should otherwise be a completely silent/background process.

[WestonThayer]

PWTEST_WATCH=1 and the s option to show & reuse the browser are so close to what would really help my workflow. I'm testing long e2e flows, long enough that writing code towards the end of a test means I have to wait a good bit when the test is re-run to reach the point I'm working on.

If the s option didn't try to "reset" the browser context, I could...

1. Make changes to the end of a long test
2. Run the test through to the end, where my changes are in watch mode
3. Interact with the browser by hand to "undo" the last few steps of the changes I'm currently working on
4. Comment out all code in my test except for the end and save

That enables me to very quickly try different changes towards the end of the test without needing to wait for dozens of steps I know will pass to complete every iteration loop.

edit: found a similar question and figured out how to hack this in https://github.com/microsoft/playwright/discussions/22883#discussioncomment-6732710.

[coaxial]

Would love a native, traditional watch mode for Playwright. I typically write code in the terminal and use tmux for subwindows. With other test runners, I can have them run in a subwindow and whenever I modify a test or part of the code, it will rerun the affected tests and show me a red/green result quickly. I can't do that with Playwright. The --ui flag means switching to another window, hiding my text editor. I'd much rather have it all in my terminal without needing to switch. Basically something like PWTEST_WATCH=1 npx playwright test but that also reruns the relevant tests when I modify the associated code; not only the test files :)

[sand4rt]

Would it be possible to run only the tests related to the open Git changes when initiating tests using PWTEST_WATCH=1, rather than executing all of them right away? Jest/Vitest uses a similar approach.

Is it also an option to add test cancelation by, for example, typing p so i can immediately search for a single test and run it without having to wait till they all complete?

[kenanAlhindi97]

I really like PWTEST_WATCH=1 It's a very productive mode but I am facing an issue, and I'm not sure if it's a bug or intended purpose :

Basically when I use "Show & reuse browser: on"

Only my setup project runs in headful browser, afterwards all tests run headlessly, any one knows what could be the issue ?

[mikestopcontinues]

Ideally, the CLI watch mode behaves exactly like vitest watch. Watch all file dependencies, only rerun the tests that change, provide some controls for initiating a rerun of (all|changed|pattern-matching) tests. That's what everyone here is missing.

[ildella]

Try setting environment variable PWTEST_WATCH=1 and running your tests. It'll engage a different version of the watch mode that should look more like the traditional jest one. Tell us what you think about it and whether it worked for you. We'll collect upvotes here to see if we want to make it official --watch.

PWTEST_WATCH=1 works perfectly.

Regardless of what the final solution will be it should be documented right away.

[GermanJablo]

We'll collect upvotes here to see if we want to make it official --watch.

I give +1 for --watch. I've had to install cross-env just to be able to use PWTEST_WATCH=1, since it's the only script I use with an environment variable.

By the way, I think your comment had a lot of upvotes, so, what do you think?

This only watches test files. Especially with component testing which are basically unit tests and follow short "Red light - Green light - Refactor" loops, it would be very useful if the CLI watched source files also the way Jest's VS Code extension does.

That's what it does.

So what do you think could be the reason why when I use PWTEST_WATCH=1 without using fixtures like page or context the watch mode works much slower than Vitest and Bun:test?