[Feature]: Add ability for watch mode to watch source code changes to make it compatible with a TDD workflow

citypaul commented 7 months ago

🚀 Feature Request

A feature that enables Playwright to automatically re-run relevant tests when implementation files or test files are saved, similar to the watch mode available in Jest or Vitest.

The --ui based watch mode is already very useful, but it currently only runs when changes to test files are detected. This would do the same thing but when source files are changed.

Example

In a typical Test-Driven Development (TDD) workflow, a developer makes changes to a test file or the corresponding implementation file and the relevant tests will re-run immediately. Upon saving these changes, Playwright's enhanced watch mode would immediately detect the update and re-run the relevant tests. This process ensures immediate feedback on the impact of changes, facilitating a more efficient development cycle.

Motivation

The primary motivation for this proposal is to enhance Playwright's utility in TDD workflows, especially for testing Next.js applications that incorporate server components. Jest and Vitest, while popular for TDD, currently lack support for server components, a gap that Playwright can fill.

By integrating an automatic test rerun feature upon file saves, Playwright would significantly improve developer experience, enabling a smoother, more effective TDD process.

This addition would not only extend Playwright's capabilities but also align it with the evolving needs of modern web development, particularly as modern frameworks like Next.js moves towards more extensive use of server components, making TDD more difficult in those frameworks using virtual DOM based testing tools like jest/vitest.

sand4rt commented 7 months ago

dgozman commented 7 months ago

@citypaul Are you interested in the watch mode for end-to-end or component tests?

citypaul commented 7 months ago

@dgozman ideally it would be both.

In the use case I've been experimenting with at the moment, I'm looking at how I can continue to have a TDD workflow using next.js with the app router.

This is made more difficult at the moment because tools like jest and vitest don't play well with react server components and server actions at the time of writing.

For most applications that I make, my daily flow is TDD based and I normally have my tests open in a terminal, updating in real time every time I hit the save button in my IDE. This gives me an excellent developer experience and works very nicely for TDD where the feedback loop is paramount.

I am working on a personal project at the moment and I decided to see how far I could push just using playwright for the majority of tests.

I've got a setup at the moment that renders entire pages and I've managed to create a mechanism to mock out requests made in server components and server actions by passing a custom header from playwright before playwright visits the page.

So far this is actually quite a nice experience (early days but it's a promising start), but my main issue at the moment is that I have to manually re-run the tests every time I make a change to my implementation code. This means I can easily come out of the TDD loop - ideally I want to know in a few seconds if I've broken something without me having to re-run anything myself.

So for this current setup, I'm using the e2e functionality (although I'm technically not doing true e2e tests as I'm mocking out server side API calls using this custom header setup).

I was considering experimenting with playwright for component testing as well - for either setup though, a TDD workflow really requires automatic feedback as quickly as possible, so I'd say in an ideal world both component testing and the e2e setup would be capable of this.

I was thinking about this a bit and I can see how it might be hard to figure out which tests to re-run in the e2e setup (probably easier in the component setup I'd imagine), so one potential option might be that you just provide an option to re-run the whole test suite when some implementation code is changed - but perhaps respecting the filters inside the UI mode. So if I filter out say just my login tests, it'll just re-run those perhaps?

I have to say the early days of my little experiment here are going quite well and I can potentially see this becoming a useful feature for a lot of people, especially if jest and vitest aren't able to solve this problem with nextjs in the near future.

I hope that helps!

dgozman commented 7 months ago

@citypaul Thank you for the explanation.

As you have guessed, watch for component tests is feasible, because it is more clear what tests depend on what source file. We are working on this, so it's coming in some future Playwright version. #23372 looks like a relevant issue here.

For e2e tests, it is indeed unclear which tests to rerun upon your source changes. The connection between the page you load in the test and the source files that it was built from relies on source maps and is broken in many cases, unfortunately. So far, I've seen workarounds using chokidar and similar tools to manually watch source files and re-run relevant tests. This is an area to explore, I'll leave the issue open to gather any feedback, but there is no immediate work planned.

citypaul commented 7 months ago

@citypaul Thank you for the explanation.

As you have guessed, watch for component tests is feasible, because it is more clear what tests depend on what source file. We are working on this, so it's coming in some future Playwright version. #23372 looks like a relevant issue here.

For e2e tests, it is indeed unclear which tests to rerun upon your source changes. The connection between the page you load in the test and the source files that it was built from relies on source maps and is broken in many cases, unfortunately. So far, I've seen workarounds using chokidar and similar tools to manually watch source files and re-run relevant tests. This is an area to explore, I'll leave the issue open to gather any feedback, but there is no immediate work planned.

If It helps, I think just having an option to re-run all e2e tests when an implementation file is changed would be ok, so long as the filter in the UI was respected in terms of choosing which tests to run.

As I'm working on a feature, I usually know which test(s) need to run, so just doing this alone would be a big help!

pavelfeldman commented 2 months ago

Why was this issue closed?

Thank you for your involvement. This issue was closed due to limited engagement (upvotes/activity), lack of recent activity, and insufficient actionability. To maintain a manageable database, we prioritize issues based on these factors.

If you disagree with this closure, please open a new issue and reference this one. More support or clarity on its necessity may prompt a review. Your understanding and cooperation are appreciated.

microsoft / playwright