Support e2e solutions like Playwright or Puppeteer

[ryall]

I'm currently switched to Playwright but basically experiencing the same issue in both counts.

When using jest-playwright-preset, auto configuration appears to throw the error: ‌[Error] Automatic Jest configuration error: Can not find any Jest runner configurations in '.'. 

It appears to be something to do with the fact that Playwright uses jest-circus as part of the preset.

Any help getting this supported or configured would be great.

[ArtemGovorov]

At the moment Wallaby doesn't support running e2e tests.

While it's not impossible to add the support for it (though we haven't estimated how much work is involved in supporting Playwright or Puppeteer specifically), we are not sure if it makes much sense to use/write typically long running E2E tests with wallaby.js (that executes tests frequently). Given their slow nature one would not probably want to run them frequently, but rather use faster unit tests with wallaby for realtime testing.

[ryall]

I understand your point about slower running tests but still it would be much faster and more convenient for me to use Wallaby even in the e2e tests than running the full suite or having to filter by name each time. Plus doesn't Wallaby do selective updates based on the test you're editing anyway? In those cases the test won't take any longer than 1-2s to run, which is still pretty fast.

[smcenlly]

@ryall - you're right in saying that Wallaby does do selective test runs based on the test/code that you're editing.

To expand a little on what @ArtemGovorov said and explain some context in terms of the challenges we have in supporting:

In the case of e2e testing frameworks, in most cases e2e tests run off the production build or require you to start your application externally before running e2e tests, (e.g. with npm start). In this case, your application source code is not run/hosted by Wallaby, which means that Wallaby is unable to add its instrumentation for code coverage (and other features). This means that Wallaby cannot determine which tests are related to your application code. In this case, Wallaby would have to fall back on re-running all tests on any code/test change as it would be unable to determine what tests are related to your most recent code changes (and vice-versa). If you consider running an entire test suite for every code change, this may take up to a few minutes (or longer).

The other thing to be aware of is that in the case of errors (which will be the normal case as you are modifying your code/tests), test runs for e2e testing frameworks usually result in a timeout instead of a fast/immediate fail like you see in a unit testing framework where an expectation fails. For example, an e2e test fails because the testing framework is waiting for a specific dom element to be rendered, when that doesn't happen after x seconds, the test fails by timeout; this means that most of the time, individual test runs are going to take 8-10 seconds to run and be performing a lot of CPU cycles with no Wallaby results being shown in your editor until 10 seconds after you stop typing; this may be less (e.g. 1-2 seconds depending on your tests) if you've finished your changes and no tests are breaking.

--

If you're interested in seeing what the experience would look like with Wallaby, you could try using cypress e2e real-world sample. If you edit a test/code file, the tests re-executed by cypress would be similar to what Wallaby could calculate. You can also see what test run timings would look like in the event of a failure in a non-trial test (e.g. change test expectation in cypress/tests/ui/new-transaction.spec.ts).

We'll continue to watch as the e2e testing landscape evolves and look forward to some of these problems being solved so Wallaby can become a more viable tool in the e2e testing context.