Support for Cypress.io

[lgandecki]

Hey @ArtemGovorov Could you please elaborate in a few sentences what do you mean by this?

[ArtemGovorov]

My thoughts were that we could create a new Wallaby test runner for Cypress, so that when writing Cypress.io tests with Wallaby one could have the same experience as writing Jest/Mocha/etc. tests with Wallaby.

[d3lm]

That'd be really cool. I thought about the exact same thing. My idea was, if Wallaby didn't support it out of the box, to create some kind of plugin that connects to Cypress and reports back the test results to Wallaby, which lines and what tests are succeeding and failing. As @ArtemGovorov said, having the same experience as writing other tests with Wallaby. So you can stay in your editor, write a test and get immediate feedback on your test. The only problem is that e2e tests are generally slower than unit tests and the feedback loop takes a little longer. But still it would be a great experience to have something like this.

[jmeischner-daparto]

Any news concerning this?

[smcenlly]

We haven't started to look at integrating Cypress with Wallaby but expect to be able to take a look over the coming months.

[xeger]

Although it's true that browser-based tests tend to be slower NodeJS tests, my team is already doing TDD using Cypress with stubbed network requests; in headless mode (which I assume Wallaby would use), startup cost is 1-3 seconds on a recent MacBook and individual test cases take 1-3 seconds. This is fast enough that I already practice TDD using Cypress' headed runner. Integrating test results into my editor would be superb!

Significant caveats I can think of: 1) Cypress input files can be written in numerous languages other than JS (e.g. Gherkin); if Wallaby relies on line numbers in test-result files, the correlation might not hold. 2) Code coverage is basically impossible to compute with browser-based tests, and I know that Wallaby prides itself on its coverage reporting. 3) Wallaby would need to use headless Cypress, which incurs a large startup cost (load Cypress modules, transpile tests, spawn browser subprocess) every time a given file is run. The ideal would be to reuse a launcher and browser process just like headed Cypress does, which saves 75% of the startup cost. 4) When Cypress tests fail, the user generally needs to see the screenshot or video in order to understand what went wrong. Artifacts are available on disk, but it might be difficult to display these rich media inline in an editor. (Perhaps a link to open in an external viewer would support?)

None of these is a showstopper, but you can see that there'd be a significant challenge to meshing Wallaby and Cypress together. Nevertheless, I'm excited about the prospect!

[arazasplunk]

• 1 In many organizations, there are separate qa and dev team. If the QA team is writing tests in cypress, it would be extremely useful to get code coverage reports that would help us write better tests to increase the test coverage.

[42tg]

It is maybe possible for Cypress to generate an istanbul report wich is shared with wallabyjs ? And limit the taskrunner to run just one worker wich execudes cypress peroiodically.

[NicholasBoll]

It is definitely possible to instrument code for Cypress. It involves instrumenting the built files Cypress is running against. https://docs.cypress.io/guides/tooling/code-coverage.html#Introduction

We did it at my last company. We didn't use Wallaby for code coverage, but used instabul + unit tests for unit test coverage and Cypress + instrumented documentation code and merged the reports in CI.

[jagreehal]

would be awesome if this happened!

[metasean]

Most of our tests are in Cypress, and we're moving toward having our unit tests in Cypress (e.g. https://github.com/cypress-io/cypress-example-recipes/tree/master/examples/unit-testing__application-code).

This and lack of Vue SFC support were the primary reasons I stopped using Wallaby and Quokka. With there now being Vue SFC support in Wallaby, this is the only reason I haven't gone back to using Wallaby.

The last update on this issue from a team member indicated that you would be looking into providing Cypress support in early 2019. It's been two years now, do y'all have any updates on if/when there might be Cypress support?

[smcenlly]

@metasean - we don't have any update on this. While we have been keeping up to date with new Cypress features, the last time we took a good look at technical integration was in mid-2019. At the time, Wallaby with Cypress was less of a priority for us because e2e tests are less suitable for Wallaby vs. faster component or unit tests.

Now that Cypress has component testing and unit testing, we're going to take another look at technical integration around July 2021. At this point, we don't know exactly what's possible, timeframe or if we'll need some support/changes from the Cypress team.

[metasean]

@smcenlly - Thank you for the update! 😄👍

I'm obviously a bit disappointed that there isn't something more definitive, but simply knowing that you'll be reviewing the possibility in July is hopeful🤞 and let's me know when to check back 📆 .

[DaniBedz]

This is definitely something that I have been interested in for a long time. I'm a huge fan of Wallaby and Quokka, it would really speed up writing e2e tests for our team to be able to leverage them as part of our process.

[quantuminformation]

Cypress support seems an order of magnitude more difficult then jest etc

[smcenlly]

We did some investigation in the last week of June and lack of multiple preprocessor is a show stopper for us right now.

There are a few other things that we weren't happy about re: integration - mostly around the high level of configuration that can be applied to cypress and we are moving towards all of our integrations being turn-key (zero configuration), which would make this a step backwards for our users. Having said this, it's not something limiting us moving forward but the preprocessor is.

On top of these things, the time it takes cypress to run tests (and please let us know if this is different for your project), the time it takes Cypress to run your tests means that the time from editing code to providing feedback will be 10+ seconds instead of almost instant with the other runners we support. We know from experience this can lead to a poor experience with Wallaby.

[greenmooseSE]

the time it takes Cypress to run your tests means that the time from editing code to providing feedback will be 10+ seconds

Maybe it could help if one could e.g. via some customizable "engine modes" e.g. "All tests", "Non Cypress Tests" (where each engine mode can filter Cypress tests based on whatever criteria, e.g. location of tests) ? Then users can do "standard" wallabyJS coding with "Non Cypress Tests" engine mode and then "finalize" work by enabling the slow Cypres tests and adding "a few" of those.