Evaluate Cypress for Web deliverables

[awentzel]

Cypress provides a comprehensive testing suite with CLI/App/Dashboard that allows testing functional UI components. The thought here is that with this we can wire up functional testing with ease easy to maintain and review issues with the team from the portal. Applitools, may also still be used on top to test Visual UI scenarios or sites.

1. Learn and implement a sample Cypress implementation and get feedback from Collaborators

Deliverables: Include a remote branch with example implementation for packages that have a UI/UX.

1. Discuss with team the folder structure
2. Discuss with team the pros/cons
3. Discuss with team if there are other requirements we should consider.

[awentzel]

Folder Structure

Recommended default though, this does not take into account a mono-repository. Will determine the proposed structure once get into the code to try this out.

Pros (Cypress.io)

Cypress.io is a free, open-source front-end testing tool. Unlike the two previous frameworks, it’s a complete testing solution. It makes no claims of being a general test automation solution because its main focus is performing end-to-end testing of a web application’s user experience. In my research, there is no true end-to-end solution. Even if there was, there is an argument to be made that it's best to choose very specific frameworks, even if they add a dependency, because you can get best testing. Could have one thing for API testing and another for UX Testing.

Cypress.io’s main selling point is its blazing speed. It achieves this speed by running inside the browser. This means that it runs in the same execution loop as your application’s source code (as opposed to Puppeteer and Selenium, which run in a different process than the browser process that renders the application code). This means that Cypress.io doesn’t need to support the control protocols that are required for inter-process communication. This also resolves any system or network latency that slows down sending and receiving commands and responses.

1. Key Features (fast to iterate, easy to write, easy to debug, quick setup)
2. Why cypress
3. Key Differences
4. Trade offs
5. Best practices
6. Works well alongside our existing Jest, Enzyme, React environment
7. Supports Circle CI, Sauce Labs (coming first and soon)
8. Cross-browser support coming soon
9. TypeScript support here
10. Supports Jquery matchers

Cons

1. Current browser support not yet complete.No current cross-browser support, not planned. In fact, it is deprioritized for Edge/Safari/Firefox browsers refer.
2. No current support for vdiff, but, planned and proposal in progress by Applitools. Once available, we could drop Applitools.
3. Cypress.io’s method makes it harder to communicate with backend servers, which will mean that you’ll need to install additional modules.
4. Second, it doesn’t support testing multiple browser tabs.
5. It can only control a single browser instance at a time.
6. Visual regression testing not built in currently although applitools has a Cypress SDK that allows for this integration and their entire system is built around comprehensive visual regression testing or Percy

Resources

• https://medium.com/@wataruoguchi/running-e2e-tests-with-selenium-and-cypress-1de69b95d936
• https://medium.com/@shivambharadwaj/https-medium-com-shivambharadwaj-why-you-should-switch-to-cypress-for-modern-web-testing-5d3739a19e6
• https://www.slideshare.net/latinov/qa-challenge-accepted-cypress-vs-selenium
• https://pusher.com/sessions/meetup/js-monthly-london/testing-made-easy-better-faster-stronger-unit-tests-and-integration-tests

User scenarios

A good example of how Jest/Cypress can be used together for functional testing. (in this example he's functionally testing the login/logout user experience in real time during development). 15 minutes for that part.I don't actually think we have a need for functional testing currently. But, hopefully this will help explain a good use case for Cypress. https://frontendmasters.com/courses/testing-react/

Spec

• Circle CI configuration details for storing artifacts
• Examples

[awentzel]

Example Implementations Cypress, React, Jest, Babel, Webpack

[awentzel]

Wiki

To install locally

$ npm install cypress --save-dev
$ npm install @cypress/webpack-preprocessor --save-dev

Configure tsconfig.json to include cypress type

  "compilerOptions": {
       "types": ["cypress"]
    }

Add new configuration file cyupress.json for cypress

{
  "projectId": "1csr87"
}

Create an .env file at root of package with the "record key". Ensure the .env is ignored by .gitignore.

CYPRESS_RECORD_KEY=<record key>

Update package.json scripts

"scripts": {
    "build:app": "webpack --progress",
    "integration-tests": "webpack --output-filename out.js --entry ./cypress/integration/*.ts",
    "integration-tests:open": "cypress open",
    "integration-tests:run": "node build/run-cypress.js",
}

Create a script file to execute both Webpack development servers. One for the App and one for Cypress testing.

const spawn = require('child_process').spawn
const webpack = require('webpack');
const webpackConfig = require('../webpack.config');
const WebpackDevServer = require('webpack-dev-server');
const envs = require('dotenv').config();
const cypress = require('cypress');

// Validate environment variables are secure and properly set
if(envs.error) {
  if(process.env.CYPRESS_RECORD_KEY.length == 0)
    throw envs.error;
  else
    console.log('Need to add CYPRESS_RECORD_KEY to CI');
}
console.log(envs.parsed);

// Start Webpack development server for the application as required by Cypress
function startServer() {
  return new Promise((resolve, reject) => {
    new WebpackDevServer(webpack(webpackConfig), {})
    .listen(7001, 'localhost', function(err) {
      resolve()
    }).on('error', (error) => {
      reject(error)
    })
  })
}

var server = startServer();

server.then(function() {

  setTimeout(() => {
    const cypress = spawn("cypress", ["run", "--record", "--key", process.env.CYPRESS_RECORD_KEY], {stdio: "inherit"});
    console.log(cypress.stdout);
  }, 10000);
})

Startup Web server for App and for Cypress and auto-run tests with dashboard integration
```bash
$ npm run integration-tests:run

Open Cypress while Web server is running for manual testing

$ npm run integration-tests:open

[awentzel]

Ready for review @chrisdholt, @nicholasrice , @janechu

[chrisdholt]

@awentzel Thanks! A few questions:

1. What determines the project Id? Is there a way that it can be more intuitive and project based?
2. Can you share some of the purpose for running two webpack dev servers simultaneously?
3. On the script definitions side of things, does build:test actually build the test or does it functionally run? I would suggest naming the npm task specific to what is happening. Typically when we use build it is packaging it up and not executing the file with webpack. Want to be sure I understand what is happening so we can be consistent.

[awentzel]

@chrisdholt

1. Project ID is created by the dashboard on Cypress upon initial sign-up, which I've already done.
2. The first Webpack dev server is to run the application (msft-docs, for example). The second is to run Cypress integration tests against that first server. It may be possible to combine, but, it depends on what we're planning on testing against. For example, we could test against development site or an already hosted production website.
3. build:test has been removed to bring inline with other tests. Code updated for package.json above.

[chrisdholt]

@awentzel I notice that Cypress stores video files in the repo locally. Is that correct? That would add a ton of weight to the repo itself. Can we store it outside the repo as an alternative? If we want the video functionality and the only way to have it is stored in the repo - I'm not sure we want to include it.

[awentzel]

@chrisdholt It does store when it runs locally during development testing run and so would be included in the source. It does delete all of these files at the start of execution so it's doesn't yet do visual regressing diff testing. When running on the CI it would be available and stored on the CI with the history of that job. While these tend to be large files, the way GitHub is architected this is not a huge perf cost when uploading and downloading. The benefits I believe would outweigh the cost. There is a feature to to turn off video feature for local testing and only use that in CI.

More details https://docs.cypress.io/guides/guides/screenshots-and-videos.html

[awentzel]

@janechu @chrisdholt @nicholasrice last chance to provide thoughts on this Proposal before it's approved. Thanks

[chrisdholt]

Don't we need approvals in order for this to be approved?

[chrisdholt]

Still curious if we need two webpack servers. Could we use one of the preprocessors instead? https://github.com/cypress-io/cypress-example-recipes/tree/master/examples/preprocessors__typescript-webpack

[chrisdholt]

@awentzel does Cypress support multiple frameworks (angular, vue, polymer)? More of a question for future proofing. I think a great thing about our unit testing with Jest and Enzyme is that it supports multiple frameworks. If Cypress supports multiple frameworks I think that's a solid in support of it.

Were there any alternatives that seemed to compete with Cypress outside of the Applitools investigation? Anything you can provide in terms of trade-offs, etc?

[awentzel]

@chrisdholt We're already using the preprocessor. Cypress is 100% JavaScript and no Selenium/WebDriver required. There is a link above for trade-offs. I reviewed the top 10 equivalents. This was the best.

During implementation, it can be investigated if 2 webpacks are really required. Cypress performs better than all the other frameworks which is one of its key selling points.

[awentzel]

TestCAFE is probably the closest competitor. The documentation, API, and gui isn't quite as good as /cypress. which literally has some of the best documentation I've seen. Very well done project. It's performance is outstanding with clear fairly straightforward implementation that doesn't require a lot of code. It's my recommendation.

[janechu]

I'm a little late jumping on here but I feel like it's a big miss to go with an E2E testing solution that misses the ability to check on safari and firefox. It's more of a E2ChromiumBasedBrowsers.

I think we need to go for a solution that doesn't box us into an API but provides more flexibility. There are too many trade-offs for me with missing these browsers.

[awentzel]

Ability to support multiple browsers is in the pipeline and was a link I provided a while back to upvote the feature. If you haven't yet done so, you should. This is by far the best framework, most flexible, most robust, and most performant of anything on the market today. It's new, a non-selenium webdriver solution that allows us to test anything in the browser, even the network itself. We can still test other browsers using Applitools during the visual regression testing until then. This is a stepping stone and we should be focused on functional regression testing which we don't have today. This steps us into a future framework that is very well supported, outstanding documentation and is as close as anything gets to E2E. No other framework comes anywhere near this close to E2E. And knowing that E2E is in the works it puts us in a place where we can have a stable API for a while to come.

[awentzel]

https://github.com/cypress-io/cypress/issues/310 to upvote cross-browser functionality.

[janechu]

@awentzel so that issue has been open since Nov 2016. If/when they get to it is hazy at best, at the bottom of their issue on the topic it has been deemed not a priority.

They have the edge in performance and ease of use, but that does not override what I consider to be a core requirement of actual cross browser testing.

Edit: Additionally, this being a Microsoft project, I would argue that it's a must to support IE and Edge for E2E testing

[awentzel]

@janechu I actually agree with their assessment. It's most important that functionality is not broken and can be quickly and accurately tested. Cypress core focus is integrated regression testing of functionality with snapshots and videos. Which we don't have today.

Applitool's core strength is testing across devices, across screens, and viewports. So if you believe that IS a requirement for FAST-DNA it can be accomplished here instead with ease and host all the images on their end. There is another proposal outstanding for Applitools. Both were intended to compliment one another given there isn't a single service out there today that does great E2E.

Furthermore, almost all current and latest releases browsers are moving towards Chromium, Chrome, , Canary, and Electron (headless by default) most choosing Blink.

I disagree that this is a Microsoft project. It's open source and thus open for anyone to use. Sure there are teams internal to Microsoft using this. But, there are teams outside Microsoft using as well.

[awentzel]

I did discover Chroma which sounds about as great as Cypress. Though it's Selenium based and takes a dependency on Storybook. We investigated Storybook and wouldn't fit into our architecture or philosophy for testing components. That said, i'm discussing with Chroma a partnership where they might build an SDK for us so we don't have to take a dependency on Storybook. Waiting to hear back from Chroma as the sales team is meeting with the engineering team to discuss further details.

[awentzel]

Recent email exchanges with Cypress have let to a discussion they that are working on an engine for FireFox and IE. I requested Edge as well.

Chroma has updated an API for us to use their platform without the dependency on Chroma. So I'll be investigating Chroma against Cypress.io. Both are JavaScript driven not Selenium.