Speed up test warmup time when large imports are being used in tests; reduce the time for "Your Tests Are Starting..."

[muratkeremozcan]

What would you like?

Can cypress have a mini internal build step for its support file and plugins (config) file. They don’t change often and Cypress could bundle and tree shake them and have one internal file to load until the next time the file changes.

While tuning up a front end app tests, I realized that large files (ex: a network helpers with many cy.intercepts) that import from other large files (ex: a test utils package that can generate many kinds of domain objects) that import from other large files (ex: faker with all locales vs faker with english only) significantly increase test warmup time (Your Tests Are Loading…). We are talking 5-10 seconds before a test starts.

This is a side issue, related to inefficient usage of Cypress test plugins, but together they are what slow down e2e test start up time by 10-20 seconds in our front-end apps.

Mind that if these large imports occur in files that run before any tests; commands.ts, e2e.ts, component.tsx, they will slow down each test.

Currently I have fine tuned the plugin imports, and I'm tackling the large files with inefficient imports

Even when we do our best though, and decrease the test warm up from 10-20 to 5-10, we will still hit a bottle neck at some point and can't optimize further.

Why is this needed?

Saving 5-10 seconds in each test from "Your Tests Are Starting..." would be a significant CI cost savings, and huge improvement for the developer experience in any test .

Other

Mind that that time does not count towards Cypress test duration, but it's real when you run a test locally, and it's real CI cost.

[lmiller1990]

I have some questions and thoughts.

Are we talking E2E or CT (or both)? They both work a bit differently, even when using webpack for both. Also, is this primarily motivated as an optimization for local usage (open mode) CI usage (run mode)?

Since CT is using a dev server, the common files (eg supportFile) should only be compiled once, at the start. Webpack Dev Server (WDS) keeps a cache of modules, it should be able to reuse one that has already been compiled. This means you should only see "Your Tests Are Starting..." once, for the first test run (well, it should only be taking a few seconds the first time - when changing to other specs, it should only need to bundle the spec file, not re-bundle the supportFile.

pluginsFile is only executed once, but that could be adding to the slowness - the double whammy of compiling a complex supportFile on top of executing a large pluginsFile.

E2E uses webpack-preprocessor, which is still webpack, but does not use a dev server - each spec is compiled "just in time" when it runs. I am not sure what this does with supportFile, but I think we have some caching logic.

I think the first thing we should do to improve performance here is find out what exactly is the bottleneck. I have done some exploration here, one thing not mentioned here is the cost of parsing large JS files. See this comment. Another idea might be not using a dev-server on CI at all, and just compiling everything and serving static files via regular HTTP. Startup time would actually be slower, but overall execution time might be faster.

[muratkeremozcan]

It can impact both e2e or CT. For our setup, E2e is the pain point. CT could initially start faster, but after the initial start, switching betweetn tests is fast as you have described.

Let me see what I can do about giving you access to the repository. Since we have an NDA, this should be possible.

[lmiller1990]

I've noticed webpack-preprocessor is massively slow. I moved one of our packages (packages/driver) to one of the community preprocessors using esbuild and it was 30% faster out of the box. I am not sure if this is an option for you, but it's night and day.

Are you on webpack 5? It's faster (but still not ⚡ fast).

[mirobo]

It can impact both e2e or CT. For our setup, E2e is the pain point. CT could initially start faster, but after the initial start, switching betweetn tests is fast as you have described.

Let me see what I can do about giving you access to the repository. Since we have an NDA, this should be possible.

I can not recommend enough trying esbuild in this case first. I had a similar problem where I imported a LOT of generated types and the built-in preprocessor was extremely slow. Once I changed the preprocessor to cypress-esbuild-preprocessor (I use this one: https://github.com/sod/cypress-esbuild-preprocessor but you could also use this: https://github.com/bahmutov/cypress-esbuild-preprocessor ) the preprocessing was "blazingly" fast :-D

Keep in mind that esbuild does not do type checking. You could add a separate step using "tsc" before running all tests..

[muratkeremozcan]

It can impact both e2e or CT. For our setup, E2e is the pain point. CT could initially start faster, but after the initial start, switching betweetn tests is fast as you have described. Let me see what I can do about giving you access to the repository. Since we have an NDA, this should be possible.

I can not recommend enough trying esbuild in this case first. I had a similar problem where I imported a LOT of generated types and the built-in preprocessor was extremely slow. Once I changed the preprocessor to cypress-esbuild-preprocessor (I use this one: https://github.com/sod/cypress-esbuild-preprocessor but you could also use this: https://github.com/bahmutov/cypress-esbuild-preprocessor ) the preprocessing was "blazingly" fast :-D

Keep in mind that esbuild does not do type checking. You could add a separate step using "tsc" before running all tests..

Spent about 20 mins, possibly works in a simpler repo but only God knows what goes on our Lerna monorepo that needs to retire.

[lmiller1990]

Basic example: very slow with a basic test import a large module, fakerjs.

https://user-images.githubusercontent.com/19196536/218920864-8e7a8aa3-7de9-457a-9139-529d10813185.mov

I ran with DEBUG=cypress:webpack* for some interesting info:

  cypress:webpack get /Users/lachlanmiller/code/dump/cypress-slow-repro/cypress/e2e/spec2.cy.js +0ms
  cypress:webpack already have bundle for /Users/lachlanmiller/code/dump/cypress-slow-repro/cypress/e2e/spec2.cy.js +0ms
GET /__cypress/tests?p=cypress/support/e2e.js 200 1.709 ms - -
  cypress:webpack finished bundling /Users/lachlanmiller/Library/Application Support/Cypress/cy/production/projects/cypress-slow-repro-ab9e04a8d953db6ec7767490f2db10f6/bundles/cypress/e2e/spec2.cy.js +433ms
Hash: 211dd11ce44bb73854f6
Version: webpack 4.46.0
Time: 431ms
Built at: 15/02/2023 1:38:26 pm
      Asset      Size  Chunks             Chunk Names
spec2.cy.js  9.39 MiB    main  [emitted]  main
Entrypoint main = spec2.cy.js
[0] multi ./cypress/e2e/spec2.cy.js 28 bytes {main} [built]
[./cypress/e2e/spec2.cy.js] 165 bytes {main} [built]
[./node_modules/@faker-js/faker/dist/esm/chunk-2ARF2KYP.mjs] 48.7 KiB {main} [built]
[./node_modules/@faker-js/faker/dist/esm/chunk-3ARMJFIB.mjs] 133 KiB {main} [built]
[./node_modules/@faker-js/faker/dist/esm/chunk-3BX74TNW.mjs] 32.7 KiB {main} [built]
[./node_modules/@faker-js/faker/dist/esm/chunk-4QDT4GR4.mjs] 229 KiB {main} [built]
[./node_modules/@faker-js/faker/dist/esm/chunk-4WRXY4YA.mjs] 25.7 KiB {main} [built]
[./node_modules/@faker-js/faker/dist/esm/chunk-572NMBBA.mjs] 114 KiB {main} [built]
[./node_modules/@faker-js/faker/dist/esm/chunk-5ZAQ2U6R.mjs] 16.2 KiB {main} [built]
[./node_modules/@faker-js/faker/dist/esm/chunk-7F72AJZR.mjs] 35.2 KiB {main} [built]
[./node_modules/@faker-js/faker/dist/esm/chunk-7M4JXUUT.mjs] 13.4 KiB {main} [built]
[./node_modules/@faker-js/faker/dist/esm/chunk-AC7CIJLS.mjs] 207 KiB {main} [built]
[./node_modules/@faker-js/faker/dist/esm/chunk-BDJXH623.mjs] 23.1 KiB {main} [built]
[./node_modules/@faker-js/faker/dist/esm/chunk-BDWC2LEF.mjs] 14 KiB {main} [built]
[./node_modules/@faker-js/faker/dist/esm/index.mjs] 2.79 KiB {main} [built]
    + 45 hidden modules
  cypress:webpack - compile finished for /Users/lachlanmiller/code/dump/cypress-slow-repro/cypress/e2e/spec2.cy.js, initial? true +10ms
GET /__cypress/tests?p=cypress/e2e/spec2.cy.js 200 445.283 ms - -

It claims to be taking Time: 431ms. It sure feels a lot longer, though. I wonder how much of the slowness is the bundling, and how much is the browser needing to pass a really large JS file.

To the original point, why are we even re-bundling the fakerjs module - I wonder if we can tell webpack to cache it, and only rebundle the test code (we can assume node_modules do not change, in the majority of cases).

Another interesting point worth noting:

 cypress:webpack webpackOptions: { mode: 'development', node: { global: true, __filename: true, __dirname: true }, module: { rules: [ [Object], [Object], [Object] ] }, resolve: { extensions: [ '.js', '.json', '.jsx', '.mjs', '.coffee' ], alias: { child_process: '/Users/lachlanmiller/Library/Caches/Cypress/12.5.1/Cypress.app/Contents/Resources/app/packages/server/node_modules/@cypress/webpack-batteries-included-preprocessor/empty.js', cluster: '/Users/lachlanmiller/Library/Caches/Cypress/12.5.1/Cypress.app/Contents/Resources/app/packages/server/node_modules/@cypress/webpack-batteries-included-preprocessor/empty.js',

Since I did not specify a file:preprocessor plugin, it will use the default one, which is https://github.com/cypress-io/cypress/tree/develop/npm/webpack-batteries-included-preprocessor. That ships babel, which might add some overhead. I'll see if I can do some benchmarks with different webpack configurations. We may even be able to use https://github.com/esbuild-kit/esbuild-loader to go faster!

[lmiller1990]

I did some quick benchmarks.

1. Defaults (webpack 4, babel, webpack-preprocessor-batteries-included)

const { defineConfig } = require("cypress");

module.exports = defineConfig({
  e2e: {
    setupNodeEvents(on, config) {
      return config;
    },
  },
});

2. webpack 5, no babel or webpack config

const { defineConfig } = require("cypress");
const webpackPreprocessor = require("@cypress/webpack-preprocessor");

module.exports = defineConfig({
  e2e: {
    setupNodeEvents(on, config) {

      on("file:preprocessor",  webpackPreprocessor({
        webpackOptions: {}
      }))

      return config;
    },
  },
});

3. esbuild preprocessor

const { defineConfig } = require("cypress");
const createBundler = require('@bahmutov/cypress-esbuild-preprocessor')

module.exports = defineConfig({
  e2e: {
    setupNodeEvents(on, config) {

      on("file:preprocessor",  createBundler())

      return config;
    },
  },
});

I ran this spec 20 times:

import { faker } from '@faker-js/faker';

describe('template spec', () => {
  it('passes', () => {
    cy.log(faker.name.firstName())
  })
})

Config File	Average time to bundle spec	Total run duration
1. defaults	255ms	40s
2. webpack 5 + no babel	233ms	41s
3. esbuild	42ms	22s

Seems like webpack does not get any faster without some significant work - either a plugin to do more aggressive caching, etc. The difference between webpack and esbuild is truly night and day - esbuild is twice as fast for run mode. I had to get this PR to use open mode, but this seems much quicker, too. https://github.com/bahmutov/cypress-esbuild-preprocessor/pull/245

Open mode

Showing how much quicker it is with esbuild - this is despite webpack-preprocessor having lots of optimizations, and esbuild-preprocessor just as a simple, unoptimized wrapper around esbuild. It's quite a bit quicker.

Webpack

https://user-images.githubusercontent.com/19196536/218920864-8e7a8aa3-7de9-457a-9139-529d10813185.mov

Esbuild

https://user-images.githubusercontent.com/19196536/218937873-47fcf473-ec3c-4e10-917a-0fb43fdb67c1.mov

@muratkeremozcan it might be worth spending a few hours to see what it would take to move your E2E tests to use esbuild. I am not sure how much faster we can make webpack - I'd even consider starting to look into using esbuild as the default for all of Cypress, it's really that much faster. This won't happen overnight, though, so trying to get esbuild going in your local environment might be a good investment.

If that's really impractical, we can explore more webpack-preprocessor improvements, but I am unsure of how much fast it will become, and how complex this is.

[lmiller1990]

Some more info, I tried esbuild for packages/driver in Nov 2022, about 15% faster out of the box. Commit: https://github.com/cypress-io/cypress/compare/lmiller/try-using-esbuild?expand=1. About 15% faster, not bad!

• Webpack (~6:30 for each container)
• esbuild (~5:58 for each container)

This is across many 5 runners, so 30s * 5 = 2.5min.

Slack thread (Cypress team only).

[muratkeremozcan]

Esbuild has worked really nicely in this sample repo https://github.com/muratkeremozcan/tour-of-heroes-react-cypress-ts.

I tried it internally, had to polyfill a few things. Getting stuck with jsonwebtoken . Reproduced the issue here https://github.com/muratkeremozcan/tour-of-heroes-react-cypress-ts/pull/219

git clone https://github.com/muratkeremozcan/tour-of-heroes-react-cypress-ts.git
cd tour-of-heroes-react-cypress-ts
git checkout test/esbuild-jsonwebtoken
yarn install
yarn cy:open-e2e

run the test "fails" to get the error crypto2.createHmac is not a function.

Wonder if enabling crypto at the polyfill package would fix it https://github.com/remorses/esbuild-plugins/blob/d9f6601a24dc4e0470046eda8c772e6523c52b96/node-modules-polyfill/src/polyfills.ts#L142

Appreciate any help on this.

[lmiller1990]

I tracked down the problem to the jwa library: https://github.com/auth0/node-jwa/blob/master/index.js#L132 (many layers down from jsonwebtoken. I tried injecting the crypto-browserify polyfill but no luck. Hmm...

[lmiller1990]

I debugged a little more - I hacked the NodePolyfills library to include the crypto polyfill, and ran into aBuffer` issue that has been discussed in various GH issues. I think the bug is upstream, we cannot fix it like this right now.

Maybe it's a sign - since this jsonwebtoken lib is really designed for Node.js usage, some options would be:

• actually use it in Node.js - use cy.task() to pass the data (sub, expiresIn) to a Task, and do the jwt.sign there. Return the token.
• use a jwt.sign function designed for browser usage. jwt.io lists some.

I personally like the first approach. I am not sure what the implications of polyfilling a security critical module like crypto is, which is why I am leaning towards just using the real Node.js implementation via a task.

Just some ideas - the whole "polyfill Node.js in the browser" in general has always felt pretty bad to me, but it's ingrained in JS culture (mainly because of browserify and webpack).

WDYT @muratkeremozcan?

[muratkeremozcan]

I debugged a little more - I hacked the NodePolyfills library to include the crypto polyfill, and ran into aBuffer` issue that has been discussed in various GH issues. I think the bug is upstream, we cannot fix it like this right now.

Maybe it's a sign - since this jsonwebtoken lib is really designed for Node.js usage, some options would be:

• actually use it in Node.js - use cy.task() to pass the data (sub, expiresIn) to a Task, and do the jwt.sign there. Return the token.
• use a jwt.sign function designed for browser usage. jwt.io lists some.

I personally like the first approach. I am not sure what the implications of polyfilling a security critical module like crypto is, which is why I am leaning towards just using the real Node.js implementation via a task.

Just some ideas - the whole "polyfill Node.js in the browser" in general has always felt pretty bad to me, but it's ingrained in JS culture (mainly because of browserify and webpack).

WDYT @muratkeremozcan?

Thank you! It worked well, here's the commit https://github.com/muratkeremozcan/tour-of-heroes-react-cypress-ts/pull/219/commits/036fb1f60b42b377e8e493196d6db57a5e798ac5.

Used cy.task, the other libs ran into similar or the same errors.

TL, DR; use polyfills and cy.task to migrate to esbuild preprocessor for Cypress.

Here is a sample PR https://github.com/muratkeremozcan/tour-of-heroes-react-cypress-ts/pull/219.

Tried this in our largest internal project, which has over 500 e2e tests. It worked really well. It will save literal days in engineer feedback time and CI minutes every year.

@lmiller1990 The workaround has worked for us. How do we go about requesting a general feature from Cypress so that esbuild is the default?

[lmiller1990]

@muratkeremozcan I made this to track: https://github.com/cypress-io/cypress/issues/25928

We have a pretty extensive internal process for major projects like this, so we will be picking it up. I'd like to see it in Q2, but cannot commit to that right now - once we have more info on the ETA, complexity, etc, I will post in #25928.

[indrif]

I added the https://github.com/bahmutov/cypress-esbuild-preprocessor plugin and that lowered the cpu usage for our cypress setup from 300% to 100% and tests are loaded and reloaded blazingly fast. 🤷 We're happy now!

[lmiller1990]

That's great to hear. I don't have a timeline right now, but if we do this, it'll likely be opt in at first, something like preprocessor: 'esbuild'. We'd likely build off Gleb's great solution.

There are known edge cases and issues to solve for -- if anyone does encounter those when trying to use esbuild, please share them here. One is that webpack v4 default polyfills don't exist, and some things won't work with cy.stub and cy.spy, since es modules are immutable by default and don't always play well with Sinon, which mutates the modules.

[rszalski]

Whoa, what an amazing find. Thank you for posting this issue and to all who did research and testing on this.

We recently switched from Create React App to Vite and as a consequences had to/wanted to also ensure Cypress can correctly preprocess files, while respecting e.g. Vite's import.meta.env. We used https://github.com/mammadataei/cypress-vite and noticed that test loading has been even slower than previously on what I guess was Webpack 4. I did not do a lot of research of why is that. Maybe bugs, maybe our misconfiguration, or maybe an old version of Rollup that this plugin is using.

To solve this we also have used https://github.com/bahmutov/cypress-esbuild-preprocessor and the speed is mind blowing. We went from legit 40-50 seconds per each test spec open to 1s or even sub 1s times.

To anyone having a similar setup, the way to make this preprocessor work with import.meta.env is to define it:

  const bundler = createBundler({
    define: {
      'import.meta.env': '{}'
    }
  });
  on('file:preprocessor', bundler);

I know it might not be the best solution but works on our tests and hopefully helps someone.