Speed up a preparation phase of the following test

[daedalius]

Hi Gleb! Yesterday I've already experimented with the v4 version and noticed that tests run slow enough.

For example: git@github.com:daedalius/cypress-react-test.git Here are 10 copies of the same test. I run them in a row:

npm run test:ci

On the timeline it takes:

• ~12 seconds for the Cypress start (it is ok. But for the cold start after Windows restarting it takes ~34 seconds)
• ~3 seconds for the first test preparation 🤔
• less than a second to run it (it is ok too)

and so on:

• ~3 seconds for next test preparation 🤔
• less than a second to run it

Total times: Windows:

1:08 for the cold start after Windows restart 0:42 for the next run

mac os:

0:38 for the cold start after a system reboot 0:34 for the next run

The output says that clean test time is 3 seconds (around ~0.3 per test). Is there any way to speed up the test preparation phase?

You know, I am in love with Cypress and this repo in particular. But it's hard to debate with colleagues who argue that Jest does 500 tests in 20 seconds.

This is maybe important:

• I've noticed in the Windows Task Manager that Antimalware Service Executable activates during the test run (especially for the cold one after restart).
• Specs: Windows: Ryzen 2600x, NVME, Windows 10 build 1809 Mac: MacBook Pro 2019, i7 6-cores 2.6 - 4.5 GHz, SSD

[bahmutov]

Great feedback @daedalius appreciate it, and we will look at the bundling and why it takes so long to start a test, so stay tuned

[bahmutov]

The support and spec bundles are extremely large. I wonder if we double pack React and other goodies

[bahmutov]

In this repo I am including testing-library/cypress for some demo tests ... and that is the cause of the giant support file, ha

[bahmutov]

Did some experiments with webpack bundling speed in https://github.com/bahmutov/webpack-bundle-experiment

Then looked the current bundling speed:

First load has support and spec files.

// support file
Hash: 64d6b36badf39ea37512
Version: webpack 4.33.0
Time: 1497ms
Built at: 05/06/2020 9:47:31 PM
   Asset      Size  Chunks             Chunk Names
index.js  14.7 KiB    main  [emitted]  main
Entrypoint main = index.js
[0] multi ./cypress/support/index.js 28 bytes {main} [built]
[./cypress/support/index.js] 43 bytes {main} [built]
[./dist/hooks.js] 1.05 KiB {main} [built]

// spec file
Hash: 79f7ea729a41d3ac5c4d
Version: webpack 4.33.0
Time: 5520ms
Built at: 05/06/2020 9:47:35 PM
          Asset   Size  Chunks             Chunk Names
counter-spec.js  3 MiB    main  [emitted]  main
Entrypoint main = counter-spec.js
[0] multi ./cypress/component/basic/counter-set-state/counter-spec.js 28 bytes {main} [built]
[./cypress/component/basic/counter-set-state/counter-spec.js] 1.13 KiB {main} [built]
[./cypress/component/basic/counter-set-state/counter.jsx] 3.97 KiB {main} [built]
[./dist/getDisplayName.js] 1.67 KiB {main} [built]
[./dist/index.js] 4.07 KiB {main} [built]
[./dist/utils.js] 2.96 KiB {main} [built]
[./node_modules/object-assign/index.js] 2.17 KiB {main} [built]
[./node_modules/prop-types/checkPropTypes.js] 3.95 KiB {main} [built]
[./node_modules/prop-types/lib/ReactPropTypesSecret.js] 311 bytes {main} [built]
[./node_modules/react-dom/cjs/react-dom.development.js] 709 KiB {main} [built]
[./node_modules/react-dom/index.js] 1.32 KiB {main} [built]
[./node_modules/react/cjs/react.development.js] 65.9 KiB {main} [built]
[./node_modules/react/index.js] 189 bytes {main} [built]
[./node_modules/scheduler/index.js] 197 bytes {main} [built]
[./node_modules/scheduler/tracing.js] 213 bytes {main} [built]
    + 2 hidden modules

So the spec is 5 seconds! and is 3 MiB

On file edit, the spec is rebuilt in 1 second

// spec file on rebuild
Hash: c1437ccddeb7749ce296
Version: webpack 4.33.0
Time: 988ms
Built at: 05/06/2020 9:48:13 PM
          Asset   Size  Chunks             Chunk Names
counter-spec.js  3 MiB    main  [emitted]  main
Entrypoint main = counter-spec.js
[0] multi ./cypress/component/basic/counter-set-state/counter-spec.js 28 bytes {main}
[./cypress/component/basic/counter-set-state/counter-spec.js] 1.12 KiB {main} [built]
[./cypress/component/basic/counter-set-state/counter.jsx] 3.97 KiB {main}
[./dist/getDisplayName.js] 1.67 KiB {main}
[./dist/index.js] 4.07 KiB {main}
[./dist/utils.js] 2.96 KiB {main}
[./node_modules/object-assign/index.js] 2.17 KiB {main}
[./node_modules/prop-types/checkPropTypes.js] 3.95 KiB {main}
[./node_modules/prop-types/lib/ReactPropTypesSecret.js] 311 bytes {main}
[./node_modules/react-dom/cjs/react-dom.development.js] 709 KiB {main}
[./node_modules/react-dom/index.js] 1.32 KiB {main}
[./node_modules/react/cjs/react.development.js] 65.9 KiB {main}
[./node_modules/react/index.js] 189 bytes {main}
[./node_modules/scheduler/index.js] 197 bytes {main}
[./node_modules/scheduler/tracing.js] 213 bytes {main}
    + 2 hidden modules

Notice that only the file itself is [built] which is good, but still requires 1 second to output the bundle.

[bahmutov]

Confirmed: if we disable inline source maps with Webpack option devtool: false, then refreshes are pretty snappy: about 50-70ms.

Also see https://github.com/bahmutov/webpack-bundle-experiment#vendor-bundles

If we want inline source maps, but split the vendor bundle (node_modules), then we can still get 20ms reload on source file change 🚀

[bahmutov]

Note: Webpack build timings in the project require https://github.com/cypress-io/cypress-webpack-preprocessor/pull/81

[bahmutov]

Looking at timings in Material UI autocomplete-spec.js

Start

DEBUG=cypress:webpack:stats npm run cy:open

Support file

Hash: 38a7a057263947b84f61
Version: webpack 4.33.0
Time: 12672ms
Built at: 05/07/2020 11:54:45 AM
   Asset     Size  Chunks                    Chunk Names
index.js  1.1 MiB       0  [emitted]  [big]  main
Entrypoint main [big] = index.js
 [36] (webpack)/buildin/global.js 920 bytes {0} [built]
 [93] multi ./cypress/support/index.js 28 bytes {0} [built]
 [94] ./cypress/support/index.js 94 bytes {0} [built]
 [95] ./dist/hooks.js 1.05 KiB {0} [built]
 [96] ./node_modules/@testing-library/cypress/add-commands.js 185 bytes {0} [built]
 [97] ./node_modules/@testing-library/cypress/dist/add-commands.js 406 bytes {0} [built]
 [98] ./node_modules/@testing-library/cypress/dist/index.js 10.8 KiB {0} [built]
 [99] ./node_modules/@babel/runtime/helpers/interopRequireDefault.js 162 bytes {0} [built]
[100] ./node_modules/@babel/runtime/helpers/extends.js 442 bytes {0} [built]
[101] ./node_modules/@babel/runtime/helpers/objectWithoutPropertiesLoose.js 399 bytes {0} [built]
[102] ./node_modules/@testing-library/dom/dist/@testing-library/dom.esm.js 63.8 KiB {0} [built]
[103] ./node_modules/node-libs-browser/node_modules/process/browser.js 4.98 KiB {0} [built]
[104] ./node_modules/@babel/runtime/helpers/esm/extends.js 477 bytes {0} [built]
[105] ./node_modules/pretty-format/build-es5/index.js 128 KiB {0} [built]
[355] ./node_modules/@testing-library/cypress/dist/utils.js 547 bytes {0} [built]
    + 341 hidden modules

Spec file

Hash: d9b89085523e89490649
Version: webpack 4.33.0
Time: 11506ms
Built at: 05/07/2020 11:54:44 AM
               Asset      Size  Chunks                    Chunk Names
autocomplete-spec.js  2.12 MiB       0  [emitted]  [big]  main
Entrypoint main [big] = autocomplete-spec.js
  [0] ./node_modules/react/index.js 189 bytes {0} [built]
  [1] ./node_modules/@babel/runtime/helpers/esm/extends.js 477 bytes {0} [built]
  [2] ./node_modules/prop-types/index.js 722 bytes {0} [built]
  [8] ./node_modules/react-dom/index.js 1.32 KiB {0} [built]
 [78] multi ./cypress/component/advanced/material-ui-example/autocomplete-spec.js 28 bytes {0} [built]
 [79] ./cypress/component/advanced/material-ui-example/autocomplete-spec.js 2.32 KiB {0} [built]
 [80] ./dist/index.js 4.07 KiB {0} [built]
 [81] ./node_modules/react/cjs/react.production.min.js 9.23 KiB {0} [built]
 [85] ./dist/getDisplayName.js 1.67 KiB {0} [built]
 [86] ./dist/utils.js 2.96 KiB {0} [built]
 [87] ./node_modules/@material-ui/core/esm/TextField/index.js 280 bytes {0} [built]
 [88] ./node_modules/@material-ui/core/esm/TextField/TextField.js 13.3 KiB {0} [built]
[197] ./node_modules/@material-ui/lab/esm/Autocomplete/index.js 1.59 KiB {0} [built]
[198] ./node_modules/@material-ui/lab/esm/Autocomplete/Autocomplete.js 28.8 KiB {0} [built]
[229] ./cypress/component/advanced/material-ui-example/top-100-movies.js 4.87 KiB {0} [built]
    + 215 hidden modules

[bahmutov]

Moved testing-library/cypress from support file into the spec that uses it

Support bundling 12 seconds -> 1.5 seconds

Hash: ebec4a4f12d424d5d37f
Version: webpack 4.33.0
Time: 1468ms
Built at: 05/07/2020 12:01:36 PM
   Asset    Size  Chunks             Chunk Names
index.js  11 KiB       0  [emitted]  main
Entrypoint main = index.js
[0] multi ./cypress/support/index.js 28 bytes {0} [built]
[1] ./cypress/support/index.js 43 bytes {0} [built]
[2] ./dist/hooks.js 1.05 KiB {0} [built]

Spec timing (same)

Hash: d9b89085523e89490649
Version: webpack 4.33.0
Time: 10614ms
Built at: 05/07/2020 12:01:45 PM
               Asset      Size  Chunks                    Chunk Names
autocomplete-spec.js  2.12 MiB       0  [emitted]  [big]  main
Entrypoint main [big] = autocomplete-spec.js
  [0] ./node_modules/react/index.js 189 bytes {0} [built]
  [1] ./node_modules/@babel/runtime/helpers/esm/extends.js 477 bytes {0} [built]
  [2] ./node_modules/prop-types/index.js 722 bytes {0} [built]
  [8] ./node_modules/react-dom/index.js 1.32 KiB {0} [built]
 [78] multi ./cypress/component/advanced/material-ui-example/autocomplete-spec.js 28 bytes {0} [built]
 [79] ./cypress/component/advanced/material-ui-example/autocomplete-spec.js 2.32 KiB {0} [built]
 [80] ./dist/index.js 4.07 KiB {0} [built]
 [81] ./node_modules/react/cjs/react.production.min.js 9.23 KiB {0} [built]
 [85] ./dist/getDisplayName.js 1.67 KiB {0} [built]
 [86] ./dist/utils.js 2.96 KiB {0} [built]
 [87] ./node_modules/@material-ui/core/esm/TextField/index.js 280 bytes {0} [built]
 [88] ./node_modules/@material-ui/core/esm/TextField/TextField.js 13.3 KiB {0} [built]
[197] ./node_modules/@material-ui/lab/esm/Autocomplete/index.js 1.59 KiB {0} [built]
[198] ./node_modules/@material-ui/lab/esm/Autocomplete/Autocomplete.js 28.8 KiB {0} [built]
[229] ./cypress/component/advanced/material-ui-example/top-100-movies.js 4.87 KiB {0} [built]
    + 215 hidden modules

[bahmutov]

In webpack options disable adding inline source maps

const webpackOptions = {
  devtool: false
}

Spec bundling time 11 -> 8 seconds

Hash: 7669b4e59b932c6f5b8d
Version: webpack 4.33.0
Time: 7759ms
Built at: 05/07/2020 12:10:12 PM
               Asset      Size  Chunks             Chunk Names
autocomplete-spec.js  1.78 MiB    main  [emitted]  main
Entrypoint main = autocomplete-spec.js
[0] multi ./cypress/component/advanced/material-ui-example/autocomplete-spec.js 28 bytes {main} [built]
[./cypress/component/advanced/material-ui-example/autocomplete-spec.js] 2.32 KiB {main} [built]
[./cypress/component/advanced/material-ui-example/top-100-movies.js] 4.87 KiB {main} [built]
[./dist/getDisplayName.js] 1.67 KiB {main} [built]
[./dist/index.js] 4.07 KiB {main} [built]
[./dist/utils.js] 2.96 KiB {main} [built]
[./node_modules/@babel/runtime/helpers/esm/defineProperty.js] 335 bytes {main} [built]
[./node_modules/@babel/runtime/helpers/esm/extends.js] 477 bytes {main} [built]
[./node_modules/@material-ui/core/esm/TextField/TextField.js] 13.3 KiB {main} [built]
[./node_modules/@material-ui/core/esm/TextField/index.js] 280 bytes {main} [built]
[./node_modules/@material-ui/lab/esm/Autocomplete/Autocomplete.js] 28.8 KiB {main} [built]
[./node_modules/@material-ui/lab/esm/Autocomplete/index.js] 1.59 KiB {main} [built]
[./node_modules/react-dom/index.js] 1.32 KiB {main} [built]
[./node_modules/react/cjs/react.development.js] 65.9 KiB {main} [built]
[./node_modules/react/index.js] 189 bytes {main} [built]
    + 218 hidden modules

Touching the file to force refresh

Hash: 7669b4e59b932c6f5b8d
Version: webpack 4.33.0
Time: 135ms
Built at: 05/07/2020 12:12:06 PM
               Asset      Size  Chunks  Chunk Names
autocomplete-spec.js  1.78 MiB    main  main
Entrypoint main = autocomplete-spec.js
[0] multi ./cypress/component/advanced/material-ui-example/autocomplete-spec.js 28 bytes {main}
[./cypress/component/advanced/material-ui-example/autocomplete-spec.js] 2.32 KiB {main} [built]
[./cypress/component/advanced/material-ui-example/top-100-movies.js] 4.87 KiB {main}
[./dist/getDisplayName.js] 1.67 KiB {main}
[./dist/index.js] 4.07 KiB {main}
[./dist/utils.js] 2.96 KiB {main}
[./node_modules/@babel/runtime/helpers/esm/defineProperty.js] 335 bytes {main}
[./node_modules/@babel/runtime/helpers/esm/extends.js] 477 bytes {main}
[./node_modules/@material-ui/core/esm/TextField/TextField.js] 13.3 KiB {main}
[./node_modules/@material-ui/core/esm/TextField/index.js] 280 bytes {main}
[./node_modules/@material-ui/lab/esm/Autocomplete/Autocomplete.js] 28.8 KiB {main}
[./node_modules/@material-ui/lab/esm/Autocomplete/index.js] 1.59 KiB {main}
[./node_modules/react-dom/index.js] 1.32 KiB {main}
[./node_modules/react/cjs/react.development.js] 65.9 KiB {main}
[./node_modules/react/index.js] 189 bytes {main}
    + 218 hidden modules

So pretty fast.

[daedalius]

Eventual Windows timings for tests in my repo:

Optimizations	Cold run (after restart)	Warm run
no optimizations	up to 1:37	0:42
+ devtool: false	1:04 - 1:30	0:37
+ no Windows Defender	0:34	0:34

Conclusions:

• Windows Defender is evil. It just randomizes timing results. 🤦‍♂️ Should we add a note about that?
• devtool: false saves ~12% of run time. 👍

Gleb, do you have any ideas of further speed up bundling? Is it possible to bundle all tests as one or bundle them in the background?

[bahmutov]

I cloned the repo and updated plugin to the latest (and removed separate webpack preprocessor, since now it is included). Find my fork in https://github.com/bahmutov/time-component-tests

Running with DEBUG=cypress:webpack:stats npx cypress run to see Webpack timings

Hash: 550d1f14959791aaf125
Version: webpack 4.43.0
Time: 988ms
Built at: 05/12/2020 4:06:13 PM
                        Asset      Size  Chunks                    Chunk Names
Datepicker.component9.spec.js  1.72 MiB       0  [emitted]  [big]  main
Entrypoint main [big] = Datepicker.component9.spec.js
  [1] ./node_modules/date-fns/esm/toDate/index.js 1.86 KiB {0} [built]
  [3] ./node_modules/react/index.js 190 bytes {0} [built]
  [6] ./node_modules/date-fns/esm/startOfWeek/index.js 2.22 KiB {0} [built]
 [11] ./node_modules/date-fns/esm/startOfDay/index.js 954 bytes {0} [built]
 [17] ./node_modules/date-fns/esm/addDays/index.js 1.17 KiB {0} [built]
 [18] ./node_modules/date-fns/esm/addMonths/index.js 1.63 KiB {0} [built]
 [19] ./node_modules/cypress-react-unit-test/dist/index.js 4.7 KiB {0} [built]
 [30] ./node_modules/react-dom/index.js 1.33 KiB {0} [built]
 [49] ./node_modules/react-datepicker/dist/react-datepicker.min.js 70 KiB {0} [built]
 [52] multi ./src/components/Datepicker/Datepicker.component9.spec.js 28 bytes {0} [built]
 [53] ./node_modules/react/cjs/react.production.min.js 6.52 KiB {0} [built]
 [57] ./node_modules/cypress-react-unit-test/dist/getDisplayName.js 1.76 KiB {0} [built]
 [58] ./node_modules/cypress-react-unit-test/dist/utils.js 3.12 KiB {0} [built]
 [59] ./node_modules/react-datepicker/dist/react-datepicker.css 510 bytes {0} [built]
[127] ./src/components/Datepicker/Datepicker.component9.spec.js + 1 modules 1.79 KiB {0} [built]
      | ./src/components/Datepicker/Datepicker.component9.spec.js 1.54 KiB [built]
      | ./src/components/Datepicker/Datepicker.jsx 232 bytes [built]
    + 113 hidden modules

WARNING in configuration
The 'mode' option has not been set, webpack will fallback to 'production' for this value. Set 'mode' option to 'development' or 'production' to enable defaults for each environment.
You can also set it to 'none' to disable any default behavior. Learn more: https://webpack.js.org/configuration/mode/

WARNING in asset size limit: The following asset(s) exceed the recommended size limit (244 KiB).
This can impact web performance.
Assets: 
  Datepicker.component9.spec.js (1.72 MiB)

WARNING in entrypoint size limit: The following entrypoint(s) combined asset size exceeds the recommended limit (244 KiB). This can impact web performance.
Entrypoints:
  main (1.72 MiB)
      Datepicker.component9.spec.js

WARNING in webpack performance recommendations: 
You can limit the size of your bundles by using import() or require.ensure to lazy load some parts of your application.
For more info visit https://webpack.js.org/guides/code-splitting/

  <Datepicker />
    ✓ renders text field. (129ms)
    ✓ renders desired placeholder text.
    ✓ renders chosen date.
    ✓ opens calendar after clicking on text field. (183ms)

Cannot find coverage file /Users/gleb/git/time-component-tests/.nyc_output/out.json
Skipping coverage report

  4 passing (464ms)

  (Results)

  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ Tests:        4                                                                                │
  │ Passing:      4                                                                                │
  │ Failing:      0                                                                                │
  │ Pending:      0                                                                                │
  │ Skipped:      0                                                                                │
  │ Screenshots:  0                                                                                │
  │ Video:        false                                                                            │
  │ Duration:     0 seconds                                                                        │
  │ Spec Ran:     components/Datepicker/Datepicker.component9.spec.js                              │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘

====================================================================================================

  (Run Finished)

       Spec                                              Tests  Passing  Failing  Pending  Skipped  
  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ ✔  components/Datepicker/Datepicker.co      528ms        4        4        -        -        - │
  │    mponent.spec.js                                                                             │
  ├────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ ✔  components/Datepicker/Datepicker.co      423ms        4        4        -        -        - │
  │    mponent10.spec.js                                                                           │
  ├────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ ✔  components/Datepicker/Datepicker.co      467ms        4        4        -        -        - │
  │    mponent2.spec.js                                                                            │
  ├────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ ✔  components/Datepicker/Datepicker.co      523ms        4        4        -        -        - │
  │    mponent3.spec.js                                                                            │
  ├────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ ✔  components/Datepicker/Datepicker.co      540ms        4        4        -        -        - │
  │    mponent4.spec.js                                                                            │
  ├────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ ✔  components/Datepicker/Datepicker.co      449ms        4        4        -        -        - │
  │    mponent5.spec.js                                                                            │
  ├────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ ✔  components/Datepicker/Datepicker.co      479ms        4        4        -        -        - │
  │    mponent6.spec.js                                                                            │
  ├────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ ✔  components/Datepicker/Datepicker.co      415ms        4        4        -        -        - │
  │    mponent7.spec.js                                                                            │
  ├────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ ✔  components/Datepicker/Datepicker.co      448ms        4        4        -        -        - │
  │    mponent8.spec.js                                                                            │
  ├────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ ✔  components/Datepicker/Datepicker.co      463ms        4        4        -        -        - │
  │    mponent9.spec.js                                                                            │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘
    ✔  All specs passed!                        00:04       40       40        -        -        - 

The same stats repeat for each spec.

[bahmutov]

Opening a spec runs slightly slower

$ DEBUG=cypress:webpack:stats npx cypress open
GET /__/ 200 5.289 ms - -
GET /__cypress/runner/cypress_runner.css 200 5.620 ms - -
GET /__cypress/runner/cypress_runner.js 200 1.798 ms - -
GET /__cypress/iframes/component/components/Datepicker/Datepicker.component.spec.js 200 4.701 ms - 795
GET /__cypress/runner/fonts/fa-solid-900.woff2 200 0.731 ms - 76120
Hash: 9ce7b994bf43d067ca67
Version: webpack 4.43.0
Time: 1890ms
Built at: 05/12/2020 4:09:48 PM
   Asset    Size  Chunks             Chunk Names
index.js  32 KiB       0  [emitted]  main
Entrypoint main = index.js
[0] multi ./cypress/support/index.js 28 bytes {0} [built]
[1] ./cypress/support/index.js 43 bytes {0} [built]
[2] ./node_modules/cypress-react-unit-test/support/index.js 255 bytes {0} [built]
[3] ./node_modules/cypress-react-unit-test/dist/hooks.js 1.29 KiB {0} [built]
[4] ./node_modules/@cypress/code-coverage/support.js 7.38 KiB {0} [built]
[5] ./node_modules/@cypress/code-coverage/support-utils.js 2.13 KiB {0} [built]

WARNING in configuration
The 'mode' option has not been set, webpack will fallback to 'production' for this value. Set 'mode' option to 'development' or 'production' to enable defaults for each environment.
You can also set it to 'none' to disable any default behavior. Learn more: https://webpack.js.org/configuration/mode/
GET /__cypress/tests?p=cypress/support/index.js-291 - - ms - -
GET /__cypress/tests?p=src/components/Datepicker/Datepicker.component.spec.js-583 - - ms - -
GET /__cypress/iframes/component/components/Datepicker/Datepicker.component.spec.js 200 1.428 ms - 795
GET /__cypress/tests?p=cypress/support/index.js-128 200 10.915 ms - -
Hash: 550d1f14959791aaf125
Version: webpack 4.43.0
Time: 2859ms
Built at: 05/12/2020 4:09:49 PM
                       Asset      Size  Chunks                    Chunk Names
Datepicker.component.spec.js  1.72 MiB       0  [emitted]  [big]  main
Entrypoint main [big] = Datepicker.component.spec.js
  [1] ./node_modules/date-fns/esm/toDate/index.js 1.86 KiB {0} [built]
  [3] ./node_modules/react/index.js 190 bytes {0} [built]
  [6] ./node_modules/date-fns/esm/startOfWeek/index.js 2.22 KiB {0} [built]
 [11] ./node_modules/date-fns/esm/startOfDay/index.js 954 bytes {0} [built]
 [17] ./node_modules/date-fns/esm/addDays/index.js 1.17 KiB {0} [built]
 [18] ./node_modules/date-fns/esm/addMonths/index.js 1.63 KiB {0} [built]
 [19] ./node_modules/cypress-react-unit-test/dist/index.js 4.7 KiB {0} [built]
 [30] ./node_modules/react-dom/index.js 1.33 KiB {0} [built]
 [49] ./node_modules/react-datepicker/dist/react-datepicker.min.js 70 KiB {0} [built]
 [52] multi ./src/components/Datepicker/Datepicker.component.spec.js 28 bytes {0} [built]
 [53] ./node_modules/react/cjs/react.production.min.js 6.52 KiB {0} [built]
 [57] ./node_modules/cypress-react-unit-test/dist/getDisplayName.js 1.76 KiB {0} [built]
 [58] ./node_modules/cypress-react-unit-test/dist/utils.js 3.12 KiB {0} [built]
 [59] ./node_modules/react-datepicker/dist/react-datepicker.css 510 bytes {0} [built]
[127] ./src/components/Datepicker/Datepicker.component.spec.js + 1 modules 1.79 KiB {0} [built]
      | ./src/components/Datepicker/Datepicker.component.spec.js 1.54 KiB [built]
      | ./src/components/Datepicker/Datepicker.jsx 232 bytes [built]
    + 113 hidden modules

WARNING in configuration
The 'mode' option has not been set, webpack will fallback to 'production' for this value. Set 'mode' option to 'development' or 'production' to enable defaults for each environment.
You can also set it to 'none' to disable any default behavior. Learn more: https://webpack.js.org/configuration/mode/

WARNING in asset size limit: The following asset(s) exceed the recommended size limit (244 KiB).
This can impact web performance.
Assets: 
  Datepicker.component.spec.js (1.72 MiB)

WARNING in entrypoint size limit: The following entrypoint(s) combined asset size exceeds the recommended limit (244 KiB). This can impact web performance.
Entrypoints:
  main (1.72 MiB)
      Datepicker.component.spec.js

WARNING in webpack performance recommendations: 
You can limit the size of your bundles by using import() or require.ensure to lazy load some parts of your application.
For more info visit https://webpack.js.org/guides/code-splitting/
GET /__cypress/tests?p=src/components/Datepicker/Datepicker.component.spec.js-726 - - ms - -
GET /__cypress/iframes/component/components/Datepicker/Datepicker.component.spec.js 200 1.471 ms - 795
GET /__cypress/tests?p=cypress/support/index.js-094 200 41.563 ms - -
GET /__cypress/tests?p=src/components/Datepicker/Datepicker.component.spec.js-740 200 41.024 ms - -
GET /__cypress/runner/fonts/fa-regular-400.woff2 200 0.827 ms - 13600
⚠️ file /Users/gleb/git/time-component-tests/.nyc_output/out.json has no coverage information
Did you forget to instrument your web application? Read https://github.com/cypress-io/code-coverage#instrument-your-application
Hash: 550d1f14959791aaf125
Version: webpack 4.43.0
Time: 1182ms
Built at: 05/12/2020 4:10:09 PM
                       Asset      Size  Chunks                    Chunk Names
Datepicker.component.spec.js  1.72 MiB       0  [emitted]  [big]  main
Entrypoint main [big] = Datepicker.component.spec.js
  [1] ./node_modules/date-fns/esm/toDate/index.js 1.86 KiB {0} [built]
  [3] ./node_modules/react/index.js 190 bytes {0} [built]
  [6] ./node_modules/date-fns/esm/startOfWeek/index.js 2.22 KiB {0} [built]
 [11] ./node_modules/date-fns/esm/startOfDay/index.js 954 bytes {0} [built]
 [17] ./node_modules/date-fns/esm/addDays/index.js 1.17 KiB {0} [built]
 [18] ./node_modules/date-fns/esm/addMonths/index.js 1.63 KiB {0} [built]
 [19] ./node_modules/cypress-react-unit-test/dist/index.js 4.7 KiB {0} [built]
 [30] ./node_modules/react-dom/index.js 1.33 KiB {0} [built]
 [49] ./node_modules/react-datepicker/dist/react-datepicker.min.js 70 KiB {0} [built]
 [52] multi ./src/components/Datepicker/Datepicker.component.spec.js 28 bytes {0} [built]
 [53] ./node_modules/react/cjs/react.production.min.js 6.52 KiB {0} [built]
 [57] ./node_modules/cypress-react-unit-test/dist/getDisplayName.js 1.76 KiB {0} [built]
 [58] ./node_modules/cypress-react-unit-test/dist/utils.js 3.12 KiB {0} [built]
 [59] ./node_modules/react-datepicker/dist/react-datepicker.css 510 bytes {0} [built]
[127] ./src/components/Datepicker/Datepicker.component.spec.js + 1 modules 1.79 KiB {0} [built]
      | ./src/components/Datepicker/Datepicker.component.spec.js 1.54 KiB [built]
      | ./src/components/Datepicker/Datepicker.jsx 232 bytes [built]
    + 113 hidden modules

WARNING in configuration
The 'mode' option has not been set, webpack will fallback to 'production' for this value. Set 'mode' option to 'development' or 'production' to enable defaults for each environment.
You can also set it to 'none' to disable any default behavior. Learn more: https://webpack.js.org/configuration/mode/

WARNING in asset size limit: The following asset(s) exceed the recommended size limit (244 KiB).
This can impact web performance.
Assets: 
  Datepicker.component.spec.js (1.72 MiB)

WARNING in entrypoint size limit: The following entrypoint(s) combined asset size exceeds the recommended limit (244 KiB). This can impact web performance.
Entrypoints:
  main (1.72 MiB)
      Datepicker.component.spec.js

Acceptable in my opinion

[bahmutov]

Disabling code coverage check (there was no instrumentation anyway) keeps the timing the same

[bahmutov]

Running with --browser chrome keeps the timing essentially the same

[bahmutov]

Of course, my old Mac Pro might be faster in this situation, but so far I don't see a show stopper in terms of bundling performance.

[daedalius]

Personally I think that ~3sec to each component test is acceptable too. But still, it is kinda weird that the actual test run time takes 10 times less than the test bundling phase (and it is a sequential process).

You may close the issue if there is nothing to improve.

[bahmutov]

@daedalius we are working on this, I think we can do much better, but nothing 100% immediate yet, so I will keep this issue open

[AndrewSouthpaw]

Just wanted to check in on this thread. I recently ran the react-todo-with-hooks unit testing example, and am still seeing a lot of overhead for each spec file. I really want this project to succeed because I love the idea of having a cypress runner for effectively unit tests, since the testing experience is so much better, but the time costs are just killing me.

For our project, where we are currently running only two small specs, the overhead is even more dramatic:

Granted, our codebase is much larger, but it's not that large, we're at ~70kloc. Turning off devtool did give us some performance gains:

But we're still looking at ~15 seconds per component spec file, which is really unsustainable for us.

Running on macOS, 2.9 GHz i9, 32 GB RAM.

[bahmutov]

@AndrewSouthpaw appreciate the timings - we are working on speeding up the bundling step. I have created prototype system for externalizing React and ReactDOM dependencies - and that really speeds things up, since every spec only has to bundle its own "small" dependencies. The Cypress team has created a different webpack preprocessor where all 3rd party dependencies are externalized. Unfortunately they got sidetracked by moving this repo, and doing some other unrelated work. Please keep an eye on this issue for updates.

[AndrewSouthpaw]

Thanks so much for the update, @bahmutov! Will keep an eye out. I appreciate all the Cypress team is doing to revolutionize frontend testing. 🙏

[oscar-b]

Hej @bahmutov! I just started playing around with component testing in Cypress but quickly ran into problems with our application. It seems like bundling takes almost 30 minutes and results in a 32Mb monster bundle. A big chunk of it seems to be caused by Font Awesome being bundled in the file, which normally is split into individual chunks for each icon and lazy loaded on demand.

  cypress:webpack finished bundling /Users/oscar/Library/Application Support/Cypress/cy/production/projects/web-d5ecf9daa93ef61ad0216aa43dc25af5/bundles/src/menu/BBBDesktopMenu.spec.tsx.js +640ms
Hash: 920ee608c7d882f4dfbb
Version: webpack 4.42.0
Time: 1528626ms
Built at: 2020-11-26 16:25:43
                     Asset      Size  Chunks             Chunk Names
BBBDesktopMenu.spec.tsx.js  32.5 MiB    main  [emitted]  main
Entrypoint main = BBBDesktopMenu.spec.tsx.js
[0] multi ./src/menu/BBBDesktopMenu.spec.tsx 28 bytes {main} [built]
[../node_modules/babel-preset-react-app/node_modules/@babel/runtime/helpers/esm/objectSpread2.js] 1.02 KiB {main} [built]
[../node_modules/babel-preset-react-app/node_modules/@babel/runtime/helpers/esm/slicedToArray.js] 397 bytes {main} [built]
[../node_modules/babel-preset-react-app/node_modules/@babel/runtime/helpers/esm/taggedTemplateLiteral.js] 225 bytes {main} [built]
[../node_modules/babel-preset-react-app/node_modules/@babel/runtime/helpers/interopRequireDefault.js] 147 bytes {main} [built]
[../node_modules/babel-preset-react-app/node_modules/@babel/runtime/helpers/interopRequireWildcard.js] 1.15 KiB {main} [built]
[../node_modules/cypress-react-unit-test/dist/index.js] 1.05 KiB {main} [built]
[../node_modules/cypress-react-unit-test/dist/mount.js] 7.61 KiB {main} [built]
[../node_modules/cypress-react-unit-test/dist/mountHook.js] 3.33 KiB {main} [built]
[../node_modules/react/cjs/react.development.js] 65.3 KiB {main} [built]
[../node_modules/react/index.js] 189 bytes {main} [built]
[./node_modules/@company/common/dist/common.es.js] 32.3 KiB {main} [built]
[./src/config/ConfigContext.tsx] 12.1 KiB {main} [built]
[./src/hooks/useApi.ts] 33.9 KiB {main} [built]
[./src/menu/BBBDesktopMenu.spec.tsx] 4.04 KiB {main} [built]
    + 3138 hidden modules

The project is using CRA and normal prod compile time is about 1 minute.

Is there anything we can do here, to make webpack behave better?

[bahmutov]

I have not seen such behavior but in general right now on demand lazy loading is not possible. The team at Cypress is working on very different bundling approach similar to webpack dev server instead of making the full bundle. What I would really really would like is an example repo showing this situation so we can validate and play with it

Sent from my iPhone

On Nov 26, 2020, at 11:20, Oscar Bolmsten notifications@github.com wrote:

 Hej @bahmutov! I just started playing around with component testing in Cypress but quickly ran into problems with our application. It seems like bundling takes almost 30 minutes and results in a 32Mb monster bundle. A big chunk of it seems to be caused by Font Awesome being bundled in the file, which normally is split into individual chunks for each icon and lazy loaded on demand.

cypress:webpack finished bundling /Users/oscar/Library/Application Support/Cypress/cy/production/projects/web-d5ecf9daa93ef61ad0216aa43dc25af5/bundles/src/menu/BBBDesktopMenu.spec.tsx.js +640ms Hash: 920ee608c7d882f4dfbb Version: webpack 4.42.0 Time: 1528626ms Built at: 2020-11-26 16:25:43 Asset Size Chunks Chunk Names BBBDesktopMenu.spec.tsx.js 32.5 MiB main [emitted] main Entrypoint main = BBBDesktopMenu.spec.tsx.js [0] multi ./src/menu/BBBDesktopMenu.spec.tsx 28 bytes {main} [built] [../node_modules/babel-preset-react-app/node_modules/@babel/runtime/helpers/esm/objectSpread2.js] 1.02 KiB {main} [built] [../node_modules/babel-preset-react-app/node_modules/@babel/runtime/helpers/esm/slicedToArray.js] 397 bytes {main} [built] [../node_modules/babel-preset-react-app/node_modules/@babel/runtime/helpers/esm/taggedTemplateLiteral.js] 225 bytes {main} [built] [../node_modules/babel-preset-react-app/node_modules/@babel/runtime/helpers/interopRequireDefault.js] 147 bytes {main} [built] [../node_modules/babel-preset-react-app/node_modules/@babel/runtime/helpers/interopRequireWildcard.js] 1.15 KiB {main} [built] [../node_modules/cypress-react-unit-test/dist/index.js] 1.05 KiB {main} [built] [../node_modules/cypress-react-unit-test/dist/mount.js] 7.61 KiB {main} [built] [../node_modules/cypress-react-unit-test/dist/mountHook.js] 3.33 KiB {main} [built] [../node_modules/react/cjs/react.development.js] 65.3 KiB {main} [built] [../node_modules/react/index.js] 189 bytes {main} [built] [./node_modules/@company/common/dist/common.es.js] 32.3 KiB {main} [built] [./src/config/ConfigContext.tsx] 12.1 KiB {main} [built] [./src/hooks/useApi.ts] 33.9 KiB {main} [built] [./src/menu/BBBDesktopMenu.spec.tsx] 4.04 KiB {main} [built]

• 3138 hidden modules

The project is using CRA and normal prod compile time is about 1 minute.

Is there anything we can do here, to make webpack behave better?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

[oscar-b]

I have not seen such behavior but in general right now on demand lazy loading is not possible. The team at Cypress is working on very different bundling approach similar to webpack dev server instead of making the full bundle. What I would really really would like is an example repo showing this situation so we can validate and play with it

I understand, I will see if I can lift out the icon loading (which I'm suspecting to be the culprit) into a separate repo. But it relies heavily on lazy loading, so I think the problem is just that it tries to bundle it all in one file. Using webpack dev server seems like a good approach, and should more closely match how CRA does it I guess.

[bahmutov]

Yeah, I have bunch of text already explaining how specs are bundled and how we can speed it up. I got to finish and put it on my blog

Sent from my iPhone

On Nov 26, 2020, at 11:31, Oscar Bolmsten notifications@github.com wrote:

 I have not seen such behavior but in general right now on demand lazy loading is not possible. The team at Cypress is working on very different bundling approach similar to webpack dev server instead of making the full bundle. What I would really really would like is an example repo showing this situation so we can validate and play with it

I understand, I will see if I can lift out the icon loading (which I'm suspecting to be the culprit) into a separate repo. But it relies heavily on lazy loading, so I think the problem is just that it tries to bundle it all in one file. Using webpack dev server seems like a good approach, and should more closely match how CRA does it I guess.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

[oscar-b]

Also to be a bit more precise, what I'm using is webpack dynamic imports, using import():

const icon = await import(
    /* webpackChunkName: "fontawesome/[request]" */
    `@fortawesome/pro-duotone-svg-icons/fa${fileName}.js`
);

[oscar-b]

@bahmutov Did some testing, and the unusual long build time is due to this code in clean-for-cypress.js:

webpackOptions.plugins.push(
  new webpack.optimize.LimitChunkCountPlugin({
    maxChunks: 1, // no chunks from dynamic imports -- includes the entry file
  }),
)

It seems webpack doesn't like that in combination with a couple of thousand dynamic imports, commenting it out and webpack builds in 17s (instead of 28m).

What's the problem with serving more than one bundle?

[taninnazar]

Hello @bahmutov , could you, please, update us how thing are going with fixing this issue ?

[bahmutov]

Things are going great and soon it will be faster than lightning

[AndrewSouthpaw]

This brings me much excitement. Thanks for all your hard work!

[denis-domanskii]

@bahmutov did you experimented with a concurrent run? I mean not cypress dashboard, but native mocha --parallel, on a single machine? I understand why Dashboard-like parallelization exists for e2e tests, but since Cypress trying to cover component, unit and api tests - probably you need to revise parallelization politic and allow it for non e2e? It's a default for Jest/Mocha/Ava and other unit-test frameworks. I have seen cypress-parallel project, but native solutions are always better :)

[denieler]

@bahmutov do you maybe have some rough estimations regarding those improvements? maybe we can help somehow?

[JessicaSachs]

Hey folks, this was fixed in the major rewrite we did to use your webpack dev server instead of rebuilding with the preprocessor from scratch every time.

Please upgrade because the performance is significantly better. From minutes to seconds for initial startup and total runtime. In open mode it’s milliseconds to switch and run tests. It’s as fast as your local dev server for initial startup and then caches everything between specs. It also uses babel cache to make incremental runs even faster. You can also disable source maps but you don’t really need to.

Headless CI will be slower than jest right now, because we bundle your application instead of mock out all of your CSS etc, but open mode is amazingly fast.

Any optimizations you add for local development will speed up your Cypress test runs. If you end up using a faster dev-server in the future (like Vite, etc) then your perf can become faster than jest.

It was released in early April with 7.0. You’ll need to read the migration guide here: https://docs.cypress.io/guides/references/migration-guide#Component-Testing

[Thaval]

Could you make this work for vue too? I started component testing today and the preparation to test one component takes more than 5 minutes. Gotta admit, it's a big projects with lots of dependencies and stuff, but component/unit testing should be fast. There needs to be some kind of caching I guess.

[JessicaSachs]

Could you make this work for vue too? I started component testing today and the preparation to test one component takes more than 5 minutes. Gotta admit, it's a big projects with lots of dependencies and stuff, but component/unit testing should be fast. There needs to be some kind of caching I guess.

Hey @Thaval, can you tell me what you mean by "the preparation to test one component"? Is this before the browser launches or during the loading spinner within Chrome?

In normal usage, we heavily cache the webpack builds + babel dependency graph across multiple launches. The performance of Cypress CT should be equal to that of your normal dev server setup. If you're stuck on the loading spinner in the browser while the component compiles and it's taking longer than your normal dev server, then your webpack setup is probably building for production, or is otherwise unoptimized.

If the browser hasn't launched yet and you're just waiting for your terminal to print anything, then it's probably due to an issue on Big Sur (or if you're not on Big Sur, then perhaps other OSs have this electron/node issue, too). To check if you have this issue, run the command DEBUG=cypress:* yarn cypress open-ct and see how long it takes before any logs print out. Another way to check if it's your OS or your build step is to try one of our boilerplates The Vite ones should start up within 1.5 seconds. The webpack ones will take a little longer (~2-5 seconds). A vanilla Vue CLI setup should take around 5-10 seconds

[Thaval]

Hi, with "preparation to test one component" I meant the build time and the startup of the browser. I just see the loading dots but nothing happens. You're absolutely right, my production build setup is not optimized and I'm not able to do so as it's not my responsibility, officially. So I will try and optimize the build. But isn't there an option to skip the build and simply call "yarn serve" or something like that? When I think about the minifying steps and such, I would not be able to debug (maybe I would need some kind of SourceMaps?).

Edit: Turned out the build process is running into a "memory allocation problem". Wow. I'm stunned :D

[Thaval]

Is it possible to tell cypress which command within package.json it should use to build?

[JessicaSachs]

Which command... what do you mean? For questions, chat on Discord

[denis-domanskii]

@JessicaSachs hey! Could you please answer the question above?